50th Annual Summer Institute of Applied Statistics 2027

The featured speaker will be Spencer Carter, Data Scientist and Statistician

Abstract:

The public and private sector alike are putting great emphasis on creating Generative Artificial Intelligence (GenAI) systems backed by Large Language Models (LLMs). Because LLMs are effective at generalization with few or zero-shot learning, it has become incredibly easy to build LLM-driven applications with little more than an API call. This ease has led some to measure these systems qualitatively on a few examples—the colloquial "vibe check". Yet these are non-deterministic models with a functionally infinite input space, for which evaluation is not trivial. This is a glowing opportunity for statisticians to play key roles in the evaluation and measurement of these systems—“evals”.

This presentation will introduce statisticians to the core concepts of LLMs and the common archetypes of GenAI application. We’ll learn about the metrics and frameworks used to evaluate LLM systems and build, then evaluate our own with an eye to how those evals allow us to improve the system’s performance. We will focus on getting our LLM system to the point where evaluating it becomes a recognizable statistical problem, allowing participants to bring their statistical machinery to bear. We will cover pitfalls and strategies for collecting and maintaining eval data, and how to build trustworthy LLM Judges to supplement limited human oversight.

Prior experience with LLMs is not needed—while we will review the inner-workings of LLMs, we will largely treat the models as black boxes, focusing instead on measuring the performance of the application holistically.

Bio:

Spencer Carter is a statistician and data scientist based in Atlanta, GA, currently an AVP of Data Science at Travelers Insurance. Spencer has worked on an array of applied problems—from vehicle repair trends, to medical billing, to web telemetry. At present, he leads technical strategy for GenAI Evaluations and model development for the Claim division. Spencer has been working on Natural Language Processing (NLP) models for 10+ years, building classical and deep learning text models for language with applications to claim notes, customer surveys, and call transcripts. With the advent of LLM paradigm, he is focused on the effective use and evaluation of LLM systems using pretrained frontier models.

To view past presenters, click here.

To register, click here.

To view two day schedule, click here.

For questions regarding SIAS, contact Jill Stevenson at (801) 422-8067 or jstevenson@stat.byu.edu.