Speaker: Klint Kanopka

Institution: Applied Statistics, Social Science, and Humanities at NYU Steinhardt

Date: 14:00 BST, 30th July, 2025

Talk Title: Reproducing Expert Judgement with Shortened Surveys using Simulated Annealing

Abstract: Surveys, screeners, and patient assessments are often shortened to decrease response burden and cost of administration. While there are many methods for shortening screeners, they often aim to reproduce the original score while inducing the smallest amount of measurement error. One consideration that is often overlooked is that these instruments are often used less for precise measurement of some latent construct, but instead for accurate prediction of an even more costly or time consuming diagnosis or expert judgment. As such, we present an alternative method that addresses these concerns through the use of a Markov Chain Monte Carlo algorithm with simulated annealing (MCMC-SA). To maintain ease of use of the eventual form, we use MCMC-SA to explore the combinatorial search space of short forms using a loss function that optimizes prediction based upon an unweighted sum score. This method is orders of magnitude more efficient than brute force search and has the advantage of optimizing for prediction of the desired outcome directly, instead of optimizing on the measurement of an intermediate outcome that may not be well aligned with the decision criteria of interest. When applied to a screener for alcohol use disorder, we demonstrate that under the unweighted sum score scoring constraint, shortened forms can more accurately predict expert judgments than full forms.

Bio: Klint Kanopka is an Assistant Professor of Applied Statistics in the department of Applied Statistics, Social Science, and Humanities at NYU Steinhardt. His work centers around joining psychometric modeling with computational methods. In practice, this includes using process data to understand how respondents interact with computerized tests, modeling features (like item order) that may hamper the comparability of scores, and using natural language processing to make sense of respondent generated text. Research-wise, he’s also interested in applications of network methods and probabilistic graphical models to problems in measurement and the social sciences.