Welcome to the Metrics and Models homepage!
Talk Title: The Paper Factory
Abstract: How can large language models (LLMs) contribute to social science research, and what parts of research remain stubbornly human? Building on existing LLM tools, we offer a multi-agent workflow capable of producing a full quantitative social science paper from an initial prompt. The workflow relies on researchers codifying their heuristics for doing data analysis, and we suggest some core design principles for researchers interested in building on this scaffolding. Using this case, we also examine what current LLM capabilities reveal about the organization of research. LLM agents can lower the cost of pursuing high-risk ideas, expand robustness and transparency, bring research to new audiences, and force scholars to articulate the heuristics that create valuable research. But they also pose challenges, both in terms of the quality of papers and in the adequacy of scientific institutions to adapt. Meeting these challenges will require not prohibition or denial, but new institutional norms that make use of these tools observable, auditable, and accountable.
Bio: Per Engzell is a sociologist at the UCL Social Research Institute, University College London. His work has been published in American Sociological Review, American Journal of Sociology, Proceedings of the National Academy of Sciences, Nature Human Behaviour, and other outlets. Substantively he is interested in social stratification and mobility in education and the labor market. Methodologically he is interested in robustness, transparency, and abductive inference. He is the current PI of an ERC Starting Grant titled “Markets and Mobility: How Employers Structure Economic Opportunity”.