Using surveys developed within social science or psychology as an instrument to study large language models (LLMs) is gaining increasing interest in the field. While originally designed for humans, questionnaires represent thoroughly curated measurement instruments consisting of individual items measuring a wide range of concepts such as political opinions or values. Measuring these theoretical concepts in LLMs remains subject to ongoing research and opens important questions concerning the validity and robustness of possible results.
One particular advantage of survey data is the availability of human responses, often representing a random sample from a well-defined population, providing a fruitful alternative to commonly used annotated data created by crowdworkers or scraped from the internet. Moreover, the data is often of high quality and provides the opportunity to compare LLMs’ responses to a representative human sample. An important methodological caveat is that surveys are designed to measure implicit concepts using sets of questions. A valid response, however, does not guarantee the existence of these concepts in an LLM.
The goal of the thesis is to understand the generalizability of survey responses. Possible research trajectories could leverage different surveys collected from the same human sample, different data sources representing the same theoretical construct, or previous research on associated behavior with survey results from human respondents (e.g., respondents answering in favour of fairness showing higher support for liberal policies).
The project includes methodological considerations inherent to survey research, such as defining the population and creating a fitting input structure of the questionnaire resulting in the most reliable results. Additional steps may include a multilingual analysis, providing translations of questionnaires and related data to different multilingual LLMs.