Using surveys developed within social science or psychology as an instrument to study large language models (LLMs) is gaining increasing interest in the field. While originally designed for humans, questionnaires represent thoroughly curated measurement instruments consisting of individual items measuring a wide range of concepts such as political opinions or values. Measuring these theoretical concepts in LLMs remains subject to ongoing research and opens important questions concerning the validity and robustness of possible results.
One particular advantage of survey data is the availability of human responses often representing a random sample from a well-defined population providing a fruitful alternative to commonly used annotated data created by crowdworkers or scraped from the internet. Moreover, the data is often of high quality and provides the opportunity to compare LLMs’ responses to a representative human sample. An important methodological caveat is that the measurement of theoretical constructs related to psychology or subjective opinions should not be mistaken for assigning human characteristics to the computational models themselves; the goal is rather to use the responses from the models to measure things like implicit biases learned from their training data.
Possible research questions could address the comparison of human responses to LLMs given different demographics, countries and / or languages. This includes methodological considerations inherent to survey research such as defining the population as well as individual in the case of LLMs and creating a fitting input structure of the questionnaire resulting in the most reliable results.
Another possible trajectory lies in the investigation of the robustness of LLMs’ responses to questionnaires given changes in the prompt. This could range from adjustments in the punctuation over full paraphrases of the questionnaire items to a multilingual analysis providing translations of questionnaires to different multilingual LLMs.