Abstract:
Many large language models score remarkably well on standardised physics tests, including on instruments the physics education research community has developed as careful diagnostics of conceptual understanding. What such a score licenses us to infer, however, is harder to pin down. The naïve reading (one option chosen, correct or incorrect, accuracy averaged across questions) imports assumptions from human psychometrics whose transfer to a non-human respondent has not yet been verified. Whether a language model is responding to the physics of a question, to the order its options happen to appear in, to the particular letters used to label them, or the attractiveness of a word in the option sentence, turns out to be a more delicate question than what is acknowledged in current evaluation practice. This furthers the question of what kind of validity protocols we should require, when giving any kind of multiple-choice test to non-human respondents.
We will serve refreshments, coffee and tea. Welcome!
The bi-weekly ODD seminar series at CCSE
The Open Discussions on Didactics (ODD) is a seminar series usually on Thursdays at 13:00-14:00 every other week (odd week numbers) at CCSE.
The seminar will be maximum one hour and is an informal arena to present and discuss learning theory, educational research and teaching experiences within computational science. To cater to the highly heterogeneous backgrounds and interests of students, teachers and researchers in our environment, we aim for seminars that introduce participants to new ideas, reflection, and discussion within a broad spectrum of educational topics.
Presentations need not be mature and polished – on the contrary, we hope that as many as possible want to share undigested observations and reflections in short presentations of varied form and topics. We hope to have enough contributions to frequently have the meetings as lightning talk sessions, where three different speakers will each give a 5-10-minute presentation followed by discussion.