AI for Qualitative Physics Education Research

Programme

Wednesday, April 22nd – Day 1

Room: Tellefsen's Tower (Physics building, 4th floor)

9:00–9:15 – Introduction

9:15–9:45 – Lightning talks 1

9:45–10:00 – Break

10:00–10:30 – Lightning talks 2

10:30–11:00 – Coffee

Big Picture: AI methods and qualitative coding

12:00–13:00 – Lunch 

Fredrikke building

13:00–15:00 – Parallel Session A: Reasoning and Labs

Room: V213 (School Lab; Physics building 2nd floor, west)

13:00–13:30 Supporting consistent thematic coding of authentic laboratory experiences in student reflections using text embeddings, Mike Verostek

Abstract: 

Written student course reflections in the form of “memos to future researchers” have served as valuable qualitative data for studying student perceptions of authenticity in course-based undergraduate research experiences (CUREs).

In our current study, which compares student outcomes in a CURE-based laboratory relevant to experimental particle physics with those in a skills-focused introductory physics lab, we collected hundreds of student memos for qualitative analysis. To facilitate analysis of this large corpus, we build on text-embedding methods developed by Odden et al. to support automated coding of student reflections. 

In this talk, I will discuss several novel approaches we have explored that use text embeddings to more consistently apply our thematic analysis framework to student responses. Brief comparisons to other LLM-assisted coding approaches will also be discussed. Adoption of these methods may allow for faster, more efficient, and more consistent thematic coding of student experiences with authenticity in introductory laboratory courses.

13:30–14:00 Evaluating Scientific Argumentation Skills in Undergraduate Lab Reports at Scale, Oxana Andriuc

Abstract: 

Evaluating the efficacy of teaching scientific argumentation skills in undergraduate courses is an important task for ensuring that we are providing a high-quality education that prepares students to be scientific leaders in an increasingly polarised society. 

We propose an approach that leverages recent advancements in computational methods to automate the extraction of text from undergraduate student lab reports and identify elements of scientific argumentation. Building on existing argumentation frameworks, we develop a new coding scheme that encompasses content labels, inter-sentence relations, and physical and logical correctness. 

Applying this method to a series of lab reports reveals insights into how students structure their arguments and the impact of teaching interventions. Finally, we explore the use of open-access Large Language Models (LLMs) towards building a fully automated pipeline for evaluating our teaching of scientific argumentation.

14:00–14:30 Co-creating Qualitative Codebooks with AI to Understand the Role of Visuals in Computational Essays, Paul Hur

Abstract: 

Computational essays (Jupyter notebooks combining code, narrative text, and visuals) make it easy for students to include plots, diagrams, and imported images, but it is often unclear how those visuals function rhetorically within a scientific explanation.

In this talk, we present insights for creating a qualitative codebook that adapts a scientific explanations framework (Claim–Evidence–Reasoning; McNeill & Krajcik, 2008) for image excerpts in undergraduate physics computational essays. We describe a three-stage codebook construction workflow (theory-grounded draft, data-calibrated refinement, and human finalization; following Zambrano et al., 2026) that supports consistent identification of Claim, Evidence, and Reasoning in visuals and their surrounding student-authored context.

The goal is to offer a transferable approach for researchers who want to treat images as analyzable components of explanation, and to discuss how qualitative codebook design can scale to multimodal student data.

14:30–15:00 Boosting vision performance with text reasoning, Giulia Polverini & Bor Gregorcic

Abstract: 

The performance of multimodal large language models (MLLMs) has been demonstrated to be prompt-sensitive. Specifically, chain-of-thought (CoT) prompting has been shown to be productive, because it pushed the models to write out a step-by-step reasoning process in their generated text, often leading to better answers.

As CoT approaches were integrated "under the hood" into newer reasoning models (RLMs), their performance improved significantly, reaching an expert-like level in many physics tasks. However, vision remains one of the main bottlenecks, due to intrinsic limits in the models' perception of visual inputs. Still, we show how text-based reasoning can partially work around these constraints. 

Using physics examples, we will present two strategies that apply the same principle of “reasoning by writing” to overcome limitations in visual perception.

13:00–15:00 – Parallel Session B: Solving Physics Problems

Room: CCSE Seminar room

13:00–13:30 Periodic Temporal Variability in GPT-4o Performance on a Physics Problem, Paul Tschisgale

Abstract: 

Large language models (LLMs) are increasingly used in research both as research tools, for example in qualitative or mixed-methods analyses, and as objects of investigation, for instance when studying their domain-specific capabilities. A largely implicit assumption underlying much of this work is that LLM performance under fixed conditions is time invariant. If an LLM’s average output quality in response to an identical prompt systematically changes over time, this assumption is violated, potentially undermining the reliability and replicability of related research findings. 

In this talk, a longitudinal study of the temporal stability of GPT-4o performance is presented. Using a fixed model snapshot, fixed hyperparameters, and an identical prompt, GPT-4o was queried via the API to solve the same multiple-choice physics problem over a period of approximately three months. Every three hours, ten independent samples were collected, resulting in a total of 6,930 model responses. 

Spectral (Fourier) analysis revealed pronounced periodic temporal variability in model performance, accounting for approximately 20% of the overall variability. Specifically, performance fluctuations can be explained by the interaction of daily and weekly rhythms. These findings highlight that, even under tightly controlled conditions, LLM performance may vary systematically over time. Implications for the reliability and replicability of research that uses or investigates LLMs are discussed.

13:30–14:00 Ways to utilize GenAI for fostering students'physics problem solving, Peter Wulff

Abstract: 

In this talk, we present the design rationale and preliminary findings from an online learning environment, WasP, that is meant to foster students' physics problem solving abilities with GenAI-based tutoring. 

We discuss implications for how GenAI can be utilized in physics education, and the challenges that we encountered in implementing this online learning environment.

14:00–14:30 Towards a hybrid framework for AI consistency: combining human coding and text embeddings in Galilean relativity,Eugenio Tufino

Abstract: 

As Large Multimodal Models (LMMs) are increasingly adopted in physics education, researchers face the challenge of assessing their reliability beyond binary accuracy. In this talk, we present a preliminary evaluation of state-of-the-art LMMs (including the latest Gemini and GPT series) using a recently validated questionnaire on Galilean relativity and relative motion. Adopting the probabilistic approach and the 'coding by meaning' methodology described by Polverini et al. (2025) in recent investigation of the Brief Electricity and Magnetism Assessment, we collected 30 independent iterations per item to capture the intrinsic variability of AI reasoning. 

To analyze this complex dataset, we propose an exploratory hybrid framework. First, we apply deductive qualitative coding to categorize reasoning failures. Second, we apply Natural Language Processing (NLP) techniques, adapting text embeddings methodologies established by Odden et al. (2024), to map the semantic space of these responses.

We utilize this approach to visualize and quantify the "semantic stability" of AI explanations across independent iterations, distinguishing stochastic hallucinations from persistent difficulties.

14:30–15:00 The Comparison between ChatGPT and Gemini in Solving Physics Problems, Muhammad Aswin Rangkuti

Abstract: 

This study compares the performance of ChatGPT and Gemini by evaluating their results on the Mechanical Waves Conceptual Survey (MWCS). The MWCS was selected because it includes graph-, image-, and text-based questions that are common in physics; as a result, the findings may offer broader insights into generative AI performance across other physics topics. 

Using a Generalized Linear Mixed Model (GLMM), a significant difference in performance was found between the two systems, with Gemini performing better overall on the MWCS. Prior publications have often reported that ChatGPT struggles with graph-based questions; in contrast, Gemini outperformed ChatGPT on these items by providing correct reasoning. Implications for supporting teaching and learning in the physics classroom are discussed based on these results.

15:00–15:30 Snack Break 

Disciplinary norms

Room: Seminar room Ø434

15:30–16:00 - > /dev/null - data minimalism for AI in physics teaching, Gerd Kortemeyer

Abstract: 

ETH Zurich has been running AI systems in physics courses for over two years, and it has been running tools for physics homework feedback and exam grading assistance for a year. A lot of colleagues have a lot of ideas what we could do with all the data that we could have amassed, and they are stunned when we tell them that we intentionally throw away everything ("> /dev/null"); not a bit of data except the cumulative token consumption across all courses. 

In this talk, I’m not exploring what we could do with the data (that part is almost too easy). I’m exploring what we should do: choices that are ethical, don’t distort student behavior, don’t widen power gaps, don’t turn “just-in-case logging” into the default, lead to actions rather than dashboards, don’t undermine instructor fairness, and don’t get in the way of actually using AI productively for teaching. What data should never exist in teaching - no matter how useful it seems?

16:00–16:30 – How Physics Educators Conceptualise AI for Qualitative Data, Philippa Petts & Peter Swift

Abstract: 

Artificial intelligence (AI) is becoming increasingly embedded in higher education, supporting assessment, feedback, learning analytics, and curriculum design. Within STEM education, most applications so far have focused on quantitative tasks as opposed to the analysis of qualitative data [1]. Disciplinary epistemology emphasises that academic disciplines are characterised by distinctive assumptions about what constitutes valid knowledge, acceptable methods, and trustworthy evidence [2]. 

Physics is commonly described as a “hard–pure” discipline, where knowledge is cumulative, highly structured, and strongly grounded in mathematical formalism and empirical testability. These characteristics shape both research practices, and pedagogical expectations and attitudes toward new technologies. Semi-structured interviews with eight higher education physics educators from diverse institutions were undertaken. Three research questions were considered covering how physics educators describe the role of AI in teaching and research; what potential uses and limitations might be seen for AI when working with qualitative data and what barriers exist with their willingness to engage with these tools. 

Thematic analysis, [3] revealed strong interest in using AI as a tool for things like automation and efficiency, but showed persistent discomfort with using AI to interpret qualitative data. Our findings show that AI adoption is shaped by disciplinary epistemology, and resistance to it use reflects concerns about evidence, interpretation, and epistemic legitimacy. 

[1] JISC, AI in tertiary education: a summary of the current state of play (2023) 
[2] T. Becher & P.R. Trowler, Academic Tribes and Territories SRHE & OUP, 2001 
[3] V. Braun and V. Clarke, Using thematic analysis in psychology Qual. Res. Psychol., 3:2, 77-101, DOI: 10.1191/1478088706qp063oa

Thursday, April 23rd – Day 2

Room: Tellefsen's Tower all day!

9:30–10:00 - When Context Matters (and It Always Does): Generative AI for Automated Survey Contextualization and Personalization, Alexandra Werth

Abstract: 

Survey items do not mean the same thing everywhere. Small changes in framing, such as course topics, teaching practices, local norms, or institution, can shift how people interpret a question and how they respond. At the same time, instructors often want surveys that clearly connect to their course and engage their students, but tailoring instruments by hand is slow and can make it hard to compare findings across contexts. In this talk, I share two strands of work. 

1. First, I present evidence that students’ responses and explanations can vary substantially with contextual framing, even when item wording is held constant.
2. Second, I describe an approach that uses generative language models to create dynamic, course-specific open-ended reflection questions at scale.

The focus is not just what these tools can generate, but what it takes to trust what they generate for both ethical and research purposes. Dynamic items raise real questions about construct validity and cross-setting comparability, and they can introduce issues such as unclear wording, double-barreled prompts, jargon, and subtle shifts in what a question is really asking. To address this, I describe how we use generative AI not only to draft prompts, but also to stress-test them before they reach students.

Using Synthetic Question–Response Analysis (SQRA), we generate synthetic response sets to candidate prompts under a range of personas (for example, students with different preparation levels, motivations, and course experiences) and then analyze those responses to surface predictable failure modes. This provides a fast, iterative loop for refining prompts and documenting design choices prior to deployment.

10:00–10:30 – Adapting traditional qualitative coding criteria to text embeddings for analysis of open-ended survey data, Mary McMullan

Abstract: 

Embedding models are a particular class of AI large language models (LLMs) optimized for classification tasks which have been usefully applied to qualitative research while still prioritizing a “human-in-the-loop” approach. 

This talk will discuss approaches to increase the level of control a researcher has when using text embeddings for analysis, focusing on adapting traditional ways of expressing coding criteria in qualitative research. Providing exemplars, defining the barrier for inclusion and exclusion criteria for multiple codes, and common misassigned codes are just a few of the possible ways researchers bring transparency and nuance to the coding process. 

Our work implements comprehensive coding criteria from a preexisting survey dataset into a validated text embedding model used for qualitative coding in PER. We measure the text embedding model’s performance and characterize the interpretability when applied to large scale analysis of qualitative survey data.

10:30–11:00 Coffee

Interpretable AI

11:00–11:30 – Combining Qualitative and Quantitative: An Approach for Human-Assisted Document Analysis with Text-Embeddings, Amanda Bowerman

Abstract: 

Research publications are continuously increasing and thus too are the need for tools that can assist researchers in analyzing the literature. Current tools lack interpretability, leaning heavily on researchers after the analysis has been completed. Keeping researchers 'out-of-the-loop' often leads to results that are outside of the scope of interest - being either too broad or too narrow and leaning too heavily on the "computer" to understand the underlying context of the analysis at hand. 

We propose a method that will keep the researcher in the "driver's seat", utilizing the advancements in Natural Language Processing (NLP) to combine aspects both qualitative and quantitative data analysis. Employing a centroid-based classification method, we are able to identify author-defined themes in the literature with the help of text-embeddings. This type of deductive analysis combined with a multi-label classification model has yet to be explored in the field of Science Education. 

By combining articles from the last 30 years from the following journals: Science Education, Journal of Research in Science Teaching, Studies in Science Education, International Journal of Science Education, and Science & Education, we were able to explore the evolution of the author-defined themes at field-level as well as the journal-level.

11:30–12:00 – Mapping 20 years of Physics Education Research using text embeddings, Helene Lane

Abstract: 

Physics education research (PER) has historically found inspiration from other fields and applied their methods. We have used text embeddings to analyse 20 years of research in the journal Physical Review Physics Education Research. 

How has PER changed over time since this journal started, and which physics topic is the most popular to conduct research on in PER?

12:00–14:00 Extended lunch & networking time. Group photo at 13.30 (We meet outside by the Physics building, main entrance)

Systematising analysis

14:00–14:30 – Applying text embeddings for the classification of quantum technology job postings, Ben Zwickl

Abstract: 

The development and application of quantum technologies is a priority in the US, EU, and more broadly, hence a highly skilled workforce is needed to support the innovation. Looking through job postings can be helpful to characterize the growth of the quantum industry and the expertise needed for available jobs, yet even the nascent quantum technology industry has over 1,000 job postings at any one time. Simple approaches to categorization using job titles are ineffective because most job titles are not standardized across companies and categorization based on educational experience offers little nuance. 

Our team of researchers from RIT and the University of Colorado Boulder have done over 40 in-depth interviews with managers and employees in the quantum industry, which have identified four major job categories (hardware, software, bridging, and public facing and business) that split into 29 distinct job profiles based on similar tasks, skills and knowledge, and experience (e.g., Quantum Hardware System Engineer, Quantum Software Engineer, Cryogenic Specialist). 

We use text embeddings and our career profiles to do deductive qualitative analysis of job postings to create a semi-automated workflow that can be regularly updated, allowing us to track changes in the quantum workforce over time, identify new job roles, and help educators maintain awareness of trends in the quantum industry.

14:30–15:00 – Comparing the Integration of Statistical and Text Embedding Methods in Systematic Literature Reviews, Emily Bolger & Dawson Kinsman

Abstract: 

About 15 years ago, education researchers conducted a systematic literature review (SLR) on change strategies for improving undergraduate STEM education instruction. Analyzing 191 articles from 1995 to 2008, the researchers identified four broad categories of change strategies through a comprehensive interdisciplinary literature review: (1) disseminating curriculum and pedagogy, (2) developing reflective teachers, (3) enacting policy, and (4) developing a shared vision. With recent developments in STEM education practices, it is imperative to conduct a follow-up SLR comparing the effects of change strategies and associated student success. 

However, with the influx of scientific articles published in recent decades, it is time consuming to conduct comprehensive SLRs without the assistance of statistical and machine learning (ML) analysis techniques. While most ML approaches can be easily written with a few lines of code, using them to extract meaningful information for literature reviews can be challenging. In this presentation, we compare the results of theme identification using Latent Dirichlet Allocation and unsupervised clustering methods with text embeddings. 

Specifically, we describe the steps and methods used to integrate human researchers' coding with the quantitative machine learning analyses and conclude with how these steps enhanced both the human and machine analyses. We believe that this mixed methods “human-in-the-loop'' approach provides valuable guidance for integrating the thoughtful application of machine learning techniques into future systematic review projects.

15:00–15:30 Snack Break

Reflecting on Qualitative Methods and AI

15:30–16:00 - Slow Research in the Context of Mixed Methods, Vashti Sawtelle & Rachel Henderson

Abstract: 

Traditionally, self-efficacy, or the confidence in one's capability to execute a task, is measured using pre/post-surveys to demonstrate shifts in students' self-efficacy. In our collaboration we draw on a mixed methods approach to examine how students’ self-efficacy fluctuates over time. Our novel design uses the Experience Sampling Method, a quantitative technique using surveys of domain-specific self-efficacy alongside reflective journal entries (driven by prompts from analysis of the surveys) from those same students. 

As a team our expertise spans the quantitative and qualitative techniques and we use these alongside a theory-driven slow research process to examine events that can have an impact on student self-efficacy. In this presentation we will briefly share our approach to this research and what we’ve learned with the methodology, and then we will invite the audience to engage with us around questions we have been considering for adapting Large Language Modeling to both the journal prompt creation and the coding of the qualitative responses.

16:00–16:30 - Where, Why and How to Keep Humans in the Loop, Noah Finkelstein

Abstract: 

An examination of why and where humans will need to essential as we conduct PER and educational practices with AI.

17:30 – Group Dinner

Friday April 24th – Day 3

Room: Tellefsen's Tower all day!

09:30–10:30  Extended Coffee & networking

Analysing rich student data

10:30–11:00 Performance and interpretability of survey data analysis using variations on text embedding qualitative coding, Benjamin D. Masucci

Abstract: 

While text embeddings have been successfully applied to journal articles and other full-length documents for classification purposes, their use in open-ended survey responses presents its own set of challenges. Unlike full-length articles, survey responses are brief and provide less context, which can cause high variance in classification performance depending on how the answer responds to the prompt. 

This presentation will explore the implementation of embeddings-based techniques on open-ended survey data. Specifically, we will examine three variations. First, we evaluate input strategies by comparing isolated responses against question-response pairs. For the latter, we compare traditional mean pooling of the final hidden state against a selective pooling approach that isolates tokens associated with the response. Second, we employ a mean-centering approach where we subtract the mean centroid from each of the embeddings, further enhancing variance among response embeddings. Third, we utilize dimensional reduction techniques such as principal component analysis during pre-processing to reduce dimensionality and noise. We compare the performance of these variations against a large set of human-coded data (~1000 responses).

11:00–11:30 – “How Do I ...?”: Procedural Questions Predominate Student-LLM Chatbot Conversations, Alexandra Neagu

Abstract: 

Providing scaffolding through educational chatbots built on LLMs has potential risks and benefits that remain an open area of research. When students navigate impasses, they ask for help by formulating impasse-driven questions. Within interactions with LLM chatbots, such questions shape the user prompts and drive the pedagogical effectiveness of the chatbot's response. 

This study focuses on such student questions from two datasets of distinct STEM learning contexts: formative self-study (in mechanical engineering), and summative assessed coursework (in computing). We analysed 6,113 messages from both learning contexts, using 11 different LLMs and three human raters to classify student questions using four existing schemas. On the feasibility of using LLMs as raters, results showed moderate-to-good inter-rater reliability, with higher consistency than human raters. The data showed that 'procedural' questions predominated in both learning contexts, but more so when students prepare for summative assessment. 

These results provide a basis on which to use LLMs for classification of student questions. However, we identify clear limitations in both the ability to classify with schemas and the value of doing so: schemas are limited and thus struggle to accommodate the semantic richness of composite prompts, offering only partial understanding the wider risks and benefits of chatbot integration. In the future, we recommend an analysis approach that captures the nuanced, multi-turn nature of conversation, for example, by applying methods from conversation analysis in discursive psychology.

11:30–12:00 – Unpacking Trends in Students’ Analogical Reasoning with Unsupervised Machine Learning, Winter Allen

Abstract: 

Analogical reasoning is a key component of scientific understanding, enabling students to connect new concepts to familiar experiences. This study investigates the extent to which students in a large enrollment physics course leverage analogies while explaining Morse potential curve in a language suitable to a second grader and in terms of their preferred everyday contexts. 

This talk presents the challenges in analyzing diverse, complex, and detailed student-generated analogies using K-Means clustering, particularly while relying solely on traditional clustering metrics such as silhouette scores and cluster distribution. The effectiveness of accounting additional cluster metrics, such as essay centroids and representative words, along with standard qualitative approaches in a systematic mixed-methods approach to mitigate the challenges are also discussed. 

The resultant clusters and sub-clusters highlight the broad emergent categories of the concepts associated with the curve and corresponding analogies employed to communicate them. Contrary to current understanding of students’ analogical reasoning in physics, we observe that students often employed similar analogical contexts in effectively communicating diverse conceptual ideas.

12:00 Lunch

Critically using AI

13:00–13:30 – AI in higher education – Fight or flight?, Jan-Fredrik Olsen

Abstract: 

It has been argued by educational researchers that dealing with generative AI in higher education is a “wicked problem”: it cannot be permanently solved, but must be managed as an enduring feature of the educational landscape. 

Any decision about how to respond to generative AI in education will have both positive and negative consequences — and importantly, choosing to do nothing is also a choice. In this talk I will frame my experiences using generative AI in teaching mathematics at the undergraduate level from an educational-science perspective, and identify decisions I made and the trade‑offs involved.

13:30–14:00 – Using locally installed LLMs to group science education research papers for a large conference, Jesper Bruun

Abstract: 

Academic conferences in science education routinely features contribution proposals (here referred to as papers) with a high degree of diversity. The grouping into sessions is usually done "by hand" by human actors. This is cumbersome work and in some cases, conference sessions become a mixture of papers which are only peripherally connected to each other - if at all. 

For a large science education conference, we wanted to explore if using locally installed large language models in combination with network analytical approaches could help mitigate this problem. Our aim was to create conference sessions that would appear as coherent as possible for the conference participant. 

This talk will show how we used keyword extraction and network analysis to operationalise coherence and produce a set of candidate sessions. The talk will explore some of the lessons learned during this process - and relate the final outcome to evaluations of the scientific programme.

14:00–14:30 - Does decomposing articles for thematic analysis reveal new trends? Michael Mingyar

Abstract: 

Thematic analysis plays a central role in understanding the evolution of trends in physics education research (PER). Prior studies relied largely on applying topic modeling to whole articles in order to identify broad trends such as an emerging focus on quantum education or student belonging. However, whole-article approaches may obscure other analytically meaningful dimensions, such as the use of specific methodological tools or research practices. 

In this study, we adopt a computational grounded theory framework to investigate how thematic structure changes when PER articles are analyzed at different levels of granularity. We perform thematic analyses on both complete articles and on isolated sections within those articles, enabling a direct comparison of the themes that emerge from each approach.

14:30–15:00 - Rethinking Qualitative PER in the Age of AI, Marcus Kubsch

Abstract:

The emergence of large language models and AI-assisted analysis tools presents both opportunities and challenges for qualitative physics education research. This talk develops a framework and research agenda for understanding how various new approaches to qualitative PER can be productively integrated, with particular attention to the coordination between human researchers and AI systems. 

Drawing on the DEFT framework (Distributing Epistemic Functions and Tasks), I argue that effective human-AI collaboration in qualitative research requires careful consideration of which epistemic functions—such as pattern recognition, interpretation, theoretical reasoning, and validity judgment—are distributed across human and artificial agents. Rather than viewing AI as either a threat to research validity or a neutral tool, I propose conceptualizing qualitative PER as operating within distributed epistemic networks where humans and AI systems contribute complementary capabilities. 

This perspective raises fundamental questions: How do we maintain interpretive depth when leveraging AI's capacity for scale? What forms of triangulation become possible—or necessary—in human-AI research partnerships? How should we document and evaluate the epistemic contributions of non-human agents? 

The talk concludes by outlining a research agenda that addresses these questions while preserving the theoretical richness that distinguishes qualitative PER.

15:00–15:30 Snack Break

15:30–16:00 – Closing, Michael Fox & Tor Ole Odden