Programme
Thursday, October 10th
Topic: Introductions, NLP and GenAI for Physics Education Research
09:00 Arrival at the CCSE
09:30 - Introductions
Activity: 24/7 Lectures
Discussion: What are the big questions we all have regarding GenAI and NLP in PER?
10:30-10:40 - Coffee break
10:40-11:40 - Tor Ole Odden (UiO): Deductive qualitative analysis at scale in PER using text embeddings
We propose a technique for performing deductive qualitative data analysis at scale on text-based data in PER. Using a natural language processing technique known as text embeddings, we create vector-based representations of texts in a high-dimensional meaning space within which it is possible to quantify differences as vector distances.
To apply the technique we have analyzed approximately 1750 articles from the physics education research conference proceedings, as well as survey data from several collaborating physics education groups.
Results show that this technique can achieve startlingly good classifications results on text data with distinct themes or topics, characterized by specific vocabulary, and it is also both scalable and flexible. However, we have also found that it may struggle with text that requires significant researcher inference to analyze, and can be difficult to benchmark absent traditional qualitatively analyzed datasets. We will discuss next steps and open questions regarding this technique, including applications to other bases of literature and exploration of the properties of embeddings spaces.
11:40-13:00 - Lunch
13:00-15:00
Peter Wulff and Fabian Kieser (PH-Heidelberg): Large Language Models and Prompt-Engineering in Science Education
Since 2022, Large Language Models (LLMs) probably need no introducton anymore. LLMs were found to excel at various tasks, even domain-specific tasks as in science education. LLMs are trained on large datasets and can then be fine-tuned or specifically prompted to be useful in assessment and/or feedback, e.g., in physics problem solving.
This talk will discuss with own research examples in what ways LLMs can be utilized (by prompt-engineering) in science education. Limitations and risks of LLMs will be critically discussed, and future directions for using LLMs in science education will be outlined.
Drew Rosen (U Edinburgh): Understanding the nature of student and (possibly) AI reasoning using dual-process theories of reasoning
An emerging body of research has shown that, even after research-based instruction, students who demonstrate correct conceptual understanding and reasoning on one task often fail to use the same knowledge and skills on related tasks. Researchers are increasingly using dualprocess theories of reasoning (DPToR) as an explanatory model that can predict student reasoning patterns. I will review the history of using DPToR in physics education research and how it is being leveraged to create interventions aimed at improving student reasoning.
Similar “reasoning” failures can be seen with NLP responses to similar physics questions. I believe that DPToR could similarly explain some of how NLP AI responds to prompts and provide insights into both uses in physics or how to build future generations of AI.
15:00-15:30 - Snack Break
15:30-16:30 - Discussion
16:30 - Group Dinner
Friday, October 11th
Topic: GenAI for Physics Education Practice
09:30-10:20 - Giulia Polverini and Bor Gregorcic (Uppsala University): Assessing the performance of LMM-based chatbots in
understanding physics representations
Our study explores the performance of Large Multimodal Model (LMM)-based chatbots in interpreting various physics representations, starting with graphs. We evaluated several chatbots, both freely available and subscription-based, using the Test of Understanding Graphs in Kinematics (TUG-K), a well-established concept inventory.
While we anticipated a potential technological divide, the only notable distinction emerged between OpenAI’s chatbots and the others. However, even the advanced ChatGPT-4o still falls short of achieving the expert-level understanding necessary for being a physics tutor. Moreover, no clear patterns emerged to explain which features of graphical representations posed more difficulty for chatbots. The most consistent finding was that all chatbots performed significantly better on linguistic tasks than on visual ones.
We are now expanding our study to encompass a broader range of physics representations. The aim is to assess the state-of-the-art chatbot performance across different representations and to identify categories of tasks that are particularly
challenging or easier for them all.
10:30-10:40 - Coffee Break
10:40-11:40 - Vidar Skogvoll and Andreas Haraldsrud (UiO): Using AI to foster learning in science in high school and university
Physics students at the University of Oslo learn programming in their first semester and go on to use it in almost every single course from then on, gaining experience as they use it in varied and increasingly sophisticated ways. But course instructors don't teach and assess programming at this increased level of sophistication. Why then, do the students improve? And which resources do they draw on in order to improve?
I have conducted interviews with computational physics masters students, and have found that when students write large codes in collaboration with others, they are motivated to improve their coding. Student are motivated both by the difficulty of the task. and from pressure from peers.
Furthermore, peers play a vital role in giving guidance and opportunities for reflection on how to write better code. In this talk, I will present how these factors and more motivate, help and hinder physics students in improving their programming skill.
11:30-13:00 - Lunch
13:00-15:00 - Gerd Kortemeyer (ETH Zurich): Ethel - a virtual teaching assistant at ETH Zurich
The talk will present experiences and results from explorations of a virtual teaching assistant, which acts as a custom chatbot, gives feedback on handwritten homework solutions, and can assist in grading handwritten exams.
Discussion!