Background and Setup
Course leader Omid Mirmotahari had prior experience with peer review tools such as Peer Grade and Canvas SpeedGrader, both of which had proven insufficient for large-scale use — Peer Grade due to GDPR issues, and Canvas due to limited functionality and poor data export options. IN1020 was therefore selected as the primary site for exploring Feedback Fruits' technical capabilities, particularly its potential for learning analytics.
For this pilot, students who had completed the two previous mandatory assignments were randomly assigned to one of three groups for the third and final mandatory assignment:
- Group 1 – Peer Assessment (PA): Students reviewed three fellow students' submissions using Feedback Fruits. No rubric or structured guidance was provided — this was a deliberate choice to study the effect of exposure to peers' work alone.
- Group 2 – Self-Assessment (SA): Students used an online rubric in Feedback Fruits to evaluate their own work against specific criteria, using a Socratic approach where they were not given correct answers but prompted to reflect.
- Group 3 – Teaching Assistant Feedback (TA): Students received written feedback from one of 18 teaching assistants within one week of first submission. This group served as the control condition, reflecting current standard practice.
All groups submitted a first draft, went through their respective feedback process, and then submitted a revised final version one week later.
The Assignment
The mandatory assignment covered three components from the course syllabus:
- A theoretical and conceptual task where students evaluated an exam question and wrote an extended solution with reasoning.
- A circuit analysis task requiring students to document circuit functionality using truth tables and Boolean expressions.
- A circuit improvement task where students proposed and justified improvements to a given circuit.
Results
All three interventions led to statistically significant improvements from first draft to final submission. The peer assessment group achieved the largest average point increase (Δ0.52 on a scale of −1 to 3), followed by teaching assistant feedback (Δ0.45) and self-assessment (Δ0.36).
On the final exam — which students completed independently after the intervention — the peer assessment group achieved the highest average score (67.08%), compared to self-assessment (62.85%) and teaching assistant feedback (62.59%). In the hardware section of the exam specifically, the peer assessment group averaged 49.67 points versus 43.84 (TA) and 46.40 (SA), with 48 points corresponding to a passing grade of C.
A key finding from the data (see Fig. 4 in the published article) is that peer assessment had the greatest positive effect on students who were initially performing at a D–C level, while teaching assistant feedback was most effective for students near the pass/fail boundary.
Student Experience
528 students participated in the study (173 TA, 183 SA, 172 PA). Course evaluations showed that negative comments related to Canvas, Feedback Fruits, or the assignment process were well below 5% of the total student population. Students critical of the platform primarily pointed to unfamiliarity with Canvas, which is not normally used in IN1020 or at the Department of Informatics more broadly.
Follow-up
The results from this pilot are published in a peer-reviewed article presented at NIKT 2024, which received the Best Paper Award at Udit 2024.
Key Figures (from published article)
Fig. 1 – Grade distribution at first submission (near-identical across groups, confirming randomization). LV = Teaching assistant, EV = self-assessment and HV = peer-assessment.
Fig. 3 – Grade distribution at final submission (improvement visible across all groups)
Fig. 4 – Effect of each intervention by initial performance level (peer review dominates in the D–C range)