Early Success Prediction in Student Codes in Computer Science Education

Presenter

Hayun Jung

Group Members

Piyush Maheshwari, Saksham Bansal

Campus

UMass Amherst

Sponsor

Andrew Lan, Department of Computer Science, UMass Amherst

Schedule

Session 2, 11:30 AM - 12:15 PM [Schedule by Time][Poster Grid for Time/Location]

Location

Poster Board A20, Campus Center Auditorium, Row 1 (A1-A20) [Poster Location Map]

Abstract

Predicting the success rate of student-written code submissions for a given problem without executing the code against test cases is a challenging task, but one that is beneficial for both instructors and students. In this study, we explore various methods to achieve this, including BERT encoding, BERT fine-tuning, and prompting approaches like Few-shot and Chain-of-Thought (CoT). Our experiments on the CSEDM Challenge dataset, consisting of 50 unique problems and 69627 student code submissions, reveal that BERT encoding has the lowest performance, while Code-BERT encodings show promise. BERT fine-tuning, particularly BERT-base-uncased, performs slightly better, which we attribute to the diverse language used in the prompt and technical constraints in the runtime environment. Among the prompting approaches, Few-shot demonstrates the lowest error, indicating the effectiveness of constructing in-context examples. However, CoT does not perform as well, partly due to the assumption that every code contains only a single bug. Overall, we find that prompting approaches show the best performance, followed by fine-tuning and then encoding approaches. Our findings provide insights into the efficacy of different methods for predicting code success rates and can inform future research in this area. This study was funded by the Undergraduate Research Volunteers program during winter break. 

Keywords

Natural Language Processing, Computer Science in Education, Machine Learning

Research Area

Computer Science

SIMILAR ABSTRACTS (BY KEYWORD)