Bilkent University
Department of Computer Engineering
M.S.THESIS PRESENTATION
Evidence-Driven, Multi-Reference Patch Correctness Assessment for Automated Program Repair Benchmarking
Sahand Moslemi Yengejeh
Master Student
(Supervisor: Asst. Prof. Anıl Koyuncu)
Computer Engineering Department
Bilkent University
Abstract: Deciding whether a patch produced by Automated Program Repair (APR) is actually correct remains a key bottleneck in the field. Hand-checking patches takes considerable effort and has well-known limitations: matching against a single reference can miss valid alternative fixes, while semantic inspection is subjective and difficult to reproduce. Many current Automated Patch Correctness Assessment (APCA) techniques are implemented as opaque predictive models that treat each candidate as if it were new, so they repeatedly re-judge patches that are semantically equivalent to those already validated. To quantify this gap, we study a large set of tool-generated patches and observe two opposing trends. Roughly 39% of the unique correct patches are syntactic clones of each other, which leaves clear room for automation. At the same time, around 65% of bugs admit more than one distinct correct fix, so any method that compares only against a single reference patch will misjudge a significant share of the space. These findings motivate Historian, a framework that uses Large Language Models to compare each new repair attempt against many historically validated patches rather than a single reference. The resulting verdicts are evidence-based and traceable to specific historical patches, and any candidate without sufficient support is conservatively held back as Unknown rather than forced into a label. Under a leave-one-tool-out protocol, Historian covers 95.0% of the patches with 88.4% accuracy, which cuts manual validation down to about 5% of the corpus. When placed in front of standalone APCA tools as an evidence-based filter, it lifts their accuracy by up to 21.8% and yields a hybrid pipeline that reaches 86.2% overall accuracy at 100% coverage. A longitudinal study of tool-generated patches over 2020-2024 further shows that redundancy among repair attempts is a recurring property: many patches keep rediscovering established fixes, which strengthens the long-term case for evidence-based assessment of APR.
DATE: July 02, Thursday @ 15:00
Place: EA 409