Missing benchmark? Contribute through GitHub or mailing list.

DBGBench

Description 291 (in)correct patches from real software professionals for 27 real bugs in C for the qualitative evaluation of automated repair techniques
URL https://dbgbench.github.io
Target C source code
Related publication Where is the Bug and How is it Fixed? An Experiment with Practitioners, by M. Boehme, E.O. Soremekun, S. Chattopadhyay, E. Ugherughe, A. Zeller. European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 2017

Codeflaws

Description 3902 bugs from Codeforces programming competition for evaluating program repair tools across different defect classes
URL https://codeflaws.github.io/
Target C source code
Related publication Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools, by S.H. Tan, J. Yi, Yulis, S. Mechtaev, A. Roychoudhury. International Conference on Software Engineering (ICSE) — Poster Track. 2017

ManyBugs

Description Automated program repair benchmark that consists of 185 defects from large popular open-source projects
URL http://repairbenchmarks.cs.umass.edu
Target C source code
Related publication The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs, by C. Le Goues, N. Holtschulte, E. K. Smith, Y. Brun, P.Devanbu, S. Forrest, W. Weimer. IEEE Transactions on Software Engineering (TSE). 2015

IntroClass

Description Automated program repair benchmark that consists of 998 defects in small student-written programming assignments
URL http://repairbenchmarks.cs.umass.edu
Target C source code
Related publication The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs, by C. Le Goues, N. Holtschulte, E. K. Smith, Y. Brun, P. Devanbu, S. Forrest, W. Weimer. IEEE Transactions on Software Engineering (TSE). 2015

Defects4j

Description A database of existing faults to enable controlled testing studies for Java
URL https://github.com/rjust/defects4j
Target Java source code
Related publication Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs, by R. Just, D. Jalali, M. D. Ernst. International Symposium on Software Testing and Analysis (ISSTA). 2014