Design a Training Dataset That Lifts a Coding Agent on a Repository-Repair Benchmark

Asked at Labelbox · System Design for ML · Difficulty: Hard · Last asked Aug 17, 2026

A research lab wants its coding model to score higher on a public benchmark of real repository issues (for example SWE-bench). Each task is a GitHub issue from an open-source project; the model must produce a patch, and hidden tests decide whether the issue is fixed. The lab hires you to deliver a training dataset that produces that improvement. Design the end-to-end plan and pipeline: where the tasks come from, what each record contains, who labels it, how it is checked, and how you prove the lift.

Loading the problem…