About this opportunity
Confidential lists this SWE-Bench Task Auditor opportunity in Verdugo City, California. Review the employer’s description below for duties, qualifications and application requirements.
Job description
Help ensure software engineering benchmark tasks are accurate, reproducible, and resistant to flawed grading. You will evaluate repository-level tasks used to train and assess advanced AI models, reviewing reference patches, test harnesses, grading integrity, and written evaluation criteria.
Key Responsibilities Assess the quality, correctness, and reproducibility of software engineering benchmark tasks.
Review repository-level tasks, reference patches, test runners, and Docker isolation.
Identify answer leakage, reward hacking risks, and grading issues.
Provide clear written feedback using defined evaluation rubrics.
Qualifications At least 3 years of professional software engineering experience.
Open-source contribution or maintainer experience, such as merged pull requests or committer and maintainer roles.
Strong ability to audit reference patches, test runners, and Docker isolation environments.
Fluency in Python and at least one of Java, Go, TypeScript, or C++.
Preferred Qualifications Familiarity with SWE-Bench Verified or similar repository-level benchmarks.
Maintainer experience with major Python open-source projects, such as Django, Flask, scikit-learn, SymPy, or pytest.
Prior experience with code review or task grading.
Work Terms Remote role open to candidates located in the United States.
Hourly engagement.
Compensation $70 to $90 per hour.
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.