About this opportunity
Confidential lists this QA Engineer opportunity in Willow Grove, Pennsylvania. Review the employer’s description below for duties, qualifications and application requirements.
Job description
Role Overview Apply software quality assurance expertise to evaluate technical AI outputs and help improve how next-generation AI systems learn, reason, and perform. This remote contract opportunity supports a high-volume project and does not require prior AI experience, but it does require prior paid human-in-the-loop AI evaluation work.
Key Responsibilities Evaluate and score AI-generated technical outputs against established quality criteria and rubrics, identifying responses that appear correct but are inaccurate or incomplete.
Design and assess thorough functional, regression, edge-case, negative, and boundary test cases.
Review bug reports and test documentation for reproducibility, completeness, and appropriate severity assessment.
Identify, isolate, and document defects with precise reproduction steps using structured tracking methods.
Provide detailed written feedback and annotations that developers can act on without further clarification.
Work with project teams to refine evaluation guidelines and continuously improve testing standards and methods.
Qualifications Professional experience as a QA Engineer, SDET, Test Engineer, QA Analyst, or in a comparable software quality assurance role.
Strong knowledge of quality assurance, software testing, test case design, bug tracking, regression testing, and manual and automated testing strategies.
Hands-on experience with automation frameworks and test management tools, such as Selenium, Playwright, Cypress, Appium, Postman, Jira, TestRail, Zephyr, BrowserStack, or similar tools.
Prior paid experience supporting AI training through human data annotation, labeling, RLHF, AI response or model evaluation, or rubric-based grading. Software QA experience alone does not meet this project requirement.
Excellent analytical and problem-solving skills, close attention to detail, and the ability to communicate complex findings clearly in written English at B2 level or above.
No formal degree is required. Demonstrable practical testing experience is prioritized.
Reliable internet access and readiness to begin promptly.
Work Terms Remote contract engagement.
Compensation $90 to $175 per hour.
Application Process Apply through the available application flow using email or a Google account. Continuing the application requires agreement to the applicable terms and privacy policies.
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.