About this opportunity
Intone Inc lists this QA Engineer for AI Products opportunity in bellevue, Washington. Review the employer’s description below for duties, qualifications and application requirements.
Job description
This is a 6-month contract position for a QA Engineer specializing in AI products, based in Bellevue, Washington. The role is hybrid and requires local candidates for in-person collaboration.
Responsibilities
Design and execute test plans for AI/ML-driven features, including model outputs, prompts, and integrated application behavior
Build and maintain automated test suites covering functional, regression, integration, and API testing
Evaluate model outputs for accuracy, consistency, bias, hallucination, and edge-case failures
Develop evaluation frameworks and golden datasets/test cases to benchmark model performance over time
Test prompt engineering changes, model version upgrades, and fine-tuning outputs for regressions
Perform adversarial and red-team style testing to surface safety, security, and robustness issues
Validate data pipelines feeding into AI models, including data quality, schema, and drift detection
Collaborate with data scientists and ML engineers to define acceptance criteria and quality metrics for models
Test latency, scalability, and reliability of AI services under load
Contribute to CI/CD pipelines, integrating automated and model-evaluation tests
Design test cases for LLM-based products such as chatbots, copilots, RAG systems, and AI agents, accounting for non-deterministic and generative outputs
Build evaluation suites, scoring rubrics, and golden datasets using AI/LLM evaluation frameworks
Validate prompt changes and model/version upgrades against baseline evaluation sets
Write evaluation scripts, test harnesses, and data validation logic
Qualifications
Required 5+ years of QA and testing experience
Hands-on experience testing LLM and generative AI products
Practical experience with AI/LLM evaluation frameworks (such as Ragas, DeepEval, LangSmith, Promptfoo, OpenAI Evals, or TruLens)
Working knowledge of evaluation metrics for generative AI, including hallucination rate, faithfulness/groundedness, relevance, answer correctness, toxicity/bias scoring, and semantic similarity measures
Strong proficiency in Python for writing test scripts and validation logic
Experience with SQL and data validation techniques
Test automation expertise
Preferred Prompt engineering or prompt testing experience
CI/CD pipeline automation experience
Exposure to Azure, Snowflake, or cloud data platforms
#J-18808-Ljbffr
Worksite address
bellevue, WA, 98009, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.