About this opportunity
Horizontal Talent lists this AI Operations Specialist opportunity in san francisco, California. Review the employer’s description below for duties, qualifications and application requirements.
Job description
AI Operations Specialist Remote 12 Months
A typical day:
Help the team decide which projects will drive the largest business impact, and make the case for the ones worth doing.
Sit with the team that owns a problem, map how the work gets done today, and work out how AI can do it and where people step in for the hard cases.
Build the AI workflow: write the prompts, set up the pipeline, and keep it at the agreed quality level, watching for drift once it’s live.
Build the test sets, measure accuracy, and calibrate LLM judges against human reviewers.
Support product teams in agreeing on what “good enough to launch” means for an AI feature, and bring the evidence for the go/no-go call.
Turn unclear policy into criteria a reviewer can apply consistently, working with product, Legal and Policy.
Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
Run labeling at scale with vendor agents, and check their work against agreement metrics and your own spot checks.
Make what you build reusable and document what you learn, so the next team gets to production faster.
Keep our methods current as models, tools and vendor agents change, and retire what stops working.
Own the workstreams you’re assigned, from setup through quality checks. Model infrastructure and product roadmaps stay with the teams that own them.
Skills
5-7 years in operations, program management, analytics, consulting or applied AI, where you were responsible for results.
You use AI every day and have built something with it that has actually moved the needle, such as a workflow, an automation or an analysis. You know where it breaks, and you keep pushing it further.
You’re comfortable with numbers: accuracy, precision and recall, sampling. SQL and notebooks help. If you haven’t used them yet, you’ll pick them up quickly with AI assistance.
You’re curious about how work actually gets done, including the awkward exceptions, and you’ve taken a messy process and made it measurable.
You’ve worked on quality in some form, such as review guidelines, QA or audits.
You write clearly and can work with business leads, PMs and engineers in the same week.
You’re fine working with limited direction, and you try things yourself instead of waiting for someone to build them. A technical background isn’t required.
You don’t need to have designed evaluations or calibrated LLM judges before. We’ll teach you, and you’ll work with a team that has done it many times.
Preferred experience
Hands-on SQL, Python or API experience.
Trust and safety, content moderation, annotation operations or other human-in-the-loop work at scale.
Fine-tuning models or calibrating LLM judges.
Background in a two-sided marketplace, travel, or another operations-heavy business.
Experience setting up intake and prioritization for a new team.
For those that join the team, we offer competitive compensation and benefits including medical, dental, vision, and retirement.
#J-18808-Ljbffr
Worksite address
san francisco, CA, 94199, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.