Job description
Description
You ship a benchmark every two to three weeks (example benchmark). Each one measures a frontier risk that nobody has measured yet. Some go public. Some go only to the labs.
Some of the benchmarks and papers are done in collaboration with the leading AI Labs and universities.
You will not write every eval yourself. Each benchmark pairs you with an in-house researcher who owns that harm area, and you get a budget for freelancers you direct. You own the taxonomy, the harness, the quality bar and the release.
The seat sits in the CTO office alongside the research lead who sets our public research agenda. Around 150 researchers here work on harms directly, and you can pull any of them onto a subject.
Key Responsibilities
1. Ship the benchmark cadence
A benchmark every two to three weeks. Size follows the subject. A chat-based taxonomy can carry 100 evals. An agentic or GRPO benchmark is closer to 20, because each one is expensive to read. Sensitivity decides what ships publicly and what goes to the labs alone.
2. Own the quality bar
The test is simple. A frontier lab reruns our set and gets our numbers. The verifiers hold, the rubrics are clear, the distribution is sane, and their subject matter experts read the taxonomy and call it novel.
That means you read the evals yourself. You can screen with a model, and you still open the file, spot the item that does not match the taxonomy, and push the researcher back on it.
3. Run the process
Hold the plan and the calendar. You are the person who keeps other researchers on timeline. You can direct two or three freelancers (SMEs) yourself ad-hoc when needed.
4. Set the roadmap with the forum
Roughly monthly you sit with the CTO and the pod and research leads. Inputs are what our research teams see, what clients are asking for, and what is moving in the news. Output is a revised release plan for the quarter, tied to the accounts we want to open.
5. Stay ahead of the curve
Around 20% of your time goes to the ecosystem. Read the research, keep contacts inside the labs, ask them what is bothering them, and travel to a couple of conferences a year. You should be talking to folks from the labs weekly.
Requirements
Requirements
PhD or Masters in computer science, machine learning or a related field, or equivalent depth from industry research.
3+ years building and running safety or security evaluations for language models in production, at an AI lab, a model provider, or a safety and security research organisation.
5+ relevant research publications in the field of AI safety and security including lead author on at least 2 of them
Strong engineer. Evaluation harnesses, distributed inference, vLLM, reading a codebase and fixing it.
You can build a taxonomy, not only score against one.
You can direct a researcher and two freelancers without managing them formally.
Strong English, written and spoken. This role runs on internal communication across time zones.
Curiosity about the harms themselves. You will be learning a new subject every three weeks.
Ideally:
Post-training experience: SFT, DPO, GRPO. Reward design for subjective and safety-relevant targets is a live problem for us.
Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
Publications at top conferences.
Willingness to present your own work on a client call. Strong communication - both verbal and written, ability to present to large and/or senior audiences
Travel to conferences at least 3 times a year
About Alice
Alice is a trust, safety, and security company built for the AI era. We safeguard the communicative technologies people use to create, collaborate, and interact- whether with each other or with machines.
Alice clients are the top 8 AI Labs in the world.
In a world where AI has fundamentally changed the nature of risk, Alice provides end-to-end coverage across the entire AI lifecycle. We support frontier model labs, enterprises, and UGC platforms with a comprehensive suite of solutions: from model hardening evaluations and pre-deployment red-teaming to runtime guardrails and ongoing drift detection.
Alice is widely considered a global leader in online safety and AI security. We have some of the most forward-thinking and passionate minds in the world working to safeguard over 3 billion users across the largest AI and tech platforms.
If you're creative and driven to secure the future of AI, we want to hear from you!
Originally posted on Himalayas
Who can apply
Eligible countries: United States. Accepted UTC offsets: UTC-10, UTC-9, UTC-8, UTC-7, UTC-6, UTC-5, UTC+14. Review the full description for employer-specific work authorization, residency and schedule requirements.