Research Scientist, Agentic Data & Benchmarking

The Agents team trains advanced agentic language models that use reasoning and tool use to complete real tasks on a computer. This is a specialist role at the center of the loop that drives those models: the data we train on and the benchmarks we measure against. 

You'll own the agentic data pipeline end-to-end — sourcing and generating high-quality trajectories, tool-use data, and RL environments — and the evaluation suite that tells us, rigorously and reproducibly, what our agents can actually do. These two halves are inseparable: benchmarks expose where models fail, and targeted data closes the gap. The agents are only as good as the data they learn from and the evals that keep us honest, and this role owns both. 

This is a research scientist position for someone who wants depth in data and measurement rather than breadth across the whole stack. You should be the kind of person who reads through datasets line by line, distrusts a metric until it's been validated, and gets satisfaction from making an eval suite that nobody questions. 

• BS, MS, or PhD (or equivalent experience) in Computer Science, Machine Learning, or a related field. 

• 2+ years of experience with a clear emphasis on evaluations and/or training-data curation for ML systems (related areas: LLM training/fine-tuning, RL, or distributed ML systems). 

• Strong Python and PyTorch development experience. 

• Demonstrated experience designing and deep-diving into evaluations, or curating and generating training datasets — ideally both. 

• Hands-on experience using LLM agents in your personal or professional work. 

• A habit of reading through raw data and trajectories to understand them and spot issues, and an instinct to distrust a metric until it's validated. 

• Experience with reinforcement learning, reward design, or RL environment construction for LLMs. 

• Background in statistics and experimental design — a feel for signal-to-noise, statistical power, and contamination in evaluations. 

• Experience with large-scale dataset sourcing, curation, and processing, including working with external vendors or domain experts. 

• Strong knowledge of the literature on agent evaluation, RL, LLM reasoning, and tool use. 

• Experience building or operating data pipelines and evaluation infrastructure reliable at scale (e.g., PyTorch, Ray). 

• Experience evaluating or generating data for software-engineering or computer-use agents. 

• Contributions to published research, public benchmarks, and/or open-source ML software. 

We encourage you to apply even if you don't meet every qualification listed. Strong candidates rarely match every line, and we'd rather hear from you than have you rule yourself out.