AI Researcher - Pre-training (f/m/d)

Your Responsibilities

• Recipe & Architecture Experiments: Design and run ablations on core elements of the training recipe (optimizers, schedules, initialization), and implement PyTorch-based architectural changes. Evaluate their effect on convergence, stability, and training efficiency.

• Scaling Experiments: Run and analyze small-scale proxy experiments, and contribute to the hyperparameter scaling laws and scale-up methods we use to predict behavior at multi-thousand-GPU scale.

• Run Monitoring & Debugging: Help monitor large-scale training runs and investigate convergence issues such as loss spikes and divergence. Work with the team to track down distributed training failures.

• Research to Practice: Follow the literature, reproduce promising results at small scale, and bring ideas worth testing to the team.

• Cross-Team Collaboration: Work with the Compute Performance, Data, Evaluation, and Post-Training teams to understand how hardware constraints and the rest of the model lifecycle shape pre-training decisions.

Core Qualifications

• You are proficient in Python and have hands-on experience with PyTorch.

• You have a strong foundation in machine learning, shown through research (e.g., a PhD, a strong MS'c thesis, or publications), open-source contributions, or substantial professional or substantial personal projects.

• You have a solid mathematical background (linear algebra, probability, optimization) and enjoy reasoning formally about why models train the way they do.

• You understand how transformers work and have pre-trained LLMs from scratch yourself, even at small scale.

• You design experiments carefully, report noisy and negative results honestly, and can explain clearly what your evidence does and does not show.

• You have a track record of picking up unfamiliar technical areas quickly and going deep, for example by reimplementing papers, ramping up on a large new codebase, or debugging a problem well outside your comfort zone.

• You write clean, maintainable code and care about reproducible experiments.

• You communicate clearly, raise problems early, and actively seek out and act on feedback.

Preferred Qualifications

• Distributed Training: Experience training models on multi-GPU or multi-node setups, or familiarity with distributed frameworks (e.g., Megatron-LM, DeepSpeed, torchtitan) and parallelism strategies (data, tensor, pipeline, expert).

• Training Dynamics: Familiarity with scaling laws, hyperparameter transfer (e.g., µP), or recent work on optimizers and training stability.

• Efficient Architectures: Exposure to Mixture-of-Experts, efficient attention variants, or other inference-efficient designs.

• Research Output: Publications at top-tier venues (NeurIPS, ICML, ICLR) or contributions to open-source ML training libraries.

• Systems Curiosity: Low-level kernel optimization is not required, but we highly value a strong curiosity about the hardware and systems constraints that shape scale.