What is an AI Evals Engineer?
EU AI Jobs ·
"Evals" is short for evaluations. More and more AI teams hire someone whose whole job is to measure whether their AI actually works. Here is what that job is, in plain words.
The short answer
An AI Evals Engineer builds the tests that show whether an AI system gives good answers.
Before a new version of an AI product goes live, their tests say whether it got better or worse. If it got worse, it doesn't ship.
Why AI needs its own kind of testing
Normal software is predictable. Give a calculator 2 + 2 and it says 4, every time. A normal test checks for exactly that answer.
AI is different in two ways:
- The same question can get a different answer each time. Both answers may be fine, or one may be wrong.
- "Good" is hard to pin down. A summary can be accurate but too long, or short but missing the one fact that mattered.
So an exact-match test is not enough. An eval checks the quality of an answer, not whether it matches word for word.
A simple example
Imagine a law firm that uses AI to summarise contracts. A wrong summary could cost a client money, so the firm hires an AI Evals Engineer, who:
- collects a few hundred real contracts, each with a summary the firm's lawyers agree is right;
- writes scoring rules with the lawyers: does the summary name the end date, the price and the notice period? Does it invent anything?
- uses a second AI model to grade thousands of answers by those rules, and checks its grades against the lawyers' own;
- runs all the tests every time the team changes the prompt or the model;
- stops a release when the scores drop, and shows the team which contracts it got wrong.
Now the team can change the AI with confidence, because they can see the effect of every change.
How is it different from other jobs?
| Role | What they work on | Main question they answer |
|---|---|---|
| QA engineer | Testing normal software | Does the feature work as specified? |
| AI Engineer | Building products on AI models | Does the product do what users need? |
| AI Evals Engineer | Measuring the quality of AI answers | Are the answers good enough, and improving? |
| Machine Learning Engineer | Training the models themselves | Is the model accurate, fast and cheap? |
Words you will see in AI Evals job ads
- Eval: a test of an AI system's answers, usually run on many examples at once.
- Golden set: a fixed collection of example questions with answers the team agrees are right. New versions are tested against it.
- Rubric: the scoring rules for what makes an answer good.
- LLM-as-a-judge: using a second AI model to grade answers by a rubric, because people can't read thousands of answers.
- Regression: a change that makes something worse that used to work.
- Hallucination: when the AI states something false as if it were true.
- Grounding: whether an answer comes from the documents it was given, rather than being made up.
- Tracing: recording every step an AI system took, so a wrong answer can be traced back to its cause.
What skills do you need?
Technical
- Solid coding, usually in Python, to build test pipelines that run on their own.
- Basic statistics: how many examples are enough, and when a change in score is real.
- Knowing how language models, RAG and agents work, and how they fail.
Product and people
- Working with experts, such as lawyers, doctors or support staff, to turn "a good answer" into clear rules.
- Explaining results so that engineers and managers can decide what to ship.
Is it right for you?
It may suit you if you like finding what's broken, you enjoy careful measurement, and you want to work close to the AI without training models. Many come from QA, data science or software engineering.
It may not suit you if you would rather build new features than test them.
Frequently asked questions
What does "evals" mean?
It is short for evaluations: tests that measure whether an AI system's answers are good enough, run on many examples at once.
How is an AI Evals Engineer different from a QA engineer?
A QA engineer checks that software does exactly what it was specified to do. An AI Evals Engineer measures the quality of answers that can differ each time, with scoring rules, sets of examples and often a second AI model as a judge.
Do AI Evals Engineers need to code?
Yes. Much of the job is building test pipelines that run on their own, usually in Python.
What is LLM-as-a-judge?
Using a second AI model to grade answers by a set of scoring rules, because people can't read thousands of answers. Its grades are checked against people's own.
Find AI jobs in Europe
Browse open AI jobs in Europe.