AI Quality Engineer

The Role

You'll work across our evaluation systems and test infrastructure — the machinery that lets us ship non-deterministic AI output with confidence. Evaluate, automate, dig, improve.

• Enhance and run our layered eval harnesses — automated checks, LLM-as-judge scoring, rubrics, targeted human review

• Maintain versioned golden datasets so evaluation stays reproducible and auditable

• Track the signals that matter for investigative output: groundedness, hallucination rate, entity resolution, source quality

• Run and improve a tiered CI model built on Python, Playwright and pytest — six authenticated personas, three browsers, real multi-tenant coverage

• Dig into flaky tests and wobbling metrics and fix them at the root

We use Claude Code by default — selector sync, failure triage and eval scoring are AI-assisted end to end. The craft is knowing when a green tick is lying.

What we're looking for

Two doors here. You're a proven senior who can do most of this today — or you have the foundations and the drive, and we'll grow you into the rest. Both doors are real.

• Strong test-automation engineering. Python (or comparable) with a modern stack — Playwright, pytest or similar — and fluent in CI/CD. Tests as infrastructure, not scripts.

• Analytically sceptical and numerate. You question baselines and never take a green tick at face value.

• You make complex things simple. The same wobbling metric explained to an engineer, a PM and a customer-facing lead — and understood by all three.

• A collaborator. Quality cuts across squads you don't manage — you build trust with the engineers whose work you're testing.

• A fast learner with proof. You've gone novice-to-competent in something genuinely hard, under real delivery pressure.

• Genuinely interested in AI quality. You can articulate why testing non-deterministic output is a different — and more interesting — problem.

• (Senior door) You've shipped LLM evals — rubric design, LLM-as-judge, regression harnesses — and raised the quality bar across teams that didn't report to you.

• Good to work with. Small team — it matters.