Talk to us
Benchmark engineering for agentic AI
The tasks, environments and audits that tell frontier labs what their agents can actually do.
Agent trajectory
$
An agent takes the shortest path to reward. If the grader can be reached, it will be reached. If the answer leaks, it will be found. So we build the task, hide the verifier, and attack it ourselves before anyone else can.
What we ship
01
Execution-based, containerised, delivered in your schema with a hidden verifier and a reference solution on every task.
02
Reward set by a verifier the policy never sees, hardened against reward hacking, calibrated by repeated trials to the band you order.
03
Your existing suite, attacked the way a capable agent would attack it. Every finding comes with a reproduction and a fix.
tests/fixturesfixedHow every task is built
We scope the capability you want measured, the harness it must land in, and what counts as passing, before anything is written.
Working engineers write the task, the verifier and the reference. The reference passes. An empty submission scores zero. Same input, same score, every run.
Our auditors attack it the way a capable agent would: grader access, answer leakage, unintended reward paths. What we find in our own work is what you never see in yours.
Difficulty is measured across repeated trials, never asserted from a read-through. The suite ships into your infrastructure with provenance per task.
Domains
Long-horizon, execution-based tasks in the fields frontier labs need measured, each written by people who work in the field.
Application development, debugging, build systems, software tooling
Software and systemsConsensus, concurrency, memory management, congestion control
Software and systemsQuery optimisation, streaming pipelines, lakehouses, temporal data
Software and systemsGPU compute, CPU vectorisation, RISC-V, numerical kernels
Software and systemsKubernetes, incident diagnosis, service reliability, infrastructure troubleshooting
Software and systemsBehavioural reconstruction, compatibility, protocol and CLI reimplementation
Software and systemsSecure APIs, denial-of-service resilience, vulnerability repair
Software and systemsDistributed training, gradient accumulation, reward-model training
AI and machine learningTokenisation, quantisation, decoding, scheduling, cache management
AI and machine learningEvaluation harnesses, programmatic grading, judge aggregation, reproducibility
AI and machine learningComputer vision, NLP, speech and audio recognition, recommendation systems
AI and machine learningNumerical analysis, optimisation, proof certificates, theorem proving
Science and dataQuantum control, trapped-ion systems, gravitational-wave analysis
Science and dataElectrochemistry, battery modelling, spectroscopy, crystallography, kinetics
Science and dataGenomics, structural variation, genome reconstruction, systems biology
Science and dataStatistical modelling, hypothesis testing, visualisation, analytical debugging
Science and dataCausal inference, panel analysis, policy evaluation, economic data
Science and dataRTL, Verilog, FPGA workflows, arithmetic circuits, communication protocols
Hardware and CADParametric modelling, sheet metal, injection moulding, additive manufacturing
Hardware and CADStructured credit, collateral management, valuation, NAV restatement
Operations and financeReinsurance, catastrophe claims, marine insurance, claims adjudication
Operations and financeSanctions screening, ownership attribution, origin determination, emissions reporting
Operations and financeSafety-case triage, adverse-event reporting workflows
Operations and financeRail-yard planning, hazardous cargo, maritime stowage, disruption handling
Operations and financeCold-chain management, traceability, allocation, sourcing
Operations and financeMarketing-mix modelling, incrementality, bidding, budget pacing, reconciliation
Operations and financeTypography, font engineering, animation, keyframe reconstruction
MediaSignal processing, mastering, music theory, counterpoint
MediaEvery task is written by someone qualified to grade it.
Working engineers and domain experts, matched to the category that fits their depth. Not a crowd.
Or write directly to team@starzo.ai