Skip to content
XocketData Labs

3,000+ vetted experts, 10,000+ more on call

Elite experts for training frontier models

Doctors, lawyers, engineers and analysts who still practise, producing the demonstrations, rubrics, environments and evals that frontier models learn from.

Book a callFor data labs
  • SFT
  • RL rubrics
  • Agent environments
  • Computer use
  • Evals

The record, since May 2026

Tasks delivered in five months.

Every task written, solved or verified by a practitioner, then checked four times before it shipped.

Accuracy on delivered work
Experts on the bench
More on call

Recent benchmark programmes

Benchmarks we have staffed recently.

Software, terminal and research tasks, written and verified by practitioners and checked four times before they shipped.

  • Software engineering tasks01

    SWE-Bench

    Tasks delivered
    Accuracy
  • Terminal and agent tasks02

    Terminal-Bench

    Tasks delivered
    Accuracy
  • Research replication tasks03

    PaperBench

    Tasks delivered
    Accuracy

Accuracy bars are drawn on a 99 to 100% scale

The problem

Today’s models are good at producing something that looks like an answer. Real work is mostly judgement: what to check first, which number looks wrong, when a rule does not apply. That judgement lives inside practitioners, not on the internet. We get it out of them and into your data.

Why it is hard

An operations problem disguised as a research problem.

Finding two hundred practising radiologists is not the hard part. Keeping quality flat while the bench turns over, for nine months, against a rubric that keeps moving, is the hard part, and it is not the thing a data lab raised money to do.

That is the part we run.

The catalogue

Five things, and nothing else.

If a request does not land in one of these we say so, rather than learning on your budget.

01 / 05Swipe
01SFT

Supervised demonstrations

Prompt and response pairs, and the reasoning that got there, written by people who do the work for a living.

  • Prompt and response pairs
  • Chain-of-thought reasoning traces
  • Multi-turn dialogues
  • Per-item author provenance
02RL

Reinforcement learning and rubrics

Expert-written prompts with the grading framework to score them, so a reward model has something defensible to learn from.

  • Prompt sets by difficulty band
  • Rubrics with anchored levels
  • Graded completions
  • Pairwise preference data
  • Inter-rater agreement per batch
03Envs

Agent environments

API and MCP environments that behave like the real tool, with task sets and the checks that say whether an agent actually finished.

  • Containerised environments
  • MCP and REST surfaces
  • Seeded fixtures and reset scripts
  • Task suites with difficulty bands
  • Programmatic verifiers
04Trajectories

Computer-use trajectories

Recorded browser and desktop work, step by step, with the intent behind each step written down.

  • Screen recordings with event streams
  • Step-level intent annotations
  • Recovery and dead-end branches
  • Desktop and browser, Windows, macOS and Linux
05Evals

Evaluation and red-teaming

Blind expert scoring, and adversarial sets written by people who know where the real edge cases hide.

  • Held-out expert evals
  • Blind double-scoring
  • Adversarial and edge-case sets
  • Failure reports with expert commentary

How we operate

Pods, not a queue.

Each project runs as pods of five: four contributors and one lead with 10+ years in that exact field. Every five pods share three seniors who audit across them, so no pod marks its own homework.

The full process
3 seniors audit every 5 pods
1 lead, 10+ years in the field4 contributorsSenior auditor

A SWE-Bench programme, for example, starts as one pod: four engineers writing and solving tasks, led by a software engineer with over a decade in production systems. We add pods as volume grows, never by stretching one.

Four checks on every task

  1. L1

    Blind peer review

    Inside the pod

  2. L2

    Lead expert re-check

    Pod lead, 10+ years in the field

  3. L3

    Senior cross-pod audit

    3 seniors for every 5 pods

  4. L4Optional

    Partner audit

    Your team, when you want it

L1 also screens for AI-written work, the bottleneck every data lab is fighting right now. A task that reads like a model wrote it goes back, whoever wrote it.

The bench

10 domains, staffed by people who still practise.

A tax expert writing transfer-pricing traces files returns for a living. The moment someone stops practising, they stop being the thing the model needs.

How we vet
Where the bench works
  • Software and tech45%
  • STEM, medicine and life sciences30%
  • Finance15%
  • Languages, video and other10%
  1. 01

    Medicine and clinical

    • Physicians
    • Nurses
    • Radiologists
    • Clinical coders
  2. 02

    Law

    • Litigation
    • Corporate and M&A
    • Intellectual property
    • Compliance
  3. 03

    Accounting and tax

    • Statutory audit
    • Direct and indirect tax
    • Transfer pricing
    • Controllership
  4. 04

    Finance

    • Equity research
    • Credit analysis
    • Trading operations
    • FP&A
  5. 05

    Software and data

    • Backend
    • Infrastructure and SRE
    • Analytics engineering
    • QA automation
  6. 06

    Engineering

    • Civil and structural
    • Mechanical
    • Electrical
    • Process and quality
  7. 07

    Supply chain and trade

    • Freight and logistics
    • Customs brokerage
    • Import and export documentation
    • Procurement
  8. 08

    Insurance

    • Underwriting
    • Claims adjudication
    • Actuarial support
  9. 09

    Pharma and regulatory

    • Clinical trial operations
    • Regulatory submissions
    • Pharmacovigilance
  10. 10

    Business operations

    • Recruiting
    • Customer operations
    • Sales operations
    • Bookkeeping

Two tracks

Behind your contract, or under ours.

Most of our work is invisible by design. A data lab wins the contract on its platform and research; we supply the bench behind it.

For data labs

Partner delivery

You have the contract. We supply the bench, work inside your spec in whichever system you choose, and stay invisible.

Read more

For frontier labs

Direct engagements

A scoped programme run end to end: experts, environments, rubrics, QA and the audit trail behind it.

Read more

Tell us the domain you are stuck staffing.

One call, and you get a straight answer about whether we can take it. If we cannot, we will say so on the call rather than three batches in.

Book a callFor data labs

Prefer to read first? See who runs Xocket Data Labs or every way to reach us.