Does the scaling law of miscalibration hold on production traffic?
Does the relationship between model scale and miscalibration reported on benchmarks also appear on real customer traffic?
Research programme
Research
All eleven papers are submitted for peer review. None has been accepted yet.
We run a small research programme on the questions our production systems force on us: how to train small models without losing the long tail, how to measure them so the number means something, and how to keep production data safe while learning from it. Every paper starts from a gap we found in the literature and an instrument we built to test it.
What we track
Where the field is moving, rate-normalised
Our arXiv tracker mirrors 152,000 AI and quantum papers with full history and detects surges rate-normalised against the field's own output. Fastest-growing clusters, February–August 2026 versus the prior six months:
- ×26Agent scaffolding and skills
- ×10On-policy distillation
- ×5.7Agent memory and its security
- ×4.3Coding agents
Rate-normalised growth, cluster level, deduplicated. Source: our tracker, corpus of 152,157 papers, 52 of 52 harvest windows complete.
Papers
Eleven questions, four themes
Each card is one paper and the question it asks. All eleven are submitted for peer review.
01Training small models
Does on-policy distillation sacrifice the long tail?
When a small model learns from a larger one on its own outputs, does it lose the rare cases?
Bounded null result
How much of the on-policy distillation literature survives a change of seed?
If the random seed changes, how many of the reported gains remain?
Two thirds of a literature's claims cannot be compared to anything: a reporting census
How many published claims report enough detail to be compared with any other result?
02Measuring models honestly
Is benchmark-contamination inflation class-structured?
Does contamination inflate scores evenly, or only for some classes of items?
Reproducibility is a property of the job: evaluation-order effects in vLLM
Does the score of a vLLM evaluation depend on the order in which the job processes its items?
Does de-telling defend or evade? What AI-text detectors actually measure
When the tell-tale features of AI text are removed, do detectors still measure authorship, or only style?
03Learning from production data safely
Scrub scope, not just scrub accuracy: field coverage and name recall in production training-data capture
When production data is scrubbed for training, are the right fields covered and are personal names caught?
Production traffic as the correctness oracle for agent-driven legacy migration
Can recorded production traffic decide whether an agent's migration of a legacy system is correct?
04Voice and matching in the field
Round trip certifies injectivity, not correctness: why a text-normalisation gate passed the defects a partner found
Why did a text-normalisation gate in a production Greek telecom voice agent pass defects that a partner later found?
How many questions does a match need? Elicitation budget as an evaluation axis
How many questions should a system ask before it proposes a match, and should that budget be scored?
How we work
Three rules
Gap first
Every week we scan the corpus for gaps, and run a control-cell test so that an empty dimension is not mistaken for an opportunity.
Instrument before claim
Every paper ships a runnable experiment and its tests. Corrections stay visible, and a falsification counts as a result.
Open by default
Preprints on arXiv, instruments released, and an AI Act (Regulation (EU) 2024/1689) dossier for every model we ship.
Collaborate with us.
We are open to joint measurements with academic groups and industry research labs.