Measure, don't believe

>_GET /blog/measure-dont-believe

Every layer so far has ended in a claim. The manifest>_Capability manifestGET /.well-known/agent.jsonThe missing middle layer between OpenAPI and schema.org — a signed business datasheet agents can parse.defined in The end of landing pages → says it does extraction at 0.90 F1. The fit response>_Verdict per requirementMeet, partial, or miss — each outcome returned with evidence, never a single aggregate score.defined in Does this actually fit? → says it meets a requirement. The quote>_Binding quoteA signed, time-limited price — not an estimate; bindable in real time with Sam's key.defined in Pricing as an API → says the price buys a certain level of service. Each one is a promise from a seller with every reason to make the promise sound clean. Blake should not take it on faith.

The stack touches ground here. Everything above needs the sandbox, the layer that turns assertions into evidence. Blake's rule for the series, measure, don't believe, stops being a slogan and becomes a procedure. A benchmark on Blake's own data beats a case study from Sam, because Sam chose the story and Blake chooses the test.

Enterprise sales knows this dance well. Nobody signs a six-figure deal without a proof-of-concept pilot, and every sales engineer has run one. The sandbox is that pilot made programmatic: weeks of coordination compressed into a benchmark run, available for a 50-dollar subscription instead of just the six-figure accounts.

A sandbox cannot behave like a free trial. A free trial protects neither side. A useful sandbox has to be neutral and mutually secured. The reason is easier to see if you look at what each agent is actually afraid of.

The clean room solves the standoff

Both fears point to the same shape: an environment where each side can verify what it needs without exposing what it must protect. In practice, that means a clean room, a trusted execution environment (TEE) such as confidential computing on AWS Nitro Enclaves or Intel TDX:

1. Blake sends its data encrypted: the brand, the competitors, the keywords, and its labeled gold set.
2. Sam's production model runs inside the enclave, where neither side can see the other's secrets.
3. Only the agreed metric leaves, such as F1, p95 latency, or coverage, alongside a signed attestation of exactly what code and model ran.

That handles both fears at once. Blake's data stays hidden and is not retained. The attestation proves Sam ran production code instead of a demo mode. Sam's model stays sealed, and rate limits, watermarking, and a neutral evaluator stop Blake from turning measurement into extraction. The sandbox becomes Sam's new demo: Blake proves the quality on its own data instead of listening to a pitch. That is the cheapest trust Sam can offer.

Ground truth keeps the number honest

To measure "sentiment accuracy is 85%," Blake needs something to measure against. A number with no reference point is just an assertion with better formatting. Ground truth can come from one of three places:

Measurement without ground truth is a promise in a lab coat.

Pinned definitions: MLPerf for SaaS

A measurement is comparable only when the test is defined. What does "coverage" mean? If "sentiment accuracy" means an F1 score, which labels and conditions define it? Without pinned metric definitions, every vendor measures "its own way," and Blake is back in the tower of Babel, unable to put Sam next to a competitor apples-to-apples. The sandbox layer needs a shared protocol, something like MLPerf for SaaS, so a result means the same thing no matter who produced it.

Dieselgate, but for SaaS

The easiest fraud here is a real number from the wrong engine. A vendor can serve a better model in the sandbox than it runs in production, the Dieselgate move. The result therefore has to be signed and portable, and it has to prove where it came from:

{
  "benchmark_id": "bench_4c2a…",
  "subject": "did:web:sam.example",
  "test": "extraction_f1_en",
  "claimed": "0.90",
  "measured": "0.91",
  "dataset": "blake_gold_set_v3 (5,000 labeled docs)",
  "environment": "production-attested",
  "attestor": "did:web:neutral-bench.example",
  "run_at": "2026-07-22T09:14:00Z",
  "signature": "ed25519:…"
}

Two defenses make this honest. Verifiable computation attests that the result came from the production model on the agreed inputs, not a tuned demo. Surprise canaries help too: Blake periodically fires the same test at the live production endpoint and compares it to the sandbox. A gap between the two is a red flag and a reputation penalty.

Proof proportional to stakes

Sandboxing costs real money: compute, data exposure, IP risk. Blake should match the depth of the test to what is on the line, exactly like the assurance tiers>_Proportional assuranceTiered proof: did:web for low stakes, full chain for contracts — assurance scales with stakes.defined in Are you really you? → in the identity post:

• A cheap synthetic test first. No data exposure, no IP risk. It screens out most candidates before anyone spends anything.
• A representative sample. For the shortlist that survives the screen.
• A full clean-room run on real data. Only for serious candidates, right before a purchase decision.

The same principle runs through the whole stack: effort proportional to risk.

The sandbox never closes

After the purchase, Blake keeps measuring. It keeps firing the benchmark at production to catch drift: a model that quietly degraded, coverage that slipped, latency that crept up. The sandbox becomes a permanent regression test against the vendor, feeding three loops at once:

• SLA. When a result drops below the threshold, credits or penalties trigger automatically, as promised back in the pricing post.
• Reputation. Blake's signed results become a signal other agents can read.
• Re-evaluation. A sustained dip below threshold can reopen the market and trigger churn.

An agent does not buy and forget. Loyalty is a function of the latest measurement, not the original pitch.

How Blake consumes it

1. It treats the self-asserted claims from the manifest as hypotheses to disprove.
2. It runs a cheap synthetic test to screen quickly.
3. For the shortlist, it runs a clean-room test on the company's data, with its own ground truth and pinned metric definitions.
4. It demands a production-attested run to block the Dieselgate move.
5. It gets a signed result, portable proof it can keep.
6. It weighs performance against price and fit, and decides.
7. After buying, it runs a canary on production that feeds SLA, reputation, and the churn decision.

Bets worth placing

The sandbox is a bet that measurement beats a story, and you can test that bet without buying the whole thesis. A synthetic benchmark on public data costs little and tells you what breaks first. Pinned metric definitions and signed results are cheap habits that pay under any future, because they discipline the product whether or not an agent ever reads them. The expensive bet is a clean-room run on the production model, worth staging only when buyers demand proof. If they do, expect canaries. A sandbox number that fails a surprise production test is worse than a lost deal. For builders, the practical openings are clean-room and confidential-computing infrastructure, neutral evaluators, industry benchmark corpora and gold sets, and anti-Dieselgate attestation that ties a result to the production model. If this market appears, the company that supplies trustworthy measurement sits between buyer and seller.