20 Twilio spec pairs: 358 findings from one engine, 142 from ours, and every disagreement named

twilio · report · 2026-08-02 · source: change database

Across 20 consecutive published pairs of the Twilio OpenAPI specs — 5 API families, from twilio-oai 1.10.0 to 1.49.0 — two engines ran head to head: oasdiff, the mature Apache-2.0 comparator run as a vendored release binary, and specdiff, the zero-dependency engine inside mendapi. oasdiff reported 358 breaking units. specdiff reported 142. Only 122 units — 34% of oasdiff's count — matched at the same normalized anchor. This is the corpus in our parity series where the two engines disagreed the most, and this write-up names every single disagreement.

The accounting identity

Units are anchor-plus-kind findings after taxonomy normalization, with 0 engine errors and 0 timeouts across all 20 pairs. Both sides decompose exactly:

- oasdiff's 358 = 122 matched + 236 oasdiff-only units - specdiff's 142 = 122 matched + 20 specdiff-only units

A 34% overlap sounds alarming until you open the 236 oasdiff-only units. That is what the rest of this page does.

236 oasdiff-only units, two buckets, zero unexplained

Every one of the 236 units oasdiff reported and specdiff stayed silent about was classified from oasdiff's own finding texts. The entire set landed in exactly two disposition buckets:

- *169 format-annotation-only changes — the dominant pattern of this corpus. Twilio's spec pipeline repeatedly toggled format annotations (date-time to date on query parameters and back) without changing the JSON wire type. A client that parses the field keeps working; an engine that treats every format flip as breaking pages you 169 times for it. - 67 object-to-any type widenings* — schemas relaxed from a typed object to an unconstrained one. Widening accepts every previously valid payload, so a false-positive-rate-first engine deliberately stays quiet.

169 + 67 = 236, real gaps: 0, accounting: 100%. To be precise about what that zero means: cite it as "0 unexplained detection gaps after disposition", not as "specdiff detects everything oasdiff does". Both buckets are deliberate silences, documented in the classification files — that is a judgment about what deserves an alert, not a detection failure on either side.

The 20 units on the other side — human-verified

specdiff reported 20 units that oasdiff did not report at the same normalized anchor. Unlike the PayPal and Vercel corpora in this series, Twilio's specdiff-only units were individually human-verified as true removals against the raw spec JSON. This is the one corpus in the series where the specdiff-only side was fully adjudicated by hand, which is why the Twilio corridor carries the strongest form of that claim — and why we still cite it per corpus instead of as a blanket accuracy statement.

Re-verified, pair for pair

All 20 pairs were re-run from the cached snapshots after ingestion: 20 of 20 pairs reproduced their baseline breaking-unit counts exactly, with zero regressions and zero new unexplained units. The recheck guards this page against silent drift of the tooling; it is a statement about these cached snapshot pairs, not a forward guarantee about future Twilio releases.

What the surface is doing right now

A separate live corridor — Twilio Verify v2, spec version 1.49.0 to the current head — produced 49 raw diff records: 21 request-prop-added, 13 response-prop-added, 11 enum-value-added, 4 path-added. Breaking: 0. The Verify surface is growing (new passkey routes among the 4 added paths), not shrinking. Growth windows and breaking windows alternate per corridor; the Stripe write-up and the OpenAI honest negative show both extremes, and we do not extrapolate any of these ratios beyond the corpus that produced them.

Why this corpus matters

The 34% overlap is the whole argument for measuring instead of trusting a number. Two reasonable engines, same specs, 358 versus 142 findings — and the difference is not accuracy, it is taxonomy plus alerting philosophy. If your monitoring pages you on all 358, 236 of those pages are format churn and type widening. Alert fatigue is not a UX bug; it starts in the diff engine.

Check your own repo

Every breaking unit from this corpus is in the change database behind this site, with a fixability verdict. To find out whether your repo is exposed — file and line number, no code leaving your machine:

npx mendapi scan --repo .

The full Twilio timeline lives on the Twilio breaking-change guide. The three-provider harness this corpus came from has its own write-up, and the engine comparison methodology is documented on the when to use mendapi page.

Detected and tracked by mendapi — Dependabot for every API you depend on. Scans run locally; your code never leaves your machine.