What 47 Stripe API versions taught us about breaking-change detection
Between v2100 and v2345, Stripe shipped 47 adjacent OpenAPI spec revisions. We replayed every one of those pairs through two diff engines, head to head: oasdiff (the mature Apache-2.0 comparator, run as a vendored release binary) and specdiff (the zero-dependency engine inside mendapi). Same specs, same corridor, no gaps. This is the full accounting.
The setup
Each pair ran both engines under a 900-second hard timeout. Every divergence and every engine error was adjudicated separately, with an independent double-sided walk of the raw specs before it was counted. Findings that were never adjudicated are never cited as an advantage. The complete per-pair reports and adjudication files are archived alongside the change database this site is generated from.
The accounting identity
The 47 pairs decompose exactly, with nothing left over:
- 39 clean pairs where both engines agreed on zero breaking changes - 1 full-parity pair where both engines reported the identical 8 breaking units - 4 pairs with measured divergences, each adjudicated to a root cause - 3 pairs where oasdiff did not complete within 15 minutes across multiple independent runs
39 + 1 + 4 + 3 = 47. Every pair is in one bucket.
The headline numbers
On the 44 pairs where both engines completed, specdiff had 0 genuine detection gaps against oasdiff after adjudication, and 0 false positives across every adjudication in the corridor. Slowest specdiff run observed in the cited adjudications: under 17 seconds on specs approaching 10 MB.
The number that matters most for anyone gating CI on a breaking-change check: 163 real breaking units in this corridor are invisible to a default oasdiff breaking gate (level 3 and above). Not because oasdiff is a weak engine. It is an excellent one. The gap splits into two mechanical causes:
- 58 units by severity classification. oasdiff detects the same removals but files optional-response-property removal and anyOf-nested request removals as level-2 warnings. Existing callers break regardless of the optional flag: when POST /v1/terminal/refunds dropped 7 request properties, every caller still sending amount or payment_intent broke, and a default gate stayed green. - 105 units by engine availability. On 3 pairs oasdiff hit the 900-second timeout on every attempt. The largest of those pairs (v2200->v2210) carried 96 adjudicated-real breaking units across 5 root-cause families; specdiff processed it in 14.6 seconds. A gate that times out is a gate that did not run.
Alert fatigue is the other failure mode
One pair (v2182->v2183) is worth calling out separately. A single $ref re-anchor in Stripe's payment_record schema fanned out to over a thousand level-3 findings in oasdiff, across 586 transitive anchors. specdiff reported the same change as 10 alerts at the canonical anchors and deliberately stopped at transitive embeds. That is not a correctness claim against oasdiff; it is a design choice about what a human on call can actually read. One root cause should page you once, not a thousand times.
What we are not claiming
Honesty notes, verbatim from the evidence file: severity divergences must be cited as "oasdiff classifies as warning", never "oasdiff missed it" — both engines detected the identical change sets. Runaway pairs are availability datapoints, not detection failures. And this corridor covers one provider and one spec group; we do not extrapolate these numbers to other providers. Each provider gets its own measured corridor before we publish claims about it.
Check your own repo against this corridor
Every breaking unit in this corridor is in the change database behind this site, with a fixability verdict. To find out whether your repo is exposed — file and line number, no code leaving your machine:
npx mendapi scan --repo .
The full Stripe timeline, including this corridor and the later v2345 to v2366 window, lives on the Stripe breaking-change guide. The engine comparison methodology is documented on the when to use mendapi page.
Detected and tracked by mendapi — Dependabot for every API you depend on. Scans run locally; your code never leaves your machine.
Related
- stripe breaking-change guide — every tracked stripe entry on one page, with fixability verdicts.
- Stripe Payment Records migration — the full migration guide with pack-backed fixes.
- All monitored providers — live change counts for stripe and 19 others.
- Migration pack catalog — every deterministic fix mendapi ships.
- Getting started — scan your repo in 30 seconds, no code leaves your machine.