monit.rs · Zenofex LLC
Our API-stability score rewarded batching breaking changes and missed their size. Deepgram was about to be named worst offender for a spec cleanup, not a break. The fix, and every corrected score.
Why every stability score on monit.rs changed today
TL;DR: Our public API-stability score (the "Steady · 85" style badge on every vendor's changelog page) was measuring the wrong thing since we launched it: it counted how many weekly snapshots contained a breaking change, not how many breaking changes there were, so a vendor who batched every break into one release scored better than one who spread the same damage over two. Before publishing the rebuilt version we hand-checked the vendor it was about to name as the worst offender — Deepgram — and found its "49 breaking changes" were a single spec-correctness fix, not damage at all. We're publishing corrected scores for all 47 tracked APIs today. Eighteen vendors with real breaking changes over the last 90 days now hold 18 distinct scores; the old formula gave those same 18 vendors only five values to share between them.
See it live: every score below is on the vendor's own page at monit.rs/changelog — e.g. Deepgram, Vercel, Cloudflare.
The badge read something like "Steady · 85" or "Volatile · 0." The formula behind it was max(0, 100 - N × 15), where N was supposed to stand in for "how much a vendor's API broke." It didn't. N was the count of weekly spec snapshots that contained at least one breaking change — not the count of breaking changes.
Specs are polled roughly once a week, so in practice the score mostly measured how many separate weeks a vendor happened to touch their spec in a breaking way. That has three consequences, and none of them are subtle once you say them out loud:
- Magnitude was invisible. A snapshot with 1 breaking change and a snapshot with 216 breaking changes counted identically: one dinged week. - Batching was rewarded. Ship every break in a single release and you lose one week's worth of penalty. Spread the same two changes across two weeks and you lose twice as much. The formula punished vendors for being incremental and rewarded them for dumping everything at once. - The scale barely used itself. Across every vendor we track, only five scores ever occurred in practice: 0, 25, 70, 85, and 100. A 100-point scale that resolves to five buckets isn't telling you much.
Under that formula, Deepgram shipped 49 flagged breaking changes across its 50-operation API in a single snapshot and scored "Steady · 85" — the second-best band we had. In the same window, Cloudflare's churn was proportionally far smaller, spread across nine separate weekly snapshots, and it scored "Volatile · 0." The metric was about to tell our users that the API with the most alarming-looking single event was more trustworthy than the one that had been shipping small, steady changes all along. That's backwards, and it was going to be a permanent, embeddable badge with a company's name on it.
So before we shipped the new methodology, we went and read all 49 of Deepgram's findings by hand.
What we found: Deepgram wasn't the problem
All 49 findings were the exact same message — Parameter removed: Authorization (header) — spread across 49 of Deepgram's 50 operations, in one snapshot. That is not 49 independent breaks. It's one change, applied consistently everywhere: Deepgram pulled Authorization out of being declared as a per-operation header parameter, almost certainly moving auth to securitySchemes, which is how OpenAPI is supposed to represent auth in the first place.
That's a spec-correctness cleanup, not a breaking change. We were about to publish Deepgram as the worst API we track in the same release where we should have been crediting them for improving their spec. We're glad we checked before that badge went out with their name on it.
The new counting rule still isn't perfect here, and it's worth saying plainly. Under least(distinct operations, distinct messages), those 49 findings collapse to a single counted action — that's the fix working as intended. But that one action is still the auth cleanup itself: it's the only breaking finding on Deepgram's books in the last 90 days, and their score of 60 is built entirely from it. Our diff engine currently flags a removed required header parameter as breaking regardless of why it was removed, and it can't yet distinguish "moved to securitySchemes" from "gone for good." So Deepgram's 60 is still a penalty for a spec improvement — a proportionate one now instead of a catastrophic one, but a penalty on a non-break all the same. Teaching the engine that distinction is a real follow-up; it isn't something this rewrite fixes.
Digging into Deepgram surfaced a second problem the old formula never had to face, because it never looked at individual findings at all. A pile of findings in one snapshot can fan out in two different, independent ways:
- One operation, many fields. Vercel restructured two response bodies in one release — 216 field-level changes on each of two operations. That's really two changes, not 432. - One field, many operations. Deepgram's Authorization removal touched 49 operations with one message. That's really one change, not 49.
Collapsing on only one of those axes still leaves the other one inflated — count by operation and Deepgram still looks 49x worse than it is; count by message and a vendor with 200 distinct one-off field removals on a single endpoint looks like a single change. So the new formula counts, per snapshot and per change kind, least(distinct operations, distinct messages). Vercel's restructure now correctly counts as 2. Deepgram's auth cleanup now correctly counts as 1.
where r is distinct breaking changes (by the counting rule above) per 100 API operations, over a trailing 90 days. Dividing by operation count means a 3,000-operation API and a 50-operation API are judged on rate, not raw count — Cloudflare's enormous surface no longer gets penalized just for being enormous, and a small API's one real mistake isn't buried in a rounding error either.
Bands: Stable 80–100, Steady 55–79, Active 30–54, Volatile 1–29. APIs with fewer than 25 tracked operations are shown as Unrated instead of a score — on a small surface, one breaking change swings the rate too far for a number to mean anything. And the score never actually reaches 0: it's a measurement, not a verdict, and a published zero next to a named company reads as the latter.
Eighteen vendors have shipped at least one real breaking change in the last 90 days. Under the old formula those 18 vendors shared just five possible scores. Today they hold 18 distinct ones. A few, as of this post:
Every one of these is live on the vendor's own changelog page right now, along with the rate and window the score is built from — not just a label.
We are not going back and rewriting earlier posts that cite v1 scores. The API drift report and our other August posts stand as dated records of what the old formula said at the time; they're marked by date and the methodology has moved on since. This post is the record of the change.
If you're a vendor and you think a score misrepresents your API — including one from before today — get in touch. That route existed before this rewrite and it still does.
The figures on this page come from the same diff engine monit.rs runs against customer endpoints. Point it at your own API and it works the same way: it learns each response schema by example, diffs every subsequent response against that baseline, and alerts you when a breaking change ships. The free tier is 3 endpoints with the full feature set and no card required. Start monitoring free.
monit.rs compares observed response fields and types with a learned baseline. Run OpenAPI contract checks against the latest stored response when you want to compare the implementation with a declared contract.
See the difference when a field disappears or changes type. Optional AI summaries suggest possible causes and next steps; review the evidence before accepting a new baseline.
monit.rs also runs a public changelog of 47+ well-known API vendors (Stripe, GitHub, OpenAI, Cloudflare, Datadog, PostHog, Vercel, Supabase, and 39 more). Every detected OpenAPI change is published at /changelog with per-vendor RSS feeds, stability badges, and side-by-side comparison pages.
See how we compare with existing monitors: vs Pingdom · vs Checkly · vs Uptime Robot · vs Statuspage.io · vs Bump.sh · vs Optic · vs Postman Monitors
3 endpoints. Full features. No credit card required.