Delivery Benchmarks
Most delivery benchmarks are a single flattering number with no method behind it. What follows is about the metrics themselves: what release cadence actually signals, what a change failure rate reveals, and what lead time and recovery time mean when read together. Reading any vendor's benchmark well also means checking for a hidden average before trusting the headline number.
What release cadence actually signals.
Release cadence measures how often a team safely ships to production, not how fast it can occasionally sprint. A team that ships every cycle without drama has already solved the harder problems: automated testing, small batch sizes and a rollback path that actually works. A cadence that looks fast but swings wildly from cycle to cycle usually means the safe parts are being skipped under pressure, not that the team got quicker.
Cadence without rollback is just risk
A team shipping constantly with no tested rollback path is moving fast toward a worse outage, not a safer one.
Batch size explains the swings
Large, infrequent releases produce cadence numbers that swing wildly, because one bad release can skew an entire quarter's figure.
Ask what 'shipped' means
A change sitting behind a feature flag for most users is a different claim from a release live for every customer.
What a change failure rate reveals.
Change failure rate is the share of releases that need a rollback, a hotfix or an emergency patch soon after shipping. It punishes exactly the shortcut a raw cadence number rewards: releasing constantly while quietly breaking things. A vendor with a fast cadence and a rising failure rate is trading tomorrow's stability for today's demo, and that trade rarely stays hidden for long.
01Watch forA single bad release skewing it
One severe incident inside a small sample can move a quarter's failure rate far more than the underlying trend.
02Watch forSeverity hidden inside a flat count
A rollback fixing a typo and one causing an outage should not count the same way toward one flat number.
03Watch forA trend dressed up as a snapshot
One good quarter proves little on its own; ask for the trend across several cycles instead of a single figure.
04Watch forFailures quietly reclassified
A rollback relabelled as a 'planned change' understates a failure rate without a single underlying fact actually changing.
What lead time and recovery time mean together.
Lead time is how long a change takes from commit to running in production. Recovery time is how long a failed release takes to fix once something breaks. Neither number means much alone, since a short lead time can simply mean testing less carefully beforehand. Read together, they describe how a team behaves under pressure: how fast it moves and how fast it recovers when moving fast goes wrong.
How to read any vendor's benchmark critically.
A median hides its own worst week, because half the underlying sample sits below it by definition. An average is worse: one catastrophic release can drag a whole quarter's number down while the same failure barely dents a median. "On-time" delivery means little once scope has quietly moved, since hitting a renegotiated date is a different claim from hitting the original one. Each figure here fixes its definition before the measurement window opens, which is the same check worth applying to anyone else's number.
01Ask for the distribution, not the median
A vendor who shows only a median is choosing not to show the tail. Ask for the slowest release next to the typical one.
02Ask what date 'on-time' compares against
A date renegotiated twice and then hit is a far weaker result than a date that never needed to move.
03Ask how a failure gets classified
A rollback quietly relabelled as routine maintenance understates a failure rate without changing what actually happened.
04Ask for a full year, not a quarter
A single strong quarter is small enough to be luck; a full year is harder to fake or cherry-pick.
Evidence that sits next to these numbers.
A benchmark is a pattern across many engagements; a case study is one instance of it with a client's name attached. The credentials record shows how the process behind these numbers is checked from outside. The company overview lists the delivery model, the evidence base and who is accountable for each part of it.
See the company overview
The delivery model, the evidence base and the people accountable for each part of it, gathered in one record.
Read the credential record
The certification and recognition record shows how this measurement discipline is itself checked from outside the company.
See a number attached to a name
A case study carries one engagement's outcome with the client named, a sharper evidence standard than an aggregate figure.
Ask for your segment's figures
An aggregate number rarely matches one exact stack or industry, so a scoping conversation gets a more relevant one.
Cadence, failure rate and recovery time, tracked release by release.
A benchmark worth trusting carries a fixed definition, a review date and, ideally, a client willing to confirm it happened. These samples show that shape now, ahead of the datasets that will replace them once sign-off clears.
Explore case studies
Release cadence held steady while change failure rate fell.
Tern tracked release cadence and change failure rate side by side across a full engagement, one for stability and one for hidden cost.
- Median release cadence
- Weekly
- Change failure rate
- Under 1 in 20 releases
Lead time and recovery time measured together, every cycle.
Koru Health's care-record platform tracked how long a change took to ship and how fast a failed release was restored.
- Median lead time, commit to production
- Under 2 days
- Median recovery time after a failed release
- Under 4 hours
On-time delivery measured against the original date, not a moved one.
Vela Ledger's platform team logged every scope change against the original committed date instead of quietly resetting the target.
- Releases hitting the originally committed date
- 9 in 10
- Scope changes disclosed before that date moved
- Every time
Their sprint report always showed the slowest release next to the median. Nobody had to ask for it.
Sample · pending client approval
Questions worth asking about any vendor's numbers.
01Why track change failure rate instead of a simple bug count?
A bug count rewards fewer logged tickets, not fewer failures reaching a client. Change failure rate measures the share of releases needing a rollback or a hotfix, which tracks what actually happens after a release ships. It is harder to improve by writing fewer tickets, which is exactly the point of choosing it over a raw count.
02Why does a median hide more risk than it reveals?
A median only tells you the midpoint of a sample, not its shape. Half the underlying releases sit below that point and half sit above it, and the worst ones can sit far below without moving the median at all. Asking for the full distribution, not just the middle value, shows the risk a median quietly hides.
03What does 'on-time delivery' mean once scope has changed?
It depends entirely on which date is being compared. Hitting a date renegotiated twice after new scope arrived is a much weaker claim than hitting the date first committed. A vendor reporting on-time delivery should also report how often the compared date moved beforehand.
04How is lead time different from release cadence?
Cadence counts how often a team ships; lead time measures how long one specific change takes to reach production after it is committed. A team can ship on a steady cadence while individual changes still sit in a queue for a long time before release. Both numbers matter, and neither substitutes for the other.
05Why does recovery time matter if failures are already rare?
A low change failure rate says little about what happens on the day a failure does occur. Recovery time measures how quickly a team restores service once something breaks, which is the number that decides how bad a rare failure actually feels. A slow recovery time turns an infrequent event into a long outage.
06Will a specific engagement's real numbers be shared before we sign anything?
Only within what that client has agreed to disclose, and that conversation usually happens once real scoping starts rather than during a first enquiry. A number from a comparable stack or industry says more than an aggregate figure ever will. A named request gets a specific answer wherever permission allows it.
07What happens when a benchmark comes in worse than the target?
It still gets published on the same schedule, because a benchmark that only appears when convenient is not a benchmark. A weak figure triggers an internal review of the engagement behind it, and the finding changes how the next cycle gets planned rather than how the number gets reported.