In the last five years since 2021, Google has ambitiously tried to push the envelope forward in defining which software metrics they think best describe the health of a company’s devops efforts. The result of their research was a treasure trove of benchmarking stats, so any company can assess whether their dev performance is industry-leading or lagging.
Research from Google on the four critical DevOps metrics, including sample values to place one's own team
Google’s team of researchers identified four determinants of success:
The first stat they recommend tracking and optimizing is “release frequency.” This is the most fundamental metric, as “releases” or “deployments” are the underpinning of all four DORA metrics. This makes sense — if you are not shipping code to production regularly, improving that opens the door to faster cycle time and repair time (discussed in more depth below).
According to Google’s benchmarking, an Elite team will make a release of their product at least once per day.
To set up release tracking, please view our Release tracking setup page.
In terms of applying Google’s release frequency benchmark to your own real world team, one must start by interpreting what constitutes a “product” or “business unit,” since companies vary widely on how many repos, projects, and team members they are grouped into. Our recommendation for simplicity is to strive to make one release per day per team. GitClear is set up to allow grouping developers, managers, and leaders/executives so it matches how employees work together in the company.
When you view Release Frequency on the Release Velocity page, you will see it grouped by team. This makes it straightforward to check the cadence of releases per functional team at the company.
This metric captures the percentage of all releases that included a defect that was later deemed “critical” or “substantive.” The highest performing teams will have critical defects found in less than 15% of their releases. A sampling of the strategies that can be used to reduce defect releases:
Thorough testing: preferably automated, but otherwise via QA. The best case outcome when a defect is introduced is that it will be found before a product release puts it in the hands of customers. A robust testing strategy is the key to getting bugs detected promptly, before a release occurs.
More frequent releases. The more time passes between releases, the more difficult it becomes to prove that there won’t be an undetected failure in how newly added systems interact.
What gets measured improves. 🙂 The act of simply making your defect release percentage transparent is a first step toward improving it.
There are two components to tracking defect release percentage. The first is tracking your release, which can be set up following the instructions on our Release Tracking setup page. The second is tracking when critical defects occur, which can be set up following the instructions on our Critical Defect Tracking setup page.
GitClear offers two options to record when a critical defect has occurred. The default option is via your issue tracker (Jira and GitHub issues currently supported). In your Issue Tracker Project settings, you can specify one or more issue tracker fields that demarcate a critical defect. By default, if you are connected to Jira and you have a. “Highest” value for your “Priority” field, we will interpret that as a critical defect. Also, if you have another field whose values contains “Critical” (e.g., a “Labels” field), that will be automatically detected as a critical defect.
If you have other conventions for designating a critical defect, you can specify this in the Issue Tracker Project settings.
The second option to designate a critical defect is to make a call to the GitClear API. Please email firstname.lastname@example.org if you would like to receive instructions on submitting critical defects via a RESTful API call.
The “Defect Release Percentage” that we show in the “Issues & Defects” tab shows what percentage of releases have subsequently had a critical defect detected in them. Note that a critical defect can span multiple repos (for example an “API” repo and a “front-end” repo), so if two repos made releases that contained the same critical defect, that will count as two defects released from the standpoint of the “Defect Release Percentage” calculation.
How many days does it take to release a fix when a critical defect is detected? Top performing teams average less than one hour between when a critical issue is detected and when there is a release that is made that deploys a fix for it.
Unlike “Release Cycle Time” and “PR Merge Cycle Time,” the MTTR measurement does not factor in business hours. For example, if a bug is filed at 12:00pm that is marked as a critical defect per Critical Defect Tracking setup, and a fix for the defect is released (as detected by Repo Release settings) the next day at 9am, then that defect would be calculated to have been repaired in (21 hours / 24 hours) = 0.875 days.
A defect is designated to have been “repaired” when a commit, PR or branch references an issue that was designated to be a critical defect, and the commit with that work is subsequently released. For example, if a critical defect is submitted to Jira at 12pm, a fix is authored at 9am, and that fix is deployed at 12pm the following day, then the “Time Until Defect Repair” would be exactly 1.0 days.
Time Until Defect Repair does not consider issue status, since developers are unreliable in the time that they will change a bugs’ status to “Resolved” relative to when that resolved work is deployed. Note this metric does not consider a defect “repaired” until the final commit that references it. So if, in the above example, subsequent to the 12pm release, another commit follows at 3pm that is released at 5pm, then the “Time Until Defect Repair” would be recalculated as 1.2 days after having previously been calculated as 1.0 days.
How many business days does the team average between when the first commit referencing an issue is authored, and the final work on the issue is released to production? Top performing teams can complete this process in less than one day.
To track “Time to Release,” you must set up Release Tracking setup and Issue Tracking. Once these have been configured, we will automatically begin to calculate the business time that has elapsed between the initial authorship of a commit, and when that commit was last released.
Time to Release is calculated in days, and considers only the “business hours” of the committer that is working on it. For example, if a developer from your team begins work on an issue at 12pm in their time zone, and a final release of that work takes place at 9pm the same day, we would calculate that issue as having a Time to Release of 0.6 days, reflecting that 5 business hours passed between when the developer began work on the commit and when that work was released (after business hours, relative to the time zone of the developer that did the work).