Are We Measuring DevOps Success or Just What’s Easy to Measure?

Some metrics are not easy to measure because they require complicated setup, their accuracy is hard to validate or they are not easy to understand. In such cases, it’s easy to fall into the McNamara fallacy: measuring what is easy, not what is important — leading to consequences that follow from this decision.

The DORA Promise: Measuring What Actually Matters

The DevOps Research and Assessment (DORA) team identified four (later added one) critical metrics that distinguish high-performing software teams:

  • Deployment Frequency
  • Lead Time for Changes
  • Time to Restore Service
  • Change Failure Rate

These metrics provide a balanced view of both velocity and stability—exactly what we need to drive meaningful improvements. But the devil is in the details. The majority of tools on the market are based on pipelines: merge request merged, new artifact created, deployment job executed. The time between these events is considered the Lead Time for Changes.

But what if the deployment consists of multiple components coming from different repositories? Where should the deployment job be located? What if there is no deployment job—if a GitOps approach is used and changes are pulled into production?

A New Approach to Measurement

Instead of relying on CI/CD vendor-proprietary approaches, we should measure deployment events based on when an artifact successfully reaches its intended environment.
For Kubernetes, this can be derived from the Prometheus kube_pod_container_info metric.
Submitting these events to an aggregation service that also tracks merge request events enables accurate calculation of DORA metrics.

By correlating these deployment events with merge request timestamps, we achieve:

  • Accurate lead time calculation regardless of deployment methodology
  • GitOps-friendly monitoring without pipeline dependencies
  • Component-level visibility through repository badges

This approach is implemented in github.com/metraction/handwheel — an open-source project that works with Devlake. It requires proper Docker image labels, but it is the least intrusive way to correlate merge requests with deployment events.
In addition, it offers badges for some DORA metrics, allowing each repository to transparently show the team how quickly a component is reaching deployment.

From Measurement to Improvement

Having transparent DORA metrics for each component would allow teams to focus on what is important: delivering value quickly to users.
It might be a huge effort to improve these metrics, but without measurement, it is hard to start any improvement at all.