KPIs: How to optimise DevOps Performance

DevOps is a success story. According to a forecast by analysts at Research and Markets, the global DevOps market is expected to be worth around USD 32 billion by 2030 (2023: USD 10.6 billion) - with an annual growth rate of 14.9 per cent. The increasing focus on automation of business processes and a growing demand for cross-departmental collaboration are cited as main reasons for this increase.

However, all that glitters in the DevOps sector is by no means gold, and the hoped-for benefits do not automatically materialise. Especially in times of cost pressure, expenditure and returns are scrutinised more critically. Many DevOps teams are realising that it is no longer enough to measure just a few metrics. They need more KPIs to prove their performance and value. Building a great engine is one side of the coin - teams need to make sure the engine is getting a company to the right destination.

DevOps KPIs from DORA to Flow

Management and optimisation are usually based on key figures. It is no coincidence that Deming or PDCA cycles are the basis for optimisation approaches in many management concepts, including ITIL. There are also tried-and-tested methods for DevOps KPIs, such as the work of Google's DevOps Research and Assessments team (DORA), which regularly publishes analyses on current implementations and best practices in the DevOps environment. DORA metrics for management can even be used in part to define a DevOps maturity level. In addition, the Flow framework defines possible metrics for measuring and improving software delivery organisations.

DevOps without comprehensive governance

Many organisations, however, act like uncoordinated tugboats pulling a freighter in different directions. Our experience shows: In eight out of ten DevOps optimisation projects, an overarching control system is either not in place or not consistently available. After all, a core principle of DevOps is self-organisation with a high degree of freedom for teams. Yet this is often equated with complete freedom, which leads to individual, team-specific approaches to defining key performance indicators and to team-specific management.

As a result, ‘lead time for change’ in team A differs from that in team B. This freedom can quickly lead to a lack of transparency and limited steering ability at portfolio or organisational level, as knowledge is located in people's heads. Depending on the control requirements of the organisation in question, this can pose a problem.

Standardise and integrate DevOps KPIs

The only sensible solution requires a good sense of balance: a stable compromise between standardisation in the methodology on one hand and maintaining a necessary degree of freedom on the other is crucial. Not everything has to be freely definable, and not everything makes sense as a standard: DevOps self-organisation, but with uniform guidelines.

First of all, it is important to integrate possible DevOps KPIs into an overarching KPI system. The aim is not to implement more KPIs, but more meaningful ones. To do this, you need to know exactly what you want to achieve or manage. Examples of strategic goals include increasing the degree of IT agility, cost efficiency or system stability. All of this can be measured in DevOps organisations, compared with other units (internal and external) and then optimised as required.

To achieve this, prerequisites for measurability and controllability must be created in DevOps teams: The central element here are usually Kanban boards, which contain various work packages or stories and are assigned to different buckets depending on their status (e.g. ideas, backlog, in progress, ready for review, done). In this context, status changes provide a good opportunity to define measurement criteria.

Standardisation of structural elements

However, structures of Kanban boards are often defined individually. The implementation of comprehensively comparable DORA or flow KPIs would therefore only be possible on the basis of uniform definitions or uniform buckets of Kanban boards, which can then serve as standardised measurement criteria. For example, no team should have problems defining a standardised bucket (e.g. in progress) for the start time of the work and for the end time of the work (e.g. done). The latter should be standardised, whether including or excluding the product owner review.

A standardised definition of when a change is a change and what counts as a ‘failure’ should also be a matter of course. To achieve this, pragmatic guidelines are needed for selected, relevant organisational aspects of DevOps teams to ensure standardised measurement points. If these requirements are met, a key performance indicator system integrated into strategic objectives can be established across several levels. This represents an effective management tool that enables the optimisation of DevOps teams within a framework of overarching organisational goals.

Optimising DevOps with a Benchmark

In this article on DevOps benchmarks, we describe how DevOps performance can be improved in a targeted manner. It is based on an assessment for an European insurance company.