Typically software development organisations measure speed and quality but not capability, customer traction, and commercial return as it applies to the market. As a result boards look at software operations in terms of cost and velocity only. Strategy and growth become disconnected resulting in longterm loss of customers and competitive positioning. These measurements also mean that CTOs and CPOs cannot describe and prove the link between investment to outcomes. What if we were able to look at product development through a lens that is familiar and gives the rest of the C Suite and the board the means to coordinate the organisation’s investment properly in order to grow the product in the market?
The Balanced Scorecard
If we are looking for a model to use, an approach elsewhere in business, familiar to most executives, has been the balanced scorecard. In 1990 a Nolan Norton Institute research project found that organisations were too focused on financial metrics, which they realised was a lagging indicator of decisions already taken. This approach was not properly describing how to affect strategic change only showing what had already happened. Robert Kaplan and David Norton famously published in the Harvard Business Review in 1992. Their thesis, inspired by Lord Kelvin, was that measurement is a precondition for improvement, so if you wanted to improve strategically you had to measure the things that would affect that improvement.
The Balanced Scorecard solved the problem using Kelvin’s principles. It retained the financial measures but also added ones that affected the organisation’s performance, points of measurement that change the way the organisation operates and lead to the lagging financial outcomes. They added three additional perspectives which were customer, internal processes and learning and growth. There is a fundamental to this approach we see in Six Sigma, where conflicting measures balance out control of a process, but the balanced scorecard expands wider than just the process.
This was not a new approach. General Electric’s 1950s framework included one financial metric and seven non-finanical ones. Peter Drucker’s management by objectives (1954) set clear objectives at every level of the organisation. Empolyee goals were aligned to the company strategy and appraised on results rather than effort. Robert Anthony (1965) divided organisations into three layers, strategic planning or organisational direction, management control which focused on resources, operational control which focused on tasks and process. Interestingly Anthony’s model is still apparent in modern target operating model design and business architecture approaches.
None of these approaches accepted dimension alone as sufficient for managing an organisation. These earlier approaches fed into later iterations of balanced scorecard use as well as the initial design.
To put some flesh on these skeletal concepts, here are some types of measures you might track in a balanced scorecard. Under the Financial metrics, return of capital employed, revenue growth, cost reduction per unit. Customer might monitor market share in target segments, customer satisfaction scores, on-time delivery rate. For Learning and Growth, employee retention rates, training hours per employee, number of employee suggestions implemented. Internal Process might look at manufacturing cycle times, defect rates, new product development times.
DORA Metrics and where they fall short
There is a standard approach to engineering measurement available today, from DORA (DevOps Research and Assessment). This focuses on 4 key metrics: deployment frequency, change lead time, change failure rate, meantime to recovery.
Looking at this measurement from a broader strategic leadership perspective, it has a narrow focus, primarily around throughput and stability. These metrics do not cover organisational capability, commerical measurement or customer outcomes. This means that you are not disinguishing between effectively delivered features and effectively delivered features that customers want.
The DORA metrics also underplay developer experience, thinking time, cognitive load, morale, ability of the development team to keep pace with and become expert in newer technologies.
Then there is a bigger problem for engineering and software development, seen in most organisations except the slickest startups, which is the issue of cross-team dependencies. At operating level all the way to board a narrow focus obscures this tension and the key improvements that need to be managed are not surfaced. Frequently this is a root cause of product portfolio issues.
Engineering leadership in recent years has realised the importance of organisation performance, talent development, and customer alignment. Domain aligned organisations have been one approach to addressing this. Taking Kelvin’s approach let’s therefore ensure we are measuring all of these factors in a balanced and complementary way.
An Engineering Balanced Scorecard covering four perspectives
In order to properly measure Engineering in a way that ensures it contributes to the organisation’s strategy in a meaningful way, we can return to Kaplan and Noton’s original perspectives, but implement them so that they are relevant to engineering and how engineering contributes to an organisation’s success.
The purpose of the Customer Experience view is to measure whether the work being done actually mattered and improved customer experience. It should measure topics such as feature use, promoter score, churn, impacts of outage.
For Internal Process we can build on the DORA metrics, focusing on cycle times, deployment frequency, the quality of code deployed and the different types of work being done. Wrapped around the DORA execution focus is a measurement of strategic vs operational investment, and technical debt reduction. This means that there is now a view of the balance of work not just the speed and stability of the work done.
Learning and Growth addresses the challenge that capability is not constant and requires investment (of time and people). Without it you cannot assess whether delivery capability is growing or degrading. This perspective should cover measurement of skills, knowledge distribution, key-person dependencies, the interaction and knowledge transfer between senior and junior staff, and succession potential.
Engineering organisations that cannot demonstrate financial contribution are treated as merely cost centres. A CTO or CPO unable to link engineering investment to financial outcomes will constantly battle for resources, or be in a spiral cost cuts. The engineering Financial view needs to understand acquisition cost, lifetime value, gross margin, and revenue from new features. A lower acquisition cost is frequently linked to the purchase and deployment ease, and the feature alignment to customer needs, lifetime value to feature richness, aligned to market niche.
A suggested set of metrics for a SaaS organisation can be seen here: https://kitneal.com/saas-cto-cpo-balanced-scorecard/.
Implementation and use of the Engineering Balanced Scorecard
The scorecard is not just four sets of measures though. It is a causal set of interlocking areas of improvement or perspectives. Or in other words, the improvement being focused on through one perspective should directly impact improvement in the next. It is important to always think of how one perspective influences another.
As the original Balanced Scorecard was adopted and improved, Kaplan and Norton developed additional tooling called a Strategy Map, in order to ensure this causal flow in the design, because without a strategy map connecting measurements, “what you’re calling a balanced scorecard is really just a list of measures”
In terms of timing relevance, Financial measures are lagging, showing the effect of what the organisation has done. Customer indicators show what is happening right now. Process metrics drive customer outcomes, and Learning and Growth activities drive better execution of processes.
Therefore, when designing and using an Engineering Balanced Scorecard, consider the following flow.
- If effective, Learning and Growth leads to stronger skills, broader knowledge distribution, lower key-person dependency and higher organisational capability
- Effective Internal Process leads to faster delivery, greater reliability, appropriate strategic investment and manageable technical debt
- Good Customer Experience leads to faster adoption, greater reliability, stronger satisfaction and improved retention
- Successful Financial Outcome leads to lower cost-to-serve, stronger lifetime value, sustainable growth and improved SaaS economics
Finally review points are important in relation to these four perspectives and the cycles may differ by perspective. Learning and Growth typically shows an impact over quarterly or half yearly period. Process impacts can be seen weekly, sprint to sprint or monthly. Financial outcomes may have seasonal factors, but commonly improve within a quarter. Customer response is often within 30 to 90 days.
When reviewing, look for the breaks in the chain, not that certain metrics have turned red. It is the relationships that show the success or failure of the investement and the operation and this is the view and insight that needs to be presented to the board and inform strategic planning.
- Kaplan, R.S. and Norton, D.P. (1992) ‘The Balanced Scorecard: Measures that Drive Performance’, Harvard Business Review, 70(1), pp. 71-79. HBS Faculty Page
- Kaplan, R.S. and Norton, D.P. (1996) The Balanced Scorecard: Translating Strategy into Action. Boston: Harvard Business School Press.
- Kaplan, R.S. and Norton, D.P. (2001) The Strategy-Focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment. Boston: Harvard Business School Press.
- Kaplan, R.S. and Norton, D.P. (2004) Strategy Maps: Converting Intangible Assets into Tangible Outcomes. Boston: Harvard Business School Press.
- Kaplan, R.S. and Norton, D.P. (2006) Alignment: Using the Balanced Scorecard to Create Corporate Synergies. Boston: Harvard Business School Press.
- Kaplan, R.S. and Norton, D.P. (2008) The Execution Premium: Linking Strategy to Operations for Competitive Advantage. Boston: Harvard Business School Press.
- Kaplan, R.S. (2010) ‘Conceptual Foundations of the Balanced Scorecard’, Harvard Business School Working Paper, No. 10-074. PDF
- Forsgren, N., Humble, J. and Kim, G. (2018) Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. Portland: IT Revolution Press.
- DORA Team (2024) State of DevOps Report 2024. Google Cloud. DORA Metrics Guide
- Balanced Scorecard Institute, ‘The Four Perspectives of the Balanced Scorecard’. Article
- Simons, R., cited in ‘What Is a Balanced Scorecard?’, Harvard Business School Online. Article
- Goodhart, C.A.E. (1975) ‘Problems of Monetary Management: The U.K. Experience’, in Papers in Monetary Economics, Vol. I. Reserve Bank of Australia. (Commonly paraphrased as: “When a measure becomes a target, it ceases to be a good measure.” The paraphrase is attributed to Marilyn Strathern, 1997.)