Three Software Development Metrics to Prevent Escaped Defects

Run-through

Metrics like code churn, code complexity, and test coverage can go a long way in diagnosing the reasons for escaped defects. The scale (developer or team) and scope (one task or many projects), as we recently discussed relative to visibility, also help in diagnosing problems in our development efforts, too. It’s important to distinguish between what’s a management issue, a new team/startup issue, and what’s a developer issue. Finally, while the symptoms are important, we want to determine likely root causes – and solutions. So, let’s give it a shot!

But first, something to brighten everyone’s day, metaphorically speaking. Software developers may resent so much attention on their coding metrics, but the reality today is that everything that can be measured is measured. According to the Bureau of Labor Statistics, the average software developer wages in the US is $52.95 an hour – enough to buy 33,898.8 matches on Amazon. There’s always going to be that one match that doesn’t light.

The Interconnectedness of Software

Our goal in most things related to software development is to make the biggest impact with the least effort. It’s a hundred times less expensive to catch a defect during design than after it goes to production. However, it’s also important to measure any changes so we can validate that performance does improve. Many metrics are linked together, directly or indirectly, proportionately or inversely, such that:

  • Improving any metric should be reflected in improvements to other metrics.
  • Fixing the root cause of a problem has a greater impact than fixing symptoms.
  • Any change that negatively impacts code quality is not an improvement.

The Pareto Principle is always helpful in understanding and prioritizing effort. Simply stated, 80% of our problems are the result of 20% of the causes. As with any troubleshooting guide, it’s best to start with the most common and easiest ones first. Is the power plugged in? 

Code churn, code complexity and test coverage are predictors of defect rates. While they are easy to spot, they are most often symptoms, not root causes where they involve a single software team. Code complexity could be a root cause if your team has taken a project over from another  team though. 

High Code Churn

Some types of code churn are good – when starting projects, working on complex problems, and small ongoing spikes as developers try to leave code in a better state than they found it. With performance analytics, you should consistently observe that code churn starts high and steadily declines. When that’s not the case, you have an issue to solve:

Diagnosing High and Unhealthy Code Churn

Scope

Symptom

Likely Root Cause

Solutions

Company or teamwide across multiple projectsMany delays, long cycle-time,
sprint planning out of whack
Ambiguous project specifications or task requirementsDefine tasks more explicitly – pair Scrum Master with an experienced developer if necessary; verify PMS training, use standups to clarify.
Teamwide on a single projectLarge spikes midstream of project or sprintOther stakeholders not educated about optionsDetail cost of changes to stakeholders to verify. In future projects, take more time to educate stakeholders of options.
Developer across multiple projectsA gradual (sometimes abrupt) decline in most metricsOver-utilized developer at risk of burnout or leaving; life issues / health.Talk with the developer to understand better – mix things up or reduce the load for burnout, examine options for retention, team or HR options for life issues.
Developer across multiple projectsLow defects but a pattern of delaysPerfectionismApplaud diligence, but coach on speed, cascade effect of their delays, and Good Enough Code.
Developer on a single projectLow velocity + higher defects or late delivery, low test coverageDeveloper working with less familiar language, complex task or high complexity codeTalk with the developer to assess if best to partner (code review / mentor) with a dev strong in the language or to reassign the task. Performance metrics can help optimize teams per task / project.
Developer on one or a few tasksHigh code complexity, late, high defects, low test coverageCustom-codingCoach on recommended resources (mature solutions). Devs like to be creative, but also a good chance they may not have known about a pre-existing solution.

High Code Complexity

High code complexity correlates closely to software that’s harder to maintain and bugs that are harder to fix. It takes longer to read, understand, and debug, thus it impacts multiple metrics. Aside from the variable complexity of software (news app vs. financial), many coding elements contributing to higher complexity are avoidable, like:

  • using comments to clarify the purpose/logic of code before writing it,
  • improving nesting sequence,
  • reducing the number of parameters,
  • and so forth.

High code complexity is a problem common to new teams owing to growth, turnover, or bringing on more inexperienced developers than can be given proper oversight. You can mitigate some of the impact with growth planning and an effective onboarding plan. The balance of the solution is a function of teamwork and time in continuously improving every developer’s skills. Utilizing the full repertoire of code reviews, mentoring, analytics-driven task assignments, standups, retrospectives works to address the symptoms and root cause.

Symptoms of high code complexity can be exhibited at every level and include:

  • Long cycle time
  • High defect rates
  • Late deliveries
  • Low test and review coverage
  • Low collaboration in the team
However, the reasons for high code complexity can be quite different:

Diagnosing High Code Complexity

Scope

Root Cause/s

Solutions

Team across many projectsHigh turnover, Fast-growing team, Large teamsAs per management (as discussed in code churn above));Growth planning, In Sprint planning, break tasks into smaller parts, Smaller, cross-functional teams.
Team on projectNew team members, new or experimental softwareAs above.
Developer on many projectsInexperienced developerMentoring
Custom study program
Developer on projectDev not fluent in programming languageUse performance metrics to help assign tasks based on language skill, Pair with a more experienced developer.
Developer on 1 or a few tasksExtra difficult task, Custom codingPair with a more experienced developer or Coach on using existing solutions

Low Test Coverage

The objective of test coverage is to find untested code. The basic notion is that low test coverage translates to high defect rates. But, test coverage doesn’t provide insight into test quality. High test coverage can result in high defect rates, too. However, in the context of your team’s historical test coverage, code complexity, and project defect rate, test coverage becomes a much more meaningful metric, a barometer for quality.

Improving test coverage isn’t a priority per se. You can perhaps achieve 100% test coverage, but it’ll take you time you don’t have and still won’t guarantee your code is defect free. Moreover, defects vary in severity, so testing all code in equal proportion falls short of objectives, too.

Diagnosing High Code Complexity

Symptoms

Root Cause/s

Solutions

Historically high test coverage + high defect rates A misguided policy for a certain level of code coverage (possibly carried over from previous teams) Coach to realign on quality and prioritizing critical code.
Low test coverage + high defect rates High code complexity, Inexperienced developers, Other factors impacting cycle time (task story points, delays in communication) Pair with a more experienced developer. Time writing tests is time not spent writing code. Anything that delays writing code, impacts tests. Test-Driven Development offers a partial remedy.

Reducing code complexity itself should make it easier for developers to write more-meaningful tests. This infers that test coverage and test quality ties into developer experience, and so it’s often an ongoing training issue – unless you’re picking up where another team left off.
To quote from Martin Fowler’s notes on code coverage, you know you have adequate test coverage when, “You rarely get bugs that escape into production, and you are rarely hesitant to change some code for fear it will cause production bugs.”

Take a look at your team’s historical test coverage rates and compare with its historical escaped defect rates. This should provide you a sense of the quality of your developer’s tests and a basis for comparing overall improvement efforts (particularly Escaped Defect Rates). Use test coverage for its intended purpose of identifying code that has not been tested. Coaching is probably best aimed at underscoring test quality while also prioritizing the testing of code according to how critical it is.

High Escaped Defect Rate

Insight from test coverage rates becomes important in helping to diagnose the root cause of escaped defect rates. High defect rate is a symptom and to reduce it, you need to identify why it’s happening first. Gitential doesn’t track escaped defect rates, but they can be tracked and tagged in Jira (or other PMS) – then compared to code churn, code complexity, and test coverage.

A client may overlook a dozen small bugs but raise hell and high water over one big one. This alone makes it a priority to aggressively exterminate big, bitey bugs. However, the time to fix all of those small ones can be equally disruptive to your team’s schedule.

Defects vary dramatically in severity, scope, and time to fix:

  • Severity – system crash or security vulnerability
  • Scope – number of users impacted (often, but not always, by device)
  • Time to fix – by story points or other measurement

So, it’s worth tagging each defect in your project management software with rating scores (1 to 5 or however you decide) for each defect. Add other tags and weight them as you see fit. Security issues might be more important for some projects than severity or scope. This gives you more data to work with to lose in on the root cause.

How can Gitential's AI Help with Escaped Defects?

At Gitential, we are in the process of upgrading our software development analytics with an AI-powered Digital Assistant. While this won’t serve to detect defects, with JIRA integration, you’ll be able to track important cost performance metrics to:

  • See how much you are spending by developer, team, and/or project for new features, bug fixes, and rework. This also extends to cost performance tracking of vendors.
  • Get immediate insights via Next Best Actions on how to improve performance at the individual developer and team level. 
  • Enhance the effectiveness of your one-on-one meetings with developers by providing much faster and more detailed feedback to help them become the best coders they can be.  This can help with OKRs, too!
We also recommend checking out AI-Enhanced Teams – Are AI Assistants Worth It? for how it helps build organizational alignment and more.

Article updated: February 25, 2022

Did you like our content?

Spread the word

Subscribe to Our Newsletter

Don't miss our latest updates.
All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.

Did you like our content?

Spread the word

Subscribe to Our Newsletter

Don't miss our latest updates. All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.