Three Software Development Metrics to Prevent Escaped Defects

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Run-through

Metrics like code churn, code complexity, and test coverage can go a long way in diagnosing the reasons for escaped defects. The scale (developer or team) and scope (one task or many projects), as we recently discussed relative to visibility, also help in diagnosing problems in our development efforts, too. It’s important to distinguish between what’s a management issue, a new team/startup issue, and what’s a developer issue. Finally, while the symptoms are important, we want to determine likely root causes – and solutions. So, let’s give it a shot!

But first, something to brighten everyone’s day, metaphorically speaking. Software developers may resent so much attention on their coding metrics, but the reality today is that everything that can be measured is measured. According to the Bureau of Labor Statistics, the average software developer wages in the US is $52.95 an hour – enough to buy 33,898.8 matches on Amazon. There’s always going to be that one match that doesn’t light.

The Interconnectedness of Software

Our goal in most things related to software development is to make the biggest impact with the least effort. It’s a hundred times less expensive to catch a defect during design than after it goes to production. However, it’s also important to measure any changes so we can validate that performance does improve. Many metrics are linked together, directly or indirectly, proportionately or inversely, such that:

Improving any metric should be reflected in improvements to other metrics.
Fixing the root cause of a problem has a greater impact than fixing symptoms.
Any change that negatively impacts code quality is not an improvement.

The Pareto Principle is always helpful in understanding and prioritizing effort. Simply stated, 80% of our problems are the result of 20% of the causes. As with any troubleshooting guide, it’s best to start with the most common ones first. Is the power plugged in? Code churn, code complexity and test coverage are predictors of defect rates. While they are easy to spot, they are most often symptoms, not root causes where they involve a single software team. Code complexity could be a root cause if your team has taken it over from another though.

High Code Churn

Some types of code churn are good – when starting projects, working on complex problems, and small ongoing spikes as developers try to leave code in a better state than they found it. With performance analytics, you should consistently observe that code churn starts high and steadily declines. When that’s not the case, you have an issue to solve:

Diagnosing High and Unhealthy Code Churn

Scope Symptom Likely Root Cause Solutions
Company or teamwide across multiple projects Many delays, long cycle-time, sprint planning out of whack Ambiguous project or task requirements Define tasks more explicitly – pair Scrum Master with an experienced developer if necessary; verify PMS training, use standups to clarify.
Teamwide on a single project Large spikes midstream of project or sprint Other stakeholders not educated about options Detail cost of changes to stakeholders to verify. In future projects, take more time to educate stakeholders of options.
Developer across multiple projects A gradual (sometimes abrupt) decline in most metrics Over-utilized developer at risk of burnout or leaving; life issues / health. Talk with the developer to understand better – mix things up or reduce the load for burnout, examine options for retention, team or HR options for life issues.
Developer across multiple projects Low defects but a pattern of delays Perfectionism Applaud diligence, but coach on speed, cascade effect of their delays, and Good Enough Code.
Developer on a single project Low velocity + higher defects or late delivery, low test coverage Developer working with less familiar language, complex task or high complexity code Talk with the developer to assess if best to partner (code review / mentor) with a dev strong in the language or to reassign the task. Performance metrics can help optimize teams per task / project.
Developer on one or a few tasks High code complexity, late, high defects, low test coverage Custom-coding Coach on recommended resources (mature solutions). Devs like to be creative, but also a good chance they may not have known about a pre-existing solution.

High Code Complexity

High code complexity correlates closely to software that’s harder to maintain and bugs that are harder to fix. It takes longer to read, understand, and debug, thus it impacts multiple metrics. Aside from the variable complexity of software (news app vs. financial), many coding elements contributing to higher complexity are avoidable, like:
  • using comments to clarify the purpose/logic of code before writing it,
  • improving nesting sequence,
  • reducing the number of parameters,
  • and so forth.
High code complexity is a problem common to new teams owing to growth, turnover, or bringing on more inexperienced developers than can be given proper oversight. You can mitigate some of the impact with growth planning and an effective onboarding plan. The balance of the solution is a function of teamwork and time in continuously improving every developer’s skills. Utilizing the full repertoire of code reviews, mentoring, analytics-driven task assignments, standups, retrospectives works to address the symptoms and root cause.

Diagnosing High Code Complexity

Scope Symptom Root Cause/s Solutions
Team across many projects Many, including:
  • Longer cycle time,
  • High defect rate,
  • Late delivery,
  • Low test and review coverage,
  • Low velocity,
  • Low collaboration
High turnover, Fast-growing team, Large teams As per management (above); Growth planning, In Sprint planning, break tasks into smaller parts, Smaller, cross-functional teams
Team on project New team members, New or experimental software
Developer on many projects Inexperienced developer Mentoring Custom study program
Developer on project Dev not fluent in programming language Use performance metrics to help assign tasks based on language skill Pair with a senior developer.
Developer on 1 or a few tasks Extra difficult task, Custom coding Pair with a senior developer Coach on using existing solutions

Low Test Coverage

The objective of test coverage is to find untested code. The basic notion is that low test coverage translates to high defect rates. But, test coverage doesn’t provide insight into test quality. High test coverage can result in high defect rates, too. However, in the context of your team’s historical test coverage, code complexity, and project defect rate, test coverage becomes a much more meaningful metric, a barometer for quality.

Improving test coverage isn’t a priority per se. You can perhaps achieve 100% test coverage, but it’ll take you time you don’t have and still won’t guarantee your code is defect free. Moreover, defects vary in severity, so testing all code in equal proportion falls short of objectives, too.

Diagnosing High Code Complexity

Symptoms Root Cause/s Solutions
Historically high test coverage + high defect rates A misguided policy for a certain level of code coverage (possibly carried over from previous teams) Coach to realign on quality and prioritizing critical code. Pair with a senior developer. Time writing tests is time not spent writing code. Anything that delays writing code, impacts tests. Test-Driven Development offers a partial remedy.
Low test coverage + high defect rates High code complexity, Inexperienced developers, Other factors impacting cycle time (task story points, delays in communication)e
Reducing code complexity itself should make it easier for developers to write more-meaningful tests. This infers that test coverage and test quality ties into developer experience, and so it’s often an ongoing training issue – unless you’re picking up where another team left off. To quote from Martin Fowler’s notes on code coverage, you know you have adequate test coverage when, “You rarely get bugs that escape into production, and you are rarely hesitant to change some code for fear it will cause production bugs.” Take a look at your team’s historical test coverage rates and compare with its historical escaped defect rates. This should provide you a sense of the quality of your developer’s tests and a basis for comparing overall improvement efforts (particularly Escaped Defect Rates). Use test coverage for its intended purpose of identifying code that has not been tested. Coaching is probably best aimed at underscoring test quality while also prioritizing the testing of code according to how critical it is.

High Escaped Defect Rate

Insight from test coverage rate becomes important in helping to diagnose the root cause of escaped defect rates. High defect rate is a symptom and to reduce it, you need to identify why it’s happening first. Gitential doesn’t track escaped defect rates, but they can be tracked and tagged in Jira (or other PMS) – then compared to code churn, code complexity, and test coverage.

A client may overlook a dozen small bugs but raise hell and high water over one big one. This alone makes it a priority to aggressively exterminate big, bitey bugs. However, the time to fix all of those small ones can be equally disruptive to your team’s schedule.

Defects vary dramatically in severity, scope, and time to fix:

  • Severity – system crash or security vulnerability
  • Scope – number of users impacted (often, but not always, by device)
  • Time to fix – by story points or other measurement

So, it’s worth tagging each defect in your project management software with rating scores (1 to 5 or however you decide) for each defect. Add other tags and weight them as you see fit. Security issues might be more important for some projects than severity or scope. This gives you more data to work with to home in even closer to the root cause.

About Gitential

Our team at Gitential constantly watches trends across the software development industry. We also get down into the code with you to find ways to make it easier for you to make your team more efficient. Please take a moment to arrange a free demo – or sign up for a free trial, no credit card is needed.

Did you like our content?

Spread the word

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Subscribe to Our Newsletter

Don't miss our latest updates.
All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.

Did you like our content?

Spread the word

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Subscribe to Our Newsletter

Don't miss our latest updates. All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.