Gitential’s Guide to the Cost of Fixing Bugs

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Run-through

How to Make a Difference with 1-to-1 Meetings

How to Make a Difference with 1-on-1 Meetings

One-on-one meetings are as much a leadership tool as they are a management tool. The main leadership requirement with 1-on-1 meetings involves a desire to make a difference. These meetings can make a GIGANTIC difference if approached properly. While we’ll cover some of the essentials for conducting 1-on-1’s, we’d like to share a few additional perspectives that you may not find elsewhere.

Read More »

“Owh, come on, it’s just a bug.”
— Johnny Rico, Starship Troopers

Bugs are our greatest threat in software development as measured by time, cost, and consequences. The Cost of Poor Software Quality in the US: A 2020 Report by Herb Krasner of CISQ estimates the cost of bugs at roughly $607 billion for the United States alone. This includes costs associated with unsuccessful projects, maintaining legacy systems, and software failures in operational systems. It doesn’t include security issues, technical debt, or a host of other “quality” issues having an overall price tag of over $2 trillion.

So, that’s the big picture. Let’s look at how much bugs are costing you, whether you need to fix all of them, and how you can prioritize which bugs to fix.

How Much Does It Cost to Fix a Bug?

It’s important to distinguish between the actual cost of fixing a bug and the consequences that may result from not fixing a bug… and then having to fix it. We discuss the consequences of bugs below. The cost to fix a bug, however, is highly relative – depending on when and where the bug is discovered. The “relative costs” in the chart below from the IBM Systems Sciences Institute (2002) still appear to hold true as a function of process:

Relative Cost To Fix A Defect Based On Where It’s Caught

 DesignImplementationTestingMaintenance
Defect Discovery Cost 1x6.5x15x100x

A developer writing code can see and rapidly correct an error. In this case, the cost is a portion of the developer’s Code Churn. The more time that elapses after the code leaves the developer’s hands, more people and time are needed to find and fix the bug.

How to Calculate the Cost of Bug Fixes

As software engineering managers, we want to know the cost of fixing bugs in our current team and project. Quantifying the cost of fixing vs. preventing bugs, by Lynda Gaines gives us a good example on how to do exactly this. She takes a look at Google’s average defect rates and costs from 2012. But, we can plug in our own numbers to determine our cost to fix a bug:
Measurement:Plug in Your Own #’s
Average Time to Fix a Bug15 hours
Average Fully-Loaded Hourly Rate of Engineer*$70.16
Base Cost to Fix a Bug$1052
* Here we’re using the US Bureau of Labor Statistics June 2021 average US-based software developer hourly rate of $52.95 for US-based developers. Fully-loaded costs can vary widely, but typically run from 25-40% of base salary, with 32.5% applied here.

The only trick here is finding your average time to fix a bug without having to do it manually…

Using Gitential’s Metrics to Determine Average Time to Fix a Bug

With our automated software development analytics you’re provided with everything needed to determine the costs of fixing bugs for your entire organization, project, team, and individual developer. The main metrics you’ll use include:

  • Code Volume: Source Lines of Code, a software metric used to measure the size of a computer program by counting the number of lines in the text of the program’s source code. 
  • Hours Spent on Bug Fixes: Total estimated coding hours spent on any bug fix that has been deployed to production through a pull request.
  • Bugs Ratio: Ratio between the number of bugs vs all pull requests.
  • Number of Bugs Fixed: Number of pull requests which have ‘bug’ or ‘fix’ in their description.

Avg Time to Fix a Bug =
# of Hours Spent Fixing Bugs / # of Fixed Bugs

The only other numbers you’ll need to calculate your average bug fix cost at each level are your fully-loaded developer wages. By diving deeper into each developer’s metrics you can identify potential root causes contributing to high defect rates. Skill in different programming languages, code complexity (and lack of test coverage), task complexity (high story points), being overloaded (utilization), and other factors could be involved. You can use this data to better align tasks to their skill or select specific partners for code reviews.

Calculating Your Annual Cost of Fixing Bugs

For a broader picture, again at your organization, project, team, and individual developer level, you can get an annualized cost based on:

SLOC produced yearly * Avg # of Bugs per 1 KSLOC * Avg Cost per Bug

Presume, in our example, that your organization is generating just 5 bugs per 1k SLOC and producing 100k SLOC yearly. That puts you over $500k for fixing bugs, requiring 3-4 full-time engineers just to fix bugs. Or, if you had 9 engineers, you’d know they’re spending at least a third of their time fixing bugs. And that is not at all unusual.

Why Is It Important to Fix Bugs?

It’s the developer’s responsibility to deliver software that:
  1. works according to specifications
  2. is on time
  3. on budget
Missing any of these three can jeopardize your job or contract as they can impact a company’s profitability (and existence). Bugs can anger end-users and cause them to cancel subscriptions or take their purchases elsewhere. Buggy software can lead to bad reviews – keeping people from downloading it in the first place. Holes in security and legal compliance can bring a ton of big, bad news and hefty fines. And for as severe as these issues can be, software is also used for: Bugs in these kinds of software can impact lives and economies. The “Y2K Bug” as it was called wasn’t one bug, but a case of the entire early software industry thinking how many bits it could save by rendering years in just two digits – like 98, 99, 00… oops. Governments and organizations globally spent $500 billion fixing that to prevent “the unknown” from happening.

Why Is It Unrealistic to Fix All Bugs?

Again, software must be delivered on schedule or risk the consequences. You probably won’t have time to fix every single bug. For that matter, you may not know about a lot of bugs until the software gets into the hands of end-users, not just with how many different devices they have, but how they actually use the software. How “some” use it may be impossible to predict. Leastwise, three things are working in your favor for “most bugs”:

  1. Tolerance. It’s generally expected by those ordering the software to be developed that there will be some bugs at and even beyond release.
  2. Fast Bug Fixing. Many non-critical bugs can pass through into production knowing that they’ll be caught and fixed often in a matter of hours.
  3. Need for a Larger Test Environment. There are over 24,000 different types of Android devices alone. Some bugs will only be uncovered through extensive use. Many software producers conduct beta tests and early releases to find bugs, determine their frequency, and get more information about them from end-users.

How Do You Prioritize Bug Fixes?

Thankfully, all bugs are not created equal. It is on the software engineering manager to prioritize which bugs should be fixed now or later, and which bugs can be safely ignored.

Technically worth an article unto itself because software can include everything from games to managing nuclear power plants. There are five major areas to consider according to a) how easy it is to fix plus b) the number of people impacted (scope) and c) degree of impact (severity):

  • Law – Privacy issues, finances, software as a medical device, etc.
  • Safety – Software for a wide-range of automated systems.
  • Security – For protection of data and infrastructure.
  • Business Requirements – Delivers what the company needs.
  • End User Satisfaction – Makes it easy for people to use and does what they ask for.

That’s five major areas, each with three dimensions, a lot to consider. Suffice that it is very much on the software engineer manager to determine the risks associated with the software they are developing.

The critical question for the manager is to ask, “Are you willing to be held liable for any consequences caused by the bug?” Ultimately, it’s the software development company that’s responsible, but that doesn’t mean CEOs and CTOs won’t be asking how or why you let a “critical bug” slip by.

Assessing Scope of Bug Impact

Beyond bugs that can cause injury and mayhem, you’re left with a lot of bugs that can impact business requirements and customer satisfaction. Knowing how many people a bug effects can help you prioritize which bugs to fix. In most cases these days, development teams are building in automated error reports to let you know how many people are being affected by a bug, and how frequently. Earlier in development, project specifications should clearly outline the families of devices the software will be used on. Sites like StatCounter can help you get a good estimate for a variety of statistics (device, OS version, device vendor, browser version, screen resolution) by state and country. It makes sense then to focus on sequentially fixing bugs that will impact the next largest group of users. It’s also worth noting that some bugs may end up being treated as features – actually working better than the original idea. Infrequent bugs experienced by small numbers of users with trivial impact may be assessed as safe to ignore. The cost to fix them exceeds the value of fixing them. Depending on your view, the Blue Screen of Death may be regarded as a bug or a “bug check” – suffice that Microsoft let this issue persist across several generations of Windows.

About Gitential

Our team at Gitential constantly watches trends across the software development industry. We also get down into the code with you to find ways for your team to improve their coding quality and… kill bugs faster! Please take a moment to arrange a free demo – or sign up for a free trial, no credit card is needed.

Did you like our content?

Spread the word

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Subscribe to Our Newsletter

Don't miss our latest updates.
All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.

Did you like our content?

Spread the word

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Subscribe to Our Newsletter

Don't miss our latest updates. All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.