Gitential’s Guide to the Cost of Fixing Bugs

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Run-through

How to Notice and Prevent Software Developer Burnout

How to Notice and Prevent Software Developer Burnout

How do you notice and prevent software developer burnout as a team lead or engineering manager? Burnout affects each person a bit differently. It often starts subtly and progresses incrementally. Developer burnout feeds into a vicious loop that can make your developers quit or even change career path. If not corrected, it can also lead to health consequences. The good news is that its onset can be identified, prevented, and rapidly alleviated. Let’s examine how.

Read More »
Effective Code Reviews

Best Practices for Effective Code Reviews

More than half (53%) of the software development teams do code reviews on a daily basis, according to SmartBear’s 2019 State of Code Review. They find, as one would expect, that more frequent code reviews have a direct correlation to higher-quality code. As their report goes on to say, “There is a compounding effect when you introduce all the benefits of code review into daily behavior. Communication improves, knowledge about the codebase is shared, and fewer bugs make it through development to QA.”

Read More »

“Owh, come on, it’s just a bug.”
— Johnny Rico, Starship Troopers

Bugs are our greatest threat in software development as measured by time, cost, and consequences. The Cost of Poor Software Quality in the US: A 2020 Report by Herb Krasner of CISQ estimates the cost of bugs at roughly $607 billion for the United States alone. This includes costs associated with unsuccessful projects, maintaining legacy systems, and software failures in operational systems. It doesn’t include security issues, technical debt, or a host of other “quality” issues having an overall price tag of over $2 trillion.

So, that’s the big picture. Let’s look at how much bugs are costing you, whether you need to fix all of them, and how you can prioritize which bugs to fix.

How Much Does It Cost to Fix a Bug?

It’s important to distinguish between the actual cost of fixing a bug and the consequences that may result from not fixing a bug… and then having to fix it. We discuss the consequences of bugs below. The cost to fix a bug, however, is highly relative – depending on when and where the bug is discovered. The “relative costs” in the chart below from the IBM Systems Sciences Institute (2002) still appear to hold true as a function of process:

Relative Cost To Fix A Defect Based On Where It’s Caught

 DesignImplementationTestingMaintenance
Defect Discovery Cost 1x6.5x15x100x

In Design, a developer writing code can see and rapidly correct an error. In this case, the cost is a perhaps some portion of the developer’s Code Churn. But, the more time that elapses after the code leaves the developer’s hands, more people and time are needed to find and fix the bug, test and validate that it’s been fixed. In the maintenance stage, that could include support for angry customers and clients. 

How to Calculate the Cost of Bug Fixes

As software engineering managers, we want to know the cost of fixing bugs in our current team and project. Quantifying the cost of fixing vs. preventing bugs, by Lynda Gaines gives us a good example on how to do exactly this. She takes a look at Google’s average defect rates and costs from 2012. But, we can plug in our own numbers to determine our cost to fix a bug:
Measurement:Plug in Your Own #’s
Average Time to Fix a Bug15 hours
Average Fully-Loaded Hourly Rate of Engineer** $68.83
Base Cost to Fix a Bug= $1032

* Here we’re using the US Bureau of Labor Statistics June 2021 average US-based software developer hourly rate of $52.95 for US-based developers. Fully-loaded costs can vary widely, but typically run from 25-40% of base salary, with 30% applied here.

The only trick here is finding your average time to fix a bug without having to do it manually… but the bigger issue is total number and types of bugs (fewer critical but hard to fix vs a horde or small fast fixes).

Using Gitential’s Metrics to Determine Average Time to Fix a Bug

With our automated software development analytics you’re provided with everything needed to determine the costs of fixing bugs for your entire organization, project, team, and individual developer. The main metrics you’ll use include:

  • Code Volume: Source Lines of Code, a software metric used to measure the size of a computer program by counting the number of lines in the text of the program’s source code. 
  • Hours Spent on Bug Fixes: Total estimated coding hours spent on any bug fix that has been deployed to production through a pull request.
  • Bugs Ratio: Ratio between the number of bugs vs all pull requests.
  • Number of Bugs Fixed: Number of pull requests which have ‘bug’ or ‘fix’ in their description.

Avg Time to Fix a Bug =
# of Hours Spent Fixing Bugs / # of Fixed Bugs

The only other numbers you’ll need to calculate your average bug fix cost at each level are your fully-loaded developer wages. By diving deeper into each developer’s metrics you can identify potential root causes contributing to high defect rates. Skill in different programming languages, code complexity (and lack of test coverage), task complexity (high story points), being overloaded (utilization), and other factors could be involved. You can use this data to better align tasks to their skill or select specific partners for code reviews.

Calculating Your Annual Cost of Fixing Bugs

For a broader picture, again at your organization, project, team, and individual developer level, you can get an annualized cost based on:

SLOC produced yearly * Avg # of Bugs per 1 KSLOC * Avg Cost per Bug

Presume, in our example, that your organization is generating just 5 bugs per 1k SLOC and producing 100k SLOC yearly. That puts you over $500k for fixing bugs, requiring 3-4 full-time engineers just to fix bugs. Or, if you had 9 engineers, you’d know they’re spending at least a third of their time fixing bugs. And that is not at all unusual.

Why Is It Important to Fix Bugs?

It’s the developer’s responsibility to deliver software that:
  1. works according to specifications
  2. is on time
  3. on budget
Missing any of these three can jeopardize your job or contract as they can impact a company’s profitability (and existence). Bugs can anger end-users and cause them to cancel subscriptions or take their purchases elsewhere. Buggy software can lead to bad reviews – keeping people from downloading it in the first place. Holes in security and legal compliance can bring a ton of big, bad news and hefty fines. And for as severe as these issues can be, software is also used for: Bugs in these kinds of software can impact lives and economies. The “Y2K Bug” as it was called wasn’t one bug, but a case of the entire early software industry thinking how many bits it could save by rendering years in just two digits – like 98, 99, 00… oops. Governments and organizations globally spent $500 billion fixing that to prevent “the unknown” from happening.

Why Is It Unrealistic to Fix All Bugs?

Again, software must be delivered on schedule or risk the consequences. You probably won’t have time to fix every single bug. For that matter, you may not know about a lot of bugs until the software gets into the hands of end-users, not just with how many different devices they have, but how they actually use the software. How “some” use it may be impossible to predict. Leastwise, three things are working in your favor for “most bugs”:

  1. Tolerance. It’s generally expected by those ordering the software to be developed that there will be some bugs at and even beyond release.
  2. Fast Bug Fixing. Many non-critical bugs can pass through into production knowing that they’ll be caught and fixed often in a couple of hours.
  3. Need for a Larger Test Environment. There are over 24,000 different types of Android devices alone. Some bugs will only be uncovered through extensive use. Many software producers conduct beta tests and early releases to find bugs, determine their frequency, and get more information about them from end-users.

How Do You Prioritize Bug Fixes?

Thankfully, all bugs are not created equal. It is on the product manager or software engineering manager to prioritize which bugs should be fixed now or later, and which bugs can be safely ignored.

Technically worth an article unto itself because software can include everything from games to managing nuclear power plants. There are five major areas to consider according to a) how easy it is to fix plus b) the number of people impacted (scope) and c) degree of impact (severity):

  • Law – Privacy issues, finances, software as a medical device, etc.
  • Safety – Software for a wide-range of automated systems.
  • Security – For protection of data and infrastructure.
  • Business Requirements – Delivers what the company needs.
  • End User Satisfaction – Makes it easy for people to use and does what they ask for.

That’s five major areas, each with three dimensions, a lot to consider. It’s important to determine the risks associated with the software being developed in advance.

In terms of managing the risk of a bug, a good question to ask is, “Are you willing to be held liable for any consequences caused by the bug?” Ultimately, it’s the software development company that’s responsible, but that doesn’t mean CEOs and CTOs won’t be asking how or why you let a “critical bug” slip by. 

Assessing Scope of Bug Impact

Beyond bugs that can cause injury and mayhem, you’re left with a lot of bugs that can impact business requirements and customer satisfaction. Knowing how many people a bug effects can help you prioritize which bugs to fix. In most cases these days, development teams are building in automated error reports to let you know how many people are being affected by a bug, and how frequently.

Earlier in development, project specifications should clearly outline the families of devices the software will be used on. Sites like StatCounter can help you get a good estimate for a variety of statistics (device, OS version, device vendor, browser version, screen resolution) by state and country. Post-release, your APM metrics can tell you exactly what to prioritize. It makes sense then to focus on sequentially fixing bugs that will impact the next largest group of users.

It’s also worth noting that some bugs may end up being treated as features – actually working better than the original idea. Infrequent bugs experienced by small numbers of users with trivial impact may be assessed as safe to ignore. The cost to fix them exceeds the value of fixing them. Depending on your view, the Blue Screen of Death may be regarded as a bug or a “bug check” – suffice that Microsoft let this issue persist across several generations of Windows.

How Can AI Help Reduce the Cost of Bug Fixes?

Gitential’s work on an AI Assistant for Software Delivery will help reduce delivery costs and the cost of bug fixes in several ways. Simply stated, an AI Assistant lets users ask questions (Google-like queries) and receive instant answers — data, supporting visuals (graphs/charts), explanation of the data, a summary and Next Best Actions

Functionally, our AI will provide engineering managers,  and developers “Insights On-Demand” – things you can do almost immediately to improve team and individual developer performance.  The AI can provide insights to C-levels, BI Specialists, and even HR managers, too, to align on organizational OKRs, optimize budget allocations, and drive hiring strategies. 

Preventing bugs in the first place is a function of optimizing teams around project specifications and requirements. Initial team selection and growth scale in complexity to the number of developers in the company (not just project or team) – and scales proportionately. This isn’t so simple, it needs to factor:

  • Each developer’s programming language skill and experience (quality)
  • Developer cost and performance (productivity)
  • Right mix of junior, mid-level, and senior developers (efficiency)
  • Relative propensity for teamwork

From a risk management perspective, bottlenecks are almost inevitable without a proper mix of experience on a team – equating to either low quality PR code reviews or delays in picking them up. Much can be said about how code complexity tends to increase dramatically as teams expand – increasing the likelihood of bugs and increasing the time/effort to find them. 

Where managers have spent hours digging through analytics to find helpful insights – AI renders actionable insights in seconds – how to help a developer reduce their defect rate or what their team needs to do to improve their MTTR. 

 

More on AI for Improving Software Delivery

We’ve prepared a lot more to explain the benefits an AI-powered Assistant can provide software development teams. The following articles are related to software delivery costs – and we hope you’ll check some of them out!

About Gitential

Gitential is an Analytics and Engineering Intelligence service provider bringing visibility and optimization highlights on teams’ productivity. Our mission is to enable faster, data-driven decisions to continuously improve software delivery team cost performance and proactive risk management.

Ready to explore different ways to improve your software projects’ efficiency? Schedule a meeting and we will be happy to listen to and discuss your needs.

Have a project but are not quite ready to contact us? See if Gitential is a fit for you!

FREE TRIAL FOR LIFE
for up to 5 repos, 10 devs and 3 months of data

Article Updated: February 07, 2022

Did you like our content?

Spread the word

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Subscribe to Our Newsletter

Don't miss our latest updates.
All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.

Did you like our content?

Spread the word

Share on facebook
Share on twitter
Share on linkedin
Share on reddit

Subscribe to Our Newsletter

Don't miss our latest updates. All About Software Engineering Best Practices, Productivity Measurement, Performance Analytics, Software Team Management and more.