How we approach bug management
The way we tackle bugs is key to keeping customers happy and developers sane.
The software developers at Atono are some of the best I’ve ever worked with. They build quickly, they build intelligently and they take calculated risks. Those risks often reap great rewards but also come with a tax in the form of defects. In a perfect world, perhaps bugs wouldn’t exist. The reality is that perfection is unlikely when taking on problems with solutions that actually challenge us, and we love a challenge! We’re going to make mistakes and those mistakes must be managed to ensure progress and success.
Triage
Atono is focused on the wellbeing of our customers. Bug triage is built into the Atono product with a customer-centric approach in which we evaluate two key factors to determine the risk rating of a particular defect.
Probability
What is the likelihood that a customer will experience the bug? Often, this is an evaluation that can be made by answering some questions:
What proportion of the user base has access to the functionality?
Is the impacted functionality frequently used?
Does the bug only occur in specific situations (i.e. an edge case)?
With this knowledge, we can assign a probability using a 5 point scale. Some examples:
Impossible: Perhaps this is a defect with our internal tooling, or non customer facing backend system.
Unlikely: Possibly of low use and an edge case.
Possible: Administrator only function, and performed rarely.
Probable: Most users use the functionality and do so often.
Certain: Unavoidable. Almost every user of the software is likely to encounter the problem.
Impact
To what degree is the customer impacted by a loss in functionality? Again, we have a 5 point scale.
Negligible: Little to no loss of functionality
Minor: Some loss but still working (i.e. misaligned labels).
Medium: Defect is more than an inconvenience but has a simple workaround.
High: Feature can still be used but has diminished capabilities. Perhaps there is a workaround but it is hard to use.
Severe: Feature is broken and there is no workaround.
Assessment Challenges
Sometimes determining the probability or impact of a bug can be difficult. Generally speaking, a well written bug can be triaged immediately, but sometimes more information is needed to better understand the scope of a problem. We use the ‘More info required’ button in Atono to assign the item back to the reporter, letting them know that we need some help rating the bug.
Rejected!
Sometimes we make the determination that we’re not going to fix a bug. This is where the Reject button comes into play. A rejected bug will be set to ‘Won’t do’ and removed from the triage list.
Some reasons we might reject a bug:
The bug is invalid. This may be the result of a misconception by the bug reporter.
Duplicate! We already have a bug in progress (or maybe even fixed!). We’ll typically comment on the rejected bug with a link to the duplicate.
Rarely, we acknowledge that the bug is valid but determine that we’re not likely to ever fix it given its risk rating versus the level of effort required by the fix. In the event that a customer finds and reports the bug, we re-open the item and triage with a probability of ‘Certain’.
Risk Rating and Prioritization
Once the probability and impact are established, we multiply them to calculate the risk rating. The risk rating provides a natural means by which to prioritize our bug fixes.
At Atono, a risk rating of 25 requires an immediate fix. Defects with ratings of 20 or higher are considered blockers for release, while defects with a rating of 16 or higher are considered very important. Below 16, defects are largely worked by risk rating order or in groupings of functional areas to make for efficient use of development time.
Triage to Triumph
When working with complicated software systems, bugs are an inevitability. How those bugs are evaluated and prioritized is important to both our customers' happiness and our developers' state of mind. The Atono triage tool has been built to mirror our internal process and streamlines the management of our bug backlog. Simple, objective evaluations of the probability and impact of a defect allows us to prioritize and fix quickly so that we can get back to building new and exciting features.