How do you know when you finished a feature? You might have a checklist, to make sure multiple people looked at all the artifacts, and that everything is checked in. (This is easier if you work as a mob/ensemble, pair, or swarm.) In addition, a product leader might approve the feature. Finally, the team, a product leader, or the customer(!) might demo the feature.
Teams have all kinds of ways to know when they finished a feature.
But, Something Happens when you release a new increment to the customer. The customer finds a problem, a defect. That defect escaped all your checklists, approvals, and demos.
Now what?
It's time to measure and learn from defect escapes.
Why Measure Defect Escapes
If you don't have many defect escapes, you might count them and deal with them as they arise. But if you are similar to one of my long-ago clients, they could plan on too many defect escapes every time they released another version of the product.
When they planned a release, they also planned to increase the number of support staff who manned the phones and email mailboxes. That's because they knew the customers would find problems the development teams had not found.
They brought me in to help them stop this craziness, so I asked to see their data.
Customer Support showed me their incoming call and email logs that showed that calls increased 20% and emails increased 25%. But the development teams? They could find the details in their defect tracking system, but they had no idea of the magnitude of the problem, where the problems lay, or how long they took to fix anything.
That's because the development teams worked by phase, not by feature. Worse, they implemented across the architecture, not through it. (See What Lifecycle or Agile Approach Fits Your Context? Part 1, Serial Lifecycles, and page down to the part about single-function teams.) Worse, they had a program, not a project. They were organized as component teams. See Component Teams Create Coupling in Products and Organizations to see the issues of single-function teams and implementation across the architecture, instead of implementation by feature.
Everyone was stressed, trying to fix the problems and respond to customers. And, because they did not release very often (just once a quarter), they had no idea when they introduced these problems.
They were literally flying blind. But what should they measure?
How to Measure Defect Escapes
First, they decided to release the entire product internally every week on a Wednesday afternoon. (They'd originally suggested Friday, but I asked them how many people liked to take off early for the weekend. They went with Wednesday.)
Just having a weekly cadence showed them problems much, much earlier. Yes, they had to spend time fixing those problems, but they were under no illusions about the quality of the product.
At first, management told them they were “too slow.” That's when I intervened and explained the difference between the illusions they'd all had before about the real progress and the actual progress they now saw. Not all the managers agreed with me, but that was okay.
Because they worked in phases, they had trouble tracking when they introduced a defect. But, here's what they could count, pre-customer-release:
- The discovery time: How many weeks (Wednesdays) since the time this code had changed to the time they discovered the defect.
- The fixing time: How long it took them to fix the defect. (See What Does It Cost to Fix A Defect.)
Then, since they still only released quarterly, they could use those same measures for post-customer-release. Now it was time to learn from their data.
Learn from Defect Escapes
Because they had pre- and post-customer release data, they expected to learn different things. And they did.
From their pre-customer release defect escape data, they learned:
- The requirements document was out of date almost as soon as the marketing people had completed it. Most of the requirements-based defect escapes were due to changed requirements.
- The detailed design and code reviews were almost a waste of time. That's because no one could imagine the real product state until QA started to test the features. (The problems of implementing across the architecture, not by feature.)
- They did not have enough hooks into the product to test enough unique features, either with an API or with manual testing. As a result, the testers did not test too many of the typical user experiences.
Because they now released weekly, internally, they were able to substantially reduce their post-release defect escapes.
However, at one point, all the teams realized they worked primarily on defects all through the release, not just after they released. But it took all those measures to realize their phases were total fiction. Finally, they moved to an incremental lifecycle and then to an iterative and incremental but not agile lifecycle.
Where Do Your Defects Escape?
Most of my products are books, workshops, and presentations. I work hard to prevent defects in all three. And I guarantee you this—you can pick up any of my books, page around a bit, and you'll find a typo. Makes me nuts. I use technical reviewers, editors, and proofreaders. And I still find typos after I released. And I'm the only developer on this product! (It's the same when I pair. We still find problems.)
I suspect that my defects escape because I am the only developer. I don't have a full, cross-functional team supporting my work.
But at most of my clients, I find this data:
- The longer the backlog or the longer the requirements document, the more problems in the backlog or the document.
- The less people collaborate, the more likely they will create defects they can't see by themselves.
- The more phases, the slower the product comes together—and the more defects people create.
- The more people implement across the architecture, the more defects they create.
Almost all of these clients think they are using an agile approach. But while they use the agile words, they still work as if they are in a serial lifecycle.
Trap Those Defects so They Can't Escape
In contrast, here's what my clients have used to reduce defect escapes:
- Frame requirements as problems to solve, not how to solve them.
- Plan just enough and then decide what to do next.
- Work as a collaborative, cross-functional team, limiting all the features in progress. That allows you to build in any review as a mob/ensemble or as a pair.
- Implement by feature, never across the architecture. (See How Long Are Your Iterations? Part 2.)
- Release internally as often as possible. I like every day, but you might need practice to get to once a week. Then release externally as often as possible, making it easy for your customers to upgrade.
Yes, that's a more agile approach than what my long-ago client did. But I don't see all teams using these approaches now.
Finish for Real
Defect escapes slow down your current product development. Worse, they can create a ton of WIP (Work in Progress) because the team has to stop what they were working on, and return to what they thought was done. That can create team morale issues, not just product or customer satisfaction issues.
Instead, focus on keeping the code (and any necessary documentation) clean. Finish and release. Continue. And, know where and when you have defect escapes, so you can choose what to do.
You'll reduce your defect escapes to some small number, so you can focus on new product development.