This is a guest post by Peter G Walen.
Good testing relies on careful thinking and interaction with the system under test, the people using it and those tasked with testing it. People will work through carefully considered scenarios, sometimes very detailed plans and occasionally “contrarian” scripts. One question, however, seems to not get asked enough at many organizations: How can this fail?
We work hard to understand the systems we work with. Diligent testing means evaluating the behavior of the system under test. We look for paths of behavior. We look for paths that would likely be encountered when the software is in use.
We do careful impact analysis and evaluations. We look for vulnerabilities in the application and the broader system. We look for potential security problems and privacy issues.
Generally, we find a lot of bugs. We can address a lot of issues with the systems we work with. Somehow when the software is pushed to production and people begin using it there are “issues.”
We know that software is never perfect or defect-free. We know problems will arise and sometimes will impact customers and users in unexpected ways.
We also know that leaving testing for the end, just before the finish date of the project often results in rushed, sub-standard work. Products are not tested as carefully when people are operating in crunch-mode. Bug “fixes” in crunch-mode tend to beget more bugs.
Instead, let us ask perhaps the hardest question we can ask: “How can this fail?”
The Beginning – Requirements
If you attend or participate in “requirements” meetings (definition, discovery, whatever) try this as an exercise. Track the items listed and agreed to as “requirements” and look to see how they are connected.
When you do that, look for gaps or leaps of faith around the current behavior of the system. One aspect of this, if the requirement states “If antimatter is set to B then X should happen.” What if the value of “antimatter” is not B but 3, what then? Are there other ways for X to happen? Can that happen under any circumstance when “antimatter” is not B?
Do the new requirements contradict how the application currently behaves? Is this intentional? Is this called out as a change? Is it possible that the new requirements set up a condition the current system, and the new requirements, do not account for?
Look for the holes. Look for the gaps. Look for the presumptions.
What presumptions are made in the requirements on purpose or about how the current system behaves? What about presumptions around existing systems the changes may interact with? I have found those presumptions and “everybody knows” assertions are good places to find problems in requirements.
Always look at how the requirements address the problem the project is supposed to solve. It is these problems in requirements that can only lead to greater problems in other areas.
The Next Step – Design
Look at how the design gets mapped out. What considerations are there? As with requirements, look for presumptions on how the current system works. Also, look for presumptions on how the individual pieces will work.
What level of consideration is given to existing vs changed vs new behavior? Are differences called out or are they fluffed over with a summary “this describes how it WILL be”?
While a final design can call out the new state, I’m a bit like the math teacher from middle and high school who insisted on seeing “all the steps” laid out to get to the answer. It might not need to be in the final copy, but it needs to be somewhere.
Why?
It is in these “thinking” steps that presumptions and assumptions will often get made. It is in these same “thinking” steps that testers can make a notable impact, before people become firmly attached to the “correctness” of their design solution.
These are easier said than done, of course. The challenge is to examine the design in ways that make the strengths and vulnerabilities understood. It is when the vulnerabilities are understood that we can then begin to look for other ways not yet considered.
The gaps that exist, and they will exist, need to be examined and discussed. Simply saying “this is missing” usually won’t work. Use this as an opportunity to learn and share, and help other people do the same. The challenge is to ask questions and discuss the team’s understanding of the area in question.
What seems an obvious hole to you might not be obvious to others. A set of gentle questions and maybe concerns can ease you into what might become an uncomfortable topic. It could also be that they know something you do not and instead of asserting “This is wrong” it might go easier if your approach is “Can you explain this please; because I don’t understand.”
Pushing Forward – Code
Examining the source code is where many organizations start with their efforts to answer the question “How can this fail?” As we have seen, there are at least two steps prior where we can look.
Code presents another potential for problems with presumptions and misunderstanding, mixed with problems from syntax and logic errors. One common trap I see is design launching immediately into code development, particularly when the person or team developing the code is the same person or team who developed the design.
Why do I call this a trap?
To be fair, it is not a trap all the time. It has the potential to become a trap when presumptions made during design are carried directly into code without careful consideration of the design and the presumptions made there.
I have seen many failures resulting from developers and designers coding while designing. Allow yourself and the project team to step back and look at the structure you are building. Without considering code, at first, ask how this addresses the problem the project is intended to solve. Too often teams will worry about the technical aspects of code while they say they are working on design. Then they find themselves solving problems that are cool (and fun) to solve, without really checking how they relate to what they are supposed to be doing.
In code, I find the simple solution – that which is easiest to understand – tends to be the most direct solution. I also find that the direct solutions tend to have fewer holes.
Looking for failure points in code often takes the form of code reviews. In some shops, pair and mob programming also helps find possible weaknesses. However, none of these will help if the failures being focused on are only coding problems.
Always look at how the code blends with the design, the requirements and the business purpose of the project.
Finally
Take a moment and see how the pieces fit together. You have a good understanding of the purpose of the project. You have reviewed and compared the requirements, the design and the code for alignment with each other and with that purpose.
You can test the requirements and the design by talking through them and examining possible impact areas. You can test the code by code reviews and pair or mob development efforts.
What is left? What haven’t we done?
Run the software, obviously. Less obviously, work with the people familiar with the business applications whenever possible. Work with them to help them understand the changes and how the software will function. Talk with them about problems they have seen and remember. Talk with them about the “normal” type of work or transactions that go through on a regular basis.
In this, I don’t mean the “experts” who helped define the requirements and the purpose of the project. I mean the people working with the system every day. Ideally, they are the same people. This is often not the case.
Get ideas from them on scenarios they have encountered. These can be the normal, simple “happy path” scenarios. Then look for unexpected twists. These can be mis-keying a value. Walking away from the screen and leaving it idle in the middle of a transaction – things that can happen to anyone. Stuff that simply happens.
Look for the cases that occur maybe a few times a year. The odd ones that “always” result in a problem. What happens now? Is it the same general problem? Is it better or worse? If this is a new system completely, does it handle the issues that plagued the old system better?
When you have done all these things to the satisfaction of the team, the business experts and you, there is one more thing to do.
Look at what has been done. Look at the results of the work. Then ask yourself two very important questions: What have we missed?
How can this fail?
Peter G. Walen has over 25 years of experience in software development, testing, and agile practices. He works hard to help teams understand how their software works and interacts with other software and the people using it. He is a member of the Agile Alliance, the Scrum Alliance and the American Society for Quality (ASQ) and an active participant in software meetups and frequent conference speaker.
