What You Don't Know May Help You

Some testers take it upon themselves to learn as much as possible about the inner workings of the system under test. This type of "gray box" testing is valuable, and most testers have the technical wherewithal to grasp much of what's going on behind the scenes. But it's important to recognize that sometimes "ignorance is strength" when it comes to finding problems that users will encounter.

When it comes to software testing, ignorance can actually be a source of strength. In particular, software testers who are not familiar with how the system has been put together can be in a better position to find certain bugs than testers who may be more familiar with the system internals.

Testers are often treated as second-class citizens, and this makes many of them eager to learn as much as they can about the software they are testing in order to prove their technical competence. This is no wrong, but we nned to recognize the maximum benefit of the inherent unfamiliarity that testers have when they are new to a project.

It is well known that the value of independent testing partly comes from having a set of fresh eyes on the software product. They are more apt to try things that might fail or notice problems that others have overlooked. Testers who are new to a project-with the least knowledge about the mechanics-bring the freshest sets of eyes.

Our software products will not be successful if we expect our users to have to understand the inner workings of our software. Therefore, testers without this understanding can teach us a valuable lesson about how our software will be used. There are several areas of testing in which "ignorant" testers can be helpful.

Usability
Testers unfamiliar with the inner workings of a product can often be very helpful in identifying usability problems. They might notice that the program flow is confusing or that terms used are hard to understand. Smart companies regularly put new employees through usability tests. This gives the company the benefit of the new employees' fresh perspectives, gives the employees insight into how the products they are going to help develop will be used, and puts everyone on notice that design is important. Even if you don't have such a program, you can still make sure that new testers are given a chance to express their observations regarding product usability.

Installation and Setup
Software installation and configuration are areas that are often handled late in the development process. As a consequence, early testers often must learn how to manually install and configure the software. Workarounds are common, perhaps requiring the manual copying of certain files or the manual creation of particular accounts or data sets. Testers who come to a project later won't have been trained to avoid these problems and are thus more likely to stumble across installation problems that had been deferred.

Error Handling
New testers are less likely to know how the software is "supposed" to be used and therefore are more likely to stumble across errors. It is important for software to handle errors appropriately (give appropriate notice of invalid input, provide options for recovery, and ensure that no data is lost). Error handling code is always a good place to look for defects. Informed testers will want to plan to exercise all error conditions, but the most important ones to check are the ones that the programmers didn't anticipate. "Uninformed testing" is one good strategy for finding them.

Program Flow
Early defects in software can train testers to avoid certain paths. I remember one product I tested where problems were likely if you closed dialogs with the x-button at the upper-right corner, rather than using the close button at the bottom. This happened often enough that the experienced testers never used the x-button. With time, many of these problems were fixed, but the testers still avoided the x-button as a matter of trained behavior. A new tester, untrained to avoid the x-button, found additional instances of these problems that had been missed.

Documentation
If you already understand the system, it is hard to read documentation as if you don't. This is why all new testers on a project should be expected to review documentation. As a new tester on a project, I once found a bad documentation bug in the product tutorial. The problem was with the instructions for setting up some data in an early chapter. They were incorrect, but the error was of no consequence until several chapters later, when further processing wouldn't give you the documented results. Even though this version of the documentation was in use in the field, everyone internally thought that someone else knew about the problem and was taking care of it.

It is certainly true that "ignorant" testers are more likely to report bugs that won't actually be fixed, either because the error is theirs or because they are simply reporting a known design problem that is not going to be fixed. Nonetheless, the value of the fresh perspective they bring makes it worth having to sort through their bug reports. It also sets up a healthier dynamic than when testers try to anticipate which problems are actually valid before they report them.

Thus we get the most out of the staff we have and not feel like they need to know more than they do in order to provide valuable contributions. That said, we can't argue against learning and increasing our knowledge. We can learn a lot about the internal workings of the systems we test and this "gray box" information is incredibly valuable to developing sound testing strategies.

When we get new testers on staff, let's make sure to benefit from the fresh eyes they bring and recognize the value of what they don't know.