Testing Strategies for Disfunctional Software Organisations - Why Doesn't it Work Better?
(Page 2 of 3 )
Testing: Why Doesn't it Work Better? Today, we live in a world in which defects in software products have become the norm. Most software products are not tested thoroughly, so both users and the organizations that write the software expect them to have bugs. Why does this condition prevail?
Waiting Until the Eleventh Hour
Most software projects don't finish on time because most of the defects testers do find before release are discovered at the eleventh hour -- that's when most testing departments receive the product for testing. And, as the time frame is very short and unrealistic, the testing department is put under tremendous pressure to release the product without testing it thoroughly.
Although some project managers understand the importance of testing earlier, many don't have a handle on how to accomplish it. Some have tried hiring a lot of testing personnel and buying expensive tools, only to eventually give up, simply because they don't see enough return on their investment.
Whose Job is it Anyway?
Another problem is assigning responsibility for testing. Inexperienced project managers often think the testing department should be fully responsible for product quality -- that developers have nothing to do with it. They don't even expect developers to test changes they make before checking in their code. This may delay defect detection until much later in the project, which results in missed deadlines.
In other cases, project managers do understand the need for some testing before developers check in changes, but they don't understand how to accomplish it without "wasting" development time. In yet other cases, managers ask developers to test before checking in changes, but they have no visibility into the amount of testing the developers actually do and no way of enforcing their requirement. They must depend on the developers to "do the right thing." As we all know from experience, unless you have the right processes in place, this strategy simply does not work.
Unrealistic Expectations
Most testers work at the application level; they don't necessarily understand the source code, so it is hard for them to understand and test all the possible failure scenarios in an application. And, even if they do understand the source code and all of these scenarios, it is very hard, expensive, and impractical to test all of them at the application level. Unfortunately, this is what most organizations ask their testers to do, and the costs are enormous.
For example, suppose you are in charge of QA for a spreadsheet application. A significant feature of a spreadsheet application is its formula-processing component, and there are several different mathematical operators one could use for this. It would take at least a few hundred tests to thoroughly test the application's formula processing capabilities. Expecting your testing team to test the application by entering all possible permutations and combinations of formulae each day would be unrealistic. The process would be very time consuming and error prone; and it simply wouldn't scale, because iterative projects increase functionality as they evolve.
Some managers invest heavily in GUI tools to automate this task, but then most of these testing teams spend their time maintaining the tool automation scripts rather than testing the application itself. GUI automation tools have their place, and they should be used with caution; they are certainly not the answer for all your testing needs. Still other managers hire lots of testing personnel for high-demand projects -- but there is still nothing to prevent developers from checking in broken code.
Making Improvements
Fortunately, there is a better way to test. It begins with investing in a component-level automated testing framework. Since developers are familiar with their code, they have the capability to test an individual component or module without having to build the entire application. They can write component-level tests only once. Then, before they check in their changes, they can simply start the automated test suite, wait for it to finish, and make sure everything is fine.
In our spreadsheet application example, developers would write component level tests for the formulae processing component just once; then, the test automation framework would run these tests each time the developer changed the source code for that component.
Most testing can be automated using this approach, which has significant advantages. First and foremost, component-level testing is very simple and inexpensive compared to testing the entire application. A significant percentage of the testing can and should be done at this level; even though it takes up a bit of a developer's time, the returns are huge. Component-level testing is much more cost effective and can be easily automated. And what does this better approach yield in the end? A development team that only checks in verified changes, and a testing team that focuses on the application as a whole, where testers can best apply their talents.
Introducing the Unit Test
Testing at the individual component level is known as "unit testing." Unit tests are designed to test a single class or component or module in isolation.
Unit testing does require developers to write a few additional lines of code, but it doesn't have to be a waste of their time. After all, unit tests actually make developers more productive by helping them detect problems early in the project lifecycle. Since unit tests are command-line driven, they don't require expensive GUI tools to automate them. In fact, with a small investment in a "unit testing framework" you can completely automate the running of unit tests. Developers have to write these tests only once. Then, they can run these tests each time they make changes to the associated class or component.
In our spreadsheet application example, developers could simply write unit tests for the formula-processing component to test it in isolation. Running these tests would be a lot faster than running a comprehensive test set from the application GUI.
Unit tests also aid developers in debugging. For the development projects I've worked on, debugging an entire application meant rebooting the machine every time the application failed. Had there been unit tests for each component, we could have debugged at the component level without rebooting at all.
Finally, unit tests help to unify project teams. Even though developers write the unit tests, there is nothing to stop the QA personnel from running them. Developers can run unit tests only for the components they are working on, and then QA personnel can run unit tests for all the components at regular project milestones, and analyze the results. Of course it is important that QA teams test the application as a whole as well, so let's consider the relationship between component testing and testing the entire application.
The Role of System Testing in the Unit Test Strategy
Testing the application as a whole -- or system testing -- is what QA teams are uniquely responsible for. In most cases, what you can test easily through unit testing requires significantly more effort and resources to test during system testing. Yet, many organizations don't adopt unit testing and put an excessive burden on their QA team, which often leads to QA team failure and poor product quality.
Unit testing is efficient, inexpensive, and simple, whereas system testing is time consuming, expensive, and often done manually -- which can lead to errors. However, both methods are necessary. The trick is to strike the right balance between the two. Early involvement by a QA architect can help project managers decide the most effective testing strategy. In our spreadsheet application example, unit tests can ensure that all the different formulae work, whereas a single system test can ensure that the formula processing component is integrated properly into the application.
If you expect drastic changes in your GUI, then a GUI automation tool may not be helpful. However, if your GUI is mature and will not change a lot, consider investing in a GUI test automation tool to automate system tests. With a unit testing framework in place you will need fewer system tests, so the time required to create and maintain the GUI test automation scripts will be minimal.
Hurdles in the Unit Testing Strategy
If unit testing is the best way to go, why don't more software development organizations adopt it? Well, there are a few hurdles managers should be aware of.
First, as I've already mentioned, in many organizations, project managers do not expect developers to write unit tests, and most QA managers don't have anything to say about how the development team works. The unit testing strategy will not work unless engineering managers, senior management, and even the developers buy into it. Here is a real world example. One of my friends who works for a large ERP vendor tells me that when his manager initially proposed unit testing, some members of his development team resisted the idea; but his manager wanted to try it because their last project had serious quality problems.
Though they were reluctant, he made the developers do unit testing. Now, a year later, my friend tells me that the only defective components they find in system tests are the ones for which they forget to write unit tests. The entire development team has bought into the idea, because unit testing is actually helping them be more productive.
Second , another hurdle is that many senior managers do not consider unit testing until too late in the lifecycle. Most poorly designed applications are the work of developers who lack a good understanding of component architectures. And poorly designed applications are hard to test, too. It pays to have your architect think about testing during the modeling stage; better yet, you should hire a QA architect early -- in the Inception phase of the project. The unit testing strategy not only helps simplify the testing process, but also forces developers to write well "componentized," reusable code.2
A third hurdle is that, quite often, existing applications are not componentized, so it would be difficult for a unit testing strategy to work. (If you want to find out how well your application is componentized, just ask your developers how easy it is to write unit tests.) In this case, a good strategy is to treat the entire application as one component and implement a command-line interface that drives the application from the command-line, without invoking the GUI.
This will help you automate testing through simple command-line scripts, as opposed to testing totally through the GUI. As command-line testing is much less expensive than GUI testing, you'll benefit greatly from this approach. And if you happen to be in this situation, I'd also recommend bringing in a testing consultant.
Finally, the setup of the unit testing infrastructure is critical to success. Without a test harness that runs the unit tests, it will be hard to sell the concept to the development team. It is important to have the harness ready before asking the developers to start writing unit tests.
Otherwise, you will be depending on individual developers and testers to do "the right thing," and this simply will not work. A test harness will give managers greater visibility into the developers' successes and failures, so they will be more likely to take unit testing seriously. A testing framework is a one-time investment, so again, it makes sense to bring in a consultant to implement it. Once the framework is in place, developers can easily write and add unit tests on the top of the framework.
A good unit testing framework should have the following features:
- Enough flexibility to allow developers to run unit tests for individual components before they check in their changes, and to allow testers to run unit tests for multiple components with a single command.
- Good analysis features (such as the ability to compare test results against a baseline, etc.).
- Good reporting features (targeted toward technical as well as managerial audiences).
- Complete automation capability for running tests.
- Command-line driven (as well as GUI driven).
- Open architecture so that it can be integrated into third-party build automation tools.
Next: Conclusion >>
More Development Cycles Articles
More By The Rational Edge
/ 9 
