Chapter 13

Chapter 13

Software Testing Strategies

CHAPTER OVERVIEW AND COMMENTS

This chapter discusses a strategic approach to software testing that is applicable to most software development projects. The recommended process begins unit testing, proceeds to integration testing, then validation testing, and finally system testing. You should emphasize the spiral—I believe it is a useful metaphor for the software engineering process and the relationship of testing steps to earlier definition and development activities.

The majority of testing that most students have done has been ad hoc. Therefore, the key concept for students to grasp is that testing must planned and assessed for quality like any other software engineering process. Students should use the Test Specification template from the SEPA web site as part of their term project.

13.1  A Strategic Approach to Software Testing

This section describes testing as a generic process that is essential to developing high quality software economically. It is important for students to understand the distinction between verification (building a product correctly) and validation (building the right product). It is also important for students to be aware that testing cannot replace attention to quality during early software engineering activities.

The role of software testing groups in egoless software development is another important point to stress with students.

Section 13.1.3 and 13.1.4 introduce some of the similarities and difference between testing for traditional software and testing in an object-oriented environment. You might want to revisit the differences between traditional and OO components, because these difference have a strong bearing on testing strategy (and tactics).

Section 13.1.5 discusses the issue of how to determine when testing has been completed. Which is an important issue to consider, if students buy into the argument that it is impossible to remove all bugs from a given program. This issue provides an opportunity to reconsider the role of metrics in project planning and software development.

13.2  Strategic Issues

Several testing issues are introduced in this section. Planning is described as being an important part of testing. Students may need assistance in learning to write testing objectives that cover all portions of their software projects.  Formal technical reviews of test plans and testing results are discussed as means of providing oversight control to the testing process. Students should be encouraged to review each other’s testing work products some time during the semester.

13.3  Test strategies for Traditional Software

Testing for tradition software begins “in the small” and moves toward “testing “in the large.” Your students should understand the “big picture” reasons for this strategy. You might also discuss the “daily build and smoke test” strategy that is used by many software product builders and is an of encountered in agile process models.

Section 13.3.1 discusses unit testing from a strategic viewpoint. Both black-box and white-box testing techniques have roles in testing individual software modules. It is important to emphasize that the white-box techniques to be introduced in Chapter 14 are most advantageous during this testing step. Students need to be aware that testing module interfaces is also a part of unit testing. Students need to consider of the overhead incurred in writing drivers and stubs required by unit testing. This effort must be taken into account during the creation of the project schedule. This section also contains lists of common software errors. Students should be encouraged to keep these errors in mind when they design their test cases.

Section 13.3.2 focuses on integration testing issues. Integration testing often forms the heart of the test specification document. Don't be dogmatic about a "pure" top down or bottom up strategy. Rather, emphasize the need for an approach that is tied to a series of tests that (hopefully) uncover module interfacing problems. Be sure to discuss the importance of software drivers and stubs (as well as simulators and other test software), indicating that development of this "overhead" software takes time and can be partially avoided with a well thought out integration strategy. Regression testing is an essential part of the integration testing process. It is very easy to introduce new module interaction errors when adding new modules to a software product. It may be wise for students to review the role of coupling and cohesion in the development of high quality software. . Don't gloss over the need for thorough test planning during this step, even if your students won't have time to complete any test documentation as part of their term projects.

13.4  Test Strategies for Object-Oriented Software

This section clarifies the differences between OOT and conventional testing with regard to unit testing and integration testing. The key point to unit testing in an OO context is that the lowest testable unit should be the encapsulated class or object (not isolated operations) and all test cases should be written with this goal in mind. Given the absence of a hierarchical control structure in OO systems integration testing of adding operators to classes is not appropriate. Students should try writing an integration test plan for an OOD based on one of the three strategies described in this section (thread-based testing, use-based testing, and cluster testing). Similarly, students should try to write a plan for validating an OOD based on the use-case scenarios defined in the OOA model.

13.5  Validation Testing

In this section validation testing is described as the last chance to catch program errors before delivery to the customer. Since the focus is on testing requirements that are apparent to the end-users, students should regard successful validation testing as very important to system delivery. If the users are not happy with what they see, the developers often do not get paid. It is sometimes worthwhile to have students test each other’s software for conformance to the explicitly stated software requirements. The key point to emphasize is traceability to requirements. In addition, the importance of alpha and beta testing (in product environments) should be stressed.

13.6  System Testing

System testing is described as involving people outside the software engineering group (since hardware engineers, system engineers or network engineers are often involved). Several systems tests are mentioned in this section (recovery, security, stress, and performance). Students should be familiar with each of them.

A thorough discussion of the problems associated with "finger pointing," possibly with excerpts from Tracy Kidder's outstanding book, The Soul of a New Machine, will provide your students with important insight.

13.7  The Art of Debugging

This section reviews the process of debugging a piece of software. Students may have seen this material in their programming courses. The debugging approaches might be illustrated by using each to track down bugs in real software as part of a class demonstration or laboratory exercise. Students need to get in the habit of examining the questions at the end of this section each time they remove a bug from their own programs.     

To emphasize how luck, intuition, and some innate aptitude contribute to successful debugging, conduct the following class experiment:

1.      Handout a 30 -50 line module with one or more semantic errors purposely embedded in it.

2.      Explain the function of the module and the symptom that the error produces.

3.      Conduct a "race" to determine:

         a) error discovery time     

         b) proposed correction time

4.      Collect timing results for the class; have each student submit his or her proposed correction and the clock time that was required to achieve it.

5.      Develop a histogram with response distribution.

It is extremely likely that you find wide variation in the students' ability to debug the problem.