Effective Engineering e-Newsletter – 6/10/2004

This is your bi-weekly e-Newsletter from Effective Engineering Consulting Services (www.effectiveeng.com).  If you would like to receive Effective Engineering e-newsletters as they are published, please send an email to e-newsletter@effectiveeng.com, and we will add you to our distribution list.  Comments and suggestions are welcome and encouraged!


eN-040610:

Development Methodology: Test –- to Verify
  By Tom Dennis – President, Effective Engineering [tdennis@effectiveeng.com]


In two previous e-Newsletters, I described the basics of development methodologies (see eN-030828 – Development Methodology Basics: Stages of Development, and eN-030911 – Development Methodology Basics: Management of Development).  Those e-Newsletters were able to cover only very broadly the basic elements of development methodologies.  The Stages of Development e-Newsletter described Requirements, Architecture, & Planning (see also eN-030925 – Development Methodology: Requirements, eN-031009 – Development Methodology: Architecture, and eN-031023 – Development Methodology: Failing to Plan Means You Are Planning to Fail!), and briefly described Build & Test (see also eN-040527 – Development Methodology: Build – to Last), and Release & Support as two other stages of development.  This and some subsequent e-Newsletters will cover in somewhat more depth Test, Release, and Support. 

The purpose of Test is to verify that products conform to their Requirements and Architecture, and that the Build process has been implemented with quality and care; it is not to attempt to Test or Inspect quality into products.  Quality must be ensured by design, and the characteristics of a Quality by Design methodology will be the topic of a subsequent e-Newsletter. 

Key people involved in Test should be actively involved in discussions during the Requirements, Architecture, Planning, and Build stages, so that design elements that may impact or be impacted by Test can be properly accounted for in advance.

There are many aspects and functions of test, some of which are carried out by the developers during the Build stage, and some of which are carried by the testers during the Test stage.  We will briefly touch on these in this discussion.

During the Build stage, it is the developers’ responsibility to Unit tests their designs.  That is, they must verify that their designs operate properly before handing them off to others.  They must verify that their design functions as specified and that all areas of the design have been tested (all possible logic combinations in a digital design; all code paths, error paths, memory leaks, exit paths in a software design; all strength, stress, heat flow aspects in a mechanical design).  Development test tools should be used extensively to help ensure all aspects are covered, and adherence to the discipline of carrying all of this out should be enforced.  Design reviews are also required to ensure that someone other than the designer concurs with the design.  This helps to avoid “forest and trees” types of problems.  Only after all of this has been done can developers’ designs be “checked in” as “ready” to be handed off to Test.  In cases where individual designs do not stand alone, but only operate in conjunction with other designs as part of “modules”, then all of the above should apply at the “module” level as well.

Once a product or elements of a product are released to Test, many methods are applied to verify the integrity of the design.  We will go into some of these in more depth in later e-Newsletters, but will briefly describe some of them here:

► Integration Testing: This is the testing of combinations of design elements as they are turned over to Test.  More and more pieces of the design get combined to begin testing the design more completely.  Such incremental testing continues up to and including the initial testing of a full product release, and beyond.  Integration testing is done in close cooperation between test and development.  Bugs are reported using the formal bug tracking system, but developers work closely in the loop with test to see and understand bugs as they are found.

► Functional Testing: This is testing of the functionality of the design of modules up to and including the full product release.  It concentrates on verifying that the specific functionality and features are operating per the requirements.

► System-Level Testing: This is testing as a customer sees the product, verifying that the product works as a complete product, with all interactions and system level functionality fully operational.  Unlike functional testing, which may concentrate on the specifics of one functional area of the product, system-level test looks at the overall operation of the product in its actual intended environment.

► Negative Testing: This is testing that deliberately introduces errors to verify that the product operates properly in the presence of such errors.

► Outside-the-Box Testing: This is performing testing that “colors outside the lines”; that is doing things that are not expected, in the way customers unfamiliar with the product may try to do.  The intent is to ensure that bad things don’t happen when users do something “wrong”.

► Regression Testing: As problems (bugs) are found and reported, developers are assigned to fix them.  Once they are fixed, regression testing is performed to both ensure that the fix actually corrects the problem, and to ensure that the fix didn’t break something else that had been working before.  All reported bugs must be regression tested and verified as fixed before a product can be released.

► Black Box Testing: Once a product has been fully integration tested, functionally tested, system-level tested, etc., while working in close cooperation with development, it is essential that all changes are made and a “clean build” of the product be put together in the same way that will be done once the product has been released to manufacture.  This “clean build” must then be tested as a “black box” that a customer would receive to verify that all tests it is subjected to operate properly and completely.  At this point, the testers work independently of development to ensure the product is truly ready for release.

► Other Testing: Many other types of testing can and are performed, including ad-hoc testing, acceptance testing, configuration testing, performance testing, usability testing, and much more.  Smaller companies may combine many of these tests to reflect the limited resources they have available, but they still need to cover the needed territory.  The specific testing methods used are a function of the product and its testing needs.


Test is a critical element of a development methodology.  Too often it is viewed as an element to correct quality problems.  If viewed that way, the development may rely on Test to find their quality problems.  This is wrong and far too late in the process.  Developers must be fully disabused of this notion.  The real purpose of Test is to verify that the design is correct, not to find the problems that development missed.  To be effective, quality must be designed in, and Test will serve to verify that designed-in quality.  When this is truly the case, the Test process can be a straightforward one.  Wouldn’t that be great!?


Copyright © 2004 Effective Engineering Consulting Services, All Rights Reserved