Testing, Testing: The Ps and Qs of SQA

Matta Saikali

This article is the first of a three-part series that applies to medium sized Software Quality Assurance (SQA) departments that have to deal with multiple projects testing internationalized and localized applications on multiple platforms with limited resources. This first article introduces general principles applicable to most testing projects.

Paragraphs in boxes will list real life challenges that my team and I have encountered.

The Need For An Independent And Isolated Test Lab

An isolated test lab is essential for any SQA department. The test lab should be independent from all other networks, especially the company network. Machines, including servers in the test lab, will crash often and other networks should not be affected by this frequent occurrence.

We once had a DHCP server that published large amounts of IP addresses every time we rebooted it. This caused the company network to crash twice before the MIS department banned us from their domain.

Furthermore, the test lab may not be as secure as other networks in the company due to testing requirements and constraints. However, connecting to other networks in the company will still be necessary. In this case a machine acting as a gateway may do the trick. The point is to aim for physical separation whenever possible.

Testers need their own stable and personal machines for recurrent tasks such as: e-mails, documentation, bug reporting, weekly reports, timesheets, etc. Testers should not use these machines to run tests although they may use them to develop test scripts and test documentation. It is actually counter productive for testers to use their own machines for testing.

The personal machine of testers should be physically outside the test lab in order to avoid confusion and hastiness.

The Test Lab Network Administrator

During the first few months, a dedicated network administrator is required to sort out all the problems of setting up the test machines and the test network(s). Some test networks will be internal to the company but others may be visible to the outside world (e.g. via the Internet) to allow customer acceptance testing. Later on, when the maintenance workload has diminished, the administrator will help analyze the operating system specific requirements.

We hired a network administrator and trained him to write operating system specific test scripts and security test cases based on product requirements Finding and developing such a unique combination of expertise proved very valuable.

Selecting Hardware For The Test Lab

The hardware in the test lab should not only be selected for its ease of maintenance or cost. For internationalized and localized products, the test lab should represent the local markets as much as possible. Hardware that is representative of the target markets must be chosen; in many cases hardware manufactured in the targeted countries is the best choice. A machine equipped for the average person in North America differs from a machine targeted for the average person in Europe or in the Far East. In particular, certain countries have their own hardware standards; examples include Japanese keyboards and Arabic OCR scanners.

Keep in mind that no matter how big the test lab is, it can never represent the market fully. Striking agreements with local distributors to use their facilities will prove helpful.

To test one of our products we used one of our partner's "laptop test lab" and "competitive test lab" which were both much bigger than our own.

In order to keep up with market trends, a special annual budget must be set aside for renewing test lab machines.

Reproducibility And The Known Initial State

Reproducibility is a fundamental concept of the development process. Relative to testing, reproducibility means firstly that the same tests performed on the same product will yield the same results; and secondly that programmers will be able to easily recreate the bugs described by testers in their bug reports.

Reproducibility is crucial for any development project whether released or still under development. Consistency in finding and reporting bugs aids the development team in releasing a more stable product. In similar fashion, once the product has been released, the reputation of the product and the company depends on the speed of response customers get when a problem arises.

To achieve reproducibility each execution of any given test must begin at the same starting point; this is called the known initial state. This known initial state is often achieved with a machine reboot (or something equivalent). The known initial state is essential to clearly and efficiently describe bugs. Bug reports can then describe the sequence of commands and actions that will reproduce the bug starting from the known initial state.

It is not uncommon, especially in the early releases of a new product, for tests to cause fatal errors in the application being tested. When such an event occurs, when a test causes an application to crash, the OS (Operating System) may be damaged or left in an unknown state. At this point, the only way to ensure a known initial state will be to somehow restore the OS itself and possibly other software packages. Given that in a typical busy test lab multiple tests are being run on multiple machines in parallel, the need to reinstall the OS and certain packages can occur very frequently.

Therefore; speed and consistency in achieving a known initial state are crucial to avoid bottlenecks in testing. In particular, for localization testing, there is a need to setup dozens of OS versions in different languages. Regular OS installs are much too long; therefore, alternate methods must be used. At least three such methods exist and can be combined creatively; we will present these methods with their pros and cons in the second article of this series: "Achieving A Known Initial State Efficiently".

Conclusion

This article sets up the groundwork for the next two articles that will cover specific testing issues that are particularly acute with internationalization and localization testing.

"Achieving A Known Initial State Efficiently" will describe how to setup machines in the lab in order to achieve speed and consistency in getting to a known initial state.

"Optimal Platform Coverage" will describe how to plan the setup and use of test machines in order to get maximum testing and optimal coverage of OS-language combinations within a limited time frame and with limited resources.

Matta Saikali is a Software Quality Assurance consultant specializing in test analysis and testing management especially in structuring and building SQA departments. Formerly an SQA director at Gemplus, Purkinje and Alis Technologies.

You can reach him at Matta.Saikali@sympatico.ca