What is a known initial state?

A known initial state (or KIS) is a preset starting point for the execution of any given test; in testing, this is often achieved with a machine reboot (or something equivalent). The KIS is essential to clearly and efficiently describing bugs. Bug reports can then describe the sequence of commands and actions that will reproduce the bug, starting from the KIS. Without a KIS, it becomes very difficult, if not impossible, to reproduce and fix bugs.

Test Lab Objectives

There are many reasons for setting up an independent test lab. Here are a few objectives that impact our discussion:

• availability and quick access to test environments,
• capability and confidence to retrieve a known initial state every time, and
• reduce the network maintenance and cost.

The most important thing is not just to achieve these objectives but also maintain them throughout the testing process.

Methods for achieving a KIS

There are many ways and available tools to help achieve the objectives of setting up an independent test lab. The test lab network administrator should choose the best solution that fits the needs of the company, keeping in mind that the test lab will support many projects with different internationalization requirements. Each test machine may be set up with any of the following methods:

Multi-Boot

MS Windows allows users to setup multiple OSs, on the same test machine, which are selectable from a menu at boot up. Changes made to any of the OSs, while loaded, will be permanently saved on that particular OS. The multi-boot method is the least resistant to crashes and requires frequent maintenance without any gain in speed or reliability. Using this method alone, ignoring any of the methods below, makes it impossible to return to a KIS with full confidence.

Virtual Machine

Each test machine is loaded with a base OS and with virtual machine software. The virtual machine software will allow the user to install many OSs as virtual OSs that can be called from the base OS. The user can now move from one virtual OS to the next by simply clicking on an icon in the loaded base OS. This is extremely helpful for developers and testers alike. The virtual machine method has the speed advantage over the other methods since it does not require a machine reboot. Virtual OSs are retrieved in amazing speed. Changes made to virtual OSs can either be saved or discarded upon termination; discarding changes will guarantee a KIS. However, if the base OS becomes corrupted, there is a chance that all the virtual OSs will be lost. Certain problems may also arise when testing hardware peripherals or when running performance tests. Furthermore, storing virtual OSs eat up lots of hard disk space.

Partition save and restore

Each test machine is first loaded with an OS, then the hard disk partition containing the OS is saved in a compressed form called an "image". This scenario is repeated for as many test OSs as required. All the images, for each machine, are then stored on a separate media in order to prevent loss of data.

Retrieving images under this method requires a machine reboot and an image restore. This is slower than using virtual machines; however, it is safer and does not cause problems when testing hardware peripherals.

Hybrids

A hybrid method is a combination of the methods listed above which will produce faster and better results. For example, the virtual machine method and the "partition save and restore" method can be combined. If ever one or more of the virtual OSs gets corrupted, the complete base OS and all the virtual OSs can be restored from the saved partition image. In a similar fashion, the multi-boot method can be combined with the "partition save and restore" method.

UNIX, Linux, and Vax Environment

For Unix, Linux, and VAX machines, the "partition save and restore" method may not be the best solution. Most of the time, to revert to a KIS, it is sufficient to wipe clean one or more directories. However, to test hardware peripherals or to recover from a system crash, reverting to a known initial state becomes more complex. System administrators may prefer to manually reinstall the OS, while testers will argue that restoring an OS image is the only sure way of reverting to the same known initial state.

Time drain

Setting up the proper environment to test internationalized/localized applications consumes a lot more time than the time required for the same task to test English only applications.

Documenting the process

For each test machine, the test lab network administrator should clearly post the instructions on how to retrieve a test environment and what test environments are available. These instructions are important since there are always newcomers, and other teams will also eventually use the test lab. A training session/document on the use of the test machines in the lab will prove helpful.

Conclusion

Whatever method is chosen, the lab administrator should keep in mind the objectives we are trying to achieve in setting up a test lab: speed and confidence in retrieving a known initial state.

Matta Saikali is a Software Quality Assurance consultant specializing in test analysis and testing management especially in structuring and building SQA departments. Formerly an SQA director at Gemplus, Purkinje and Alis Technologies.

You can reach him at Matta.Saikali@sympatico.ca