What is a test platform?

In order to understand the complexity of platform coverage in a test lab, we have to define what constitutes a platform on a test machine. A platform is the combination of many things on a test machine such as operating system, language, browsers, security protocols, network cards, etc… Let us consider a realistic testing problem where we have to test an application on 4 different operating systems (OS), with 8 different languages, 4 different browser versions, 2 different security protocols, and 3 different network cards. The resulting number of platforms (4*8*4*2*3) is 768, which is a complete but unpractical result. Reducing the number of platforms to a realistic number involves making many decisions on what not to test and what is an acceptable risk. For example the testing team may decide not to test the network cards with every possible language and browser, or they may decide not to test every browser with every language etc… The testing team will eliminate platforms until a minimal acceptable number of platforms is reached. Whatever minimal number is reached, it is not realistic to assume that each test machine will be loaded with all the possible platforms. Therefore, a plan is required in order to decide how many machines are needed in the test lab and how these test machines will be loaded with the different platforms.

Paving the way for localization testing

An internationalization test suite consists of all the test cases used to verify the internationalized application. It is recommended that the internationalization test suite contain all the test cases of the original (English) product.

Applications should go through internationalization testing before they are subjected to localization testing. Internationalization testing consists of fully testing a non-localized application over English operating systems, and some selected non-English operating systems, after the application has been internationalized. Localization testing consists of testing the localized application over non-English platforms. The main benefit of internationalization testing is to reduce the amount of localization testing required.

Since we are dealing with multiple languages, it is often not practical to use the complete internationalization test suite for localization testing. The localization test suite is made up of test cases developed to cover locale specific issues and a few test cases selected from the internationalization test suite.

Since the localization test suite is a fraction of the internationalization test suite, it is safe to assume that the test lab should have more English dedicated operating systems than any other specific language.

Populating the test lab

How many machines do we need in the test lab?

Each test machine should have a base operating system from which users can access the other installed platforms on the same machine. It is also recommended that at least one platform per machine be reserved for special tests/requirements. Special tests include interaction tests with other software/hardware, and tests that require an elaborate setup of other applications. A special requirement may come from developers to setup debugging tools on certain machines to find strange bugs "that do not occur on a developer's machine". The number of machines required for internationalization and localization testing may be calculated using the formula:

Number of Test Machines = (number of target languages) * (number of OS) * (number of machines required to efficiently run a test suite) / (number of platforms loaded per machine)

The number of machines required to efficiently run a test suite is usually proportional to the number of testers assigned to the project. It is by trial and error that you discover the optimal number of platforms to load per machine. Many factors will influence the optimal number of platforms per machine such as CPU speed, RAM capacity, hard disk capacity and speed, as well as the strategy used to establish the known initial state (link to previous article).

From the above formula we can come up with the number of client machines required in a localization test lab.

To continue with our example, let us assume that 3 machines are required to efficiently run a localization test suite, and that each machine can hold up to 10 different platforms where one platform is reserved for the base platform, one platform is reserved for special tests, and one platform is reserved for internationalization testing. This leaves us with only 7 possible platforms per machine. From these numbers we get: nTestMachines = 12, which is the number of client machines required. We deducted 1 from the languages since English has been taken care of.

The number of servers in the test lab should be calculated on the basis of the number of client machines and mostly on the specific testing requirements.

Determining platform priority

The testing project stakeholders (customer support, product manager, development manager, etc.) assign a coefficient for all the items in the variables that are considered when determining what constitutes a platform. If we consider that a platform is the combination of an operating system, a language, a browser, a security protocol, and a network card, then we have to assign a coefficient for every operating system, language, browser, security protocol, and network card. A simple way to assign coefficients to variables is to calculate the market share percentage of each item in that variable class. As the testing department evolves, a more elaborate scheme may be used to determine coefficients such as incorporating a technical risk factor (probability of a bug occurring).

In the platform priority matrix, the higher the number the more important the platform is.

Platform deployment plan

A platform deployment plan identifies what platforms are loaded on what test machines. At this point we should have established the number of client test machines required and drafted a platform priority matrix. The client machines in the test lab should be numbered from 1 to n. The number of machines required to efficiently run a localization test suite determines on how many machines the same platform will be installed. The platforms are installed starting with the highest priority, as determined by the platform priority matrix, down to the lowest priority.

Conclusion

With the setup described above, test lab users will be able to easily retrieve platforms for testing and debugging. Platform scheduling conflicts will be minimized. This same method can be repeated on a periodic basis to allow the test lab to adapt to new company priorities and projects.

Matta Saikali is a Software Quality Assurance consultant specializing in test analysis and testing management especially in structuring and building SQA departments. He was formerly an SQA director at Gemplus, Purkinje and Alis Technologies.

Matta will be conducting an Internationalization & Localization Testing workshop at the LISA Forum Europe in Heidelberg, Germany in November. Click here for more information.