In this article, I outline some of the basic issues that need to be addressed in the development of the basic International English version of the product together with some of the techniques and tools that are available. In a way, this is an abbreviated attempt to answer the question "What do U.S. software companies mean when they talk about Software Quality Assurance?".

There is enormous literature for this field based on considerable work in the last ten years; Reference 1 contains a 14 page bibliography, for example. Rather than repeat the contents of such books and papers, I shall refer mostly to my own experience with manufacturers of shrink-wrap packages. Large companies, particularly those that produce hardware as well as software may use more advanced techinques than the ones I describe below.

U.S. software manufacturers demonstrated considerable resistance to creating formal testing procedures for their products; at first sight it appeared that formal quality control or assurance would cost money, delay releases [1] and the only immediate results from that expense were criticisms of the developers' efforts and sometimes of the developers themselves. What value did they add to the manufacturer in return for the expense? It is not possible to predict exactly what cost savings will result from the fact that events do not happen and it is certainly easy to reject fuzzy figures.

Yet as PCs proliferated and mass markets developed, the benefits of distributing software that contained as few bugs as possible did become more evident. Such benefits showed up as noticeable reductions in the numbers of calls for help to the manufacturer (who eventually set up formal Technical Support departments too) and noticeably fewer costly shipments of repaired products to many customers. And since many competing products contain the same features more or less, then the characteristic of quality is one practical way in which one manufacturer's product can be said to stand out from another's. Whatever the benefits are exactly, by now most U. S. software manufacturers have created Quality Assurance departments to test their products before releasing them to market.

I shall refer to the group formally testing software as either the QA group or the Testing group. Technical Documentation staff write Help and documentation; they need to operate the product so that they can describe it and thus they act as informal testers. Technical Support staff need to learn about the product and in the process of their learning, towards the end of development, test the product . Usually, they pass information about anomalies that they detect to a tester who writes a report, adds it to the Anomaly Tracking System (see below) and then monitors work on the anomaly.

However, no department whose function it is to detect software bugs determines the level of quality in an organization. Instead, they provide the following kind of statistical information to management:

• The number of anomaly reports opened
• The number of anomaly reports closed
• The general time to find an anomaly
• The number of anomalies found per person-hour of testing
• The effectiveness of the testing
• The percentage of tests applied that the Software Product passes and fails

At the same time, programmers provide information about how difficult it has been to repair anomalies, and their opinion about the software's current state of repair. Technical Support and Marketing people give their opinions about how important various open anomalies are. And everyone- programmers, testers, technical writers and so on-who has worked extensively with the code can give subjective impressions. Over the months spent programming, testing and writing documentation for the product, people do develop a feel for such fuzzy concepts as the program's stability.

These statistics and these impressions help the manufacturer to decide when a Software Product is ready for release. Note that the decision to release is a business one, not just a technical one. By that I mean that the manufacturer's management considers data about the state of the market, reports on the status of the product, projected costs of further development and testing, potential lost income, estimated costs of future Technical Support and so on in addition to the statistics. All these factors must be weighed and balanced against each other. It is a case of "on one hand, …, on the other hand…". But still, the manufacturer's management makes the decision to ship. And consequently, management sets the acceptable level of quality for the organization.

For a working definition, we shall say that an error is a mismatch between a program and its specification. Of course, a program written to a bad specification is a bad program, and there are other definitions, but I will let Plato and Socrates deal with such statements as "There can never be an absolute definition for bugs, nor an absolute determination of their existence".

2. U.S. DOMESTIC QUALITY ASSURANCE

There are five simple principles in quality assurance as practiced in the U.S.:

1. Have someone other than the author check out the object to be tested. This provides a fresh point of view and probably a different way of looking at how the object is behaving.
2. Verify any and all claims and features. Thus skepticism (as well as criticism) is a useful personal attribute for any tester, unlikeable as it may be. Sometimes wishful thinking creeps into the claims, sometimes people make mistakes, sometimes things get overlooked. Even if some feature appears to be obvious and trivial, it is quite possible that the feature will not work correctly in some particular computer environment although it works in many others. Testers do not try to make the product work, they try to make it crash.
3. Have testers concentrate on finding anomalies. In any normal project, this will keep them busy enough. It is not easy to maintain a formal testing program if testers are continually asked to spend their time on other tasks.
4. Start testing the product as early as feasible in the development. The earlier an anomaly is detected, the easier and cheaper it is to fix it. It is impossible to overstate how important this is.
5. Be aware that you can never be sure that you have found all the (fatal and major) errors in a product. At best, you can only have proceeded in an intelligent manner.

As in all software projects, there is never enough time to do everything that is desired. The experience of testers manifests itself in the apt choice of a limited set of program features to test in a limited set of environments.

2.1. Theory and Experience

2.1.1. Numbers and Quantities

All software released to the user contains anomalies that either have not been detected or have been detected and not repaired (is there any other product that you buy that you expect to have bugs?). In my thirty years' experience in the software industry, I know of only two programs that are now without errors. Both are scientific, with a lot of number-crunching. One was written in BASIC by one man over a period of three or four years. Despite the code's Byzantine and sphagetti-like structure and thirteen levels of subroutine, the author instantaneously can locate the section of the code for any specific purpose. His relationship to the code strikes me as obsessive. The second is an in-house proprietary statistical program that was written thirty years ago in FORTRAN with some Assembler subroutines. While it has been ported from platform to platform, continual use by several high-level mathematicians has located by now what appears to be all the bugs.

I have heard the following numbers bruited about in Software Engineering circles although object-oriented programming, with its tendency to reuse code, mitigates these figures somewhat:

1. Traditional programming with no formal QA produces 50 anomalies per KLOC thousand lines of code.
2. Adding some formality in QA (code inspection and reviews, an anomaly tracking system) reduces this by a factor of 5, i.e., to 10 anomalies per KLOC.
3. Formal test planning plus Coverage Analysis reduces this even more, to 1 anomaly per KLOC.

It is claimed that space-flight software operates at .5 anomalies per KLOC. The quality assurance effort to reach this level of quality may require as many as 1.5 testers per coder! The .5 bugs/KLOC remaining must be subtle since the more obvious ones will have been detected and repaired. I have seen estimates that the U.S. Star Wars project would have required thirty million lines of code, presumably presenting us earth-bound mortals with fifteen thousand undetected errors controlling all that hardware orbiting the earth above our heads!

It is unclear to me how these numbers are obtained. Intuitively, they appear to be the right order of magnitude and their relationship one to the others appears reasonable. However, I have never seen an article whose methodology was convincing. After all, in their studies how do the authors detect all the errors that they claim remain in the software under study, something they must do in order to count them?

While these kinds of numbers may be useful to developers, they are not useful to end-users. What they would really want to see, I think, is the deterministic analog of a statistic like the Mean Time Between Failure. But what that is, what it means, how to measure it and lastly how it relates to the number of undetected errors per KLOC, no one knows.

Thinking along these lines leads one to suggest that programmers seed the code with deliberate errors and then management can estimate the total number of errors in the code by computing the fraction of the seeded errors detected by the Quality Assurance effort and applying that fraction to the number of unseeded errors.

Relying very loosely on statistical large number theory, the argument goes like this: Let w be the (unknown) total number of unseeded errors in the code to be detected. Suppose that at some point testing has detected x seeded errors and y errors in total. Then obviously (y-x) is the number of unseeded errors detected. If the total number of seeded errors is z, then w = z * (y - x) / x

The remaining number of unseeded errors to detect is:
w - (y - x) = (z * (y - x) / x )- (y - x)
= (y - x) ((z / x) - 1)
= (y - x) (z - x ) / x

The fraction of testing work completed is x / z. If H is the total number of testing person-hours already spent, then the remaining effort is H * (z - x) / x person hours.

Some of the assumptions underlying this arithmetic are that errors, both seeded and unseeded are distributed randomly throughout the code and that the collection of detected seeded and unseeded errors constitute a random sample. How valid are these assumptions? Validity changes with perspective; the farther away from the details, the better they are! The closer one gets to the details, the more concern there is for cause and effect; the farther away one is from the details the more useful statistical science is.

It is reported that during the five years development of MS-Windows NT, a project that cost $150 million, one team of programmers coded during the day and another team tried to break the code at night. Thirty thousand errors were detected.

My experience with word processors and desktop publishers, projects involving three programmers, five testers, and a project manager in a nine month substantial upgrade is that one would find about two thousand plus or minus five hundred errors.

2.2. Basic Definitions

This section contains general basic information about conventional QA.

"Anomaly", "Bug", "Defect" and "Error" are synonyms. An "Enhancement" or feature that someone thinks that the product lacks can be considered to be a type of anomaly.

Upon detection of an anomaly, it is written up in an anomaly report. Anomaly reports must contain the following information, at least:

• Date
• Release ID
• Environment
• Configuration
• Summary of Anomaly
• Name of Discoverer or tester responsible for monitoring work on the anomaly
• An estimate of the severity of the anomaly
• A detailed description of how to reproduce the anomaly
• A description of the actions taken on the anomaly with dates
• Names of the persons taking the actions
• Verification of repair, if appropriate
• Status of the Anomaly
• Resolution
• Priority

There should be a separate report for each anomaly.

Anomaly reports are entered into an Anomaly Tracking System. A QA group department must have a way of keeping track of errors that is accessible to all concerned. Anomalies, in a way, have their own lifetime: they are detected, evaluated and disposed of in one way or another. The Anomaly Tracking System allows all actions concerning the anomaly during its lifetime to be recorded.

Properly designed, the system allows for input from everyone involved: from the people who find the errors, the decision-maker's decisions and whether or not the error has been fixed and verified. It is a way for all concerned to communicate with each other. Manufacturers also use the anomaly tracking system to tally potential enhancements to the Software Product. They use these records to review potential new features considering product updates.

It is customary for testers to be assigned ownership of all errors, no matter who originates the error. The tester monitors the report and verifies whatever action is taken concerning the bug.

Sometimes an anomaly is easy to detect and to replicate. At the beginning of a project, most detected anomalies are easy to find. Quite a few are not only simple but dramatic. As development progresses, detected anomalies become more complex and more difficult to replicate. The number of steps needed to replicate the anomaly becomes bigger and bigger. At the time a flaw is evident, it may be impossible to recall exactly all the steps that led up to the flaw, since some of the steps may have been taken unconsciously: for example, switching on the machine, setting session parameters or testing some seemingly unrelated part of the program.

Severity: Many software developers group anomalies into four classes representing how serious they consider them to be. For example:

1. Trivial: Trivial anomalies can be nothing more than a misspelled (but not misunderstood) word in an obscure message.
2. Minor: Minor anomalies are not classifiable in one of the other groups.
3. Major: Major anomalies are serious deficiencies or failures in the program's operation.
4. Fatal: Fatal anomalies involve an important missing feature or destruction of data or a system crash.

There is also an ISO (International Organization for Standardization) standard listing six severities: critical, major, minor, warning, clear and unknown.

The severity of an anomaly is not to be confused with how much work is necessary to repair it. The effect of a fatal error can be mitigated by the existence of a work-around. The repair may be as simple as an addition to the documentation warning the user not to do something and describing another way of doing whatever (s)he wants to do.

The Anomaly Tracking System is the major source of information on project progress and status that is available to project management, especially during the later stages of development.

The status of the anomaly report can be open or closed:

Open: There is no resolution of the problem yet.

Closed: There is resolution of the problem.

At any time, it should be feasible to determine the total number of open and closed reports, or to accumulate other statistics such as the number of events in a certain time period. In a mechanized database, it is possible to generate many different types of reports.

During development, or perhaps afterwards, many actions may be taken concerning an anomaly (for example, there could be a claim that it has been repaired, but a tester could not verify it; or it could be fixed, the fix could be verified and then later programming work could undo the fix). Therefore, the status will vary and it is useful to have an open-ended text field in which all these steps can be recorded. As a rule, only the tester who monitors progress on an anomaly can officially close the report. On the other hand, the resolution of the anomaly is more complex and is handled by the developers, except for the value of "Fixed", which programmers can change back to "Pending."

Here is a list of typical values for a resolution field:

1. Pending: The anomaly report is still under review (also, the status is open).
2. Fixed: Developers believe they have repaired the anomaly. The tester will verify the fix and close the report if (s)he agrees. Otherwise, the tester sets this back to "pending."
3. Irreproducible: Programmers cannot replicate the steps for the anomaly. This tells the tester to provide more detail or to close the report.
4. Deferred: This will not be repaired in this release. The report will be closed.
5. As Designed: The feature observed is actually part of the product, not an anomaly. This means that the tester reporting the anomaly did not recognize some behavior as correct. This is not rare since test scenarios can lead one to obscure situations.
6. Can't Be Fixed
7. Withdrawn: The tester reporting the anomaly is withdrawing the report.
8. Need More Information: The developers need more information.
9. Disagree with Suggestion
10. Duplicate: This is the same as another bug.

If the development effort is large enough, then the project team may assign a priority to the report. For example, they may rate it on a scale from one to five.

A test case is a detailed experiment that verifies that a certain feature works or not. A test cycle refers to making all planned tests. A test log records the results of test cases. A test scenario is similar to a test bed or test harness, but on a smaller scale. It usually involves setting the stage so that one can run a specific test. A test script is the written procedure for a test case. A test suite is a collection of test cases, usually automated.

2.3 Developmental Process

Kaner (reference 1) presents the most realistic portrait of the developmental process that I've seen, one that correlates highly with my own experience. Often, as deadlines approach, the developmental department resembles a zoo. That description may seem a little extreme, but there is no doubt that there is never enough time for everyone to do everything they would like to do and certainly not enough time to do it in a reasoned and calm manner. Unfortunately, this is true for everyone engaged in development: coders, testers, marketers, writers and technical support staff.

One of the first things that a Project Manager does at the beginning of the project to develop a new release of a product is to look at the Anomaly Tracking System's report for the previous release. It is reasonable to assess those anomalies detected in the previous version whose repair was deferred and then to decide again whether to repair them or avoid them in the contemplated new release. And, of course, there are the suggested enhancements to consider.

Among the development milestones four are of major interest to the QA group. These are alpha, beta, the User Interface (UI) Freeze and Final. At alpha, the software code contains every feature planned. Features may not be complete or may not work well, but they are there. Some manufacturers require that all known fatal errors be repaired before declaring alpha. At beta, the product's operation is acceptable enough that it can be released to outsiders for examination. After the UI freeze, which occurs after alpha and before beta, there will be no changes in design of the user interface. At final, the product is released to the manufacturing group for duplication and assembly of the entire package.

Testing should have started long before alpha and should continue as long as it is feasible, even after final. The focus of programming work after alpha is fixing errors. Concurrent with this work there is further testing, reporting new anomalies and in many cases fixing them as well. At beta, the product is released to outsiders for examination; close customers or clients will undertake to examine the program, reporting errors and opinions about the program. Normally clients will try to use the program during their regular work.

Normally, Marketing or Technical Support staff coordinates the beta testing. But, the QA group continues testing as well. Just before Final there may be many engineering releases of the product, as many as a new release every day. At Final, I have seen the entire manufacturer's staff try out the product for a day or two. Often, a few people are designated to try Out-of-the-box tests where they pretend to be new users who want to install and use the product.

2.4. Techniques

There are several techniques and approaches in QA and the level of sophistication among them varies considerably. Of course, testers or Quality Assurance staff need to be or to become familiar with the product and its designated platforms. If some particular platform is complex, then using even the simplest QA techniques requires a high degree of expertise and computer know-how. But whatever you use, there is no way to be sure that you have not missed (major or fatal) bugs.

Common to all techniques and approaches is the concept and practice of test planning. I have seen this take many forms. For example, I have seen all the testers for a product brainstorm in a conference room for a week, with someone writing the results of their deliberations on sheets of butcher paper hanging from the walls of the entire development department. This served to familiarize them all with the product specifications and provided each tester with a draft zero or high-level plan for their specific functional area.

More usually, testers develop their own plans in the form of tables listing features to test and environments. They discuss these with other testers and with the Lead Tester (QA on QA!) in developing their plans. Sometimes, the agreed- upon plans are modified as a result of discoveries that are made during product development.

For automated QA which I discuss below, these plans are a first step in developing a test suite. In that case, the next step is to program the set of tests.

There are different types of techniques in use. In this section, I discuss some of the more important variations that I have seen. These variations include static and dynamic, black box and white box, and automated and manual. The use of one variation does not preclude the use of the other; often one can be used to expand on issues raised by the other.

2.4.1. Static and Dynamic Tests

Static Tests

In static quality assurance, everything boils down to a formal review of documents by someone other than the author. Reviews may be manual or automatic. In a manual review, formality increases its efficiency; the reviewer should make a formal report of the review. Such reviews can take the form of walkthroughs in which the author guides others through code or design or specification documents, or someone can run a desk check of the document, keeping track of the questions, issues and potential problems. Documents can include specifications, design and program listings or more conventional documentation.

Code Inspections, Code Reviews, Code Walkthroughs, Design Reviews, Editing of Help documentation, and Editing and Proofing the manual and other documentation are typical activities in Static Testing.

Not all reviews have to be manual. It is possible to perform automated reviews by using a code analyzer, a program that scans the code under test. In various forms, such programs can compute metrics such as logical complexity, and review variables, scan for unbalanced parentheses and duplicate hot keys. Some code analyzers scan source code and others scan compiled code. They can be used over and over again, each time that the engineers produce a new release.

Examples of issues that a code analyzer might check are spelling, punctuation, menus, dialogs, string tables, and measures of complexity.

There is a question in my mind as to how useful measures of complexity are. It is possible, I suppose, that management within a particular manufacturer can get used to what are acceptable values of many of the complexity metrics for the typical programming for a product. Consequently, deviations from these values may signal problems. On the other hand, they do not get to the essence of a program. There are analogous metrics for text like the number of verbs, the average lengths of sentences, average lengths of words and so on, but they do not distinguish between good and bad poetry and or between poetry and technical material in a meaningful way. Some attribute akin to meaning should be involved and that may well be beyond the capability of modern software analyzers.

The advantage of static testing is that it enables the testing process to begin very early in the development process. This is important. As I mentioned above, the earlier that a bug is discovered, the cheaper and easier it is to fix it.

Dynamic Tests

On the other hand, once code has been compiled and linked, then it can be executed. Dynamic testing takes place when code can be executed and when on-line Help is operative. Simply stated, it consists of executing the code and verifying that it responds to inputs as specified. This type of testing is normally what comes to mind when testing software is mentioned.

Dynamic tests include Beta tests, Boundary tests, Integration tests, Modular tests (Touch tests, Validation tests), Out-of-the-Box tests, Performance tests, Quick tests, Stress tests and System tests to name the most frequently used. See reference 1 or other testing books for a discussion of these.

2.4.2. Black Box or User Tests and Glass or White Box Tests

The imagery in this distinction refers to the fact that you can see inside a glass box and cannot see inside a black box. White is also the opposite of black. Consequently, glass box tests are far more thorough. In fact, they can be vital in certain cases.

Is it important to see inside the code? It could be if there is a chance that the program under test has been programmed fraudulently and part of the QA function is to test that the program is resistant to fraud. Thus, in addition to verifying that the program works as specified, it is important to verify that no additional unknown features have been included.

Consider the case of a program designed to officially tally the results of elections and suppose that someone has inserted a section of code that executes in a specified time interval only (i.e., for a few days after election day). Invoking the program for a test outside of that time interval (before election day) will not show the fraudulent activity at all, unless someone thinks to change the system date in the test.

Simple-minded user testing would consist of nothing more than running a few sample ballots against the program before election day, more to check installation than anything else. However, glass box testing with its more rigorous and disciplined approach would stand a good chance of detecting fraudulent code, if it existed, just by its very nature. Good thinking and good test planning may have led black box testers to the same point, but the chances are of their doing so are so much smaller.

Consider the sheer mass and volume of on-line transaction system programs, such as the systems used for large payroll systems or large banking systems. Couple that with the fact that these systems account for enormous quantities of money. The potential for getting away with fraud is magnified because there are so few people who know the code intimately and who can question the purpose of a few lines code buried here and there in a mass of legitimate code.

In my experience, thorough testing of large-scale on-line systems is incredibly demanding. The challenge is usually to find a suitable test harness. For example one large real-time information data base in the U.S. operates on twelve large mainframes all the time except for fifteen minutes early on Sunday morning. Some financial systems in California process two million checks per day; some insurance claims processing systems get 140,000 claims per day.

Black Box or User Tests

Black box tests are like psychological stimulus-response experiments. The user or tester provides input or a stimulus, and with no knowledge of the software's specific inner workings, observes the output or response. As a rule, testers need considerable ingenuity to test all responses for one stimulus and all the stimuli for one response. An example of a case where there can be more than one response to a specific stimulus is the response to pressing a dead key on an overseas keyboard.

Testers can develop considerable intuition or a sixth sense about the object under test from having used similar programs previously or they may derive it from their programming knowledge. But they usually do not know specific program instructions in the object under test.

Glass or White Box Tests

On the other hand, glass box testing requires considerable knowledge of the specific program. Having someone else other than the author learn or inspect the program is one of the major protections against fraud. Some consider glass box testing to be part of development; in my experience I have seen QA departments use it, and there it certainly should be if there is any possibility of fraud.

The most significant component of glass box testing is Coverage Analysis. Other components are memory management and stack management. The purpose of coverage analysis is to measure the proportion of the code under test that is invoked during execution on a set of test cases. The proportion of code features invoked may be the percentage of:

Paragraphs invoked (in COBOL programs), subroutines invoked (S0), subroutine calls invoked (S1), lines invoked, blocks invoked, branches in the code tested (C1), and paths in the code tested (Ct).

These variants of coverage measures are in increasing order of power and complexity. All variants allow the tester to trace the execution steps in the program, recording these steps usually in a trace file.

There are three steps necessary to perform Coverage Analysis:

• Instrumentation
• Execution of the tests
• Reporting (accumulative)

In instrumentation, a program called an instrumenter inserts identifying code into the source code under test. For example, in branch coverage analysis, the instrumenter inserts a print statement and identifier into each branch, including implicit "else's" and branches involved in loops. This is somewhat similar to what a debugger does with breakpoints. The resulting instrumented code has the same basic logic as before but it is somewhat larger and it will execute a little more slowly.

The instrumented code is then compiled and executed, usually against a test suite. During execution, each time that a branch is invoked, the additional code from the first step writes a record to a trace file recording the hit. Naturally, branches that are not invoked leave no trace.

A reporting program analyses the trace file from the previous step. Some branches may be invoked thousands of times and it is common to combine individual hits into one record stating the branch number and the number of invocations. This means usually that the trace details showing the sequence of execution are lost, but the main purpose anyway is to record hits and misses. There is considerable similarity between the functionality of a coverage analyzer and a hot-spot profiler if this program counts hits.

It is very useful if the reporting program can accumulate the results of many separate test runs. In this case, one must take care that the branch identifier is consistent, something that may not be the case if there is a new engineering release.

The significant topics in the trace file report are the listings of branches that were hit (invoked) and those that were missed (not invoked). The next step is to develop further tests designed to invoke the branches that were missed and it is at this point that you would expect to uncover any fraudulent code. The program can then be executed again under these new tests and the accumulative coverage examined. Needless to say, the actual results of the test itself need to be verified as well. It is one thing to execute the branch or line of code; it is another thing to verify that it produces the correct result.

In my experience with branch coverage analysis, it is easy to obtain 30% coverage. A little thought easily raises this to 50%. Devising tests to raise the coverage to 85% can be knotty and complex, and usually requires the cooperation of the program's author to devise test cases that will execute the previously missed branches. Since most software code often includes dead code or tests for unusual situations that cannot be repeated under test situations, one rarely sees 100% coverage.

Clearly, there are different levels of assurance when confronted with tests that produce a coverage of 50% in comparison with tests that produce 85%.

2.4.3. Automated and Manual Tests

Testing, by its very nature, involves substantial repetition and therefore it is natural to consider using automation. When engineers produce a new release, many previous tests must be performed again to verify that specific bugs have been repaired and, perhaps more importantly, that nothing else has been broken. It is just as important to verify that tests that were successful on the previous engineering release are still successful with the new release. Reapplying old tests is known as regression testing.

More motivation for automation lies in the fact that many tests need to be repeated on hardware and software from different manufacturers. Though some features may be trivial, verifying them on such platforms as DOS can be a combinatorial nightmare. There are so many varieties of nearly everything in the DOS world: CPUs, displays, versions of the operating system and particularly printers, to name a few.

To cope with this, many shrink-wrap package manufacturers maintain testing laboratories with mixes of different manufacturers' equipment. Network products make similar setups necessary because it is necessary to link many different computers together with the network. The equipment in a lab is a substantial investment mitigated somewhat in cases of popular products because manufacturers will lend equipment to the developer so that it is clear that the popular product works on their equipment. A million dollars worth of equipment in a lab doesn't look like very much, but to paraphrase Senator Everett Dirksen, take a million here and a million there and pretty soon you're talking about real money.

The variety of equipment in a lab is a concrete and powerful indicator for developing automated testing procedures as it make it so obvious that many tests will have to be repeated on many different platforms.

Manual Tests

These terms refer to the process of invoking execution of a program under test, providing input by hand via keyboard, mouse or microphone and observing output on the screen, printer or loudspeaker. Comparing the output with the expected output verifies the program's correctness.

It is customary to plan the invocations by writing test plans and preparing test scripts to follow.

Automated Tests

Automating tests is expensive in both cost and time to develop the automation program. The tests are designed to run unattended, allowing for overnight and weekend work, freeing up testers' time for other tests and so it is necessary to produce a log of test results for later examination. But often, the precise format of the files and displays is not known until development reaches the User Interface freeze stage. Therefore development of automated tests can not be completed until that development stage is reached, not leaving much time for development.

Verifying the result of tests involves comparing either file outputs or screen displays or both. The pass or fail decision depends on a "diff", comparing files or displays with the canon or desired output. A point to note has to do with the "diff": in some cases parts of the output, for example system dates and times, can change considerably without changing the substance of the output in any material fashion. So the "diff" procedure should be able to exclude some parts of the output.

In the typical case, it is expected that an equality of output with previous or baseline output would be a successful result. This is only true when there has been no change in the code producing the result. If a bug has been fixed, then success could mean that the two outputs must be different. Neither may yet be the correct output, but they should be different. And the new output can serve as the baseline for the next round of tests. The very first baseline has to obtained manually or on a special run in which the comparison function is turned off.

There are two basic approaches to automating tests, but the most commonly seen is a mixture of the two. One approach is to record keystrokes and mouse movements so that they may be played back later. The second is to program the desired key strokes and mouse movements in a master program that drives the object under test. Sometimes, the test driver is on a master computer driving the software under test in operation on a different computer and sometimes the driver resides on the same computer.

When recording keystrokes and mouse movements, it is customary to save the recordings in files that can be edited. Thus these files serve as the basis for further tests. Similarly, master programs driving the object under test are available for modification. In combining the two approaches, the master program may invoke keystroke and mouse movement files as procedures or subroutines. From this point of view therefore, development of automated test suites is a straightforward programming project in itself, albeit a specialized one.

Output may be unexpected. For example, suppose that a test causes the system to crash. Then in order to proceed to the next test, and to perform it in a known context, it will be necessary to reconstruct all the steps that constitute that context. If a test is designed to invoke say the second level of subroutine, then it is necessary to invoke the first level before the second. But if invocation of the first level fails, then testing the second level is not possible. So, part of the programming for construction of an automated regression test includes setting up a test scenario. And if that scenario fails somewhere in construction, then one must pass to the next scenario.

This might mean - as has been my experience - downloading test data and source code from network archives, compiling and linking the code, then executing it in newly created directories. When a test fails catastrophically, the next set of tests may require starting all over with the blank computer and perhaps reconfiguring it. This can involve complicated programming.

Another major challenge with automated tests is how to handle timing. Synchronizing input and output events is tricky at times. If test playback is too fast for the system under test, then input can be lost when input buffers overfill. Thus subsequent tests are useless because it is far from clear what the system under test perceives its input to be. The simpler an environment is, the more straightforward timing is, but in complex environments such as networks, considerable sophistication may be required.

Since 1992, Quality Assurance tool manufacturers as a whole have placed increasing emphasis on object-oriented testing tools. The purpose of these tools is to improve the efficiency of testing windows. The focus of these tools is to treat the components of windows as identifiable objects, addressing them by their identifiers rather than by their physical location on the screen display. This is necessary as the physical placement of a window depends heavily, perhaps unpredictably on what other events are taking place in the computer and in the case of multitasking or client-server systems, one will not know what else is going on. Retrieving objects is necessary for making comparisons.

Whether automated or manual, tests can still be black box or white box. They are dynamic, of course. Automated tests can also include coverage analysis.

2.5. Types of Errors

The types of errors to look for in U. S. programs are:

• User interface errors (functionality, communication, command structure, missing commands, performance, and output)
• Error handling
• Boundary-related errors
• Calculation errors
• Initial and later states
• Control flow errors
• Errors in handling or interpreting data
• Race conditions
• Load conditions
• Hardware
• Source and version control
• Documentation
• Tester errors.

3. QA TOOLS

It is reasonable to ask to see results of tests that manufacturers have run on their own products.

3.1. Static analysis

• AppPolish - Encore Systems.
• INCA - Powerbase Systems.
• STW/Advisor - Software Research.
• VistaTest - Veritas Software.

3.2. Coverage analysis

• TCAT - Software Research.
• VistaTest - Veritas Software.

3.3. Automation

Atron Evaluator - Eastern Systems Automated

Test Facility - Softbridge, Inc.

Autotester (Plus) - AutoTester Inc.

Ferret - Tiburon Systems

MS Test - Microsoft Corp.

QA Partner - Segue Software.

STW/Regression - Software Research.

Test Runner/Win Runner X-Runner - Mercury Interactive.

Virtual User (VU) - Apple Computer.

VistaReplay - Veritas Software.

3.4 Miscellaneous

• XL8 - Globalware, Inc.

4. REFERENCE

[1]. Kaner, Cem, Jack Falk and Hung Quoc Nguyen. Testing Computer Software. 2nd ed. New York: Van Nostrand Reinhold, 1993. ISBN 0-442-01361-2

ABOUT THE AUTHOR:

Emmanuel Uren is the senior author of Software Internationalization and Localization, An Introduction published by Van Nostrand Reinhold in August 1993. Anglo-Maltese, he attended Manchester University, Yale and Stanford. He has lived and worked in the United Kingdom, Quebec, Venezuela and Mexico as well as the U. S. Over the last thirty years, he has worked in computer software in many roles, most recently in Software Quality Assurance, and in Internationalization and Localization.

---

FOOTNOTE 1.

From time to time one reads or hears that a product's release is being delayed for a considerable amount of time. And this is considered to be bad news or even a disaster. Financially, that may be so for the manufacturer. However, in a curious back-handed manner, this is a tribute to the manufacturer who is unwilling - for whatever reason to release an error-prone product into the marketplace. My initial reaction to this type of news is to think that this product may be worth investigating for purchase when it finally makes it to market because the manufacturer has taken a long time to fix bugs and then verify the fix.

---

Emmanuel Uren
83 Seward Street
San Francisco CA 94114
USA
voice/fax:+1(415)-552-9085
Internet: euren@netcom.com