To support buyers in this process, we have put together some of the latest data available from LISA Members. Companies such as Grisoft, IBM, Symantec and WhP share their experiences, along with their implemented best practices, in this area.

Editor’s Note: The following is an excerpt from Outsourcing Best Practices: QA of Localized Software. If you are a LISA Sponsor, Corporate or Introductory Member, the publication is available as a free PDF download at http://www.lisa.org/products/obp/?from=gi. Others may order the publication at the same URL.

To support the simultaneous shipment of products, more and more software companies are outsourcing their QA function. The Symantec QA Team, one of the leading software and product testing groups in the world, facilitated a standing room only workgroup session by sharing the challenges they have faced and what they have learned as they have moved from an in-house localization QA model to a substantially outsourced one.

Based on a questionnaire distributed prior to the LISA Forum Europe in Warsaw last November, Tommy McShane, Ken O’Brien and Brian Kirwan of Symantec and the audience shared outsourcing best practices on the following topics: the business model for QA outsourcing, challenges faced and overcome, and the processes and technologies used to manage the new outsourced model with service providers. (Read Symantec Discusses Outsourcing QA for Localized Products: Trials and Tribulations for more details.)

There were 113 respondents to the Symantec QA Survey. To review the questions, please click here.

The Symantec QA Process Today

Over the last two years, Symantec has aggressively pursued a QA outsourcing strategy. At its peak, the Dublin QA group employed 60 people. Today, it has 22. All processes and procedures have been significantly modified to facilitate the outsourcing strategy. The internal QA role at Symantec today is to carefully monitor its QA Vendors throughout the entire project lifecycle. There are three main internal roles now: QA Manager, QA Supervisor and Project Lead. (See the slides for a detailed organizational chart.)

Between 2004 and today, the Symantec QA Group has gone from testing around 250 releases to almost 700. And it has done this without increasing headcount due to its outsourced model.

SQA (Symantec QA) Framework 4.0 is a structured approach to how Symantec executes its software localization testing. It includes templates for all project documents for managing, testing, auditing and evaluating projects.

Based on a review of mature consumer projects, Symantec confirmed that its localization testing was only uncovering 5% functionality bugs vs. 95% UI (user interface) bugs. Since then, it has significantly reduced functional QA and increased UI testing. The internal QA team is flexible about the breakdown and continues to monitor the balance since enterprise software products require more functional testing for localized versions.

Symantec provides two days of training for its QA Vendors. Staff must pass the assessment with a minimum of 70% to guarantee that they have at least reached a baseline level. The QA group has invested a lot of resource into this system and maintains a database of the results. They also began running QA Audits 6 months ago, and it’s working very well for both the company and for the Vendors. It is transparent, so the Vendors know exactly how they are being evaluated.

Being a security company, Symantec is very careful with access, so the system is based on a site-to-site VPN. Their QA Vendors are basically satellite offices. Data flows back and forth, straight into Symantec’s QA lab – this is very valuable around the time of ship since multiple builds and final CD checks can be performed with no dependence on FTP.

The key to all of this is the relationships that Symantec has built with its QA Vendors. The company has worked very hard to form productive partnerships with a small number of Vendors with whom it has worked for years, e.g., there are 2-3 in Eastern Europe. These relationships are built on transparency and honesty, and the Vendors are always encouraged to provide feedback and input at all stages throughout the process.

As part of this program, Symantec is now rolling out an awards program to recognize its Vendors’ contributions. The awards include one for innovation and for the “best bug.”

Tools Used During the QA Process

Symantec has invested significantly in tools over the last 15 years. There are now two designated developers for QA tools who focus on UI and build acceptance automation. The latter is key because certifying fully testable builds in up to 16 languages for consumer products requires a lot of time. They now have a tool that greatly speeds up the process. The Warsaw team has developed a UI checking tool.

The rest of the workgroup session focused on three areas from the questionnaire:

• Outsourcing requirements
• The QA process itself
• Implementing technologies to manage the QA process

LISA QA Model 3.1
http://www.lisa.org/products/qamodel/?from=gi

Taking Software to the World: Results of the LISA Global Software Survey
http://www.lisa.org/products/surveys/gssurvey.html/?from=gi

LISA Best Practice Guide: Quality Assurance – The Client Perspective
http://www.lisa.org/products/bestPractice/?from=gi