INSIDER: Why did you approach LISA about running the “Outsourcing QA Software Testing Workgroup” during the LISA Forum Europe in Warsaw in November? What do you hope to learn during the session?

The whole idea for this session grew out of a 2-hour benchmarking session with Filenet about a year ago. We ended up sharing with them the code for one of our internal training programs, and we adapted several of their ideas on how to work more effectively with our service providers.

We have a high degree of insight into what has gone well and what hasn’t in our whole QA outsourcing experience in Software Engineering and Documentation Services for Symantec here in Dublin. We have modified all of our processes, including our in-house ones, to make them more suitable for outsourcing. So now, what we’re looking for is to share best practices in this area with others. What has worked for them? What hasn’t? Why? What tools and processes might we share?

For example, in the area of technology, most of our processes for Engineering and Doc Services are based on a high degree of web-based technology and workflow systems. However, we haven’t yet implemented anything like this for QA – we’re still using a manual system. We are looking to learn from someone who has already done this. Perhaps another organization already has a clever, outsourced QA workflow system in place that it would be willing to share with others. Either to leverage or to plagiarize, perhaps?!

Openness and sharing are what we’re hoping to foster as the tenor of this session. We will be very open about sharing what Symantec has done. Our attitude is that we’re all in this together, so we need to share our ideas, tools and processes publicly. If we can achieve this level of transparency by all participants, we will all benefit greatly. It will come down to people being willing to share their experiences with others.

INSIDER: What do you need from our readers to make the workgroup session successful?

Of paramount importance is their feedback to the pre-session questionnaire. We encourage both buyers and sellers of services to participate, so that everyone can benefit from this session. Sooo, readers, please stop reading this article, and click here now to fill out the survey! We will review every answer and distill all your observations into concise answers, which we will use as the basis for this session.

Second, we are depending on the participants to be open and frank about describing their challenges and the solutions that they have tried – whether or not they have worked.

Here are just a few issues that we think will come up:

• Has anyone else gone all the way like AOL and outsourced all of their localized QA?
• Where are people outsourcing these tasks? We find Eastern Europe to be very competitive, with very smart and enthusiastic team members. However, others are using China. We would be interested in knowing why they are using China rather than India, and vice-versa. How have they overcome the language and time zone issues in these regions?
• What technologies/tools (e.g., web-based, workflow systems, content management systems) are companies using to enable them to manage the entire QA process?
• How do companies provide training and support to their outsourced teams? What metrics do they have in place for checking quality and ROI?
• How much testing is done on localized software, and how does one decide the amount of testing needed?

INSIDER: Where is Symantec right now along the continuum of outsourcing its localized software QA?

We have gone about halfway along the continuum. Our QA Group for localized products has gone from 60 people (including both temporary and full-time staff) to a core team of about 22. This team now drives what’s being done with our outsourced teams in Eastern Europe – specifically Estonia, the Ukraine and Warsaw. Our internal quality control people now carefully monitor our service providers. The role of our internal team has shifted significantly – we’re now in the business of “watching the watchers.”

We still maintain a modest-sized group in-house because we’re not yet ready to fully outsource everything. We believe that there is still a high level of risk, so we want to keep our “hands on the rudder,” so to speak. We are very interested to find out how other software companies have solved the balance between in-house and out-of-house when it comes to this area.

It’s important to point out that here in Dublin we have only outsourced our consumer boxed products, the ones that are basically sold to 1 person/1 PC. This scenario only presents a modest level of challenge for outsourcing our localized QA. With the more sophisticated enterprise products, it is a whole different challenge with hardware labs to set up to support the installation of realistic test environments.

Size and complexity matter.

We have done some of this enterprise software testing, but it has only been for products that can be set up in about 15 minutes. Now that our world has changed, based on our merger with VERITAS, so has the playing field. Some of the QA for the VERITAS product line is outsourced to China for Simplified Chinese and Japanese, but we have encountered a lot of problems. Size and complexity matter when it comes to QA for localized products.

Our plan is to progress along the continuum without reinventing the wheel as we look at possibly outsourcing more of our QA for localized products at the enterprise level. However, we prefer to benchmark ourselves and to adapt what others have already tested and know that work.

INSIDER: What has Symantec done right as it travels along this continuum?

First, a bit of background on how we are organized, to provide readers with some perspective.

About 500 people report into our Vice President of Localization, who reports into the CTO (Chief Technical Officer). Symantec is organized along business lines, so there are very few places where we wouldn’t be biased, so the CTO’s organization supplies us with a much-needed neutral place.

Now, to answer the question that you asked …

We have worked very, very hard to build up very good, long-term relationships with a small group of high-quality service providers. I can’t say that we’re truly an icon of how it should be done, but our QA outsourcing strategy has been very successful so far, and we are very happy with it.

In the old days, it was popular to go with the large, multi-language vendors (MLVs) for reliability and economies of scale. However, we found that this model didn’t really work for Symantec, even when we sourced what we considered to be a reasonable amount of business with one MLV. They needed guarantees for workloads that we just couldn’t provide. And we found ourselves requiring the flexibility to ramp up and down that the MLVs couldn’t provide.

We’re not bullies!

We have had much more success with smaller service providers (usually with around 50-70 full- and part-time employees) who have a much stronger interest in retaining our business. If I call with an issue, I am guaranteed an immediate response.

Critical to this whole process is transparency, i.e., we encourage our service providers to “confess” if they know that there is a problem through weekly audits and through rerunning test cases. We do find things that have been missed. But, we work together to solve the issues. This includes talking with key service providers to discuss how the relationship is progressing from their point of view.

We’re not bullies, though. Together, we have built very comfortable relationships that now provide tailored and efficient service that meet our needs.

At the same time, you don’t want any one service provider to have 90% of their business tied to you. We find that about 25-35% works for us. It’s big enough, and the service provider will pay us a high rate of attention.

We work on the basis of a continuous improvement outlook. Even if something works really well, we still review it every 18 months to see if it can be made better.

INSIDER: What would you do differently, if you could?

We would have moved to an outsourcing strategy much more quickly. In the old days, we used to commandeer entire meeting rooms. In fact, a whole new room was built specifically for us to ramp up with temporary staff, and we actually rented space in another building. However, the down side is that all of these people had to be (re-)recruited, (re-)interviewed and constantly managed, housed and fed!

In hindsight, we can see how ridiculous it became. We eventually realized that we were only letting go of a small amount of work, even though our service providers had proved themselves quite capable. We have now finally outsourced entire product lines – even the “crown jewels” of the consumer lines are now outsourced. This would have been heresy just a few years ago

INSIDER: How will the world of QA and outsourcing be different two years from now?

It’s already different! There has already been a paradigm shift in the outsourcing world. Just a few years ago, we would have had significant numbers of QA staff brought in from local/Dublin service providers. Today, all of our outsourced QA work is done outside of Ireland in either Eastern Europe or China. That’s the trend – to do less and less internally, and to do more and more externally (and more cheaply).

The key is to figure out where the cost-benefit ratio kicks in. We have very, very little frontline QA done in-house at Symantec now – only the products that require a really expensive setup are done in-house. As long as such a huge world of low-cost talent is available, we will continue to do it out-of-house. No one talks about investing in Dublin anymore – Symantec alone has an installation in India that is set to grow from 1,700 to 2,500 employees, and we are opening a second site with room for 2,500. Then, there’s China, where we are set to hire 500 people, with plans to double in size each year.

Ireland has become a knowledge management clearinghouse for the rest of the world.

What the Irish do best is to manage from cradle to grave – we are good organizers and implementers. Knowledge management tasks will continue to be sourced here in Ireland, as it functions as a clearinghouse for the rest of the work to be done in India, China and Eastern Europe.

In QA, two years is a long time. I expect that we will see a lot more use of distributed QA systems, since the web allows for a different way of doing things. Perhaps even a workflow system for freelancers to enable them to sign out their work from all over the world. QA may eventually become commoditized, at least for consumer products. And technology will continue to transform how everything gets done. Eyes are still needed to review software screens, but there will be a huge shift in the industry if and when the technology is created to put it all through an automated system.

Outsourcing will continue to expand, and technology will continue to enhance what we can accomplish with less time and at less cost. I’m not a visionary, but I’m hoping to meet some people in Warsaw in November who can fill me in on the future!

---

Tommy McShane has worked for Symantec for fourteen years, five of which were spent in Tokyo managing Asia-Pacific and Japanese localization. Based in Dublin for the last four years, McShane currently wears two hats, one as Director of Quality Assurance across all Symantec sites and the other as Director of Localization for Eastern Europe.