Software testing in a virtual environment

A while ago I answered the following question on SearchSoftwareQuality.com’s Ask The Software Quality Expert: Questions & Answers.

What is the likelihood of capturing accurate load testing results in a virtual test environment? We use LoadRunner/PerformanceCenter for performance testing. Our company is in the process of making use of virtualization. It seems this may be ideal for functional test environments, but not for performance test environment. What is your opinion?

Here is a clip from my answer:

Some research on the topic has found that virtual environments don't scale as well as non-virtual environments. In a study by BlueLock, a company that provides IT infrastructure as a service, they found that "the number of simultaneous users that could be handled by the virtualized server was 14% lower than the number of simultaneous users being handled by the traditional server configuration."

This is consistent with my experience testing financial service applications in virtual environments. Scott Barber, fellow SearchSoftwareQuality.com expert, recently gave a talk on the topic. You can find the PowerPoint slides for the talk here . In those slides he points out some other common challenges with working in a virtual environment.

If you don't have much choice, or if you have a lot of pressure to make it work, I would recommend that you perform a comparison performance test to prove out the new platform. If you can do that successfully, you'll have some confidence that the platform is comparable. But just be aware that over time, as the application changes and the server configurations change (both virtual and the physical servers in production) your comparison will become outdated. It may happen faster than you might think.

As Scott points out in his talk, the problem isn't necessarily virtualization. It's that we don't always pay attention to all the other factors that affect performance. Differences in software and hardware configurations, network devices, geographic location, firewalls and other security measures, and a host of other factors all affect performance. Virtual environments often only make it more complex to track everything since they introduce their own overhead, rely on different network devices, and can reside in different physical locations.

You can find the full posting here.