Performance testing cannot be just an afterthought, or performed by just any person who has been given access to a performance testing tool. It requires strategic planning and experienced testers to execute correctly. The results also need to be reviewed and interpreted by seasoned testers and engineers. Without a solid plan and expert resource, a performance test will just be a waste of time and money, providing little to no meaning.

A comprehensive, successful performance test starts with a well-defined plan. Here’s how QA Mentor lays the groundwork, executes the plan, and reviews the results.

Scope It

As with any plan, you need to know what it is you hope to achieve before you can determine how to achieve it, and whether or not you’ve met the goal at the end. Thus, the determination of success requires a plan or a goal. We first work with you and your team to identify the problem you are trying to solve. Are you planning on moving your application to new servers and need to validate their performance? Are you getting pages of timeout errors on your access logs and want to find out why? Do you want to offload server SSL and compression? Or, are you building a new system from scratch? Whatever the problem or goal is, we work with you to lay it out in simple terms. Success criteria is outlined and all of this is independent of any vendor specification.

Plan It

Once we know what we need to do, then we can devise the plan to accomplish it. This step happens automatically after the goals have been determined. The plan should be as simple as possible, in clear and concise terms. It should include all of the specific tests necessary for achieving the goals, any configuration settings that will need to be checked or modified, and all of the tools that will be used for the test.

Performance Testing Should

QA Mentor Performance Testing Methodology

One of the first steps in this planning is to build an accurate workload model. This model will be representative of user behavior and user load. If the workload model is incorrect, then the results of the performance test could be misleading, so it’s very important for those planning the tests to understand the current or expected usage and utilization of the system and its resources. For instance, some functions, such as administration ones, are rarely used and don’t generally need to be included in the performances tests. QA Mentor uses a risk-based approach to select what should be tested and what can be left out.

Everything is laid out in simple terms in the test plans, including but not limited to:

– Connections that will be used
– Number of servers and their specifications
– Number of requests, users, or transactions
– Testing tools
– Test scenarios
– Ramp up period
– File sizes
– Think time
– Failure criteria
– Success criteria
– Reporting tools

Script It

The testers writing the scripts buckle-down and learn the system thoroughly at this stage. They make sure the system can be tested using the chosen tool, while learning the business processes. Regardless of the tool chosen to script and execute the tests, the most crucial factor is the simulation. Testers spend time to make sure the scripts provide complete coverage, simulate the load accurately, and that they run correctly after all coding changes. After spending time in the system, our performance testers will be able to identify possible problem areas due to noting response times for each step of the scripts. Test data is created or identified at this phase as well.

Do It

We’ve planned the execution, next we execute the plan. On a system being newly built, the performance testing is phased in conjunction with the development.

Phase I – Unit Test – Generally this phase of performance testing is completed by developers after they complete components and before integrating them. Performance testing at this phases uses tools such as a memory profiler, code profiler, and coverage profiler. Each of these tools can help identify an area at risk for significant issues when many users are accessing it at the same time. Testing and fixing at this stage is much easier and less expensive.

Phase II – Application Integration Load Test – After the functional integration test has been completed and any functional issues are fixed, the integrated load test can begin. This phase is a full load test using the planned number of users to simulate an expected production load. For best results, the test is performed twice. The first time through, the test is executed with minimal monitoring. At this point, we just want to make sure it can hold up to an expected load. The second time through we add detailed monitoring. At this point we should be able to identify any bottlenecks and begin performance tuning.

Phase III – Production Staging Load Test – At this point, the team has moved the application to an environment that mimics production and performed functional testing. This phase of load testing is where we really see how the application handles in a shared environment where resources are in competition. We turn up the heat in this phase and make sure the system can handle the expected production traffic.

Phase IV – Capacity Assessment – Once we’ve come to this point, we know that the application or system meets expectations, but we want to go one step further. This phase determines the system’s breaking point. By assessing the capacity early, you can set points in the future when you know new hardware will need to be added in the future, or plan for new architecture. During this test we steadily increase the load on the system until resources are saturated, response times slow, and data transfer rates drop. The information obtained during this test will aid in capacity planning for the future of the application.

Review It

After the test, the team will inevitably have a great deal of data to sort through. It’s crucial to have experienced performance testers and engineers on the team for this step. Inaccuracies are much easier to spot if you have the experience and know what you’re looking for. Results should be reviewed in the context of the goals and the method of testing. Each person on the team will be given the opportunity to review and discuss the outcome of the tests. Any questionable results will be flagged for deeper evaluation, and it’s possible that tests may need to be re-run. This is also the point where it’s determined if performance tuning is necessary and a strategic plan may be outlined for hardware or software changes to be followed by a second round of tests.