Load Testing Web Services

Syscon September Edition of SOA Web Services Journal has a number of good pieces on web services testing. I found Load Testing Web Services interesting, especially in its description of important parameters in Load Testing web services.

Load Testing Metrics and Parameters
The results obtained by load testing Web Services can potentially be reflected in terms of the following parameters.

• Response time: It's the most important parameter to reflect the quality of a Web Service. Response time is the total time it takes after the client sends a request till it gets a response. This includes the time the message remains in transit on the network, which can't be measured exclusively by any load-testing tool. So we're restricted to testing Web Services deployed on a local machine. The result will be a graph measuring the average response time against the number of virtual users.
• Number of transactions passed/failed: This parameter simply shows the total number of transactions passed or failed.
• Throughput: It's measured in bytes and represents the amount of data that the virtual users receive from the server at any given second. We can compare this graph to the response-time graph to see how the throughput affects transaction performance.
• Load size: The number of concurrent virtual users trying to access the Web Service at any particular instance in an interval of time.
• CPU utilization: The amount of CPU time used by the Web Service while processing the request.
• Memory utilization: The amount of memory used by the Web Service while processing the request.
• Wait Time (Average Latency): The time it takes from when a request is sent until the first byte is received.

We have deployed WS-SOA Gateways at many locations worldwide, and most of the deployments are 1-u Appliances with dual CPUs and crypto accelerators. Most of the load requirements center around message sizes. I have yet to see a deployment that comes close to harnessing the 1000-2000 TPS capacity of appliances when small to mid size documents are invovled ranging from 1K-100K. Where things get interesting is in the > 2G range of SOAP Attachments. In such deployments we typically see the back end application server croak on such documents once the gateway has processed it and given it a clean bill of health. And by the way, only at one deployment, did number of concurrent users even matter. Web Services are typically used for Application-to-Application communication where 1000's of concurrent connections are unnecessary, unlike in B-to-C deployments.

What's even more interesting is the lack of SOA Testing Tools that can handle such large documents without choking. Of course you can always write your own testing scripts or propose to do so if you are in the consulting business ;-)