Sunday, October 29, 2006

Load Testing Web Services

Syscon September Edition of SOA Web Services Journal has a number of good pieces on web services testing. I found Load Testing Web Services interesting, especially in its description of important parameters in Load Testing web services.

Load Testing Metrics and Parameters
The results obtained by load testing Web Services can potentially be reflected in terms of the following parameters.

  • Response time: It's the most important parameter to reflect the quality of a Web Service. Response time is the total time it takes after the client sends a request till it gets a response. This includes the time the message remains in transit on the network, which can't be measured exclusively by any load-testing tool. So we're restricted to testing Web Services deployed on a local machine. The result will be a graph measuring the average response time against the number of virtual users.
  • Number of transactions passed/failed: This parameter simply shows the total number of transactions passed or failed.
  • Throughput: It's measured in bytes and represents the amount of data that the virtual users receive from the server at any given second. We can compare this graph to the response-time graph to see how the throughput affects transaction performance.
  • Load size: The number of concurrent virtual users trying to access the Web Service at any particular instance in an interval of time.
  • CPU utilization: The amount of CPU time used by the Web Service while processing the request.
  • Memory utilization: The amount of memory used by the Web Service while processing the request.
  • Wait Time (Average Latency): The time it takes from when a request is sent until the first byte is received.

We have deployed WS-SOA Gateways at many locations worldwide, and most of the deployments are 1-u Appliances with dual CPUs and crypto accelerators. Most of the load requirements center around message sizes. I have yet to see a deployment that comes close to harnessing the 1000-2000 TPS capacity of appliances when small to mid size documents are invovled ranging from 1K-100K. Where things get interesting is in the > 2G range of SOAP Attachments. In such deployments we typically see the back end application server croak on such documents once the gateway has processed it and given it a clean bill of health. And by the way, only at one deployment, did number of concurrent users even matter. Web Services are typically used for Application-to-Application communication where 1000's of concurrent connections are unnecessary, unlike in B-to-C deployments.

What's even more interesting is the lack of SOA Testing Tools that can handle such large documents without choking. Of course you can always write your own testing scripts or propose to do so if you are in the consulting business ;-)

Thursday, October 19, 2006

ChoicePay: Rising above the SOA Testing Challenge

Recently, Salman Akhtar, CEO Techlogix, published a good case study regarding SOA Testing Challenges and how ChoicePay, an electronic billing company, overcame these challenges "without throwing more bodies at the problem."

The article, published by SOA /WS Journal is a good read for anyone concerned about SOA Tesing:

Monday, October 16, 2006

Web Services Testing NetWeaver Application Server, Java EE 5 Edition

SAP has recently released one of the first Java EE 5 compliant application servers. I tested this server's web services capabilities and found it to be easy to use. SAP NetWeaver Application Server, Java EE 5 Edition install contains the application server, the NetWeaver DeveloperStudio and MaxDB database. My biggest challenge was installation, but SAP Developer Network (SDN) came to the rescue and after my installation was complete, I was off to the races.

Details of the evaluation are published on

You can also see details of the installation and test process at:

Give NetWeaver Application Server a spin - you'll be publishing web services in no time.

Monday, October 02, 2006

How to use Amazon EC2 WSDL

If you are interested in testing the Amazon EC2 WSDL API as well as the S3 WSDL interface, you can go to the URL below for details on how to setup SOAPSonar, a web services testing tool by Crosscheck Networks.

Loading up X.509 certificates in SOAP headers for authentication to Amazon EC2 is the most interesting part of this article.