SOA + Utility Computing

As Amazon EC2 exploded onto the IT scene summer 2006 with a beta "by invitation only," we were fortunate to get an account and by August 2006 start pounding on the first practical Utility Computing infrastructure that was more than a bunch of slides and marketing mumbo-jumbo.

Every astute marketing team has at some point used Utility, Grid and Virtualization in their power point slides and brochure-ware. I have been at many conferences and on many calls where I kept hearing such terms but couldn't figure out what all this hoopla was all about, I don't think the presenters knew either ;-) If I can't touch it, I don't understand it. If I can't play with it, it is just vaporware. So, I kept my eye on Grid computing but never really saw a commercial (non-academic) benefit of using grids and utility computing. As much as I am into Astronomy, I couldn't see myself ever using an academic grid to compute the Milky Ways expansion rate or some other esoteric problem.

Finally, Amazon EC2 exploded on the scene with a startling characteristic: a full web services-based provisioning interface. Starting up and terminating Linux instance is as simple as a SOAP call.

This characteristic is the most significant and disruptive aspect of the Amazon EC2 platform. All Hardware and Software product vendors take notice: Your products should be Amazon-aware for true scalability.

So we did what any geek would do, played with the EC2 platform day and night and then published a first paper on EC2: "Amazon EC2 and Oracle SOA Suite a Strong Combo" on Dr. Dobbs.

A few weeks later, SAP has taken notice. Perhaps it had something to do with the emails that we sent to 40 SAP folks about this concept. A couple of significant announcements/interviews from SAP this week:

InfoWorld: Q&A: SAP chief developer heads 'clouds'
Business Week: Opening Up to Collaboration (interview with Shai Agassi, President SAP Product Development)

We sincerely hope that large vendors such as SAP, Oracle, BEA, IBM, and Microsoft take advantage of platforms like EC2 for both their SaaS offerings as well as making their software components EC2-aware. We believe SOA Testing Tools will have a significant role to play in making SOA components run smoothly on Utility Computing Platform such as EC2.

Mashup Camp, MIT

I enjoyed attending a few sessions at Mashup Camp. Overall, there is significant innovation and electricity around what the Mashup community is putting together. I enjoyed Jinesh Varia's (Amazon) presentation on building your own YouTube (iTube!). The use case involved uploading-> storing(S3)-> queuing(SQS)-> processing(EC2)-> hosting(ECS)-> Filtering(MTurk)-> Searching(perhaps Hadoop on EC2) video/images. The use case was intriguing since it logically and convincingly utilized parts of the Amazon Infrastructure. This highlights the point in my earlier blog that the cost of innovation is dramatically lower and even when scalability is required, the cost will align with specific computation & storage needs.

I also enjoyed looking at Mashups developed by LignUp, VOIP-SOA company. LignUp provides a comprehensive platform for controlling and integrating VOIP traffic within an enterprise network. The use case that I found most intriguing is having voice tied into CRM systems. This would give a fuller picture on communication between a company and its customers beyond just email archives.

I bumped into Julio Burgo from Carmun, a web portal for students to share and store academic research papers. I love their tag line: "Students of the world Unite!"

For a list of all the attendees, see Mashup Attendees.

Amazon EC2 and Oracle SOA Suite a Strong Combo

With Mashup Camp (Boston) right around the corner, I was happy to see Dr. Dobbs Journal publish an article that I have collaborated on with David Shaffer, Sr. Director Product Mgmt, Oracle Integration and Rizwan Mallal, from Crosscheck Networks, R&D. The article is written to drive home an important set of shifts that are a result of Amazon's Utility Compute Platforms, EC2, S3, SDS, etc. and it's intersection with web services products such as Oracle SOA Suite 10g.

In this article, we focused on how the Oracle SOA Suite can be used beyond its conventional business process role and more as a core SOA infrastructure traffic management platform for content-based routing, load balancing and fail over type functions for Amazon EC2 Linux instances.

http://www.ddj.com/dept/webservices/196900803

Prediction: In the next 2-3 years, perhaps sooner, Amazon EC2 in combination with SOA Suites will cause a fundamental shift in corporate computing. Also, EC2 & SOA will unleash unprecedented low cost innovation where startups will get off the ground with far lower capital than that required in the pre-Amazon and pre-web services era.

How to use Amazon EC2 WSDL

If you are interested in testing the Amazon EC2 WSDL API as well as the S3 WSDL interface, you can go to the URL below for details on how to setup SOAPSonar, a web services testing tool by Crosscheck Networks.

Loading up X.509 certificates in SOAP headers for authentication to Amazon EC2 is the most interesting part of this article.

http://www.crosschecknet.com/web_services_testing_tools_amazon_ec2.php

Amazon EC2 and S3 SOAP API

For the SOAP/WSDL-types out there thinking of building mashups, you can use SOAPSonar Enterprise Edition from Crosscheck Networks for using and testing the EC2 and S3 WSDL APIs simultaneously. SOAPSonar handles X.509 Certificates and easily communicates with both S3 and EC2.

I used SOAPSonar for cleaning up my buckets that were populated with image files from EC2.

Floating through the Amazon Elastic Compute Cloud

I have been at it again - OCD. My friend "lent" me his AWS account and I have been obsessively experimenting with Amazon's Elastic Compute Cloud or EC2. You can provision a Linux instance in a few minutes by choosing from Amazon's default image menu -- which I see growing, or you can build your own private image by either modifying a default image or build a brand new image on your Linux machine.

I love the fact that I can organize a bunch on different images on Amazon Simple Storage Service (S3) and provision a machine whenever I need to.

My QA team is responsible for testing our SOA/WebServices Gateway with a number of integration points such as Identity Server (LDAP, AD, SiteMinder, ClearTrust, TAM) for Web Service Authentication and Authorization, Database Servers (MySQL, Oracle, DB2, etc.) for Archiving and a number of App/Web Servers (Apache, WebLogic, WebSphere, etc.). And all of this across different versions.

Needless to say, our testing infrastructure is large, complex and expensive. It is going to save us a "crap" load of time & money by storing images on S3 and provisioning the "required" images when needed. I can hear the silence of all those sever fans in our labs now. The noise reduction itself is worth a premium.

And when my Field Engineers have to demonstrate how well our product (Forum Sentry) integrates with the IT assets in a corporate ecosystem, all they have to do is instantiate the clean, tuned, tested components imaged on S3. No more running around the day before trying to install WebSphere, or TAM. I will pay to see someone install some of these components on their demo laptops within a day.

Next steps -- using EC2 WSDL API along with the S3 WSDL API.

Stay tuned, my OCD stems from the thrill of being able to start 10 instance of MySQL with one command (ec2-run-instances).