Architecture and Scaling Cloud Applications

OK, quickly, you've got a new app that has gone "off the charts", it's hosted on EC2 and you want to be able to scale in order to meet demand.

What do you do?

While this is a great situation, too often the answer technical people come up with is either:
#1 (customer answer) We need to get someone else to build this us, our IT guys don't know what they're doing.
#2a (developer answer) Rewrite the app in (erlang, scala, ruby, java, C#) because our code sucks and isn't scalable
#2b (developer answer) Switch to (Oracle, DB2, MySql, MongoDB, Terracotta, Spring, EJB3) because (Oracle, DB2, MySql, MongoDB, Terracotta, Spring, EJB3) doesn't scale well
#3a (infrastructure answer) We need to buy more EC2 instances and "scale out"
#3b (intrastructure answer) We need to bring it in-house and we'll get the biggest baddest server you can buy
#4 (architect answer) Where's the bottleneck?

OK, I know #4 isn't really an answer, but it illustrates the problem. The architect's job is to fully understand the problem and help guide discussion about what possible solutions are. I've met a lot of architects and usually can tell what their background is after a discussion like this. The well rounded folks will typically ask a lot of questions before jumping to a conclusion about the best answer. The folks who grew up in the business will choose #1, Developers #2, Infrastructure #3, and rarely you'll get a #4.

That having been said, the best answer is probably to get ahead of the problem and build scalability into your design. For dynamic web applications this means you should adopt the following positions:

#1 My application might be hosted on a laptop with an in memory database, or it might be hosted on 52 app servers with 12 database instances - my design shouldn't require large changes to accomodate either.
#2 My URL scheme matters (even down to DNS) and should reflect unique identities of resources that my application deals with (see REST AND use proper http verbs where applicable.
#3 Don't panic

So what does this really mean?

For one, the days of writing a J2EE app, and/or SOAP as the best way to do things are past. While J2EE can scale up easily and theoretically even scale out, it's just too dang expensive and difficult to get that to work well (unless you like throwing money at problems). Why?

First off, most J2EE containers (and applications) are designed around the idea that you have a big chunk of shared memory and multiple processors available. As an example, many of the shared resources rely on thread pooling. This has some serious drawbacks when you start getting into extremely high load. All those thread pools need overhead to manage each other's state and that management affects your entire application as all those CPUs start to melt down doing thread management activities.

Second, SOAP is basically a more web centric platform independent replacement for CORBA... Because of it's design, it requires a bit more work than simple URL parsing to be able to scale it. Add to this the problem of xml marshalling and you'll quickly devlolve into a big mess of performance and scalability problems.

These problems aren't the only ones, nor are they limited to J2EE and web services (.Net suffers from similar problems), but they are indicative of a bigger problem brewing in web application development and architecture. To get ahead of this, start thinking about what is is REALLY NECESSARY to run your application. It may be surprising how much extra "stuff" you've added in "just in case" that is not only hampering development tempo, but also potentially slowing you down at runtime.