DevOps Zone is brought to you in partnership with:

I am an experienced software development manager, project manager and CTO focused on hard problems in software development and maintenance, software quality and security. For the last 15 years I have been managing teams building electronic trading platforms for stock exchanges and investment banks around the world. My special interest is how small teams can be most effective at building real software: high-quality, secure systems at the extreme limits of reliability, performance, and adaptability. Jim is a DZone MVB and is not an employee of DZone and has posted 101 posts at DZone. You can read more from them at their website. View Full User Profile

Developing and Testing in the Cloud

  • submit to reddit
There’s a lot of hype around “the Cloud” and what it can do.

One of the things that I am interested in is Cloud solutions that can help small software companies, and especially to kickstart software startups. Good tools that development teams can take advantage of to build and test their own stuff, without all of the hassle and expense of internal IT, buying and provisioning their own gear, setting up the tools and systems and networks and finding someone who understands all of it well enough to do it right and to keep it running. I want to find Cloud-based technology that can do for software teams what does for customer-service businesses, so that software teams can get started off quickly, and doing things right from the start.

Software teams need a core set of shared tools and capabilities:
  • Source code management / version control
  • Bug tracking
  • Collaboration and shared documentationBuild and Continuous Integration
  • Source code scanning and static analysis
  • Unit and functional testing
  • System, load and stress testing
  • Code deployment.
Can the Cloud provide all of this in an effective way?

Managing and Building Code in the Cloud

I keep running into people using GitHub to host their source code repositories. Not just people working on Open Source projects, but companies using GitHub to manage commercial projects in hosted private repositories, a service used by a number of startups. In GitHub, developers have a platform for managing code with the Git distributed version control system, and they get access to a good set of tools for a development team, especially distributed teams: admin functions to control permissioning, wikis, a bug tracking system and an online code review tool.

One alternative to GitHub is BitBucket which uses the Mercurial DVCS. Like GitHub, BitBucket can be used to manage Open Source and private projects, and it looks like it offers a similar set of management capabilities and tools. GitHub is free for Open Source projects and cheap for small teams while BitBucket’s pricing is based on the number of users, small teams (up to 5 users) are free for open or closed source.

Another On Demand SCM platform built on Mercurial is Fog Creek Software’s Kiln which integrates nicely with Fog Creek’s bug tracking and planning system FogBugz.

Atlassian, which bought BitBucket last year also offers Jira Studio a comprehensive hosted development platform centered on a Subversion code repository integrated with the rest of Atlaissian’s strong development toolset: Jira bug tracking, FishEye for code searching, Confluence wiki, Greenhopper for Agile team planning, Elastic Bamboo for build and Continuous Integration on Amazon EC2 and Crucible for online code review. That’s almost everything that a team needs (all that is missing is static analysis and a functional and load testing platform). Presumably the same integrated development tool support will extend to BitBucket over time, although there are already some overlaps between BitBucket and Jira Studio.

And there’s CloudBees Dev@cloud which offers a choice of secure Git, SVN or Maven repositories, and Continuous Integration through a hosted Jenkins server.

Spring Source’s Code2Cloud might be a good option especially for Java / Spring developers when it is ready – the details are fuzzy at the moment.

Google Code is cool, but for now is only available for Open Source projects.

For enterprises, IBM recently announced Smart Business Development and Test Cloud and IBM Smart Business Development and Test on the IBM Cloud (also known as SmartCloud Enterprise - I’m not sure what the difference is between them, I got too worn out reading through the marketing speak) , with all sorts of IBM technology and partner tools. This stuff isn’t for the faint of heart, or the small of wallet.

Continuous Integration in the Cloud

Continuous Integration is a problem that development teams need to solve before they get too far along, and setting up a CI server and keeping it running isn’t trivial. It’s another good fit for the elastic On Demand model of the Cloud, paying for more infrastructure only when you need it.

Besides Atlassian’s hosted end-to-end platform, and CloudBees’ Jenkins as a Service which can be used to build code hosted on a repository accessible through the Internet, there are a few options for CI in the Cloud (courtesy of Pascal Thivent on Stack Overflow):

For Open Source projects there is CodeBetter based on the TeamCity build server. For proprietary projects there is Mike CI which is hosted on Amazon EC2, and includes interfaces to Subversion, Git and Mercurial repositories; and CI Foundry. However, I don’t know how real these services are. Mike CI was down the last couple of times I went to check on it, and CI Foundry’s home page includes this not-very-convincing testimonial:

What people are saying...
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris quis ligula vel ligula varius cursus tristique eget sem. Mauris sagittis, ante id imperdiet auctor, sem mi condimentum neque, quis feugiat quam dui dictum risus.
- Chris Read
Continuous Deployment expert

Thivent suggests that you can roll-your- own CI solution on Amazon EC2 or maybe use something like “CI in a box”. Using an On Demand platform like EC2 makes sense for CI, rather than provisioning your own gear: it’s a bit more work for you to do the setup, but you can rely on Amazon’s infrastructure to reduce costs and simplify operations.

Bug Tracking in the Cloud

There are some decent and reasonably priced Cloud-based bug tracking systems suitable for small teams, including Fog Creek Software’s FogBugz and Atlassian’s Jira Hosted option both of which are also included in their hosted code management suites.

Testing in the Cloud

The Cloud has a lot of promise for app testing, especially for large-scale performance and load testing. You’re building a cool new Web platform that needs to support 10,000 or 100,000 concurrent sessions according to your business plan. Before your investors put more money in, they want to see just how real your software is, how far away you are from delivering something that could work. Of course you don’t have the money or time or IT skills to build a large-scale test center of your own at this point, and there’s a lot of waste in doing this – you need a lot of gear, but you won’t need to use all of it very often.

You can spin up a test farm on EC2 surprisingly quickly, even a big one – and as long as you don’t need to run the tests for a long time and you don’t ask for too high a service level, it won’t cost you much at all. You pay for what you need when you need it.

To run stress tests you need more than just the app deployed in a farm. You also need a scalable stress test harness to drive load scenarios and to measure the results. There are some viable options available in the Cloud today: SOASTA with its CloudTest solution for load testing and extreme stress testing, scaling from smallish sites to really big; HP Load Runner in the Cloud running on EC2; JMeter in the Cloud; LoadStorm and BrowserMob.

There are also some other interesting test capabilities, including technology from Sauce Labs which lets you run Selenium tests on Web apps in the Cloud or manually test your app against different Cloud-based instantly-available browsers, all with video records of failed tests and other problems found. Simple, useful and cool.

A different take on Cloud-based testing that could work for startups and smaller companies is crowd-sourced testing services like uTest, where you get people “in the Cloud” to test your app (web apps, desktop apps and mobile apps), including functional coverage and regression testing (you give them tests, they run them and maybe help improve them), exploratory testing, usability testing, load testing and test planning and management. Like other Cloud-based On Demand solutions, you pay for work when you need it. uTest has a community of something like 40,000 testers available, although it’s not clear how many of them are much good.

Mob4Hire is another crowd-sourced solution specifically for mobile apps – they offer functional testing and usability testing and other services like market research.

Static Analysis in the Cloud

Some of the leading static analysis suppliers have also gone to the Cloud. HP offers Fortify on Demand to scan code for security vulnerabilities, and IBM has Rational Appscan OnDemand which includes assistance from an experienced team to help you interpret the results and plan remediation. Both of these solutions are targeted more towards the enterprise than small teams.

Veracode, which runs binary code analysis for security vulnerabilities and now also offers web vulnerability scanning, is only available through the Cloud. They offer a free trial scan of small Java apps for XSS and SQL Injection flaws (two of the most common and fatal web application vulnerabilities).

For startups that care about building reliable and secure apps (and most of them should, especially Web 2.0 startups and mobile app developers) it probably makes more sense to look at simpler developer IDE-based tools like Findbugs and Google’s CodePro Analytix (both free) or Klocwork Solo for Java which is priced per user.

The Tradeoffs

Managing your code, building it and testing it in the Cloud has a number of clear advantages:
  • Faster time to market – you can get access to the infrastructure and tools quickly, with minimal hassle and setup time.
  • Savings on cost and Capex – especially for load testing – you don’t have to pay upfront, you only have to pay for what you use when you use it, so you can better manage your cash flow.
  • Convenience – you don’t have to provision and operate the gear yourself and add to your operational headaches.
  • Access to specialist skills – your Cloud provider will know more about these technologies and how to solve a specific problem much better than you can afford to, so you don’t need to contract or hire gurus to help solve problems or make sure everything is setup properly, or waste time figuring it out yourself.
  • Service levels – they’ll have more people to help keep things running, to make sure code and data are backed up, systems are cleaned up and monitored and patched. They’re less likely to lose your code and data than you are.
When it comes to security, the common argument for working with a Cloud-based provider is that they will be more responsible and serious about security than most of their customers will be or can be. Because the of economies of scale, they can invest in doing things right, and because they have to be secure to compete. GitHub for example looks like they have a good security program in place and they are hosted by Rackspace so the data center, network and servers will be setup and taken care of much better than a small company, especially a startup, can afford to or would know how to do.

The concern with using a Cloud platform like this comes down to some fundamental points:

1. It’s a shared platform, used by a lot of companies. The more successful the platform is, the more customers and interesting and valuable data / IP that they manage, the more attractive a target they are to bad guys. A small company by itself is an easy target, but not especially interesting or easy to find. But a platform that holds data for a lot of small, innovative companies? Yummy….[pronounced in a deep, growly voice with a foreign accent]

2. You take on some important risks any time that you outsource something that is core and critical to your business. Your data / code / customer base is more important to you than it is to whoever you outsource this responsibility to. If something goes badly wrong, if you can’t get an important change or an important bug fix out to a customer because the Cloud service isn’t working, or if the integrity or confidentiality of your code is compromised, they will be sad, and they will lose a customer (you)... but you may go out of business.

You can ask for, and pay for, good SLAs - but Amazon’s major EC2 outage earlier this year showed that even the best providers can’t meet their SLA commitments. While Amazon did a lot of things right in handling and recovering from that outage, it affected a lot of customers for too long.

3. There’s more to a secure online platform than using SSL and network firewalls and running in a good data center - pretending otherwise is at best naïve. As a customer, you need to be confident in how the Cloud services provider designed and built the software architecture and platform to protect the confidentiality and integrity of your IP and data, in their multi-tenant partitioning scheme and software security controls, and in their SDLC. Security vulnerabilities in application code are a serious source of risk to businesses on the Web, and most companies still do a poor job of building apps in a secure way.

DropBox, a Cloud-based document storage facility is an example of a platform that seemed to be secure and confidential, until somebody started to look deeper into it, and they continue to run into problems with basic security issues Unfortunately, none of the hosted platforms provide any kind of statement on their secure SDLC that I could find, on what steps they take to ensure that the platform is designed and implemented safely.

Testing in the Cloud, especially load testing for Web apps, looks to be a no-brainer – the business case is too compelling to ignore. Many of the other tools have a lot of promise: something like Atlassian’s end-to-end hosted studio would save a lot of time and trouble for a small company, and the upfront cost savings are real.

It comes down to how much trust you can put into your Cloud provider and their SLAs for availability and support, and whether in the end you can afford to trust your core IP to somebody else. 37 Signals, which builds and operates its own Cloud solutions for project management and document sharing, doesn’t think it is a good idea:

We host all the source code for our applications internally for obvious security reasons. That’s not to say Github’s private repository hosting isn’t a good option, especially if you want a hassle-free setup. It’s just not for us.
JH 22 Aug 08


For some companies, especially startups, IP embodied in their code is critically important – it may be all that they have. Ironically, it’s these companies that are most likely to use a Cloud platform like GitHub or Jira Studio or BitBucket. Deciding whether to trust your future to the Cloud is not an easy decision to make. But as the technology continues to get better, it’s a question that is worth asking.

Published at DZone with permission of Jim Bird, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Keith Barret replied on Wed, 2011/07/13 - 5:14am

Thanks for this extensive article which gives a broad view on the cloud ecosystem evolved around software development.

Nevertheless I'm always puzzled how any company feels so safe to deliberately hand out their very own properties to a third party. The Time magazine made a really good summary statement which might explain the seductional aspect of cloud computing:

Telling consumers their data is in the cloud is like telling a kid his dog has gone to doggie heaven.

I do think cloud computing is an interesting options. But only in form of private cloud computing, where you keep the ultimate control over you own assets. 

Vijayanathan Na... replied on Thu, 2011/08/25 - 11:37pm

This is indeed is a very good and comprehensive perspective on the business benefits and challenges that organizations could face on their way to cloud adoption. There is no denying that the advent of the cloud, and its promises of significant cost savings and business agility, is going to change the business model for software development and testing drastically across organizations. While the blog does speak a lot about cloud adoption in small and medium firms, I strongly believe that the cloud is very relevant for large enterprises too. There has been a lot of concerns raised with regards to data security and privacy when it comes to cloud adoption by larger enterprises. I believe that these concerns are due to over cautious stand taken by enterprises. These concerns can be addressed, provided the enterprise or organization has done its homework thoroughly before embarking on the cloud adoption journey. This homework requires the organization to have a very clear sense of it's cloud adoption strategy, clear processes in place to identify the right cloud service provider, focus on governance model and SLAs establishment to ensure complete and clear accountability of the cloud service provider. To make this journey a lot more easier and streamlined, I would recommend that organizations/ enterprises should consult and partner with large IT service providers who can help them in this entire exercise and address all the cloud related concerns and challenges.

According to our research and multiple discussions with fortune 500 clients and prospects, we believe that large organizations/ enterprises need to begin cloud adoption with their QA and development environments. We believe these areas offer the least business risk, highest ROI possible and faster time to market, making them the ideal stepping stone for organizations to launch into enterprise wide adoption of Cloud.

For more information on Infosys’ work in the testing space, visit

Sirikant Noori replied on Sat, 2012/03/31 - 8:51am

Thank you Andre, as always, I learn some good stuff from you. I didn't know about Lapse+ (the last time I had checked, Lapse had lapsed, so it's good to see the project moving forward). I checked out O2 earlier, it's not the kind of tool that mere mortal developers are going to understand or use, although it is definitely powerful. I wasn't up on Checkmarx On Demand, thank you for that too.


Nobel Boy replied on Sat, 2014/04/05 - 3:31am

 A fascinating discussion is worth comment. I feel that you should write extra on this subject, it may not be a taboo subject but typically individuals are not sufficient to talk on such topics.

Binary options trading

mivpl jaipur replied on Fri, 2014/05/09 - 5:45am

 This homework compels the union to receive a mere evident intuition of it's confuse adoption plan, hurdle processes in arrange to associate the privilege blur employ provider.penny auction for cars 

Jessica Trishy replied on Thu, 2014/06/19 - 3:07pm

 This homework compels the union to receive a mere evident intuition of it's confuse adoption plan, hurdle processes in arrange to associate the privilege blur employ provider.

ремонт редуктора маз


ремонт редуктора заднего моста краз

Olympia Bruce replied on Sat, 2014/08/23 - 2:05am

I look "Cloud Computing" as one of the helpful source in offices. Through this, any one can share files to anyone. Now, anyone out of the home, can order assignment writing service by the help of "The Cloud"

Bertha Flores replied on Tue, 2014/09/23 - 2:26am

 well this service would be really helpful for me i can do my essay or can order essay services by the help of "Cloud" thanks for the useful info.

Jessica Trishy replied on Fri, 2014/11/14 - 3:21pm

 well this service would be really helpful for me i can do my essay or can order essay services by the help of "Cloud" thanks for the useful info.

read more.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.