Cloud Zone is brought to you in partnership with:

EXECUTIVE SUMMARY Craig S. Dickson is a software development professional with over 15 years of experience. He has proven leadership experience in both domestic and multi-national start-up and Fortune 500 corporations in the United States, Australia and Europe. Craig specializes in enterprise Java development and cloud architecture and holds multiple certifications including Sun Certified Architect for JavaEE and Certified Scrummaster. Craig brings specific expertise in enterprise software architecture and design, refining development processes and building development teams around Agile software engineering principles. Educated in Australia, Craig holds a BSc(Hons) in Computer Science. He is based in Huntington Beach, CA, and Brisbane, Australia. SPECIALTIES Enterprise Java - Software Development Best Practices - Software Development Team Leadership - Cloud Computing Craig is a DZone MVB and is not an employee of DZone and has posted 20 posts at DZone. You can read more from them at their website. View Full User Profile

Degraded Amazon EC2 Instance

10.26.2011
| 7324 views |
  • submit to reddit

I received an email from today that I have never seen before. Here it is in its entirety:

Hello,

We have noticed that one or more of your instances is running on a host degraded due to hardware failure.

i-XXXXXXXX

The risk of your instances failing is increased at this point. We cannot determine the health of any applications running on the instances. We recommend that you take appropriate action.

If your instance was launched from an EBS-backed AMI, issuing a stop and start from the Management Console will migrate your instance to new hardware and help avoid any unforeseen downtime.

For more options to stop and start your instance please see:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/starting-stopping-instances.html

If your instance was launched from an instance store-backed AMI, you should launch a replacement instance from your most recent AMI and migrate all necessary data to the replacement instance.

Should have you have any additional questions, we offer AWS Basic Support via our Community Forums for free, or Premium Support for one-on-one assistance direct from an AWS Developer Support Engineer at http://aws.amazon.com/support.

Sincerely,

The Amazon Team


So I dutifully went and followed the instructions and stopped and started (not just rebooted) the specified instance using the EC2 Web Management Console.

PROBLEM: The instance came back up as expected in the Web Management Console, however I could not ping it or SSH to it or connect to it in any way for that matter using my name. I could however connect to it using the Amazon assigned public name. It took me a few minutes to figure it out (all the while my site was down of course), but I eventually noticed that the Elastic IP address assigned to that instance was no longer shown in the instance details view. I went over to the Elastic IP management screen and sure enough that Elastic IP address was shown as not being associated with any instances. I reassigned the Elastic IP address to the instance and a few moments later, everything was back up and running.

CONCLUSION: This scenario is exactly why you need to be using an Elastic Block Storage (EBS) backed EC2 instance for any of your important servers, so in the event that the hardware fails, your actual server image is still safe and can be restored on other hardware. It also proves that while “the cloud” is awesome, it can fail and you need to be prepared for it. Also, one last curious piece about Elastic IP addresses becoming disassociated with instances – not sure if this is related to the hardware failure, or to the stop/start of the instance, but definitely something to keep an eye out for in the future.

References
Published at DZone with permission of Craig Dickson, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Amara Amjad replied on Sun, 2012/03/25 - 2:46am

Hi Craig,

I had the exact same problem a while back. EBS is definitely the way to go!

Cheers!

Joay Sim replied on Sat, 2013/02/16 - 9:51am

 I learn some new stuff from it too, thanks for sharing your information.

          Go to Astpl

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.