Cloud Zone is brought to you in partnership with:

I'm a Brooklyn-based entrepreneur. I'm passionate about turning data into understanding. George is a DZone MVB and is not an employee of DZone and has posted 9 posts at DZone. You can read more from them at their website. View Full User Profile

How to Notify/Email Yourself When an EC2 Instance Terminates

04.26.2013
| 10140 views |
  • submit to reddit

I make pretty heavy use of EC2 spot instances, which as you know can terminate at any time with no warning.

In order to get my spots back up ASAP, I’d like be notified when they terminate.

This turned out to be much harder than I expected. I thought I’d be able to add a simple script that would send me an email when the instance shuts down (Amazon is nice enough to send a shutdown command on termination instead of just pulling the plug.)

But that approach has a couple of problems.

First, you can’t easily send email from EC2 instances (because of spammers) and have to manually get instances whitelisted by elastic IP which is a pain.

Second, it’s not the easiest thing to write a shutdown hook in Linux.

So, here’s a solution for both of those problems.

The email problem

Amazon has a service called CloudWatch that seems great for this, except that it can only monitor metrics emitted by running instances. So you can’t set it to alert you on a system shutdown, nor on a metric polling failure because it randomly misses packets all the time.

So the solution is a different Amazon service called SNS (simple notification service) that will let you trigger an event that can be configured to send you an email. So we’re going to write a script that tells SNS to send us an email.

(SNS is free for ~200k requests/month, so unless you’re planning on doing something nuts this approach should have no marginal costs.)

To do that, you first need to set up a “topic” in SNS. Go to the SNS dashboard and

1) click “create topic”.
2) Create a topic called “instance_down” or whatever you like.
3) Click the topic and click “create subscription”
4) Choose protocol “email” and enter your email address as the endpoint

To use SNS in a script, we need the AWS command line tools.

If you already have pip installed, just

    pip install awscli

Then:

    touch ~/.awsconfig
    emacs ~/.awsconfig

Make it say:

[default]
AWS_ACCESS_KEY_ID=« YOUR AWS ID»
AWS_SECRET_ACCESS_KEY=« YOUR AWS SECRET KEY»
region=« YOUR REGION »

Create an init.d script:

    sudo emacs /etc/init.d/ec2-shutdown

Make it say:

    #! /bin/sh
    ### BEGIN INIT INFO
    # Provides: ec2-terminate
    # Required-Start: $network $syslog
    # Required-Stop:
    # Default-Start:
    # Default-Stop:
    # Short-Description: restart
    # Description: send termination email
    ### END INIT INFO
    #

    export AWS_CONFIG_FILE={{ YOUR CONFIG FILE }}
    export AWS_DEFAULT_REGION={{ YOUR REGION }} # config file not picking up region for some reason
    sudo -E aws sns publish —topic-arn {{ YOUR SNS ARN }} —message “ec2 ser\
    ver {{ YOUR IDENTIFIER }} went down at $(date)”
    sleep 3 # make sure the message has time to send

    exit 0

That script will tell SNS to send you a email saying your server is down and giving the time. You can customize the message however you like.

The Shutdown Script Problem

So how do we make this run on shutdown?

We’re going to use init.d scripts. It’s taken me a little while to get my head around how this works, but in a nutshell…

Linux has a concept of “runstates”, which include things like “shutdown” and “logged in”. You can tell Ubuntu to run shell scripts when it changes into a runstate by placing scripts in certain folders in side of /etc. The two states that concern us are rc0 and rc6, i.e. “shutdown” and “reboot”, which correspond to folders /etc/rc0.d and /etc/rc6.d respectively.

Ubuntu has a command line tool, update-rc.d that will automatically symlink scripts into the appropriate folders depending on the CL paramaters you pass it. It’s all a bit complicated, but all you need to do here is:

    sudo update-rc.d ec2-shutdown start 10 0 6 .

(This says run the script on entering run states 0 and 6, shutdown and reboot. And put it 10th in the list of things to do)

And that should do it! Now your instance should send you an email whenever it shuts down, terminates or reboots. I’m not sure how limit to just EC2 termination, but please comment if you know!

If you have any problems, leave a comment.

And if you find this useful, consider following me on Twitter.

Published at DZone with permission of George London, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)