Allan Feid - devops

Doing something useful with your SNMP traps

Allan Feid — Fri, 25 Apr 2014 01:14:44 +0000

Attachment	Size
slack_pic.jpg	7 KB

Sometimes when you're trying to automate all the things, you have to revisit old technologies that you have long forgotten. In the networking world, SNMP is king for monitoring and metric gathering. I haven't seen a lot of information out there about setting up SNMP traps in a way that allows you to do what you want. Sure there's plenty of Network Management Systems out there, but I don't need all the bells and whistles.

I recently spent some time figuring out how to audit Juniper gear for configuration changes, get notified of flapping ports on the network, and check for failed SSH login attempts. I considered reinventing the wheel here, but came across snmptt. While the project is on Sourceforge, and uses CVS, it still looks actively maintained. I looked into the configuration, and there are a ton of options with plenty of flexibility.

Quick primer on net-snmp

If you've never used the net-snmp tools, here's a quick guide to getting started. First, make sure you have the MIBs downloaded for your device and placed into /etc/snmp/mibs. Second, add the following to /etc/snmp/snmp.conf:

mibdirs +/etc/snmp/mibs
mibs +ALL

This sets up net-snmp to work with your custom MIBs, feel free to puruse their documentation if you want to know more.

Get your mibs in the snmptt format

In order to get snmptt configured with your MIBs, they include a useful tool called snmpttconvertmib. It's quick enough to use like so:

mkdir /tmp/snmptt_configs
for mibfile in /etc/snmp/mibs/*.mib; do
    snmpttconvertmib --in $mibfile \
                     --out /tmp/snmptt_configs/$(basename $mibfile) \
                     --net_snmp_perl
done
cat /tmp/snmptt_configs/* > /etc/snmp/snmptt_vendor.conf

This takes all of those mib files and lets you translate them into simple text strings. The conversion isn't perfect, but it's easy to go in and tweak the ones you care about and execute actions.

Intercept the traps and send them to snmptt

I'm not going to go over setting up snmptt in daemon mode, you can follow their documentation, or if you're on CentOS yum install snmptt to get init scripts. Once you do have your daemon running, you have to set up snmptrapd to execute snmptthandler like so:

# cat /etc/snmp/snmptrapd.conf
traphandle default /usr/sbin/snmptthandler
disableAuthorization yes

To send SNMP traps to snmptrapd from a Juniper device, you need a configuration like this:

    set snmp trap-group default version v2
    set snmp trap-group default categories chassis
    set snmp trap-group default categories link
    set snmp trap-group default categories routing
    set snmp trap-group default categories startup
    set snmp trap-group default categories configuration
    set snmp trap-group default categories chassis-cluster
    set snmp trap-group default targets $snmptrapd_ip

    set event-options policy login-traps events sshd_login_failed
    set event-options policy login-traps events login_failed
    set event-options policy login-traps events login_pam_authentication_error
    set event-options policy login-traps then raise-trap

A couple tips about snmptt

If you're receiving a lot of traps, I recommend setting threads_enable = 1 in your snmptt.ini file. This allows EXEC statements to be launched in a new thread, preventing a build up of traps while you wait for the previous EXEC. I also recommend net_snmp_perl_enable = 1 which will convert integers to their proper names if you have net-snmp set up properly. In the example below, this is what converts a 2 into the word "cli," which is way more informative to us humans.

Do something useful with your traps

You can store all these traps in MySQL, which I highly recommend for analytics and historical reporting. (Protip: use DATETIME rather than VARCHAR to query by time easily). But what if you have a bunch of Juniper gear and happen to use Slack? I find it useful to know when people are committing changes to the network, especially since we have a handful of engineers all working on different tasks, and I can't keep up with all the network changes.

First you have to know which trap you really care about. For me it's jnxCmCfgChange, which gets triggered on configuration changes. Looking through my previously converted mib to snmptt config, I found the event and changed it to look like this:

EVENT jnxCmCfgChange .1.3.6.1.4.1.2636.4.5.0.1 "Status Events" Normal
FORMAT Configuration change by $4 (reason: $5, from: $3)
EXEC send_slack_notification "$A - Configuration change by $4 (reason: $5, from: $3)"
..snip (took out description for simplicity)..

In my snmptt log, anytime this gets fired I get a line like this:

10.1.10.1 - Configuration change by afeid (from: cli, reason: look a change!)

You may have noticed the EXEC statement. That launches an arbitrary command and can use the same string format as the FORMAT string. So for me, I created a dumb little script to post a message to a slack channel like so:

#!/usr/bin/env ruby
require 'uri'
require 'net/https'

TOKEN='yourtokenhere'
CHANNEL='#yourchannelhere'
USERNAME='SNMP Traps'

def usage
  puts "#{$0} message"
end

text = ARGV.join(' ')
usage if text.nil?

uri = URI.parse("https://slack.com/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true

params = URI.encode("token=#{TOKEN}&channel=#{CHANNEL}" +
                    "&username=#{USERNAME}&text=#{text}")
request = Net::HTTP::Post.new("/api/chat.postMessage?#{params}")
http.request(request)

The end result:

Now this is a very simple use case, there are plenty of powerful things you can do like create alerts in nagios using passive checks or trigger a pager when an interface goes down. What do you do with your SNMP traps? I'd be interesting in finding out.

Create an inventory tool using MCollective's registration feature

Allan Feid — Sat, 08 Oct 2011 22:05:09 +0000

I started working at a company (Shutterstock) that had MCollective implemented a few months ago. We use it to perform various functions across a wide range of servers. Things such as rolling the application servers to pick up new code, clearing caches, checking replication lag on databases, telling our puppetmaster to sign certificates, and even doing code deployment. There are endless possibilites with this system, and it is so simple to write your own agent/client utilities, that I'm not sure why more people aren't using MCollective. One of the things we were lacking was a central inventory of our systems in an easy to view manner. Thankfully, R.I. Pienaar's MCollective has a great feature called registration. Essentially, it causes your nodes to blast out to an ActiveMQ/rabbitmq topic at a set interval, and any node configured to listen can read that data and perform actions based on that data. With that in mind, I set out to create a database of information.

Setting up a registration plugin

Taking a look at this registration plugin, you can begin to see how easy generating your data set is. From this example, you get MCollective's agentlist, facts, and classes sent during registration. I wanted to also report on virtual machines running on hypervisors with an in-house libvirtd agent. To achieve that, your agent needs to have a simple method that can be called without the request/reply data structures. An example of that would be this:

module MCollective
  module Agent
    class Custom<RPC::Agent

      def some_method
        # gather some data and return it
      end

    end
  end
end

Once your agent has a method that can simply be called, you can call this method inside your registration plugin with something like this:

module MCollective
  module Registration
    class Meta<Base
      def body
        result = {
          :agentlist => [],
          :facts => [],
          :classes => []
        }

        cfile = Config.instance.classesfile
        if File.exists?(cfile)
          result[:classes] = File.readlines(cfile).map {|i| i.chomp}
        end

        result[:agentlist] = Agents.agentlist
        result[:facts] = PluginManager["facts_plugin"].get_facts

        if Agents.agentlist.include?("custom")
          # because of how PluginManager works, you need to 
          # append _agent to the agent's name
          result[:custom] = PluginManager["custom_agent"].some_method
        end

        result
      end
    end
  end
end

This way your nodes can report back with literally any information that you'd like to include. I've had great success reporting on my hypervisor's available disk, memory, and CPU utilization as well as what virtual machines are running.

What do you do with this data?

With inspiration from this mongodb registration agent, I decided MongoDB was a pretty good way to store data that was dynamicly generated. I wanted to make it easy to store/retrieve data as I saw fit, so using Mongoid seemed like the way to go. Something as simple as this is all you really need:

class Node
  include Mongoid::Document
  include Mongoid::Timestamps
  field :hostname, :unique => true, :type => String
  index :hostname
  validates_uniqueness_of :hostname
end

Improving performance with a queue

After running registration for a bit, memory usage by mcollective began to skyrocket on the machine processing the data, and eventually I started seeing timeouts or pool limits being hit on my mongo connection. I was able to talk a bit with R.I. Pienaar on IRC, and he informed me that every request that a node recieves spawns a new thread for an agent to run in. This is not scalable for the registration process, and once you hit over about 100 nodes, you run into problems since your machine is constantly opening up new threads to handle registration data. He did offer a solution that he uses personally and has suggested to others. That is to use a queue rather than a topic, and to have a simple daemon dedicated to processing registration data. There's some sample code available on his website, and I'm sure this will be included in the main application at some point. In order to get this working I did need to change my ActiveMQ configuration a bit, but the change was simple and just required adding a new line similar to this:

<authorizationEntry queue="mcollective.>" write="admins" read="admins" admin="admins" />

Basically the mcollective user that you use from the client side needs to be able to read/write to the new queue. This also requires that your registration plugin sends the result out to the queue rather than a topic. Another simple change:

module MCollective
  module Registration
    class Meta<Base
      def body
        ...
        PluginManager["connector_plugin"].connection.publish("/queue/mcollective.registration", result.to_json)
        nil
      end
    end
  end
end

By having the body method return nil, you won't send anything out to the standard registration topic, which would be redundant in this case.

The end result

With a bit of tweaking, I was able to get my registration daemon/agent saving data into mongo in less than a second per request recieved. Once I had all that data in place, I was able to implement a slick interface using Sinatra and Bootstrap that gives me good insight into our inventory. I'll probably go more into the frontend interface when it becomes more mature, but once you have all this data, it's up to you to decide what you want to do with it.

Tracking stats in your application

Allan Feid — Fri, 22 Apr 2011 17:59:26 +0000

Inspired by Etsy's post on keeping track of everything, I set out to achieve similar functionality on a Rails application. Essentially, you run Graphite somewhere and send UDP packets from your application to StatsD. When you have Graphite and StatsD running, you can really get some awesome stats from your application and at the same time, since it's using UDP, the overhead is very minimal.

Getting your graphite on

I'm using Ubuntu 10.10 in this example, because it has a whisper package available, but you should be able to install this on any distro as all the sources are available.

# apt-get install -y bzr python-cairo-dev python-django python-twisted python-whisper libapache2-mod-wsgi

Once you have the necessary packages installed, you can easily grab the graphite source from launchpad, verify you have all the dependencies and finally install the software.

# bzr branch lp:graphite
# cd graphite
# ./check-dependenicies.py
...
# python setup.py install

Graphite installs itself to /opt/graphite, with plenty of example configurations. You will need to copy a few in place to get started.

# cd /opt/graphite/conf
# cp carbon.conf.example carbon.conf
# cp storage-schemas.conf.example storage-schemas.conf
# cp graphite.wsgi.example graphite.wsgi

I added the following to my storage-schemas.conf which provides some sane defaults for use with StatsD, provided by Etsy:

[stats]
priority = 110
pattern = ^stats\..*
retentions = 10:2160,60:10080,600:262974

The Graphite source comes with an example virtual host definition for Apache under the examples directory. I simply copied this over to /etc/apache2/sites-available/graphite, but had to comment out the WSGCISocketPrefix line. After the virtual host definition is in place, you will need to set up the initial database, then change up some permissions so Apache can write to the necessary directories.

# cd /opt/graphite/webapp/graphite
# python manage.py syncdb
# chown -R www-data:www-data /opt/graphite/storage/
# a2ensite graphite
# service apache2 restart

The final step is to start up carbon which is graphite's data aggregator.

# cd /opt/graphite/
# ./bin/carbon-cache.py start

If all went well, you should now have graphite up and running at http://stats.example.com. At some point I really need to look into writing a Chef cookbook for this, but looks like dje has written one already. I haven't checked it out yet, but I'm sure it's at the very least a good starting point.

A simple node.js daemon: StatsD

You can either compile node.js from source, or grab it from a PPA. The version included in the Ubuntu repositories are too old to run StatsD.

# add-apt-repository ppa:richarvey/nodester
# apt-get update
# apt-get install nodejs

With the required dependencies in place, clone the repo and move it to a more suitable location.

# git clone https://github.com/etsy/statsd.git
# mv statsd /opt/
# mkdir /etc/statsd

Create your configuration file, in this case graphite is running on the same machine as statsd:

# cat /etc/statsd/config.js
{
  graphitePort: 2003
, graphiteHost: "localhost"
, port: 8125
}

So now you will probably want to use upstart to integrate the service properly in Ubuntu. Create a simple upstart service definition as follows:

# cat /etc/init/statsd.conf
description "statsd"
author      "rdio"

start on startup
stop on shutdown

script  
    export HOME="/root"
    exec sudo -u nobody nodejs /opt/statsd/stats.js /etc/statsd/config.js
end script

Now you can easily start statsd and verify it is running:

# start statsd
# status statsd
statsd start/running, process 22307
# netstat -nulp | grep nodejs
udp        0      0 0.0.0.0:8125            0.0.0.0:*                           22307/nodejs

At this point you will be able to configure your application to send data at your StatsD instance. There's plenty of code out there for doing this in pretty much every language. In fact, when you cloned the StatsD repository, it came with PHP and Python examples. Below I'll show you how to implement a statsd client in a Rails application.

Some basic stats in a Rails application

To start out, you need to tell Rails you need the dawanda-statsd-client gem, so add the following in your environments.rb:

config.gem 'dawanda-statsd-client', :lib => 'statsd/client'

The default methods are pretty good, but getting your connection information in needs a bit of work so we override the config method in initializers/statsd.rb:

class Statsd
  class << self
    def config
      environments = YAML.load_file("#{Rails.root}/config/statsd-client.yml") || {}
      environments[Rails.env]
    end
  end
end

Next you'll want to keep your StatsD connection information in a separate yaml file. You can define different settings per the running Rails environment. Set up config/statsd-client.yml:

production:
  host: stats.example.com
  port: 8125

Now you're ready to increment any arbitrary stat at some place in your application:

Statsd.increment('user.logins')
Statsd.timing('user.upload', end_time - start_time) # time is tracked in ms

What stats to track?

Well that's entirely up to you, but ideally you would be tracking as much as humanly possible. Things that you don't even know will matter, may matter at some point, so having a history of stats will help you troubleshoot some issues. You could track every CRUD operation in your models, specific actions in your controllers, or even user page load times. You will definitely want to keep track of timings of any external API calls and If you're using Capistrano to do deployments, it's easy to add a simple increment every time you do a deployment. This will allow you to overlay your arbitrary stats with code deployments. This is awesome for seeing the effects of your modifications and can be invaluable for showing progress to your management types.

Using Amazon's CloudFormation, cloud-init, chef and fog to automate infrastructure

Allan Feid — Thu, 03 Mar 2011 02:08:59 +0000

Amazon recently announced their new CloudFormation API. This allows you to create what they call "stacks," which will bring up and provision various AWS resources. You are able to configure security groups, EC2 instances, RDS instances, and elastic load balancers just to name a few. With this level of support and ease of declaration, it makes you wonder how much longer things like RightScale are going to last.

In this post, I'm going to cover a couple different topics. Since there's a lot going on here, I may be brief to explain certain aspects. Hopefully I do a good job of getting the basic idea down, but just in case, here's a quick overview of what I'm attempting to accomplish:

create EC2 user data that can be used with cloud-init
have the user data kick off chef-solo to configure an instance
setup a simple CloudFormation template for easy launching
use fog to launch a CloudFormation template

EC2 user data and cloud-init

User data can be associated with EC2 instances on boot. The data can really be any arbitrary value. It is up to you to handle what is done with this data, however, the Ubuntu developers came up with a pretty cool concept: cloud-init. With the cloud-init package, you can format your user data to their specs, and have the instance perform various actions. This becomes extremely powerful. One of the most widely used features, is to setup a User-Data Script, which in cloud-init, means starting your userdata with "#!", the shebang. A user-data script will run around the same time as rc.local in the boot process making it a great place to perform service configuration.

What if you want more control, rather than relying on scripts? Some of the really cool parts of cloud-init include being able to install packages, specifying that you want apt-get to update, and even upgrade the system on boot. With the multi-part support, you use mime-types to combine multiple files together. This allows you to have both a user-script and a cloud-config portion of your EC2 user data, amongst other things.

With knowledge of cloud-init's user-scripts in mind, I started off with this basic script:

#!/bin/bash
log='/tmp/init.log'
apt-get update &>> $log
apt-get install -y ruby ruby1.8-dev build-essential wget libruby-extras libruby1.8-extras git-core &>> $log
cd /tmp
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz &>> $log
tar zxf rubygems-1.3.7.tgz &>> $log
cd rubygems-1.3.7
ruby setup.rb --no-format-executable &>> $log
gem install ohai chef --no-rdoc --no-ri --verbose &>> $log
mkdir -p /var/chef/cache
git clone -b deploy git://github.com/crazed/cookbooks.git /var/chef/cookbooks &>> $log
mkdir /etc/chef
cat << EOF > /etc/chef/solo.rb
file_cache_path "/var/chef/cache"
cookbook_path "/var/chef/cookbooks"
json_attribs "/etc/chef/node.json"
log_location "/var/chef/solo.log"
verbose_logging true
EOF
cat << EOF > /etc/chef/node.json
{
  "www": {
    "document_root": "/srv/http",
    "server_name": "localhost"
  },
  "mysql": {
    "user": "user",
    "password": "user",
    "hostname": "localhost",
    "database": "drupal",
    "rootpw": "root"
  },
  "drupal": {
    "version": "7.0",
    "modules": [ "pathauto", "token" ]
  },
  "run_list": [ 
    "recipe[php-fpm]",
    "recipe[mysql::new_database]",
    "recipe[drupal::install]",
    "recipe[drupal::nginx-config]"
  ]
}
EOF
chef-solo &>> $log

This was cool and all, but I wanted to really separate my node attributes and solo configuration from the user-script. So I stumbled upon the part-handler portion of cloud-init. This allows you to write a bit of python to handle additional mime-types inside the EC2 user data. Here's a very basic chunk that adds support for text/chef-attributes and text/chef-solo mime types:

#part-handler
# vi: syntax=python ts=4

def list_types():
    # return a list of mime-types that are handled by this module
    return(["text/chef-attributes", "text/chef-solo"])

def handle_part(data,ctype,filename,payload):
    # data: the cloudinit object
    # ctype: '__begin__', '__end__', or the specific mime-type of the part
    # filename: the filename for the part, or dynamically generated part if
    #           no filename is given attribute is present
    # payload: the content of the part (empty for begin or end)
    if ctype == "__begin__":
       print "my handler is beginning"
       return
    if ctype == "__end__":
       print "my handler is ending"
       return

    if ctype == "text/chef-attributes":
        print "handling %s " % ctype
        import os
        attribs = '/etc/chef/node.json'
        d = os.path.dirname(attribs)
        if not os.path.exists(d):
            os.makedirs(d)
        f = open(attribs, 'w')
        f.write(payload)
        f.close()
        return

    if ctype == "text/chef-solo":
        print "handling %s" % ctype
        import os
        solo = '/etc/chef/solo.rb'
        d = os.path.dirname(solo)
        if not os.path.exists(d):
            os.makedirs(d)
        f = open(solo, 'w')
        f.write(payload)
        f.close()
        return

There's probably more effecient ways of doing the above, but this is all still a work in progress for me. Either way, continuing with the original script, I can also pull out the package installation and add this to a cloud-config portion of my user data using the following:

#cloud-config
packages:
- ruby
- ruby1.8-dev
- build-essential
- wget
- libruby-extras
- libruby1.8-extras
- git-core

Note the #cloud-config line at the begining. This tells cloud-init to handle data in the file as part of the cloud-config specification. Now that I have a cloud-config portion installing some required packages, and a part-handler to handle additional mime-types, I can put my chef node attributes and chef solo configuration into their own files.

# cat node.json
{
  "www": {
    "document_root": "/srv/http",
    "server_name": "localhost"
  },
  "mysql": {
    "user": "user",
    "password": "user",
    "hostname": "localhost",
    "database": "drupal",
    "rootpw": "root"
  },
  "drupal": {
    "version": "7.0",
    "modules": [ "pathauto", "token" ]
  },
  "run_list": [ 
    "recipe[php-fpm]",
    "recipe[mysql::new_database]",
    "recipe[drupal::install]",
    "recipe[drupal::nginx-config]"
  ]
}

# cat solo.rb
file_cache_path "/var/chef/cache"
cookbook_path "/var/chef/cookbooks"
json_attribs "/etc/chef/node.json"
log_location "/var/chef/solo.log"
verbose_logging true

I'm not going to go over the node.json part all that much, but basically my chef cookbooks will read this data in order to determine how to configure my instance. I want to set my http document root to /srv/http, setup a new mysql database, and install drupal 7.0 with the pathauto, and token modules. The solo configuration points out where I will be storing my cookbooks, which my stripped down user-script will be checking out from my github repository:

#!/bin/bash
log='/var/log/cloud-init.log'
cd /tmp
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz &>> $log
tar zxf rubygems-1.3.7.tgz &>> $log
cd rubygems-1.3.7
ruby setup.rb --no-format-executable &>> $log
mkdir -p /var/chef/cache
git clone -b deploy git://github.com/crazed/cookbooks.git /var/chef/cookbooks &>> $log
gem install ohai chef --no-rdoc --no-ri --verbose &>> $log
chef-solo &>> $log

Now that I have all these files, I need to combine them using mime-multiparts. There's a script availabe in the cloud-utils package, but you won't really need to install this on your workstation/laptop. So just use bzr to checkout the latest version:

# bzr branch lp:ubuntu/cloud-utils
# cd cloud-utils
# ./write-mime-multipart --output=combined-userdata.txt \
part-handler.py:text/part-handler \
node.json:text/chef-attributes \
solo.rb:text/chef-solo \
cloud-config.txt \
user-script.sh

Note that I don't specify the mime-types for user-script.sh or cloud-config.txt, as they begin with the appropriate lines which cause cloud-init to handle them properly.

CloudFormation templates

So now that you have this slick user-data file, you're going to want to put more than one of these together to create a full infrastructure. This is where the CloudFormation "stacks" come into play. They are JSON based templates. This example will create you an EC2 Security Group and an EC2 instance.

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description": "Create a fully functioning Drupal installation." 
  "Parameters": {
    "InstanceType": {
      "Description": "Type of EC2 instance to launch",
      "Type": "String",
      "Default": "m1.small"
    },
    "WebServerPort": {
      "Description": "TCP/IP port fo the webserver",
      "Type": "String",
      "Default": "80"
    },
    "KeyName": {
      "Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances",
      "Type": "String"
    }
  },

  "Mappings": {
    "AWSInstanceType2Arch": {
      "t1.micro"    : { "Arch": "64" },
      "m1.small"    : { "Arch" : "32" },
      "m1.large"    : { "Arch" : "64" },
      "m1.xlarge"   : { "Arch" : "64" },
      "m2.xlarge"   : { "Arch" : "64" },
      "m2.2xlarge"  : { "Arch" : "64" },
      "m2.4xlarge"  : { "Arch" : "64" },
      "c1.medium"   : { "Arch" : "32" },
      "c1.xlarge"   : { "Arch" : "64" },
      "cc1.4xlarge" : { "Arch" : "64" } 
    }, 
    "AWSRegionArch2AMI": {
      "us-east-1": { "32": "ami-3e02f257", "64": "ami-3202f25b" }
    }
  },

  "Resources": {
    "web1": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "InstanceType": { "Ref": "InstanceType" },
        "SecurityGroups": [ { "Ref": "web" }, "default"  ],
        "KeyName": { "Ref": "KeyName" },
        "ImageId": { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },
                            { "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ] } ] },
        "UserData": { "Fn::Base64": " SEE BELOW " }
      }
    },

    "web": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "Web security group, allows port 80 in",
        "SecurityGroupIngress": [ {
          "IpProtocol": "tcp",
          "FromPort": "22",
          "ToPort": "22",
          "CidrIp": "0.0.0.0/0"
        },
        {
          "IpProtocol": "tcp",
          "FromPort": { "Ref": "WebServerPort" },
          "ToPort": { "Ref": "WebServerPort" },
          "CidrIp": "0.0.0.0/0"
        } ]
      }
    }
  }

}

Taking a look at the template, you'll see it gets a bit messy in the user data portion of my instance. Essentially you need to make sure your userdata is compliant with the JSON format. To do that, I used this simple hackish ruby script:

# cat json_encode.rb
#!/usr/bin/ruby
require 'rubygems'
require 'json'

if ARGV.size != 1
  puts "Usage: #{$0} <file>"
  exit 1
end

def escape(string)
  parse = JSON.parse({ 'json' => string }.to_json)
  parse['json']
end

data = ''
File.open(ARGV[0]) {|f| data << f.read}
p escape(data)

# ./json_encode.rb combined-userdata.txt
...

Manipulating stacks with fog

Now you have a stack template that's fairly powerful. You get to configure a few things, such as what EC2 keypair to use and what instance type you want for the stack. You can even configure what web port will be opened on security group. How do you actually use this data? While Amazon provides command line api tools, I decided to add CloudFormation support to the fog gem and use this to create a basic tool for myself. Realistically, you could take fog and build a nice interface to launch or build your own templates using your own chef cookbooks/node attributes. As of this writing, CloudFormation is not included in the released fog gem, so you're going to have to checkout the latest code from github:

# mkdir projects && cd projects
# git clone git://github.com/geemus/fog.git

With everything in place, here's a simple script that will allow you to launch a stack:

#!/usr/bin/env ruby
require 'optparse'
require "#{ENV['HOME']}/projects/fog/lib/fog"

options = {}
parser = OptionParser.new do |opts|
  opts.banner = "Usage: #{$0} -n STACK_NAME -t TEMPLATE_FILE [options]"

  options[:name] = nil
  opts.on('-n', '--name NAME', 'Required name attribute') do |name|
    options[:name] = name
  end

  options[:parameters] = {}
  opts.on('-p', '--parameters KEY=VALUE', 'Set the parameters for a stack template', 
    '(can be comma separated key value pairs)') do |params|
    params.split(',').each do |param|
      k,v = param.split('=')
      raise "Invalid parameter definition" unless v
      options[:parameters][k] = v
    end
  end

  options[:template] = nil
  opts.on('-t', '--template FILE', 'Set the template file for this stack') do |file|
    options[:template] = file
  end
end

parser.parse!
[:name, :template].each do |p|
  raise "Missing required parameter: --#{p}" unless options[p]
end

config = YAML::load(File.open(ENV['HOME']+'/.fog'))[:default]
template_body = ''
File.open(options[:template]) {|f| template_body << f.read}

cf = Fog::AWS::CloudFormation.new(
  :aws_access_key_id => config[:aws_access_key_id],
  :aws_secret_access_key => config[:aws_secret_access_key]
)

def aws_params(hash)
  r = Hash.new
  c = 1
  hash.each do |k,v|
    r["Parameters.member.#{c}.ParameterKey"] = k
    r["Parameters.member.#{c}.ParameterValue"] = v
    c += 1
  end
  r
end
cf.create_stack(options[:name], {'TemplateBody' => template_body}.merge!(aws_params(options[:parameters])))

You can use it as follows:

# ./create-stack.rb --name drupal --parameters KeyName=allan,InstanceType=m1.large --template drupal_dev.template

This will upload your template definition and have Amazon read and launch the various resources that you have defined. As these resources are launched, you can get status messages using the DescribeStackEvents API call. Again, here's a simple script that's using fog to pull down these messages:

#!/usr/bin/env ruby
require 'rubygems'
require 'yaml'
require "#{ENV['HOME']}/projects/fog/lib/fog"

if ARGV.size != 1
  puts "Usage: #{$0} <stack name>"
  exit 1
end

config = YAML::load(File.open(ENV['HOME']+'/.fog'))[:default]
cf = Fog::AWS::CloudFormation.new(
  :aws_access_key_id => config[:aws_access_key_id],
  :aws_secret_access_key => config[:aws_secret_access_key]
)

events = cf.describe_stack_events('StackName' => ARGV[0]).body['Events']
events.each do |event|
  puts "Timestamp: #{event['Timestamp']}"
  puts "LogicalResourceId: #{event['LogicalResourceId']}"
  puts "ResourceType: #{event['ResourceType']}"
  puts "ResourceStatus: #{event['ResourceStatus']}"
  puts "ResourceStatusReason: #{event['ResourceStatusReason']}" if event['ResourceStatusReason']
  puts "--"
end

And what the output will look like:

# ./describe_stack_events.rb drupal
Timestamp: 2011-03-02T17:13:08Z
LogicalResourceId: drupal
ResourceType: AWS::CloudFormation::Stack
ResourceStatus: CREATE_COMPLETE
--
Timestamp: 2011-03-02T17:13:05Z
LogicalResourceId: web1
ResourceType: AWS::EC2::Instance
ResourceStatus: CREATE_COMPLETE
--
Timestamp: 2011-03-02T17:12:27Z
LogicalResourceId: web1
ResourceType: AWS::EC2::Instance
ResourceStatus: CREATE_IN_PROGRESS
--
Timestamp: 2011-03-02T17:12:27Z
LogicalResourceId: web
ResourceType: AWS::EC2::SecurityGroup
ResourceStatus: CREATE_COMPLETE
--
Timestamp: 2011-03-02T17:12:23Z
LogicalResourceId: web
ResourceType: AWS::EC2::SecurityGroup
ResourceStatus: CREATE_IN_PROGRESS
--
Timestamp: 2011-03-02T17:12:21Z
LogicalResourceId: drupal
ResourceType: AWS::CloudFormation::Stack
ResourceStatus: CREATE_IN_PROGRESS
ResourceStatusReason: User Initiated
--

Again you could simply use the provided Amazon tools, as they are probaly more feature-complete, but sometimes you want to be able to query for specific bits of data. Learning how to use fog will save you tons of time compared to parsing data that comes out of the Amazon tools. Fog really lets you do some powerful things with the Amazon APIs and is always gaining more features.

Current shortcommings and summary

This setup has a few issues that I currently don't know how to solve. Let's say you were to launch a stack that included two EC2 instances, a load balancer, and an RDS instance. During the creation phase, you want to import a basic DB schema into your RDS instance. Since you can't get shell on the RDS instance, you will have to use one of your EC2 instances to import the schema. You have a race condition here, how do you make sure your RDS instance is available before your EC2 instance attemps to import data? Right now the only thing I can think of is to have your chef recipe gracefully handle retries. It seems we need a way for resources of a stack to depend on other resources before being launched, or a notification system that could potentially kick off the chef run (SNS?).

Either way, I think CloudFormation, chef and fog can be used in conjuction with cloud-init to create quick deployments that are OS independent. This allows you to maintain your automation portions separately from your images, which as a consequence, promotes the use of vanilla images. With these basic ideas, you can easily swap out the node.json file for your own, and replace the user-script in such a way that it checks out your own cookbooks. The possibilities are really endless. Currently cloud-init supports Puppet out of the box, and I'd like to see some chef support here as well.

Scaling your application in the cloud

Allan Feid — Wed, 16 Feb 2011 02:55:26 +0000

Recently, I've been standing up a lot of infrastructure in Amazon's EC2. While auto-scaling the infrastructure aspects is fairly simple using configuration automation utilities such as Chef or Puppet, getting developers to understand these concepts may be an interesting task. I will try to outline most of the common issues I see when uplifting applications to an elastic cloud environment.

Rules to live by

Infrastructure automation is critical

You need to know how to build the infrastructure that runs your application, and you need to know how to do this repeatedly. Using tools such as Chef or Puppet will allow you to do this efficiently and in a self-documenting way that also provides revision control to your software configurations. Being able to do this allows you to quickly bring up additional server resources as your application demands it.

Revision control is not optional

Choose your favorite source control system and learn how to use every aspect of it. These tools were meant to keep track of your code changes. When dealing with the cloud, you're going to need to keep track of what is in you production environment vs your staging environment.

One of the biggest changes with a cloud based infrastructure, is that the underlying resources will be changing. There will be monitors that detect high traffic, CPU usage, memory usage, or anything really. This will cause additional virtual machines (instances) to launch, configure themselves (see above), check out the latest version of your code from your repository, and finally add itself to a load balancer. When your application's server infrastructure can build itself, you better be able to checkout the latest revision from your deployment branch and have it run with no human interaction.

Don't rely on traditional shared storage

I've seen many legacy applications rely on shared storage of some sort. This will not work in most cloud environments, and definitely will not work in Amazon. As of this writing, Amazon's Elastic Block Storage (EBS) volumes can only be attached to one server instance. This means you cannot utilize a clustered filesystem and standing up your own NFS server leaves for a single point of failure.

The solution? Put all of your site content that isn't revision controlled into an asset host or content delivery network (CDN). When on Amazon, this means using Amazon's Simple Storage Service (S3). By doing this you are storing your precious images, movies, and other content on storage that is 99.999999999% available and accessible anywhere over HTTP. As an added bonus, you will be serving up images faster and can achieve better edge network performance by enabling CloudFront, Amazon's CDN service.

Relational databases aren't evil

There's a lot of hype over NoSQL databases such as MongoDB and CouchDB. They are touted as being "webscale" because you can add and remove instances from the database cluster as needed. These are great if you're starting a new application and have decided you need these features, but what if you have existing data in MySQL?

Amazon's Relational Database Service (RDS) supports MySQL databases that are both redundant and scalable to your needs. You can easily grow the database storage size, cpu, and memory resources. They also perform automatic backups in a timeslot that you configure. This makes transitioning from traditional database infrastructure to the cloud fairly painless. They have also announced plans to support Oracle 11g in the future on the same platform. So while scaling relational databases are not as trivial as using MongoDB, Amazon has really stepped up to the plate by taking care of this problem for you.

Summary

By utilizing asset hosts, maintaining an automated server build, and keeping strict revision control you will be setting yourself up for success in today's cloud environment, however, I cannot stress the importance of at least having a staging environment. Have unit test and force all commits to be tested, then deployed to your staging environment. This allows you to quickly catch any code issues. An added benefit to your staging environment, is that you can test infrastructure changes before deciding to put the changes in the wild.

Setting up your own git repository

Allan Feid — Thu, 03 Feb 2011 02:27:31 +0000

Setup a bare repository

The first thing you'll need to do is create a new bare repository. A bare repository is one that does not contain a working directory, essentially it just contains everything in your standard .git directory. I'll be serving all my repositories from /srv/git, and assuming all the users you want to give access to this repository are in the git group.

# mkdir /srv/git
# git init --bare --shared=group /srv/git/my-awesome-application.git
# chgrp -R git /srv/git/my-awesome-application.git

If you're not going to be sharing this with multiple users, you won't need the --shared=group. Later down the road, if you forgot to specify this option, all it really does is the following:

# chmod -R g+rws /srv/git/my-awesome-application.git

Then, you either need to change the default umask of each user to 002, or you can make /srv/git/my-awesome-application.git/config look like this:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        sharedrepository = 1

The key is the sharedrepository = 1 directive. This can be changed to a variety of options, but when set to true or group it ensures all file pushes will use a sane umask of 002. You can find more information about this setting in the git-config(1) man page.

Configuring user access

I personally never setup a git repository without using ssh keys. So have your committers send your their public keys and create users for them. Add these keys to their authorized_keys file, and have them confirm they can ssh in without any issues. Once this is done, you can have them add a new remote repository to git as follows:

# git remote add origin user@crazed.pl:/srv/git/my-awesome-application.git

That's it, they're ready to start pushing code in the normal ways:

# git push origin master

Allowing read only access

Sometimes it's necessary to allow read only access to your repositories, such as for use with Capistrano or performing continuous integration. In this case, using git-daemon is easy and provides faster cloning/pulls than using HTTP or SSH. On Ubuntu, you can install the following package, which sets you up with a runit configuration:

# apt-get install -y git-daemon-run

There's one thing you'll likely want to change in this daemon's arguments, and that is the --base-path option. I use the following in /etc/sv/git-daemon/run:

#!/bin/sh
exec 2>&1
echo 'git-daemon starting.'
exec chpst -ugitdaemon \
  /usr/lib/git-core/git-daemon --verbose --base-path=/srv/git /srv/git

If you took a look at the git-daemon(1) man page, you will see that the base path essentially changes the root for git requests. So, in reality your application lives at /srv/git/my-awesome-application.git, but when pulling over the git protocol, it's as easy as this:

# git clone git://crazed.pl/my-awesome-application.git

However, before you can even do that, there is one magic step that makes this all work.

# touch /srv/git/my-awesome-application.git/git-daemon-export-ok

Touching this file tells the git-daemon that you meant for this repository to be publicly accessible (assuming you have the proper ports open, tcp/9418).

Providing a web front end

So your boss wants to see what you've been doing, but has no idea what git is. Why not give him a simple web interface? Luckily, Ubuntu again has a package for this:

# apt-get install gitweb

Gitweb is a simple web interface written in Perl and used as a CGI script. There may be better interfaces out there, but if it's good enough for kernel.org, it's good enough for me. To get this working, there is one simple change to the configuration file /etc/gitweb.conf:

# path to git projects (<project>.git)
$projectroot = "/srv/git";

The Ubuntu package comes with a simple Apache configuration to get this working at fqdn.com/gitweb:

Alias /gitweb /usr/share/gitweb

<Directory /usr/share/gitweb>
  Options FollowSymLinks +ExecCGI
  AddHandler cgi-script .cgi
</Directory>
root@ip-10-117-65-141:~# vi /etc/gitweb.conf
root@ip-10-117-65-141:~# cat /etc/apache2/conf.d/gitweb 
Alias /gitweb /usr/share/gitweb

<Directory /usr/share/gitweb>
  Options FollowSymLinks +ExecCGI
  AddHandler cgi-script .cgi
</Directory>

That's it, you now have a private repository that can be shared amongst multiple users, as well as a web interface and read-only access to those that need it.

Continuous integration with git, Capistrano, and Integrity

Allan Feid — Thu, 03 Feb 2011 01:36:45 +0000

What is continuous integration (CI)?

When working in an environment with multiple developers, the practice of continous integration can be tremendous for catching integration errors quickly. This means, as soon as code is checked in to your central repository, integration will be verified by an automated build / test suite. This requires that you have a single source repository and are writing proper tests for your application.

Automated builds / tests with Integrity

Integrity will build your code, then run tests to make sure everything works as expected. Installation instructions can be found on their homepage, but once you do have it setup, it is quite easy to add extremely useful functionality. What I have used, is the email on completion functionality. To get this up and running, you simply need the following line in your init.rb file:

require "integrity/notifier/email"

Once you have added a new project, the repository URI, branch, and name should be straight forward to fill out. Your build script is what will be evaluated for a return value of 0, indicating everything went well and good. You can use this section to do some really cool things, what I've done is take a rails app and run some tasks using rake. Then using &&, on success of the tests, it will launch a deploy using capistrano.

rake RAILS_ENV=test gems:build db:migrate \
&& spec ./spec --format specdoc && cap -f /srv/app/capfile deploy

I heavily advise against using this straight to your production web server, in this example I'm having cap deploy to my staging servers. This is a great way to go from successful tests to deployment on your staging or QA servers. Now, to complete the full automated build to deploy, you need a way for your code check-ins to perform an action.

Using git hooks to initiate a build

Inside every git repository, there is a subdirectory called hooks. This directory contains a series of specially named scripts that will get executed at various points whne interacting with git. For a full explanation of the scripts, check out the githooks(5) man page.

#!/bin/bash
echo "Received commit to branch: $1"
if [[ "$1" =~ ^(.*)/master$ ]]; then
  echo "Starting Integrity build..."
  curl -F 'build=build' http://ci.example.com/project/builds 2>/dev/null
else
  echo "Current branch is not master, Integrity will not be started."
fi

Sending a simple POST request to your integrity server at the /:project/builds URI will cause integrity to do its thing. This script checks the branch that was commited to, and only kicks off the POST request if it was master.

There you have it, a full commit, test, and deployment all automated. This forces your developers to use version control and write good tests, because as soon as you commit something, it will be tested and if it passes it's going straight to your servers. As a sysadmin, I've learned a lot about rails applications by setting something like this up.

Allan Feid - devops

Doing something useful with your SNMP traps

Quick primer on net-snmp

Get your mibs in the snmptt format

Intercept the traps and send them to snmptt

A couple tips about snmptt

Do something useful with your traps

Tags:

Create an inventory tool using MCollective's registration feature

Setting up a registration plugin

What do you do with this data?

Improving performance with a queue

The end result

Tags:

Tracking stats in your application

Getting your graphite on

A simple node.js daemon: StatsD

Some basic stats in a Rails application

What stats to track?

Tags:

Using Amazon's CloudFormation, cloud-init, chef and fog to automate infrastructure

EC2 user data and cloud-init

CloudFormation templates

Manipulating stacks with fog

Current shortcommings and summary

Tags:

Scaling your application in the cloud

Rules to live by

Infrastructure automation is critical

Revision control is not optional

Don't rely on traditional shared storage

Relational databases aren't evil

Summary

Tags:

Setting up your own git repository

Setup a bare repository

Configuring user access

Allowing read only access

Providing a web front end

Tags:

Continuous integration with git, Capistrano, and Integrity

What is continuous integration (CI)?

Automated builds / tests with Integrity

Using git hooks to initiate a build

Tags: