Allan Feid - sysadmin

Create an inventory tool using MCollective's registration feature

Allan Feid — Sat, 08 Oct 2011 22:05:09 +0000

I started working at a company (Shutterstock) that had MCollective implemented a few months ago. We use it to perform various functions across a wide range of servers. Things such as rolling the application servers to pick up new code, clearing caches, checking replication lag on databases, telling our puppetmaster to sign certificates, and even doing code deployment. There are endless possibilites with this system, and it is so simple to write your own agent/client utilities, that I'm not sure why more people aren't using MCollective. One of the things we were lacking was a central inventory of our systems in an easy to view manner. Thankfully, R.I. Pienaar's MCollective has a great feature called registration. Essentially, it causes your nodes to blast out to an ActiveMQ/rabbitmq topic at a set interval, and any node configured to listen can read that data and perform actions based on that data. With that in mind, I set out to create a database of information.

Setting up a registration plugin

Taking a look at this registration plugin, you can begin to see how easy generating your data set is. From this example, you get MCollective's agentlist, facts, and classes sent during registration. I wanted to also report on virtual machines running on hypervisors with an in-house libvirtd agent. To achieve that, your agent needs to have a simple method that can be called without the request/reply data structures. An example of that would be this:

module MCollective
  module Agent
    class Custom<RPC::Agent

      def some_method
        # gather some data and return it
      end

    end
  end
end

Once your agent has a method that can simply be called, you can call this method inside your registration plugin with something like this:

module MCollective
  module Registration
    class Meta<Base
      def body
        result = {
          :agentlist => [],
          :facts => [],
          :classes => []
        }

        cfile = Config.instance.classesfile
        if File.exists?(cfile)
          result[:classes] = File.readlines(cfile).map {|i| i.chomp}
        end

        result[:agentlist] = Agents.agentlist
        result[:facts] = PluginManager["facts_plugin"].get_facts

        if Agents.agentlist.include?("custom")
          # because of how PluginManager works, you need to 
          # append _agent to the agent's name
          result[:custom] = PluginManager["custom_agent"].some_method
        end

        result
      end
    end
  end
end

This way your nodes can report back with literally any information that you'd like to include. I've had great success reporting on my hypervisor's available disk, memory, and CPU utilization as well as what virtual machines are running.

What do you do with this data?

With inspiration from this mongodb registration agent, I decided MongoDB was a pretty good way to store data that was dynamicly generated. I wanted to make it easy to store/retrieve data as I saw fit, so using Mongoid seemed like the way to go. Something as simple as this is all you really need:

class Node
  include Mongoid::Document
  include Mongoid::Timestamps
  field :hostname, :unique => true, :type => String
  index :hostname
  validates_uniqueness_of :hostname
end

Improving performance with a queue

After running registration for a bit, memory usage by mcollective began to skyrocket on the machine processing the data, and eventually I started seeing timeouts or pool limits being hit on my mongo connection. I was able to talk a bit with R.I. Pienaar on IRC, and he informed me that every request that a node recieves spawns a new thread for an agent to run in. This is not scalable for the registration process, and once you hit over about 100 nodes, you run into problems since your machine is constantly opening up new threads to handle registration data. He did offer a solution that he uses personally and has suggested to others. That is to use a queue rather than a topic, and to have a simple daemon dedicated to processing registration data. There's some sample code available on his website, and I'm sure this will be included in the main application at some point. In order to get this working I did need to change my ActiveMQ configuration a bit, but the change was simple and just required adding a new line similar to this:

<authorizationEntry queue="mcollective.>" write="admins" read="admins" admin="admins" />

Basically the mcollective user that you use from the client side needs to be able to read/write to the new queue. This also requires that your registration plugin sends the result out to the queue rather than a topic. Another simple change:

module MCollective
  module Registration
    class Meta<Base
      def body
        ...
        PluginManager["connector_plugin"].connection.publish("/queue/mcollective.registration", result.to_json)
        nil
      end
    end
  end
end

By having the body method return nil, you won't send anything out to the standard registration topic, which would be redundant in this case.

The end result

With a bit of tweaking, I was able to get my registration daemon/agent saving data into mongo in less than a second per request recieved. Once I had all that data in place, I was able to implement a slick interface using Sinatra and Bootstrap that gives me good insight into our inventory. I'll probably go more into the frontend interface when it becomes more mature, but once you have all this data, it's up to you to decide what you want to do with it.

Tracking stats in your application

Allan Feid — Fri, 22 Apr 2011 17:59:26 +0000

Inspired by Etsy's post on keeping track of everything, I set out to achieve similar functionality on a Rails application. Essentially, you run Graphite somewhere and send UDP packets from your application to StatsD. When you have Graphite and StatsD running, you can really get some awesome stats from your application and at the same time, since it's using UDP, the overhead is very minimal.

Getting your graphite on

I'm using Ubuntu 10.10 in this example, because it has a whisper package available, but you should be able to install this on any distro as all the sources are available.

# apt-get install -y bzr python-cairo-dev python-django python-twisted python-whisper libapache2-mod-wsgi

Once you have the necessary packages installed, you can easily grab the graphite source from launchpad, verify you have all the dependencies and finally install the software.

# bzr branch lp:graphite
# cd graphite
# ./check-dependenicies.py
...
# python setup.py install

Graphite installs itself to /opt/graphite, with plenty of example configurations. You will need to copy a few in place to get started.

# cd /opt/graphite/conf
# cp carbon.conf.example carbon.conf
# cp storage-schemas.conf.example storage-schemas.conf
# cp graphite.wsgi.example graphite.wsgi

I added the following to my storage-schemas.conf which provides some sane defaults for use with StatsD, provided by Etsy:

[stats]
priority = 110
pattern = ^stats\..*
retentions = 10:2160,60:10080,600:262974

The Graphite source comes with an example virtual host definition for Apache under the examples directory. I simply copied this over to /etc/apache2/sites-available/graphite, but had to comment out the WSGCISocketPrefix line. After the virtual host definition is in place, you will need to set up the initial database, then change up some permissions so Apache can write to the necessary directories.

# cd /opt/graphite/webapp/graphite
# python manage.py syncdb
# chown -R www-data:www-data /opt/graphite/storage/
# a2ensite graphite
# service apache2 restart

The final step is to start up carbon which is graphite's data aggregator.

# cd /opt/graphite/
# ./bin/carbon-cache.py start

If all went well, you should now have graphite up and running at http://stats.example.com. At some point I really need to look into writing a Chef cookbook for this, but looks like dje has written one already. I haven't checked it out yet, but I'm sure it's at the very least a good starting point.

A simple node.js daemon: StatsD

You can either compile node.js from source, or grab it from a PPA. The version included in the Ubuntu repositories are too old to run StatsD.

# add-apt-repository ppa:richarvey/nodester
# apt-get update
# apt-get install nodejs

With the required dependencies in place, clone the repo and move it to a more suitable location.

# git clone https://github.com/etsy/statsd.git
# mv statsd /opt/
# mkdir /etc/statsd

Create your configuration file, in this case graphite is running on the same machine as statsd:

# cat /etc/statsd/config.js
{
  graphitePort: 2003
, graphiteHost: "localhost"
, port: 8125
}

So now you will probably want to use upstart to integrate the service properly in Ubuntu. Create a simple upstart service definition as follows:

# cat /etc/init/statsd.conf
description "statsd"
author      "rdio"

start on startup
stop on shutdown

script  
    export HOME="/root"
    exec sudo -u nobody nodejs /opt/statsd/stats.js /etc/statsd/config.js
end script

Now you can easily start statsd and verify it is running:

# start statsd
# status statsd
statsd start/running, process 22307
# netstat -nulp | grep nodejs
udp        0      0 0.0.0.0:8125            0.0.0.0:*                           22307/nodejs

At this point you will be able to configure your application to send data at your StatsD instance. There's plenty of code out there for doing this in pretty much every language. In fact, when you cloned the StatsD repository, it came with PHP and Python examples. Below I'll show you how to implement a statsd client in a Rails application.

Some basic stats in a Rails application

To start out, you need to tell Rails you need the dawanda-statsd-client gem, so add the following in your environments.rb:

config.gem 'dawanda-statsd-client', :lib => 'statsd/client'

The default methods are pretty good, but getting your connection information in needs a bit of work so we override the config method in initializers/statsd.rb:

class Statsd
  class << self
    def config
      environments = YAML.load_file("#{Rails.root}/config/statsd-client.yml") || {}
      environments[Rails.env]
    end
  end
end

Next you'll want to keep your StatsD connection information in a separate yaml file. You can define different settings per the running Rails environment. Set up config/statsd-client.yml:

production:
  host: stats.example.com
  port: 8125

Now you're ready to increment any arbitrary stat at some place in your application:

Statsd.increment('user.logins')
Statsd.timing('user.upload', end_time - start_time) # time is tracked in ms

What stats to track?

Well that's entirely up to you, but ideally you would be tracking as much as humanly possible. Things that you don't even know will matter, may matter at some point, so having a history of stats will help you troubleshoot some issues. You could track every CRUD operation in your models, specific actions in your controllers, or even user page load times. You will definitely want to keep track of timings of any external API calls and If you're using Capistrano to do deployments, it's easy to add a simple increment every time you do a deployment. This will allow you to overlay your arbitrary stats with code deployments. This is awesome for seeing the effects of your modifications and can be invaluable for showing progress to your management types.

Using Amazon's CloudFormation, cloud-init, chef and fog to automate infrastructure

Allan Feid — Thu, 03 Mar 2011 02:08:59 +0000

Amazon recently announced their new CloudFormation API. This allows you to create what they call "stacks," which will bring up and provision various AWS resources. You are able to configure security groups, EC2 instances, RDS instances, and elastic load balancers just to name a few. With this level of support and ease of declaration, it makes you wonder how much longer things like RightScale are going to last.

In this post, I'm going to cover a couple different topics. Since there's a lot going on here, I may be brief to explain certain aspects. Hopefully I do a good job of getting the basic idea down, but just in case, here's a quick overview of what I'm attempting to accomplish:

create EC2 user data that can be used with cloud-init
have the user data kick off chef-solo to configure an instance
setup a simple CloudFormation template for easy launching
use fog to launch a CloudFormation template

EC2 user data and cloud-init

User data can be associated with EC2 instances on boot. The data can really be any arbitrary value. It is up to you to handle what is done with this data, however, the Ubuntu developers came up with a pretty cool concept: cloud-init. With the cloud-init package, you can format your user data to their specs, and have the instance perform various actions. This becomes extremely powerful. One of the most widely used features, is to setup a User-Data Script, which in cloud-init, means starting your userdata with "#!", the shebang. A user-data script will run around the same time as rc.local in the boot process making it a great place to perform service configuration.

What if you want more control, rather than relying on scripts? Some of the really cool parts of cloud-init include being able to install packages, specifying that you want apt-get to update, and even upgrade the system on boot. With the multi-part support, you use mime-types to combine multiple files together. This allows you to have both a user-script and a cloud-config portion of your EC2 user data, amongst other things.

With knowledge of cloud-init's user-scripts in mind, I started off with this basic script:

#!/bin/bash
log='/tmp/init.log'
apt-get update &>> $log
apt-get install -y ruby ruby1.8-dev build-essential wget libruby-extras libruby1.8-extras git-core &>> $log
cd /tmp
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz &>> $log
tar zxf rubygems-1.3.7.tgz &>> $log
cd rubygems-1.3.7
ruby setup.rb --no-format-executable &>> $log
gem install ohai chef --no-rdoc --no-ri --verbose &>> $log
mkdir -p /var/chef/cache
git clone -b deploy git://github.com/crazed/cookbooks.git /var/chef/cookbooks &>> $log
mkdir /etc/chef
cat << EOF > /etc/chef/solo.rb
file_cache_path "/var/chef/cache"
cookbook_path "/var/chef/cookbooks"
json_attribs "/etc/chef/node.json"
log_location "/var/chef/solo.log"
verbose_logging true
EOF
cat << EOF > /etc/chef/node.json
{
  "www": {
    "document_root": "/srv/http",
    "server_name": "localhost"
  },
  "mysql": {
    "user": "user",
    "password": "user",
    "hostname": "localhost",
    "database": "drupal",
    "rootpw": "root"
  },
  "drupal": {
    "version": "7.0",
    "modules": [ "pathauto", "token" ]
  },
  "run_list": [ 
    "recipe[php-fpm]",
    "recipe[mysql::new_database]",
    "recipe[drupal::install]",
    "recipe[drupal::nginx-config]"
  ]
}
EOF
chef-solo &>> $log

This was cool and all, but I wanted to really separate my node attributes and solo configuration from the user-script. So I stumbled upon the part-handler portion of cloud-init. This allows you to write a bit of python to handle additional mime-types inside the EC2 user data. Here's a very basic chunk that adds support for text/chef-attributes and text/chef-solo mime types:

#part-handler
# vi: syntax=python ts=4

def list_types():
    # return a list of mime-types that are handled by this module
    return(["text/chef-attributes", "text/chef-solo"])

def handle_part(data,ctype,filename,payload):
    # data: the cloudinit object
    # ctype: '__begin__', '__end__', or the specific mime-type of the part
    # filename: the filename for the part, or dynamically generated part if
    #           no filename is given attribute is present
    # payload: the content of the part (empty for begin or end)
    if ctype == "__begin__":
       print "my handler is beginning"
       return
    if ctype == "__end__":
       print "my handler is ending"
       return

    if ctype == "text/chef-attributes":
        print "handling %s " % ctype
        import os
        attribs = '/etc/chef/node.json'
        d = os.path.dirname(attribs)
        if not os.path.exists(d):
            os.makedirs(d)
        f = open(attribs, 'w')
        f.write(payload)
        f.close()
        return

    if ctype == "text/chef-solo":
        print "handling %s" % ctype
        import os
        solo = '/etc/chef/solo.rb'
        d = os.path.dirname(solo)
        if not os.path.exists(d):
            os.makedirs(d)
        f = open(solo, 'w')
        f.write(payload)
        f.close()
        return

There's probably more effecient ways of doing the above, but this is all still a work in progress for me. Either way, continuing with the original script, I can also pull out the package installation and add this to a cloud-config portion of my user data using the following:

#cloud-config
packages:
- ruby
- ruby1.8-dev
- build-essential
- wget
- libruby-extras
- libruby1.8-extras
- git-core

Note the #cloud-config line at the begining. This tells cloud-init to handle data in the file as part of the cloud-config specification. Now that I have a cloud-config portion installing some required packages, and a part-handler to handle additional mime-types, I can put my chef node attributes and chef solo configuration into their own files.

# cat node.json
{
  "www": {
    "document_root": "/srv/http",
    "server_name": "localhost"
  },
  "mysql": {
    "user": "user",
    "password": "user",
    "hostname": "localhost",
    "database": "drupal",
    "rootpw": "root"
  },
  "drupal": {
    "version": "7.0",
    "modules": [ "pathauto", "token" ]
  },
  "run_list": [ 
    "recipe[php-fpm]",
    "recipe[mysql::new_database]",
    "recipe[drupal::install]",
    "recipe[drupal::nginx-config]"
  ]
}

# cat solo.rb
file_cache_path "/var/chef/cache"
cookbook_path "/var/chef/cookbooks"
json_attribs "/etc/chef/node.json"
log_location "/var/chef/solo.log"
verbose_logging true

I'm not going to go over the node.json part all that much, but basically my chef cookbooks will read this data in order to determine how to configure my instance. I want to set my http document root to /srv/http, setup a new mysql database, and install drupal 7.0 with the pathauto, and token modules. The solo configuration points out where I will be storing my cookbooks, which my stripped down user-script will be checking out from my github repository:

#!/bin/bash
log='/var/log/cloud-init.log'
cd /tmp
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz &>> $log
tar zxf rubygems-1.3.7.tgz &>> $log
cd rubygems-1.3.7
ruby setup.rb --no-format-executable &>> $log
mkdir -p /var/chef/cache
git clone -b deploy git://github.com/crazed/cookbooks.git /var/chef/cookbooks &>> $log
gem install ohai chef --no-rdoc --no-ri --verbose &>> $log
chef-solo &>> $log

Now that I have all these files, I need to combine them using mime-multiparts. There's a script availabe in the cloud-utils package, but you won't really need to install this on your workstation/laptop. So just use bzr to checkout the latest version:

# bzr branch lp:ubuntu/cloud-utils
# cd cloud-utils
# ./write-mime-multipart --output=combined-userdata.txt \
part-handler.py:text/part-handler \
node.json:text/chef-attributes \
solo.rb:text/chef-solo \
cloud-config.txt \
user-script.sh

Note that I don't specify the mime-types for user-script.sh or cloud-config.txt, as they begin with the appropriate lines which cause cloud-init to handle them properly.

CloudFormation templates

So now that you have this slick user-data file, you're going to want to put more than one of these together to create a full infrastructure. This is where the CloudFormation "stacks" come into play. They are JSON based templates. This example will create you an EC2 Security Group and an EC2 instance.

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description": "Create a fully functioning Drupal installation." 
  "Parameters": {
    "InstanceType": {
      "Description": "Type of EC2 instance to launch",
      "Type": "String",
      "Default": "m1.small"
    },
    "WebServerPort": {
      "Description": "TCP/IP port fo the webserver",
      "Type": "String",
      "Default": "80"
    },
    "KeyName": {
      "Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances",
      "Type": "String"
    }
  },

  "Mappings": {
    "AWSInstanceType2Arch": {
      "t1.micro"    : { "Arch": "64" },
      "m1.small"    : { "Arch" : "32" },
      "m1.large"    : { "Arch" : "64" },
      "m1.xlarge"   : { "Arch" : "64" },
      "m2.xlarge"   : { "Arch" : "64" },
      "m2.2xlarge"  : { "Arch" : "64" },
      "m2.4xlarge"  : { "Arch" : "64" },
      "c1.medium"   : { "Arch" : "32" },
      "c1.xlarge"   : { "Arch" : "64" },
      "cc1.4xlarge" : { "Arch" : "64" } 
    }, 
    "AWSRegionArch2AMI": {
      "us-east-1": { "32": "ami-3e02f257", "64": "ami-3202f25b" }
    }
  },

  "Resources": {
    "web1": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "InstanceType": { "Ref": "InstanceType" },
        "SecurityGroups": [ { "Ref": "web" }, "default"  ],
        "KeyName": { "Ref": "KeyName" },
        "ImageId": { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },
                            { "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ] } ] },
        "UserData": { "Fn::Base64": " SEE BELOW " }
      }
    },

    "web": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "Web security group, allows port 80 in",
        "SecurityGroupIngress": [ {
          "IpProtocol": "tcp",
          "FromPort": "22",
          "ToPort": "22",
          "CidrIp": "0.0.0.0/0"
        },
        {
          "IpProtocol": "tcp",
          "FromPort": { "Ref": "WebServerPort" },
          "ToPort": { "Ref": "WebServerPort" },
          "CidrIp": "0.0.0.0/0"
        } ]
      }
    }
  }

}

Taking a look at the template, you'll see it gets a bit messy in the user data portion of my instance. Essentially you need to make sure your userdata is compliant with the JSON format. To do that, I used this simple hackish ruby script:

# cat json_encode.rb
#!/usr/bin/ruby
require 'rubygems'
require 'json'

if ARGV.size != 1
  puts "Usage: #{$0} <file>"
  exit 1
end

def escape(string)
  parse = JSON.parse({ 'json' => string }.to_json)
  parse['json']
end

data = ''
File.open(ARGV[0]) {|f| data << f.read}
p escape(data)

# ./json_encode.rb combined-userdata.txt
...

Manipulating stacks with fog

Now you have a stack template that's fairly powerful. You get to configure a few things, such as what EC2 keypair to use and what instance type you want for the stack. You can even configure what web port will be opened on security group. How do you actually use this data? While Amazon provides command line api tools, I decided to add CloudFormation support to the fog gem and use this to create a basic tool for myself. Realistically, you could take fog and build a nice interface to launch or build your own templates using your own chef cookbooks/node attributes. As of this writing, CloudFormation is not included in the released fog gem, so you're going to have to checkout the latest code from github:

# mkdir projects && cd projects
# git clone git://github.com/geemus/fog.git

With everything in place, here's a simple script that will allow you to launch a stack:

#!/usr/bin/env ruby
require 'optparse'
require "#{ENV['HOME']}/projects/fog/lib/fog"

options = {}
parser = OptionParser.new do |opts|
  opts.banner = "Usage: #{$0} -n STACK_NAME -t TEMPLATE_FILE [options]"

  options[:name] = nil
  opts.on('-n', '--name NAME', 'Required name attribute') do |name|
    options[:name] = name
  end

  options[:parameters] = {}
  opts.on('-p', '--parameters KEY=VALUE', 'Set the parameters for a stack template', 
    '(can be comma separated key value pairs)') do |params|
    params.split(',').each do |param|
      k,v = param.split('=')
      raise "Invalid parameter definition" unless v
      options[:parameters][k] = v
    end
  end

  options[:template] = nil
  opts.on('-t', '--template FILE', 'Set the template file for this stack') do |file|
    options[:template] = file
  end
end

parser.parse!
[:name, :template].each do |p|
  raise "Missing required parameter: --#{p}" unless options[p]
end

config = YAML::load(File.open(ENV['HOME']+'/.fog'))[:default]
template_body = ''
File.open(options[:template]) {|f| template_body << f.read}

cf = Fog::AWS::CloudFormation.new(
  :aws_access_key_id => config[:aws_access_key_id],
  :aws_secret_access_key => config[:aws_secret_access_key]
)

def aws_params(hash)
  r = Hash.new
  c = 1
  hash.each do |k,v|
    r["Parameters.member.#{c}.ParameterKey"] = k
    r["Parameters.member.#{c}.ParameterValue"] = v
    c += 1
  end
  r
end
cf.create_stack(options[:name], {'TemplateBody' => template_body}.merge!(aws_params(options[:parameters])))

You can use it as follows:

# ./create-stack.rb --name drupal --parameters KeyName=allan,InstanceType=m1.large --template drupal_dev.template

This will upload your template definition and have Amazon read and launch the various resources that you have defined. As these resources are launched, you can get status messages using the DescribeStackEvents API call. Again, here's a simple script that's using fog to pull down these messages:

#!/usr/bin/env ruby
require 'rubygems'
require 'yaml'
require "#{ENV['HOME']}/projects/fog/lib/fog"

if ARGV.size != 1
  puts "Usage: #{$0} <stack name>"
  exit 1
end

config = YAML::load(File.open(ENV['HOME']+'/.fog'))[:default]
cf = Fog::AWS::CloudFormation.new(
  :aws_access_key_id => config[:aws_access_key_id],
  :aws_secret_access_key => config[:aws_secret_access_key]
)

events = cf.describe_stack_events('StackName' => ARGV[0]).body['Events']
events.each do |event|
  puts "Timestamp: #{event['Timestamp']}"
  puts "LogicalResourceId: #{event['LogicalResourceId']}"
  puts "ResourceType: #{event['ResourceType']}"
  puts "ResourceStatus: #{event['ResourceStatus']}"
  puts "ResourceStatusReason: #{event['ResourceStatusReason']}" if event['ResourceStatusReason']
  puts "--"
end

And what the output will look like:

# ./describe_stack_events.rb drupal
Timestamp: 2011-03-02T17:13:08Z
LogicalResourceId: drupal
ResourceType: AWS::CloudFormation::Stack
ResourceStatus: CREATE_COMPLETE
--
Timestamp: 2011-03-02T17:13:05Z
LogicalResourceId: web1
ResourceType: AWS::EC2::Instance
ResourceStatus: CREATE_COMPLETE
--
Timestamp: 2011-03-02T17:12:27Z
LogicalResourceId: web1
ResourceType: AWS::EC2::Instance
ResourceStatus: CREATE_IN_PROGRESS
--
Timestamp: 2011-03-02T17:12:27Z
LogicalResourceId: web
ResourceType: AWS::EC2::SecurityGroup
ResourceStatus: CREATE_COMPLETE
--
Timestamp: 2011-03-02T17:12:23Z
LogicalResourceId: web
ResourceType: AWS::EC2::SecurityGroup
ResourceStatus: CREATE_IN_PROGRESS
--
Timestamp: 2011-03-02T17:12:21Z
LogicalResourceId: drupal
ResourceType: AWS::CloudFormation::Stack
ResourceStatus: CREATE_IN_PROGRESS
ResourceStatusReason: User Initiated
--

Again you could simply use the provided Amazon tools, as they are probaly more feature-complete, but sometimes you want to be able to query for specific bits of data. Learning how to use fog will save you tons of time compared to parsing data that comes out of the Amazon tools. Fog really lets you do some powerful things with the Amazon APIs and is always gaining more features.

Current shortcommings and summary

This setup has a few issues that I currently don't know how to solve. Let's say you were to launch a stack that included two EC2 instances, a load balancer, and an RDS instance. During the creation phase, you want to import a basic DB schema into your RDS instance. Since you can't get shell on the RDS instance, you will have to use one of your EC2 instances to import the schema. You have a race condition here, how do you make sure your RDS instance is available before your EC2 instance attemps to import data? Right now the only thing I can think of is to have your chef recipe gracefully handle retries. It seems we need a way for resources of a stack to depend on other resources before being launched, or a notification system that could potentially kick off the chef run (SNS?).

Either way, I think CloudFormation, chef and fog can be used in conjuction with cloud-init to create quick deployments that are OS independent. This allows you to maintain your automation portions separately from your images, which as a consequence, promotes the use of vanilla images. With these basic ideas, you can easily swap out the node.json file for your own, and replace the user-script in such a way that it checks out your own cookbooks. The possibilities are really endless. Currently cloud-init supports Puppet out of the box, and I'd like to see some chef support here as well.

Improving varnish hit rates for Drupal 7

Allan Feid — Mon, 21 Feb 2011 19:25:55 +0000

I've recently moved my site to a new VPS and decided to upgrade to Drupal 7 at the same time. It was an interesting move, since a lot of the modules I was using have not yet been ported to D7. One of the main reasons I moved over was for native external caching support, meaning I no longer needed to depend on Pressflow. I ran into a few issues actually getting this to work properly with Varnish, so I wanted to cover how I got this working.

Changes to Drupal's settings.php

Just like in Pressflow, you need to make sure the site is aware it is being accessed from behind a reverse proxy:

$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

The next option needed is specific to Drupal 7, and there's a bug report to clarify the relation between external caching and page_cache_invoke_hooks. This was a confusing and frustrating part of getting Drupal to set the Cache-Control headers properly.

$conf['page_cache_invoke_hooks'] = FALSE;

When you set page_cache_invoke_hooks to false, the cache settings in the UI will be taken into account when sending out the appropriate cache related headers. Without this, you will see Cache-Control: max-age=0, which basically tells Varnish not to cache anything. The bug report explains more in depth as to why this happens.

A basic VCL for Drupal

Cookies are usually the biggest issue when it comes to properly caching a website. To find out what cookies were being sent to my backend from Varnish, I started with no custom configuration and picked a URL to test against.

varnishlog -b -o TxURL /content/scaling-your-application-cloud

Visit the URL path, and look for a line similar to the following:

11 TxHeader     b Cookie: __utmz=148462695.1292604478.3.2.utmcsr=linkedin.com|utmccn=(referral)
|utmcmd=referral|utmcct=/profile/edit; Drupal.toolbar.collapsed=0; has_js=1; 
__utma=148462695.668390275.1288311613.1298230598.1298310610.28; __utmc=148462695; __utmb=148462695.11

These cookies will cause bad cache hit rates because of the Vary: Cookie header, which tells a compliant reverse proxy, to vary the cache based off of the Cookie header. You can see how this will become an issue, because every browser and client is going to send a different Cookie header. To remove them, you need a line similar to this in the vcl_recv portion of your configuration:

set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js|Drupal.toolbar.collapsed)=[^;]*", "");

Most of this comes from the default Pressflow VCL, but a new addition in Drupal 7 is the Drupal.toolbar.collapsed cookie. This cookie seems to be set in order to control toolbar visibility, per the module documentation.

My full VCL is as follows:

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

sub vcl_recv {
  remove req.http.X-Forwarded-For;
  set req.http.X-Forwarded-For = client.ip;

  // from https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow?focusedCommentId=13828160
  // Remove has_js, Google Analytics __* and Drupal.toolbar.collapsed cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js|Drupal.toolbar.collapsed)=[^;]*", "");
  // Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
  // Remove empty cookies.
  if (req.http.Cookie ~ "^\s*$") {
    unset req.http.Cookie;
  }

  // fix compression per http://www.varnish-cache.org/trac/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
        # No point in compressing these
        remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
        set req.http.Accept-Encoding = "deflate";
    } else {
        # unkown algorithm
        remove req.http.Accept-Encoding;
    }
  }

}

sub vcl_hash {
  if (req.http.Cookie) {
    set req.hash += req.http.Cookie;
  }
}

Once I dropped this VCL in, I started seeing much higher hit rates, which can be viewed on my munin stats page. Prior to implementing the VCL, I was seeing low rates (< 10%) but didn't have the time to troubleshoot until this weekend. I learned a bunch about using varnishtop and varnishlog to find signs as to why my requests weren't being properly cached. I highly recommend checking out the documentation on achieving higher hit rates.

Scaling your application in the cloud

Allan Feid — Wed, 16 Feb 2011 02:55:26 +0000

Recently, I've been standing up a lot of infrastructure in Amazon's EC2. While auto-scaling the infrastructure aspects is fairly simple using configuration automation utilities such as Chef or Puppet, getting developers to understand these concepts may be an interesting task. I will try to outline most of the common issues I see when uplifting applications to an elastic cloud environment.

Rules to live by

Infrastructure automation is critical

You need to know how to build the infrastructure that runs your application, and you need to know how to do this repeatedly. Using tools such as Chef or Puppet will allow you to do this efficiently and in a self-documenting way that also provides revision control to your software configurations. Being able to do this allows you to quickly bring up additional server resources as your application demands it.

Revision control is not optional

Choose your favorite source control system and learn how to use every aspect of it. These tools were meant to keep track of your code changes. When dealing with the cloud, you're going to need to keep track of what is in you production environment vs your staging environment.

One of the biggest changes with a cloud based infrastructure, is that the underlying resources will be changing. There will be monitors that detect high traffic, CPU usage, memory usage, or anything really. This will cause additional virtual machines (instances) to launch, configure themselves (see above), check out the latest version of your code from your repository, and finally add itself to a load balancer. When your application's server infrastructure can build itself, you better be able to checkout the latest revision from your deployment branch and have it run with no human interaction.

Don't rely on traditional shared storage

I've seen many legacy applications rely on shared storage of some sort. This will not work in most cloud environments, and definitely will not work in Amazon. As of this writing, Amazon's Elastic Block Storage (EBS) volumes can only be attached to one server instance. This means you cannot utilize a clustered filesystem and standing up your own NFS server leaves for a single point of failure.

The solution? Put all of your site content that isn't revision controlled into an asset host or content delivery network (CDN). When on Amazon, this means using Amazon's Simple Storage Service (S3). By doing this you are storing your precious images, movies, and other content on storage that is 99.999999999% available and accessible anywhere over HTTP. As an added bonus, you will be serving up images faster and can achieve better edge network performance by enabling CloudFront, Amazon's CDN service.

Relational databases aren't evil

There's a lot of hype over NoSQL databases such as MongoDB and CouchDB. They are touted as being "webscale" because you can add and remove instances from the database cluster as needed. These are great if you're starting a new application and have decided you need these features, but what if you have existing data in MySQL?

Amazon's Relational Database Service (RDS) supports MySQL databases that are both redundant and scalable to your needs. You can easily grow the database storage size, cpu, and memory resources. They also perform automatic backups in a timeslot that you configure. This makes transitioning from traditional database infrastructure to the cloud fairly painless. They have also announced plans to support Oracle 11g in the future on the same platform. So while scaling relational databases are not as trivial as using MongoDB, Amazon has really stepped up to the plate by taking care of this problem for you.

Summary

By utilizing asset hosts, maintaining an automated server build, and keeping strict revision control you will be setting yourself up for success in today's cloud environment, however, I cannot stress the importance of at least having a staging environment. Have unit test and force all commits to be tested, then deployed to your staging environment. This allows you to quickly catch any code issues. An added benefit to your staging environment, is that you can test infrastructure changes before deciding to put the changes in the wild.

Setting up your own git repository

Allan Feid — Thu, 03 Feb 2011 02:27:31 +0000

Setup a bare repository

The first thing you'll need to do is create a new bare repository. A bare repository is one that does not contain a working directory, essentially it just contains everything in your standard .git directory. I'll be serving all my repositories from /srv/git, and assuming all the users you want to give access to this repository are in the git group.

# mkdir /srv/git
# git init --bare --shared=group /srv/git/my-awesome-application.git
# chgrp -R git /srv/git/my-awesome-application.git

If you're not going to be sharing this with multiple users, you won't need the --shared=group. Later down the road, if you forgot to specify this option, all it really does is the following:

# chmod -R g+rws /srv/git/my-awesome-application.git

Then, you either need to change the default umask of each user to 002, or you can make /srv/git/my-awesome-application.git/config look like this:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        sharedrepository = 1

The key is the sharedrepository = 1 directive. This can be changed to a variety of options, but when set to true or group it ensures all file pushes will use a sane umask of 002. You can find more information about this setting in the git-config(1) man page.

Configuring user access

I personally never setup a git repository without using ssh keys. So have your committers send your their public keys and create users for them. Add these keys to their authorized_keys file, and have them confirm they can ssh in without any issues. Once this is done, you can have them add a new remote repository to git as follows:

# git remote add origin user@crazed.pl:/srv/git/my-awesome-application.git

That's it, they're ready to start pushing code in the normal ways:

# git push origin master

Allowing read only access

Sometimes it's necessary to allow read only access to your repositories, such as for use with Capistrano or performing continuous integration. In this case, using git-daemon is easy and provides faster cloning/pulls than using HTTP or SSH. On Ubuntu, you can install the following package, which sets you up with a runit configuration:

# apt-get install -y git-daemon-run

There's one thing you'll likely want to change in this daemon's arguments, and that is the --base-path option. I use the following in /etc/sv/git-daemon/run:

#!/bin/sh
exec 2>&1
echo 'git-daemon starting.'
exec chpst -ugitdaemon \
  /usr/lib/git-core/git-daemon --verbose --base-path=/srv/git /srv/git

If you took a look at the git-daemon(1) man page, you will see that the base path essentially changes the root for git requests. So, in reality your application lives at /srv/git/my-awesome-application.git, but when pulling over the git protocol, it's as easy as this:

# git clone git://crazed.pl/my-awesome-application.git

However, before you can even do that, there is one magic step that makes this all work.

# touch /srv/git/my-awesome-application.git/git-daemon-export-ok

Touching this file tells the git-daemon that you meant for this repository to be publicly accessible (assuming you have the proper ports open, tcp/9418).

Providing a web front end

So your boss wants to see what you've been doing, but has no idea what git is. Why not give him a simple web interface? Luckily, Ubuntu again has a package for this:

# apt-get install gitweb

Gitweb is a simple web interface written in Perl and used as a CGI script. There may be better interfaces out there, but if it's good enough for kernel.org, it's good enough for me. To get this working, there is one simple change to the configuration file /etc/gitweb.conf:

# path to git projects (<project>.git)
$projectroot = "/srv/git";

The Ubuntu package comes with a simple Apache configuration to get this working at fqdn.com/gitweb:

Alias /gitweb /usr/share/gitweb

<Directory /usr/share/gitweb>
  Options FollowSymLinks +ExecCGI
  AddHandler cgi-script .cgi
</Directory>
root@ip-10-117-65-141:~# vi /etc/gitweb.conf
root@ip-10-117-65-141:~# cat /etc/apache2/conf.d/gitweb 
Alias /gitweb /usr/share/gitweb

<Directory /usr/share/gitweb>
  Options FollowSymLinks +ExecCGI
  AddHandler cgi-script .cgi
</Directory>

That's it, you now have a private repository that can be shared amongst multiple users, as well as a web interface and read-only access to those that need it.

Continuous integration with git, Capistrano, and Integrity

Allan Feid — Thu, 03 Feb 2011 01:36:45 +0000

What is continuous integration (CI)?

When working in an environment with multiple developers, the practice of continous integration can be tremendous for catching integration errors quickly. This means, as soon as code is checked in to your central repository, integration will be verified by an automated build / test suite. This requires that you have a single source repository and are writing proper tests for your application.

Automated builds / tests with Integrity

Integrity will build your code, then run tests to make sure everything works as expected. Installation instructions can be found on their homepage, but once you do have it setup, it is quite easy to add extremely useful functionality. What I have used, is the email on completion functionality. To get this up and running, you simply need the following line in your init.rb file:

require "integrity/notifier/email"

Once you have added a new project, the repository URI, branch, and name should be straight forward to fill out. Your build script is what will be evaluated for a return value of 0, indicating everything went well and good. You can use this section to do some really cool things, what I've done is take a rails app and run some tasks using rake. Then using &&, on success of the tests, it will launch a deploy using capistrano.

rake RAILS_ENV=test gems:build db:migrate \
&& spec ./spec --format specdoc && cap -f /srv/app/capfile deploy

I heavily advise against using this straight to your production web server, in this example I'm having cap deploy to my staging servers. This is a great way to go from successful tests to deployment on your staging or QA servers. Now, to complete the full automated build to deploy, you need a way for your code check-ins to perform an action.

Using git hooks to initiate a build

Inside every git repository, there is a subdirectory called hooks. This directory contains a series of specially named scripts that will get executed at various points whne interacting with git. For a full explanation of the scripts, check out the githooks(5) man page.

#!/bin/bash
echo "Received commit to branch: $1"
if [[ "$1" =~ ^(.*)/master$ ]]; then
  echo "Starting Integrity build..."
  curl -F 'build=build' http://ci.example.com/project/builds 2>/dev/null
else
  echo "Current branch is not master, Integrity will not be started."
fi

Sending a simple POST request to your integrity server at the /:project/builds URI will cause integrity to do its thing. This script checks the branch that was commited to, and only kicks off the POST request if it was master.

There you have it, a full commit, test, and deployment all automated. This forces your developers to use version control and write good tests, because as soon as you commit something, it will be tested and if it passes it's going straight to your servers. As a sysadmin, I've learned a lot about rails applications by setting something like this up.

Cooking with Chef

Allan Feid — Sun, 09 Jan 2011 19:01:56 +0000

I wanted to take some time to explain one of my favorite configuration management tools. That is Chef. I've been utilizing it pretty heavily over the passed few months and thought I'd share some of what I learned. I'm by no means an expert, but I've been managing my own chef-server as well as writing plenty of my own recipes.

Part 1: Cookbooks and Recipes

It is important to understand what Recipes and Cookbooks really are. A cookbook is a collection of Recipes, Attributes, Templates, Files, and possibly even Resources and Providers. On the file system, it looks like this:

dpadmin@chef-east:~/cookbooks$ ls haproxy/
attributes  metadata.json  metadata.rb  README.rdoc  recipes  templates

Each piece of your cookbook will be under the sub-directory related to what that piece represents. So for example, your template files will go under templates, recipes under recipes, etc. The main pieces of your cookbooks will be the Recipes. This is where you define what will happen when applied to a node. Let's start with something basic, like installing and starting HAproxy:

package "haproxy" do
  action :install
end

service "haproxy" do
  supports :restart => true, :status => true, :reload => true
  action [:enable, :start]
end

More information can be found on Opscode's Wiki for Resources, which you should get familiar with because it really helps you configure individual resources and understand what is going on. In the above example, you have two Resources: a Package and a Service resource. The package resource does nothing more than install a particular package. The service resource defines a service that we want to control in our recipes. In this case it is haproxy (a load balancer). We tell chef that haproxy supports the restart, status, and reload actions (think init scripts). Then we tell the service to enable itself (chkconfig, update-rc.d, etc), then start the service.

Now that we have the required package installed and service configured, we want to start using templates for the configuration files. Here's a template resource for haproxy.cfg:

template "/etc/haproxy/haproxy.cfg" do
  source "haproxy.cfg.erb"
  owner "root"
  group "root"
  mode "644"
  variables(:servers => server_list)
  notifies :reload, resources(:service => "haproxy")
end

In this resource definition, you can see we specify the full path to the file we will be templating (/etc/haproxy/haproxy.cfg) and the source of the template (haproxy.cfg.erb). Here's a great explanation of template location specificity, but for now we will focus more on the Recipe aspect rather than Templates. The owner, group, and mode directives here specify the permissions the template will have. The variables are exactly what they are called, variables. These variables are passed to the Erubis template engine to be parsed. In this case we are passing a list of servers called :servers and defining it as the contents of server_list (an array, for those familiar with programming concepts). Finally, since the template is the configuration for a service, we need to tell our haproxy service to reload its configuration. The notifies directive can be used in any Resource definition and allows you to tell another resource to perform some sort of action. Here we tell haproxy to reload (/etc/init.d/haproxy reload).

Part 2: Templates

Templates are part of Cookbooks, but really need their own section because they can be confusing if you don't know how they are parsed. Take a look at this template for haproxy.cfg:

dpadmin@chef-east:~/cookbooks$ cat haproxy/templates/default/haproxy.cfg.erb 
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        maxconn 4096
        #debug
        #quiet
        user haproxy
        group haproxy

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option  forwardfor
        retries 3
        option  redispatch
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

# Set up application listeners here.
listen application *:80
        mode http
        balance roundrobin
        stats enable
        stats auth myuser:mypass
        <% @servers.each do |name,ip| -%>
        server <%= name %> <%= ip %> check
        <% end -%>

For the most part this configuration is pretty static. The only rendered piece is at the bottom:

        <% @servers.each do |name,ip| -%>
        server <%= name %> <%= ip %> check
        <% end -%>

As you saw in the template definition in the previous section, we pass the variable :servers, which in the template file is called an instance variable. In Ruby, instance variables are prefixed by an @. The rest is Ruby syntax, but I will do my best to explain what's going on here. A hash (key->value store), has various methods (functions). In this case we are calling the each method, which will loop through each key->value pair in the hash. The |name,ip| part gets assigned the key and hash values, then we output these values by calling them directly inside the loop. This allows us to generate a final product that looks like this:

        server web2.example.com 10.204.39.249 check
        server web1.example.com 10.203.83.213 check

As you can see, our @servers hash had two elements. The values of each element were as follows:

{
  "web2.example.com" => "10.204.39.249",
  "web1.example.com" => "10.203.83.213"
}

So to wrap it up, the Templates get passed variables from Recipes. These variables can be anything including hashes, arrays, strings, and integers. Then the ERB template engine is in charge of parsing the data to come up with our final result.

You may be wondering where we got server_list to begin with. I left it out of the previous examples as it's necessary to understand the guts of Chef before moving towards where data is stored. The real power of Chef is being able to separate all of your data from the actual working parts. This allows easy re-use of the same recipes across any number of different data stores. In this case, we would be talking about each config environment having its own data to be used with any pre-written recipes.

Part 3: Understanding where data is stored

Chef stores a wealth of information that is indexed and search-able from within recipes. When you are writing your recipes, you don't want to keep variables that are statically assigned. This makes them less re-usable and way too specific. So, continuing with the HAproxy example, I will show you where server_list (the hash of servers to generate our config) came from. In a Data Bag Item, we have the following JSON data:

...
  "haproxy": {
    "eip": "204.236.219.111",
    "production_role": "prod-web"
  },
  "common_names": {
    "lb1": "lb1.example.com",
    "web1": "web1.example.com",
    "web2": "web2.example.com"
  },
...

Then, within our recipe file we have the following logic to generate the server_list hash:

# get our config data bag item
config_info = data_bag_item("configs", node.default[:cid])

# pull out the common_names hash for later use
common_names = config_info['common_names']

# assign a new hash that we will store hostnames to ip addresses
server_list = Hash.new

# basic logging
Chef::Log.debug("Searching for role #{config_info['haproxy']['production_role']}")

# search for nodes that have the "config_info['haproxy']['production_role']" role applied to them,
# loop through each one and assign any matches to the server variable
search(:node, "role:#{config_info['haproxy']['production_role']}").each do |server|
  Chef::Log.debug("Found a server: #{server} IP: #{server.cloud.private_ips.first}")

  # add to our hash. use the common name for the server as key and the private ip as the value
  server_list[common_names[server.name]] = server.cloud.private_ips.first
  Chef::Log.info("Added #{common_names[server.name]} => #{server_list[common_names[server.name]]}")
end

In this case we retrieve some data from a Data Bag Item. The Data Bag being "configs" and the item being node.default[:cid]. Here we access a custom attribute called :cid. So when running this recipe against a group of nodes with the config ID of "example," the Data Bag Item is configs[example]. Now, inside that data bag item, we have a bunch of key->value pairs (think hashes). We access the information stored under 'common_names', 'haproxy', and whatever else we want to store. All of this is just being pulled from an arbitrary JSON data store that Chef refers to as a Data Bag.

When it comes down to it, what we are doing is searching for any nodes that have the role "prod-web" applied to them. I created this role for any production web servers that example.com will be load balancing between. So when our HAproxy recipe is applied to a different node, it generates the configuration file based off of which servers have that role applied to them. Data Bags are not the only place information can be stored. There are three main places we can store data:

Data Bag Items
Role Attributes
Node Attributes

In the HAproxy example, we make use of all three of the above. The "common_names" for each host in a solution and which role to search for are stored in a Data Bag Item. The config identifier (cid) is being stored in a Role Attribute, and our IP address information comes from Node Attributes. All of this data is easily viewable using the knife command line tool.

Data Bags

For Chef, there's two parts to this. There's the Data Bag, then the Data Bag Item. A data bag can store multiple items. Each Item is nothing more than an arbitrary JSON data store. This is a great place to store information that will be used in various recipes. I've been using it as a central configuration for various services. What this means is, I have the configs data bag, and each Item is called by the configs ID. Inside each data store, I keep configuration variables and general data for any recipes that will be applied to nodes in that config's environment. To give you a brief idea, here's what example.com has:

{
  "ldap": {
    "bindpw": "xxxxxxxxxxxxxxxxxxx",
    "binddn": "cn=proxy,ou=services,dc=example,dc=cid",
  },
  "common_names": {
    "lb1": "lb1.example.com",
    "web1": "web1.example.com",
    "web2": "web2.example.com"
  },
  "hosts": {
    "us-east-1": [
      "10.204.39.249 web2.example.com web2",
      "10.203.83.213 web1.example.com web1",
      "10.203.29.220 lb1.example.com lb1"
    ]
  },
  "id": "example",
  "aws": {
    "aws_access_key_id": "XXXXXXXXXXXXXXXXXXXX",
    "aws_secret_access_key_id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  },
  "apache": {
    "vhosts": {
      "www.example.com": {
        "DocumentRoot": "/ebs/sites/example.com"
      },
      ...
    }
  },
  "haproxy": {
    "eip": "204.236.219.111",
    "production_role": "prod-web"
  }
}

You sort of have to think of this data store as a central place to manipulate the entire environment. Rather than thinking about individual nodes, think of configuring everything in one place. Accessing data from a data bag in a recipe, requires that you use:

data = data_bag_item('configs', 'example')
puts data['haproxy']['production_role'] # prints prod-web to stdout

From there you will have a nice hash with all of the JSON data in that data bag item.

Role Attributes

Again stored in JSON, Role Attributes come from your Role definitions. When you define a role, it will look like this:

{
    "name": "base",
    "default_attributes": {
      "cid": "example"
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ldap::auth]",
      "recipe[ldap::sudoers]",
      "recipe[sshd]",
      "recipe[zabbix]",
      "recipe[clients::hosts]",
      "recipe[clients::hostname]",
      "recipe[postfix]"
    ],
    "description": "",
    "chef_type": "role",
    "override_attributes": {
    }
}

Don't let the JSON make you think creating new roles is difficult. Creating a role with a properly configured knife client is pretty straight forward:

# knife role create <some_role>

This will give you a basic role with no run_list and no default_attributes. All I do for this role (base), is define what recipes to run, and set the config ID that we use in various recipes. Accessing the data from these attributes is done the same way as the Node Attributes. Really we can just call them Attributes, but because they are technically stored in different places I wanted to cover them separately.

Node Attributes

If you're thinking that this is stored using JSON, you are correct. Each node, upon being bootstrapped, gets populated with a large data store. This contains everything from the contents of /etc/passwd, to the amount of bytes transmitted on eth0. Each time a node checks in with the server, this data gets updated. You can modify any data for a node directly using knife, and to give you an example of the extensive information stored by Chef, here's what the ec2 section looks like:

      "ec2": {
        "public_hostname": "ec2-174-129-1.compute-1.amazonaws.com",
        "placement_availability_zone": "us-east-1a",
        "block_device_mapping_root": "/dev/sda1",
        "instance_id": "i-d1535fbb",
        "instance_type": "m1.large",
        "local_ipv4": "10.204.39.249",
        "block_device_mapping_ephemeral0": "/dev/sdb",
        "reservation_id": "r-940419ff",
        "public_ipv4": "174.129.1.1",
        "local_hostname": "ip-10-204-39-249.ec2.internal",
        "kernel_id": "aki-2407f24d",
        "hostname": "ip-10-204-39-249.ec2.internal",
        "ami_id": "ami-6006f309",
        "userdata": null,
        "security_groups": [
          "web"
        ],
        "block_device_mapping_ami": "/dev/sda1",
        "ami_manifest_path": "(unknown)",
        "ami_launch_index": "0"
      },

Accessing any of these attributes from recipes can be done in the following manner:

# the node's name, ex: web1
node.name

# the AMI a node is running, ex: ami-6006f309
node.ec2.ami_id

# node's uptime, ex: "16 days 16 hours 35 minutes 23 seconds"
node.uptime

# distro, ex: "ubuntu"
node.platform

# version of php on the instance, ex: "5.3.2-1ubuntu4.5"
node.languages.php.version

# gcc version
node.languages.c.gcc.version

# number of packets sent/received 
node.network.interfaces.eth0.tx.packets
node.network.interfaces.eth0.rx.packets

As long as you have a node object, you can access any of this data via the object's methods. When you are in a recipe, you can use the variable node as it is defined automatically by Chef.

Capistrano for the sysadmin

Allan Feid — Sun, 09 Jan 2011 18:43:34 +0000

There's always been a fine line between a good sysadmin and a programmer. Capistrano is software built for Ruby on Rails to make deploys across multiple servers easy, however, the potential is much greater than just code deployment. I have recently grown in love with how easily I can run commands across groups of servers, and setting up Capistrano is not a difficult task. If you've ever configured password-less SSH access, you can get this up and running without much difficulty.

Have a centralized ssh-key

# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.

You have an SSH key, now you just need to install it on each server you would like to manage. One of the things I like to do, is setup an rsync share to pull down important config files. This can also be easily accomplished using an NFS mount. Either way after you have password-less SSH access to all the servers from one location, this will be your controlling server. You won't need to login to the other boxes individually from this point forward.

Security note: in this example, I'm using root logins. I recommend setting PermitRootLogin to without-password if you want to use root access, however, I really do not recommend using root directly, as it becomes a liability issue when you have more than one admin. The use of a normal user account plus sudo is highly recommended.

Install the software

Depending on what OS you're on, these steps may be different, but the end goal is the same: get RubyGems installed. On CentOS/RHEL the following works:

# yum install -y ruby ruby-devel ruby-rdocs ruby-ri

Since the version of ruby included on CentOS/RHEL is a bit dated, we have to use an older version of RubyGems (1.3.5).

# wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
# tar xzf rubygems-1.3.5.tgz
# cd rubygems-1.3.5/
# ruby setup.rb
# gem install capistrano

Configuring Capistrano

At this point, a Rails developer would cd into their code directory, run capify and configure Capistrano's deploy scripts to their needs. Going over code deployments is an entirely different process, which I'll save for another time. As a sysadmin, we can skip running capify and just make our own cap file. Think of a cap file as a basic config file, except written in Ruby. You get the power of a scripting language built right into your config file. This truly gives you endless options and the ability to tie into anything you'd like (think database, or ldap). Here I'll show you a basic "cap file" that will do what sysadmins want: make managing multiple servers easy.

# cd /etc/manage
# ls -l 
-rw-r--r-- 1 root root 2739 Jul 20 18:14 capfile
drwxr-xr-x 2 root root 4096 Jul 19 12:15 roles/

# cat capfile
def getrole(role)
        f = open('/etc/manage/roles/' + role)
        f.readlines
end

role(:web) { getrole('web') }
role(:db) { getrole('db') }

desc "Show Memory"
task :show_mem do
  run "grep -i memtotal /proc/meminfo"
end

What I like to do is create a directory (/etc/manage/), then populate this with central management type configurations. In this case, it would be my cap file, and a directory called roles. The roles directory just contains files with lists of servers, for example:

# cat /etc/manage/roles/web
web1
web2
web3

So this would be the resolvable names of all my web servers. The cap file uses a basic function called getrole() to read the file declared by role inside the roles directo One would be to cd into /etc/manage and the other would be to declare the cap file from the command line with the -f parameter. The first thing you can look at, is the output of -T.

# cap -T
cap invoke     # Invoke a single command on the remote servers.
cap shell      # Begin an interactive Capistrano session.
cap show_mem   # Show Memory

This will show you all the different tasks you have available. There are two built in ones (invoke, shell) and one we defined in our cap file (show_mem). This is a very basic example, but will suite a wide range of needs. The most useful, in my opinion (at least at this stage), is the shell. This is where you have the power to run commands across "roles" of servers.

# cap shell
cap> df -h
 ** [out :: web1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web1] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web1] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web2] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web2] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web2] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web3] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web3] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web3] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: db1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: db1] /dev/cciss/c0d0p1     128G  2.6G  119G   3% /
 ** [out :: db1] tmpfs                 2.0G     0  2.0G   0% /dev/shm
cap> with web
cap> grep admin /etc/passwd
 ** [out :: web1] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web2] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web3] admin:x:500:500:Admin user:/home/admin:/bin/bash
cap> with db
cap> !show_mem 
 ** [out :: db1] MemTotal:      3985972 kB

When you first enter the shell, it will be set to execute tasks or commands across ALL roles. Using the with directive, you can specify which role you want to use for the subsequent commands. Finally, adding a bang (!) in front, tells Capistrano to execute a task rather than standard shell commands. Hopefully I've given you enough information to apply Capistrano to your own environments. Think about commands you often execute, create a new task and you will find yourself saving a lot of time. Even troubleshooting and adding new functionality across servers becomes much quicker when you can access everything from one location.

Central login management with OpenLDAP

Allan Feid — Sun, 28 Mar 2010 17:18:58 +0000

Over the weekend I spent some time learning how to configure OpenLDAP to provide a central login server at home. This is pretty useful since I was tired of either syncing UID/GID's across servers or setting directories to 777 on my NFSv4 server. I chose Fedora 12 as my OS for the server, and will show how easy it is to configure authentication on both Fedora and OpenSolaris.

Some information about the network

I'll give you a quick overview of how the network looks from home.

DNS: ns1.crazy.lan (172.16.1.3)
LDAP: ldap1.crazy.lan (10.10.1.1)
Client 1: test.crazy.lan (10.10.2.1)
Client 2: virt.crazy.lan (192.168.1.8)

I'm not going to go over the importance of DNS (both forward and reverse lookups should work!) right now, but you will save yourself plenty of headaches by making sure DNS is setup properly on your network.

Basic configurations

Install the needed software!

For Fedora 12, installing OpenLDAP is easy:

# yum install -y openldap*

This will get you everything you need to get started.

There are a few things you need to change in order to really start using OpenLDAP. Everything is all in one file slapd.conf. Prior to editing the file, you'll want to generate a password to use in this configuration.

# slappasswd
New password: 
Re-enter new password: 
{SSHA}fFVfwsztHn+xuQPjN/q/urdNC+V0G+dW

Copy the new hash as you will need to paste it into the configuration file. Add or modify the following in /etc/openldap/ldap.conf:

suffix      "dc=crazy,dc=lan"
rootdn      "cn=Manager,dc=crazy,dc=lan"
rootpw      {SSHA}fFVfwsztHn+xuQPjN/q/urdNC+V0G+dW

# ACL's to use
loglevel acl stats
access to dn.base="" by * read
access to dn.base="cn=Subschema" by * read
access to *
        by self write
        by * read
        by anonymous auth

That's all you need there for a basic configuration. This will not be using TLS/SSL, I will cover that in a later post. It's always good to start with the basics so that you can spend more time learning the protocol. Once you've gotten that down hardening a service is easy.

Now we just need to finish up and start the service. Since slapd.conf is deprecated in favor of the slapd.d/ directory, we can use the configuration file to generate the appropriate directory structure. The huge advantage of this style is dynamic backend configuration changes. Yes, that means persistent dynamic changes. No more restarting your directory server to make changes to things such as your logging level.

# cd /etc/openldap
# rm -rf slapd.d
# mkdir slapd.d
# slaptest -f slapd.conf -F slapd.d
# chown -R ldap:ldap slapd.d
# chkconfig slapd on
# service slapd start

You can test for functionality by running the following:

ldapsearch -x -h localhost

If all went well you will see some output and no errors.

Configure rsyslog for logging

If you're not using Fedora, or rsyslog, you'll want to lookup the specifics for your logging daemon's configuration. OpenLDAP sends it's logging output to the local4 log facility. For rsyslog, just add the following line to /etc/rsyslog.conf:

local4.*                                                -/var/log/ldap

Then restart your logging service.

service rsyslog restart

Importing some data to LDAP

The LDIF formatted file is where things start to get a little bit tricky. Don't worry about it so much as there are plenty of good GUI clients for LDAP, but we do need to get a few things imported.

# crazy.lan
dn: dc=crazy,dc=lan
objectClass: dcObject  
objectClass: organization
o: Crazy Organization  
dc: crazy

# Manager, crazy.lan   
dn: cn=Manager,dc=crazy,dc=lan
objectClass: organizationalRole
cn: manager

# People, crazy.lan
dn: ou=People,dc=crazy,dc=lan
objectClass: organizationalUnit
ou: People

# Group, crazy.lan
dn: ou=Group,dc=crazy,dc=lan
objectClass: organizationalUnit
ou: Group

# users, Group, crazy.lan
dn: cn=users,ou=Group,dc=crazy,dc=lan
objectClass: posixGroup
objectClass: top
cn: users
gidNumber: 2000

# myaccount, People, crazy.lan
dn: uid=myaccount,ou=People,dc=crazy,dc=lan
uid: myaccount
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
gidNumber: 2000
uidNumber: 3000
homeDirectory: /home/myaccount
loginShell: /bin/bash
userPassword: {SSHA}cc696ZrGCPmgx0/CD2zGJrMqR3tb8iMk
cn: Allan Feid

I saved the above as import.ldif. You will probably want to generate a new password with slappaswd to use with the userPassword section above. Now to import, it all I have to do is this:

ldapadd -f import.ldif -D "cn=Manager,dc=crazy,dc=lan" -W

This will prompt you for your rootdn password that you created earlier using slappasswd. Now you have the following basic hierarchy configured:

Base Domain: crazy.lan
- Organizational Unit: People
  - User: myaccount
- Organizational Unit: Group
  - Group: users

User accounts will go into the People OU, and group accounts will be assigned the Group OU. This is a pretty standard configuration as the pam_ldap plugin will look in these OU's later for user/group information.

Client configuration

Fedora 12

Redhat has really made this easier than it used to be. Their authconfig command is pretty straight forward, the following commands will get you the nss_ldap, pam_ldap and ldap utility commands on your client:

# yum install -y openldap-clients
# authconfig --enableldap \
    --enableldapauth \
    --disablenis \
    --enablecache \
    --ldapserver=ldap1.crazy.lan \
    --ldapbasedn=dc=crazy,dc=lan \
    --updateall

That's it. You will now see the myaccount user if you run getent.

# getent passwd |grep myaccount
myaccount:x:3000:2000:Allan Feid:/home/myaccount:/bin/bash

OpenSolaris

This should also work for plain Solaris, but I prefer OpenSolaris and haven't tested on Solaris 10. This, just like Redhat, is very easy to configure, but does require one extra step. The ldapclient command by default copies /etc/nsswitch.ldap to /etc/nsswitch.conf. Which is fine, but the ldap nsswich configuration by, by default, uses ldap for hostname lookups. This is probably not desirable in most cases as that's what we use DNS for. So just change the following lines in /etc/nsswitch.ldap:

hosts:      files dns
ipnodes:    files dns

Then just a simple command will get you up and running:

# ldapclient -v manual -a defaultSearchBase=dc=crazy,dc=lan \
    -a domainName=crazy.lan \
    -a defaultServerList=ldap1.crazy.lan
# getent passwd |grep myaccount
myaccount:x:3000:2000::/home/myaccount:/bin/bas

General Configuration for client's using pam_ldap/nss_ldap

If you aren't using Fedora or OpenSolaris, anything that uses pam_ldap/nss_ldap can be configured pretty easily. The main configuration file for these two is /etc/ldap.conf. All you need is to make sure you have the following lines:

base dc=crazy,dc=lan
uri ldap://ldap1.crazy.lan/

Then you will have to configure nss by changing the following in /etc/nsswitch.conf:

passwd:     files ldap
shadow:     files ldap
group:      files ldap

You will also need to modify your PAM configuration. Generally you will want to edit /etc/pam.d/system-auth but this may very from OS to OS. Always check with the documentation!

auth        sufficient    pam_ldap.so use_first_pass

Use ADS, make your life easier!

I highly recommend Apache Directory Studio as a great LDAP client. It is cross platform (java based) and having a GUI for ldap sometimes just makes more sense as you get to visualize your hierarchy easily. Setup a new connection and login as cn=Manager,dc=crazy,dc=lan using your rootdn password. You can now easily create a template user to be copied for new users, add new groups, add OU's, etc all from a comfortable GUI.

In upcoming posts, I plan on covering enabling TLS encryption, and eventually working on Kerberos integration with LDAP for a true Single SIgn-On (SSO) environment. If all goes well I'll share how I got it all working.

Allan Feid - sysadmin

Create an inventory tool using MCollective's registration feature

Setting up a registration plugin

What do you do with this data?

Improving performance with a queue

The end result

Tags:

Tracking stats in your application

Getting your graphite on

A simple node.js daemon: StatsD

Some basic stats in a Rails application

What stats to track?

Tags:

Using Amazon's CloudFormation, cloud-init, chef and fog to automate infrastructure

EC2 user data and cloud-init

CloudFormation templates

Manipulating stacks with fog

Current shortcommings and summary

Tags:

Improving varnish hit rates for Drupal 7

Changes to Drupal's settings.php

A basic VCL for Drupal

Tags:

Scaling your application in the cloud

Rules to live by

Infrastructure automation is critical

Revision control is not optional

Don't rely on traditional shared storage

Relational databases aren't evil

Summary

Tags:

Setting up your own git repository

Setup a bare repository

Configuring user access

Allowing read only access

Providing a web front end

Tags:

Continuous integration with git, Capistrano, and Integrity

What is continuous integration (CI)?

Automated builds / tests with Integrity

Using git hooks to initiate a build

Tags:

Cooking with Chef

Part 1: Cookbooks and Recipes

Part 2: Templates

Part 3: Understanding where data is stored

Data Bags

Role Attributes

Node Attributes

Further Reading

Tags:

Capistrano for the sysadmin

Have a centralized ssh-key

Install the software

Configuring Capistrano

Tags:

Central login management with OpenLDAP

Some information about the network

Basic configurations

Install the needed software!

Configure rsyslog for logging

Importing some data to LDAP

Client configuration

Fedora 12

OpenSolaris

General Configuration for client's using pam_ldap/nss_ldap

Use ADS, make your life easier!

Tags: