Continuous integration with git, Capistrano, and Integrity

Allan Feid — Thu, 03 Feb 2011 01:36:45 +0000

What is continuous integration (CI)?

When working in an environment with multiple developers, the practice of continous integration can be tremendous for catching integration errors quickly. This means, as soon as code is checked in to your central repository, integration will be verified by an automated build / test suite. This requires that you have a single source repository and are writing proper tests for your application.

Automated builds / tests with Integrity

Integrity will build your code, then run tests to make sure everything works as expected. Installation instructions can be found on their homepage, but once you do have it setup, it is quite easy to add extremely useful functionality. What I have used, is the email on completion functionality. To get this up and running, you simply need the following line in your init.rb file:

require "integrity/notifier/email"

Once you have added a new project, the repository URI, branch, and name should be straight forward to fill out. Your build script is what will be evaluated for a return value of 0, indicating everything went well and good. You can use this section to do some really cool things, what I've done is take a rails app and run some tasks using rake. Then using &&, on success of the tests, it will launch a deploy using capistrano.

rake RAILS_ENV=test gems:build db:migrate \
&& spec ./spec --format specdoc && cap -f /srv/app/capfile deploy

I heavily advise against using this straight to your production web server, in this example I'm having cap deploy to my staging servers. This is a great way to go from successful tests to deployment on your staging or QA servers. Now, to complete the full automated build to deploy, you need a way for your code check-ins to perform an action.

Using git hooks to initiate a build

Inside every git repository, there is a subdirectory called hooks. This directory contains a series of specially named scripts that will get executed at various points whne interacting with git. For a full explanation of the scripts, check out the githooks(5) man page.

#!/bin/bash
echo "Received commit to branch: $1"
if [[ "$1" =~ ^(.*)/master$ ]]; then
  echo "Starting Integrity build..."
  curl -F 'build=build' http://ci.example.com/project/builds 2>/dev/null
else
  echo "Current branch is not master, Integrity will not be started."
fi

Sending a simple POST request to your integrity server at the /:project/builds URI will cause integrity to do its thing. This script checks the branch that was commited to, and only kicks off the POST request if it was master.

There you have it, a full commit, test, and deployment all automated. This forces your developers to use version control and write good tests, because as soon as you commit something, it will be tested and if it passes it's going straight to your servers. As a sysadmin, I've learned a lot about rails applications by setting something like this up.

Capistrano for the sysadmin

Allan Feid — Sun, 09 Jan 2011 18:43:34 +0000

There's always been a fine line between a good sysadmin and a programmer. Capistrano is software built for Ruby on Rails to make deploys across multiple servers easy, however, the potential is much greater than just code deployment. I have recently grown in love with how easily I can run commands across groups of servers, and setting up Capistrano is not a difficult task. If you've ever configured password-less SSH access, you can get this up and running without much difficulty.

Have a centralized ssh-key

# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.

You have an SSH key, now you just need to install it on each server you would like to manage. One of the things I like to do, is setup an rsync share to pull down important config files. This can also be easily accomplished using an NFS mount. Either way after you have password-less SSH access to all the servers from one location, this will be your controlling server. You won't need to login to the other boxes individually from this point forward.

Security note: in this example, I'm using root logins. I recommend setting PermitRootLogin to without-password if you want to use root access, however, I really do not recommend using root directly, as it becomes a liability issue when you have more than one admin. The use of a normal user account plus sudo is highly recommended.

Install the software

Depending on what OS you're on, these steps may be different, but the end goal is the same: get RubyGems installed. On CentOS/RHEL the following works:

# yum install -y ruby ruby-devel ruby-rdocs ruby-ri

Since the version of ruby included on CentOS/RHEL is a bit dated, we have to use an older version of RubyGems (1.3.5).

# wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
# tar xzf rubygems-1.3.5.tgz
# cd rubygems-1.3.5/
# ruby setup.rb
# gem install capistrano

Configuring Capistrano

At this point, a Rails developer would cd into their code directory, run capify and configure Capistrano's deploy scripts to their needs. Going over code deployments is an entirely different process, which I'll save for another time. As a sysadmin, we can skip running capify and just make our own cap file. Think of a cap file as a basic config file, except written in Ruby. You get the power of a scripting language built right into your config file. This truly gives you endless options and the ability to tie into anything you'd like (think database, or ldap). Here I'll show you a basic "cap file" that will do what sysadmins want: make managing multiple servers easy.

# cd /etc/manage
# ls -l 
-rw-r--r-- 1 root root 2739 Jul 20 18:14 capfile
drwxr-xr-x 2 root root 4096 Jul 19 12:15 roles/

# cat capfile
def getrole(role)
        f = open('/etc/manage/roles/' + role)
        f.readlines
end

role(:web) { getrole('web') }
role(:db) { getrole('db') }

desc "Show Memory"
task :show_mem do
  run "grep -i memtotal /proc/meminfo"
end

What I like to do is create a directory (/etc/manage/), then populate this with central management type configurations. In this case, it would be my cap file, and a directory called roles. The roles directory just contains files with lists of servers, for example:

# cat /etc/manage/roles/web
web1
web2
web3

So this would be the resolvable names of all my web servers. The cap file uses a basic function called getrole() to read the file declared by role inside the roles directo One would be to cd into /etc/manage and the other would be to declare the cap file from the command line with the -f parameter. The first thing you can look at, is the output of -T.

# cap -T
cap invoke     # Invoke a single command on the remote servers.
cap shell      # Begin an interactive Capistrano session.
cap show_mem   # Show Memory

This will show you all the different tasks you have available. There are two built in ones (invoke, shell) and one we defined in our cap file (show_mem). This is a very basic example, but will suite a wide range of needs. The most useful, in my opinion (at least at this stage), is the shell. This is where you have the power to run commands across "roles" of servers.

# cap shell
cap> df -h
 ** [out :: web1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web1] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web1] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web2] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web2] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web2] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web3] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web3] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web3] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: db1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: db1] /dev/cciss/c0d0p1     128G  2.6G  119G   3% /
 ** [out :: db1] tmpfs                 2.0G     0  2.0G   0% /dev/shm
cap> with web
cap> grep admin /etc/passwd
 ** [out :: web1] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web2] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web3] admin:x:500:500:Admin user:/home/admin:/bin/bash
cap> with db
cap> !show_mem 
 ** [out :: db1] MemTotal:      3985972 kB

When you first enter the shell, it will be set to execute tasks or commands across ALL roles. Using the with directive, you can specify which role you want to use for the subsequent commands. Finally, adding a bang (!) in front, tells Capistrano to execute a task rather than standard shell commands. Hopefully I've given you enough information to apply Capistrano to your own environments. Think about commands you often execute, create a new task and you will find yourself saving a lot of time. Even troubleshooting and adding new functionality across servers becomes much quicker when you can access everything from one location.

Allan Feid - Capistrano