Allan Feid - script

Cluster Management with FreePanel

Allan Feid — Fri, 05 Mar 2010 17:16:09 +0000

I haven't had much time to continue work on FreePanel, but the Perl modules I've written have proven to be useful in cluster management. Take a look at the following diagram as an example scenario:

There's two key points to take away from that diagram.

physical servers are load balanced using a single VIP (Virtual IP)
the servers share access to a single storage mount

The idea behind load balancing is that requests go to the individual servers based on which can handle the traffic. A hardware appliance takes care of the actual load balancing work, but this could be done using some software just as easily.

So what does this have to do with FreePanel?

If you're constantly needing to add additional domains to your cluster, doing this manually becomes tiresome, especially when in most cases your application needs the same Apache configuration per domain. Take the below basic Virtual Host config:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ServerAlias www.example.com
        ErrorLog /home/sites/logs/example.com.error
        CustomLog /home/sites/logs/example.com.access combined
        DocumentRoot /home/sites/example.com/web

        
                Options Indexes FollowSymLinks
                AllowOverride None
                Order allow,deny
                Allow from all

Each of your 50 or so hosted applications needs a similar configuration, just with a different domain name. That's where FreePanel can make your life easier. Take the above example and we'll form a template that can be used with FreePanel's Apache module.

>
        ServerAdmin webmaster@
        ServerName 
        ServerAlias www.
        ErrorLog /home/sites/logs/.error
        CustomLog /home/sites/logs/.access combined
        DocumentRoot /home/sites//web

        /web/>
                Options Indexes FollowSymLinks
                AllowOverride None
                Order allow,deny
                Allow from all

As you can expect, this module can parse the template file and replace certain items with what arguments are being passed. We'll take a look at the addSite method of FreePanel::Admin::Apache.

sub addSite {
        my ($self, $domain, $ip_addr) = @_;
        $self->logger("function: addSite($domain, $ip_addr) called.", $self->FUNC_CALL);

        my $vhost_dir = $self->getVhostDir();
        my $vhost_templ = $self->getVhostTemplate();
        my $check = $self->getValidateObj();

        if ($check->is_active($domain, $vhost_dir)) {
                $self->logger("vhost config already exists for $domain.", $self->ERROR);
                return 0;
        }

        open (VHOST, '>', $vhost_dir."/$domain");
        open (VHOST_T, ') {
                        s//$domain/;
                        s//$ip_addr/;
                        print VHOST;
                }
        }
        else { 
                while () {
                        s//$domain/;
                        s//\*/;
                        print VHOST;
                }
        }

        close VHOST_T;
        close VHOST;

        return 1;
}

Nothing really that crazy going on here, it just opens up the template file -- defined in your config file -- then replaces <DOMAIN> and <IP_ADDR> with the values you pass. This file is written out to a directory -- also defined in the config file -- and if no IP address is passed, we default to use *. This could be easily modified to include additional values, but I won't go into that now.

Okay, how do I use it?

#!/usr/bin/perl
use strict;
use FreePanel::Admin::Apache;

if ($#ARGV  [ip address]\n";
        exit;
}

my $domain = $ARGV[0];
my $ip_addr = $ARGV[1];

my $apache = new FreePanel::Admin::Apache;

$apache->addWebDir($domain)
        or die "Error adding web directory for $domain";
$apache->logger("web directory for $domain has been created.", $apache->INFO);

if ($ip_addr) {
        $apache->addSite($domain, $ip_addr)
                or die "Error creating vhost with $ip_addr for $domain";
        $apache->logger("vhost config created for $domain using $ip_addr.", $apache->INFO);
}
else {
        $apache->addSite($domain)
                or die "Error creating vhost for $domain";
        $apache->logger("vhost config craeted for $domain.", $apache->INFO);
}

# restart apache, still implementing FreePanel::Admin::Apache->restart
system('apachectl restart');

With the above you have a basic script for adding new domains fairly quickly. This will also allow you to keep track of what has been added and when since everything you do using FreePanel is logged to a file you chose. Here's a sample log output:

03-07-2010 11:59:36 :: [ DEBUG ] function: addWebDir(example.com)
03-07-2010 11:59:36 :: [ INFO ] web directory for example.com has been created.
03-07-2010 11:59:36 :: [ DEBUG ] function: addSite(example.com) called.
03-07-2010 11:59:36 :: [ INFO ] vhost config craeted for example.com.

Your level of logging is also defined by the configuration file. So with this basic script you get a log keeping track of what sites have been added and when as well as taking on the tedious work of copying a configuration file and switching a few values to meet the new domain's needs. There's a bit of documentation lacking, but some will soon be added to the FreePanel Wiki.

A case study: three web servers (clustered) using SSL for all domains

Recently, I was dealing with a client that had three web servers all behind a load balancer (Cisco CSS). This was a basic CRM software that required SSL for each virtual host (domain). This was troublesome because you need a unique IP address for each SSL site, and since all of their clients were using SSL, this meant every single domain had to have its own IP. This also means that each virtual host will be listening on a private IP behind the CSS. What this means is you will see something like this:

Web01:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ...

Web02:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ...

Web03:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ...

Being a lazy systems admin, I certainly did not want to manually create three configurations each time I wanted to add a new domain. So I used FreePanel::Admin::Apache and some ssh keys to carry out the creation of each. Using the addSite method, I passed the last octet of each domain rather than the full IP, and had the template using something similar to this:

Of course, changing the first three octets based on which server this template was being used on. I had a simple script similar to the one above that took the input for domain name and IP address, and a main script which I ran from an admin box. The main script was in charge of creating SSH connections and launching the helper script on each server with the proper arguments. This main script also remotely restarted Apache after creating each site.

While this example certainly isn't very elegant as password-less root SSH keys were required, it did however prove to be a very useful use-case for our Apache module. Solutions to problems such as this are why I work on FreePanel, because we all know that lazy syadmins are usually the best sysadmins.

Perl: wrapper for virt-install to provide iSCSI storage pool creation

Allan Feid — Wed, 03 Mar 2010 16:42:28 +0000

So I've been working on a setup for virtualization, where currently I'm using one large iSCSI target from an OpenSolaris machine. It occurred to me that there is a downside to this. I cannot use zfs snapshots for individual machines and backups/restores will be more difficult. I also did not want to be hassled with having to connect the iscsi targets using iscsiadm every time I spin up a new VM.

Luckily libvirt provides iSCSI storage pool methods which will connect the targets and manage them for you. This makes life much easier for me, however, virt-install does not have the functionality to install on these. Me being more proficient in Perl, I decided to take up the task of creating a simple wrapper. To use it, first you need this template XML file called "disk.xml."

      $NAME
                  <host name="$PORTAL"/>
          <device path="$TARGET"/>
        
        /dev/disk/by-path

Here's my script that I call virt-iscsi-install, and it's pretty easy to use. Not the greatest solution as it does not do much for error checking, but for now this is what I'll be using. I may consider venturing into some Python and patch virt-install to include similar functionality as the libvirt library is well documented and not hard to understand.

#!/usr/bin/perl
use Sys::Virt;
use Getopt::Long;

usage() if ( ! GetOptions(
        'name|n=s'      => \$name,
        'ram|r=s'       => \$ram,
        'target|t=s'    => \$target,
        'portal|p=s'    => \$portal,
        'mirror|m=s'    => \$mirror
)) ;

#print "name: $name\nram: $ram\ntarget: $target\nportal: $portal\n";

sub usage {
        print "usage: $0 --name  --ram  --target ".
                " --portal  --mirror \n";
        exit;
}

my $path = createPool($target,$portal,$name);
print "Volume: $path\n";
print "Launching virt-install..\n";
system ("virt-install -n $name -r $ram --location $mirror --file $path --vnc --bridge=br0");

sub createPool {
        my ($target, $portal, $name) = @_;
        my $vmm = Sys::Virt->new(address => 'qemu:///system');


        open my $fh, ') {
                s/\$NAME/$name/;
                s/\$TARGET/$target/;
                s/\$PORTAL/$portal/;
                $xml = "$xml $_";
        }
        close $fh;
        #print $xml;
        my $pool = $vmm->define_storage_pool($xml);
        $pool->create();

        if (!$pool->get_autostart) {
                print "setting pool to auto-start\n";
                $pool->set_autostart(1);
        }

        my @volNames = $pool->list_volumes;
        foreach my $vol (@volNames) {
                return $vol->get_path;
        }
}

Parse Airodump-NG Output with Perl

Allan Feid — Wed, 01 Apr 2009 03:32:51 +0000

I had to enumerate all the wireless access points around school today. So I booted up BackTrack 4 on my Eee PC and ran airodump-ng to create a list of ESSIDs, their corresponding BSSIDs, and the channels they were running on. Start off by running airodump-ng to save the data:

airodump-ng ath0 -w enum

That saves all the data to enum-01.txt, .csv, and .cap files. The .txt file itself is difficult to read, so I wanted to make a perl script to parse the data to a much more readable format. This is what I came up with:

#!/usr/bin/perl -w 

my @data;
my @bssid;
my @fts;
my @lts;
my @channel;
my @speed;
my @privacy;
my @cipher;
my @authentication;
my @power;
my @num_beacons;
my @num_ivs;
my @lan_ip;
my @idlen;
my @essid;



# read the airodump file
open FILE, "" or die $!;

while ("); ?>) {
	# lines are put in default $_ var, split reads this by default	
	@data = split(/,\s*/);

	push(@bssid, $data[0]);
	push(@fts, $data[1]);
	push(@lts, $data[2]);
	push(@channel, $data[3]);
	push(@speed, $data[4]);
	push(@privacy, $data[5]);
	push(@cipher, $data[6]);
	push(@authentication, $data[7]);
	push(@power, $data[8]);
	push(@num_beacons, $data[9]);
	push(@num_ivs, $data[10]);
	push(@lan_ip, $data[11]);
	push(@idlen, $data[12]);
	push(@essid, $data[13]);

}

close(FILE);

# get unique essids

my $essid_search;	# easier to remember/type than $essid[$i];
my @unique_essids;	# array of unique essids
my $unique_essid;	# var used in foreach loop as the current unique essid
my $check_unique;	# bool, if true (1) essid is unique

foreach ($i = 0; $i <= scalar(@essid); $i++) {
	$essid_search = $essid[$i];

	# if blank essid go to next
	if (!$essid_search) {	
		next;
	}

	# loop through and see if it's a unique essid
	$check_unique = 1;
	foreach $unique_essid (@unique_essids) {
		if ($essid_search eq $unique_essid) {
			$check_unique = 0;			
			last;
		}
	}
	
	# if unique, add it to @unique_essids
	if ($check_unique == 1) {
		push(@unique_essids, $essid_search);
	}
}

# look at all the essid, then print out corresponding bssid and channel
my $essid_match;
foreach $essid_match (@unique_essids) {

	print "Matching for $essid_match\n";
	print "-----------------------------------------------------\n";
	print "\tBSSID\t\t\tCHANNEL\tPRIVACY\n";
	print "-----------------------------------------------------\n";

	for ($i = 0; $i <= scalar(@essid); $i++) {
		if (($essid[$i]) && ($essid[$i] eq $essid_match)) {
			print "\t".$bssid[$i]."\t".$channel[$i]."\t".$privacy[$i]."\n";
		}
	}
	
	print "=====================================================\n\n\n";
}

Nothing fancy going on here and you'll have to change the open FILE section to correlate to the name of your capture file. This also ignores blank ESSIDs, as for my project I did not need them. Either way quick and easy and much better looking. I'll probably expand on this for future use but this is where I'll leave it for now.

Automated Backup w/ rsync over SSH

Allan Feid — Wed, 26 Nov 2008 01:47:52 +0000

There are four main sections to getting this up and running

Introduction
Installing/Configuring Required Software

OpenSSH
rsync
smbfs (optional)

Auto-mounting a Windows share (optional)

Creating a script to do the work for you

Automating the backup using cron jobs

Introduction

Before going over everything here, I'd like to point out why this information is useful and define a few terms. Using rsync to create a backup server solution is entirely free and conserves bandwidth. The reason it conserves bandwidth is rsync itself only sends the parts of each file that have changed. FTP sends the entire file each time even if only one word was changed in a file. Sending all of this traffic over SSH ensures security and also protects you from having to run a full rsync server. Key Terms:

Backup server: the PC that will be issuing all the rsync commands and storing the backups
Host computer: the PC where we will be getting the files from

With all of that out of the way, it's just a matter of setting everything up.

Installing/Configuring Required Software

In order to get things going you'll need to install a few pieces of software. In my situation I had a Windwos share that I wanted to back my home folder up to. In order to play around with Windows shares you'll need to also install the samba client.

OpenSSH

Simple to install with one line:
sudo apt-get install openssh-server

Once this is done you will be able to connect but it requires a password entry. This is sort of a problem when we're looking at automating the entire backup process so in order to solve that problem we'll use ssh keys instead. From your backup server we'll need to create an ssh key by issuing the following:

ssh-keygen -t dsa

Now you can either copy the key manually in some fashion or do it over an SSH connection with this:

scp ~/.ssh/id_dsa.pub username@x.x.x.x:./id_dsa.pub

Note that username is the local account on the host and x.x.x.x is the IP of the host. What this will do is copy the public key from the server to our host. After that we need to tell our host PC to use that key as authorization instead of asking for a password.

cd .ssh
touch authorized_keys2
chmod 600 authorized_keys2
cat ../id_dsa.pub >> authorized_keys2
rm ../id_dsa.pub

Here you can see we just change directory to .ssh and use touch to create the file authorized_keys2. Once the file is created lock it down and only allow the current user to change the file and finally add the public key then remove the original file containing the key. When you finish this try it out. If you're able to connect to your host pc from the backup server over SSH with no password everything has been setup correctly.

rsync & smbfs

These two are simple to install with just one line:

sudo apt-get install rsync 
sudo apt-get install smbfs

Keep in mind that smbfs is not required it just depends on your setup. If you would like to backup to/from a Windows share then you will want this along with the next section. Otherwise you can skip down to automating the backup using cron jobs.

Auto-mounting a Windows share (optional)

To mount a windows share everytime you boot up you will need to modify your /etc/fstab file. You only need to add one line at the end of this file like this:

//x.x.x.x/Backups /media/Backup cifs iocharset=utf8,credentials=/root/.smbcredentials,uid=1000 0 0

Of course you must replace x.x.x.x with the IP of your Windows computer and "Backups" with the share name. You will also need to create the file /root/.smbcredentials.

.smbcredentials

username=MyUsername@domain.com
password=MyPassword

This is a security risk because anyone with root priviledges can read your username/password in plain text so we must lock this down as much as possible.

sudo chown root /root/.smbcredentials
sudo chmod 600 /root/.smbcredentials

Also as a side note, you won't need "@domain.com" if your PC is not part of a domain. Once all this is down you can mount everything by issuing this:

sudo mount -a

You should now be able to access the share by browsing /media/Backups.

Creating a script to do the work for you
We're all lazy. Especially with making sure we have up-to-date backups. So to make it easier I've compiled a simple script that will ping the host to make sure it's alive and then execute the rsync command.

rsync_backup.sh

#!/bin/bash
awk '	function pinger(count,ip) {
		command = "ping -c "count " " ip		
		while (( command | getline res )> 0 ) {	
			if ( res ~ /0 received|100% packet loss/ ) {
				close(command)
				return 100			 
			}
		}
	close(command)
	return 0
	}	
	BEGIN { 	
	IP="x.x.x.x"	
	if ( pinger(2,IP) == 0 ) {
		system("rsync -raz --delete --rsh=ssh user@x.x.x.x:/home/user /media/Backup")
	}
}'

This will of course need to be modified to work with your setup. Replace x.x.x.x with your hosts IP and /home/user with the directory you would like to backup.

rsync -raz --delete --rsh=ssh user@x.x.x.x:/home/user /media/Backup

Here you can modify the actual rsync command used. The options r, a, & z are used for recursive, archive, and compress the data. The delete option deletes files that no longer exist on the source (in this case /home/user) and rsh tells rsync to use ssh. Take a look at the man page for further information about available rsync commands.

Automating the backup using cron jobs

To run this script everyday at 11PM you can simply add a cron job.

crontab -e

The above starts your default text editor in the file that holds cron jobs. The next line is what should be added to run the script at a certain time everyday.

0 23 * * * /path/to/rsync_backup.sh

Take a look at the crontab man page for a more detailed explanation of how cronjobs work.

Summary

Once this is all setup it's easy to modify to backup multiple PCs from the single backup server. For Windows PCs you can create a share or use the administrative share (\\comp\c$\path\to\files) and auto-mounting in order to back up PCs without installing SSH or rsync. This concept also works for Linux environments but does require a full samba install and configuration. That's the beauty of rsync, you can really do whatever you would like with it. If you have any questions or feel stumped feel free to contact me.

Back to my projects

Allan Feid - script

Cluster Management with FreePanel

So what does this have to do with FreePanel?

Okay, how do I use it?

A case study: three web servers (clustered) using SSL for all domains

Tags:

Perl: wrapper for virt-install to provide iSCSI storage pool creation

Tags:

Parse Airodump-NG Output with Perl

Tags:

Automated Backup w/ rsync over SSH

Tags: