Allan Feid - perl

Cluster Management with FreePanel

Allan Feid — Fri, 05 Mar 2010 17:16:09 +0000

I haven't had much time to continue work on FreePanel, but the Perl modules I've written have proven to be useful in cluster management. Take a look at the following diagram as an example scenario:

There's two key points to take away from that diagram.

physical servers are load balanced using a single VIP (Virtual IP)
the servers share access to a single storage mount

The idea behind load balancing is that requests go to the individual servers based on which can handle the traffic. A hardware appliance takes care of the actual load balancing work, but this could be done using some software just as easily.

So what does this have to do with FreePanel?

If you're constantly needing to add additional domains to your cluster, doing this manually becomes tiresome, especially when in most cases your application needs the same Apache configuration per domain. Take the below basic Virtual Host config:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ServerAlias www.example.com
        ErrorLog /home/sites/logs/example.com.error
        CustomLog /home/sites/logs/example.com.access combined
        DocumentRoot /home/sites/example.com/web

        
                Options Indexes FollowSymLinks
                AllowOverride None
                Order allow,deny
                Allow from all

Each of your 50 or so hosted applications needs a similar configuration, just with a different domain name. That's where FreePanel can make your life easier. Take the above example and we'll form a template that can be used with FreePanel's Apache module.

>
        ServerAdmin webmaster@
        ServerName 
        ServerAlias www.
        ErrorLog /home/sites/logs/.error
        CustomLog /home/sites/logs/.access combined
        DocumentRoot /home/sites//web

        /web/>
                Options Indexes FollowSymLinks
                AllowOverride None
                Order allow,deny
                Allow from all

As you can expect, this module can parse the template file and replace certain items with what arguments are being passed. We'll take a look at the addSite method of FreePanel::Admin::Apache.

sub addSite {
        my ($self, $domain, $ip_addr) = @_;
        $self->logger("function: addSite($domain, $ip_addr) called.", $self->FUNC_CALL);

        my $vhost_dir = $self->getVhostDir();
        my $vhost_templ = $self->getVhostTemplate();
        my $check = $self->getValidateObj();

        if ($check->is_active($domain, $vhost_dir)) {
                $self->logger("vhost config already exists for $domain.", $self->ERROR);
                return 0;
        }

        open (VHOST, '>', $vhost_dir."/$domain");
        open (VHOST_T, ') {
                        s//$domain/;
                        s//$ip_addr/;
                        print VHOST;
                }
        }
        else { 
                while () {
                        s//$domain/;
                        s//\*/;
                        print VHOST;
                }
        }

        close VHOST_T;
        close VHOST;

        return 1;
}

Nothing really that crazy going on here, it just opens up the template file -- defined in your config file -- then replaces <DOMAIN> and <IP_ADDR> with the values you pass. This file is written out to a directory -- also defined in the config file -- and if no IP address is passed, we default to use *. This could be easily modified to include additional values, but I won't go into that now.

Okay, how do I use it?

#!/usr/bin/perl
use strict;
use FreePanel::Admin::Apache;

if ($#ARGV  [ip address]\n";
        exit;
}

my $domain = $ARGV[0];
my $ip_addr = $ARGV[1];

my $apache = new FreePanel::Admin::Apache;

$apache->addWebDir($domain)
        or die "Error adding web directory for $domain";
$apache->logger("web directory for $domain has been created.", $apache->INFO);

if ($ip_addr) {
        $apache->addSite($domain, $ip_addr)
                or die "Error creating vhost with $ip_addr for $domain";
        $apache->logger("vhost config created for $domain using $ip_addr.", $apache->INFO);
}
else {
        $apache->addSite($domain)
                or die "Error creating vhost for $domain";
        $apache->logger("vhost config craeted for $domain.", $apache->INFO);
}

# restart apache, still implementing FreePanel::Admin::Apache->restart
system('apachectl restart');

With the above you have a basic script for adding new domains fairly quickly. This will also allow you to keep track of what has been added and when since everything you do using FreePanel is logged to a file you chose. Here's a sample log output:

03-07-2010 11:59:36 :: [ DEBUG ] function: addWebDir(example.com)
03-07-2010 11:59:36 :: [ INFO ] web directory for example.com has been created.
03-07-2010 11:59:36 :: [ DEBUG ] function: addSite(example.com) called.
03-07-2010 11:59:36 :: [ INFO ] vhost config craeted for example.com.

Your level of logging is also defined by the configuration file. So with this basic script you get a log keeping track of what sites have been added and when as well as taking on the tedious work of copying a configuration file and switching a few values to meet the new domain's needs. There's a bit of documentation lacking, but some will soon be added to the FreePanel Wiki.

A case study: three web servers (clustered) using SSL for all domains

Recently, I was dealing with a client that had three web servers all behind a load balancer (Cisco CSS). This was a basic CRM software that required SSL for each virtual host (domain). This was troublesome because you need a unique IP address for each SSL site, and since all of their clients were using SSL, this meant every single domain had to have its own IP. This also means that each virtual host will be listening on a private IP behind the CSS. What this means is you will see something like this:

Web01:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ...

Web02:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ...

Web03:


        ServerAdmin webmaster@example.com
        ServerName example.com
        ...

Being a lazy systems admin, I certainly did not want to manually create three configurations each time I wanted to add a new domain. So I used FreePanel::Admin::Apache and some ssh keys to carry out the creation of each. Using the addSite method, I passed the last octet of each domain rather than the full IP, and had the template using something similar to this:

Of course, changing the first three octets based on which server this template was being used on. I had a simple script similar to the one above that took the input for domain name and IP address, and a main script which I ran from an admin box. The main script was in charge of creating SSH connections and launching the helper script on each server with the proper arguments. This main script also remotely restarted Apache after creating each site.

While this example certainly isn't very elegant as password-less root SSH keys were required, it did however prove to be a very useful use-case for our Apache module. Solutions to problems such as this are why I work on FreePanel, because we all know that lazy syadmins are usually the best sysadmins.

Perl: wrapper for virt-install to provide iSCSI storage pool creation

Allan Feid — Wed, 03 Mar 2010 16:42:28 +0000

So I've been working on a setup for virtualization, where currently I'm using one large iSCSI target from an OpenSolaris machine. It occurred to me that there is a downside to this. I cannot use zfs snapshots for individual machines and backups/restores will be more difficult. I also did not want to be hassled with having to connect the iscsi targets using iscsiadm every time I spin up a new VM.

Luckily libvirt provides iSCSI storage pool methods which will connect the targets and manage them for you. This makes life much easier for me, however, virt-install does not have the functionality to install on these. Me being more proficient in Perl, I decided to take up the task of creating a simple wrapper. To use it, first you need this template XML file called "disk.xml."

      $NAME
                  <host name="$PORTAL"/>
          <device path="$TARGET"/>
        
        /dev/disk/by-path

Here's my script that I call virt-iscsi-install, and it's pretty easy to use. Not the greatest solution as it does not do much for error checking, but for now this is what I'll be using. I may consider venturing into some Python and patch virt-install to include similar functionality as the libvirt library is well documented and not hard to understand.

#!/usr/bin/perl
use Sys::Virt;
use Getopt::Long;

usage() if ( ! GetOptions(
        'name|n=s'      => \$name,
        'ram|r=s'       => \$ram,
        'target|t=s'    => \$target,
        'portal|p=s'    => \$portal,
        'mirror|m=s'    => \$mirror
)) ;

#print "name: $name\nram: $ram\ntarget: $target\nportal: $portal\n";

sub usage {
        print "usage: $0 --name  --ram  --target ".
                " --portal  --mirror \n";
        exit;
}

my $path = createPool($target,$portal,$name);
print "Volume: $path\n";
print "Launching virt-install..\n";
system ("virt-install -n $name -r $ram --location $mirror --file $path --vnc --bridge=br0");

sub createPool {
        my ($target, $portal, $name) = @_;
        my $vmm = Sys::Virt->new(address => 'qemu:///system');


        open my $fh, ') {
                s/\$NAME/$name/;
                s/\$TARGET/$target/;
                s/\$PORTAL/$portal/;
                $xml = "$xml $_";
        }
        close $fh;
        #print $xml;
        my $pool = $vmm->define_storage_pool($xml);
        $pool->create();

        if (!$pool->get_autostart) {
                print "setting pool to auto-start\n";
                $pool->set_autostart(1);
        }

        my @volNames = $pool->list_volumes;
        foreach my $vol (@volNames) {
                return $vol->get_path;
        }
}

Announcing FreePanel

Allan Feid — Fri, 15 Jan 2010 17:40:38 +0000

What's been keeping me busy? Well I've been working for a managed hosting company (DataPipe) and that has been the biggest source of time suck for me. Outside of that I've been working on some Perl modules to make administrating load balanced servers as easy as possible. For now, read the source if you want to know more:

FreePanel

Parse Airodump-NG Output with Perl

Allan Feid — Wed, 01 Apr 2009 03:32:51 +0000

I had to enumerate all the wireless access points around school today. So I booted up BackTrack 4 on my Eee PC and ran airodump-ng to create a list of ESSIDs, their corresponding BSSIDs, and the channels they were running on. Start off by running airodump-ng to save the data:

airodump-ng ath0 -w enum

That saves all the data to enum-01.txt, .csv, and .cap files. The .txt file itself is difficult to read, so I wanted to make a perl script to parse the data to a much more readable format. This is what I came up with:

#!/usr/bin/perl -w 

my @data;
my @bssid;
my @fts;
my @lts;
my @channel;
my @speed;
my @privacy;
my @cipher;
my @authentication;
my @power;
my @num_beacons;
my @num_ivs;
my @lan_ip;
my @idlen;
my @essid;



# read the airodump file
open FILE, "" or die $!;

while ("); ?>) {
	# lines are put in default $_ var, split reads this by default	
	@data = split(/,\s*/);

	push(@bssid, $data[0]);
	push(@fts, $data[1]);
	push(@lts, $data[2]);
	push(@channel, $data[3]);
	push(@speed, $data[4]);
	push(@privacy, $data[5]);
	push(@cipher, $data[6]);
	push(@authentication, $data[7]);
	push(@power, $data[8]);
	push(@num_beacons, $data[9]);
	push(@num_ivs, $data[10]);
	push(@lan_ip, $data[11]);
	push(@idlen, $data[12]);
	push(@essid, $data[13]);

}

close(FILE);

# get unique essids

my $essid_search;	# easier to remember/type than $essid[$i];
my @unique_essids;	# array of unique essids
my $unique_essid;	# var used in foreach loop as the current unique essid
my $check_unique;	# bool, if true (1) essid is unique

foreach ($i = 0; $i <= scalar(@essid); $i++) {
	$essid_search = $essid[$i];

	# if blank essid go to next
	if (!$essid_search) {	
		next;
	}

	# loop through and see if it's a unique essid
	$check_unique = 1;
	foreach $unique_essid (@unique_essids) {
		if ($essid_search eq $unique_essid) {
			$check_unique = 0;			
			last;
		}
	}
	
	# if unique, add it to @unique_essids
	if ($check_unique == 1) {
		push(@unique_essids, $essid_search);
	}
}

# look at all the essid, then print out corresponding bssid and channel
my $essid_match;
foreach $essid_match (@unique_essids) {

	print "Matching for $essid_match\n";
	print "-----------------------------------------------------\n";
	print "\tBSSID\t\t\tCHANNEL\tPRIVACY\n";
	print "-----------------------------------------------------\n";

	for ($i = 0; $i <= scalar(@essid); $i++) {
		if (($essid[$i]) && ($essid[$i] eq $essid_match)) {
			print "\t".$bssid[$i]."\t".$channel[$i]."\t".$privacy[$i]."\n";
		}
	}
	
	print "=====================================================\n\n\n";
}

Nothing fancy going on here and you'll have to change the open FILE section to correlate to the name of your capture file. This also ignores blank ESSIDs, as for my project I did not need them. Either way quick and easy and much better looking. I'll probably expand on this for future use but this is where I'll leave it for now.

Allan Feid - perl

Cluster Management with FreePanel

So what does this have to do with FreePanel?

Okay, how do I use it?

A case study: three web servers (clustered) using SSL for all domains

Tags:

Perl: wrapper for virt-install to provide iSCSI storage pool creation

Tags:

Announcing FreePanel

Tags:

Parse Airodump-NG Output with Perl

Tags: