Allan Feid - Linux

Create an inventory tool using MCollective's registration feature

Allan Feid — Sat, 08 Oct 2011 22:05:09 +0000

I started working at a company (Shutterstock) that had MCollective implemented a few months ago. We use it to perform various functions across a wide range of servers. Things such as rolling the application servers to pick up new code, clearing caches, checking replication lag on databases, telling our puppetmaster to sign certificates, and even doing code deployment. There are endless possibilites with this system, and it is so simple to write your own agent/client utilities, that I'm not sure why more people aren't using MCollective. One of the things we were lacking was a central inventory of our systems in an easy to view manner. Thankfully, R.I. Pienaar's MCollective has a great feature called registration. Essentially, it causes your nodes to blast out to an ActiveMQ/rabbitmq topic at a set interval, and any node configured to listen can read that data and perform actions based on that data. With that in mind, I set out to create a database of information.

Setting up a registration plugin

Taking a look at this registration plugin, you can begin to see how easy generating your data set is. From this example, you get MCollective's agentlist, facts, and classes sent during registration. I wanted to also report on virtual machines running on hypervisors with an in-house libvirtd agent. To achieve that, your agent needs to have a simple method that can be called without the request/reply data structures. An example of that would be this:

module MCollective
  module Agent
    class Custom<RPC::Agent

      def some_method
        # gather some data and return it
      end

    end
  end
end

Once your agent has a method that can simply be called, you can call this method inside your registration plugin with something like this:

module MCollective
  module Registration
    class Meta<Base
      def body
        result = {
          :agentlist => [],
          :facts => [],
          :classes => []
        }

        cfile = Config.instance.classesfile
        if File.exists?(cfile)
          result[:classes] = File.readlines(cfile).map {|i| i.chomp}
        end

        result[:agentlist] = Agents.agentlist
        result[:facts] = PluginManager["facts_plugin"].get_facts

        if Agents.agentlist.include?("custom")
          # because of how PluginManager works, you need to 
          # append _agent to the agent's name
          result[:custom] = PluginManager["custom_agent"].some_method
        end

        result
      end
    end
  end
end

This way your nodes can report back with literally any information that you'd like to include. I've had great success reporting on my hypervisor's available disk, memory, and CPU utilization as well as what virtual machines are running.

What do you do with this data?

With inspiration from this mongodb registration agent, I decided MongoDB was a pretty good way to store data that was dynamicly generated. I wanted to make it easy to store/retrieve data as I saw fit, so using Mongoid seemed like the way to go. Something as simple as this is all you really need:

class Node
  include Mongoid::Document
  include Mongoid::Timestamps
  field :hostname, :unique => true, :type => String
  index :hostname
  validates_uniqueness_of :hostname
end

Improving performance with a queue

After running registration for a bit, memory usage by mcollective began to skyrocket on the machine processing the data, and eventually I started seeing timeouts or pool limits being hit on my mongo connection. I was able to talk a bit with R.I. Pienaar on IRC, and he informed me that every request that a node recieves spawns a new thread for an agent to run in. This is not scalable for the registration process, and once you hit over about 100 nodes, you run into problems since your machine is constantly opening up new threads to handle registration data. He did offer a solution that he uses personally and has suggested to others. That is to use a queue rather than a topic, and to have a simple daemon dedicated to processing registration data. There's some sample code available on his website, and I'm sure this will be included in the main application at some point. In order to get this working I did need to change my ActiveMQ configuration a bit, but the change was simple and just required adding a new line similar to this:

<authorizationEntry queue="mcollective.>" write="admins" read="admins" admin="admins" />

Basically the mcollective user that you use from the client side needs to be able to read/write to the new queue. This also requires that your registration plugin sends the result out to the queue rather than a topic. Another simple change:

module MCollective
  module Registration
    class Meta<Base
      def body
        ...
        PluginManager["connector_plugin"].connection.publish("/queue/mcollective.registration", result.to_json)
        nil
      end
    end
  end
end

By having the body method return nil, you won't send anything out to the standard registration topic, which would be redundant in this case.

The end result

With a bit of tweaking, I was able to get my registration daemon/agent saving data into mongo in less than a second per request recieved. Once I had all that data in place, I was able to implement a slick interface using Sinatra and Bootstrap that gives me good insight into our inventory. I'll probably go more into the frontend interface when it becomes more mature, but once you have all this data, it's up to you to decide what you want to do with it.

Capistrano for the sysadmin

Allan Feid — Sun, 09 Jan 2011 18:43:34 +0000

There's always been a fine line between a good sysadmin and a programmer. Capistrano is software built for Ruby on Rails to make deploys across multiple servers easy, however, the potential is much greater than just code deployment. I have recently grown in love with how easily I can run commands across groups of servers, and setting up Capistrano is not a difficult task. If you've ever configured password-less SSH access, you can get this up and running without much difficulty.

Have a centralized ssh-key

# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.

You have an SSH key, now you just need to install it on each server you would like to manage. One of the things I like to do, is setup an rsync share to pull down important config files. This can also be easily accomplished using an NFS mount. Either way after you have password-less SSH access to all the servers from one location, this will be your controlling server. You won't need to login to the other boxes individually from this point forward.

Security note: in this example, I'm using root logins. I recommend setting PermitRootLogin to without-password if you want to use root access, however, I really do not recommend using root directly, as it becomes a liability issue when you have more than one admin. The use of a normal user account plus sudo is highly recommended.

Install the software

Depending on what OS you're on, these steps may be different, but the end goal is the same: get RubyGems installed. On CentOS/RHEL the following works:

# yum install -y ruby ruby-devel ruby-rdocs ruby-ri

Since the version of ruby included on CentOS/RHEL is a bit dated, we have to use an older version of RubyGems (1.3.5).

# wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
# tar xzf rubygems-1.3.5.tgz
# cd rubygems-1.3.5/
# ruby setup.rb
# gem install capistrano

Configuring Capistrano

At this point, a Rails developer would cd into their code directory, run capify and configure Capistrano's deploy scripts to their needs. Going over code deployments is an entirely different process, which I'll save for another time. As a sysadmin, we can skip running capify and just make our own cap file. Think of a cap file as a basic config file, except written in Ruby. You get the power of a scripting language built right into your config file. This truly gives you endless options and the ability to tie into anything you'd like (think database, or ldap). Here I'll show you a basic "cap file" that will do what sysadmins want: make managing multiple servers easy.

# cd /etc/manage
# ls -l 
-rw-r--r-- 1 root root 2739 Jul 20 18:14 capfile
drwxr-xr-x 2 root root 4096 Jul 19 12:15 roles/

# cat capfile
def getrole(role)
        f = open('/etc/manage/roles/' + role)
        f.readlines
end

role(:web) { getrole('web') }
role(:db) { getrole('db') }

desc "Show Memory"
task :show_mem do
  run "grep -i memtotal /proc/meminfo"
end

What I like to do is create a directory (/etc/manage/), then populate this with central management type configurations. In this case, it would be my cap file, and a directory called roles. The roles directory just contains files with lists of servers, for example:

# cat /etc/manage/roles/web
web1
web2
web3

So this would be the resolvable names of all my web servers. The cap file uses a basic function called getrole() to read the file declared by role inside the roles directo One would be to cd into /etc/manage and the other would be to declare the cap file from the command line with the -f parameter. The first thing you can look at, is the output of -T.

# cap -T
cap invoke     # Invoke a single command on the remote servers.
cap shell      # Begin an interactive Capistrano session.
cap show_mem   # Show Memory

This will show you all the different tasks you have available. There are two built in ones (invoke, shell) and one we defined in our cap file (show_mem). This is a very basic example, but will suite a wide range of needs. The most useful, in my opinion (at least at this stage), is the shell. This is where you have the power to run commands across "roles" of servers.

# cap shell
cap> df -h
 ** [out :: web1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web1] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web1] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web2] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web2] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web2] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web3] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web3] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web3] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: db1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: db1] /dev/cciss/c0d0p1     128G  2.6G  119G   3% /
 ** [out :: db1] tmpfs                 2.0G     0  2.0G   0% /dev/shm
cap> with web
cap> grep admin /etc/passwd
 ** [out :: web1] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web2] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web3] admin:x:500:500:Admin user:/home/admin:/bin/bash
cap> with db
cap> !show_mem 
 ** [out :: db1] MemTotal:      3985972 kB

When you first enter the shell, it will be set to execute tasks or commands across ALL roles. Using the with directive, you can specify which role you want to use for the subsequent commands. Finally, adding a bang (!) in front, tells Capistrano to execute a task rather than standard shell commands. Hopefully I've given you enough information to apply Capistrano to your own environments. Think about commands you often execute, create a new task and you will find yourself saving a lot of time. Even troubleshooting and adding new functionality across servers becomes much quicker when you can access everything from one location.

Central login management with OpenLDAP

Allan Feid — Sun, 28 Mar 2010 17:18:58 +0000

Over the weekend I spent some time learning how to configure OpenLDAP to provide a central login server at home. This is pretty useful since I was tired of either syncing UID/GID's across servers or setting directories to 777 on my NFSv4 server. I chose Fedora 12 as my OS for the server, and will show how easy it is to configure authentication on both Fedora and OpenSolaris.

Some information about the network

I'll give you a quick overview of how the network looks from home.

DNS: ns1.crazy.lan (172.16.1.3)
LDAP: ldap1.crazy.lan (10.10.1.1)
Client 1: test.crazy.lan (10.10.2.1)
Client 2: virt.crazy.lan (192.168.1.8)

I'm not going to go over the importance of DNS (both forward and reverse lookups should work!) right now, but you will save yourself plenty of headaches by making sure DNS is setup properly on your network.

Basic configurations

Install the needed software!

For Fedora 12, installing OpenLDAP is easy:

# yum install -y openldap*

This will get you everything you need to get started.

There are a few things you need to change in order to really start using OpenLDAP. Everything is all in one file slapd.conf. Prior to editing the file, you'll want to generate a password to use in this configuration.

# slappasswd
New password: 
Re-enter new password: 
{SSHA}fFVfwsztHn+xuQPjN/q/urdNC+V0G+dW

Copy the new hash as you will need to paste it into the configuration file. Add or modify the following in /etc/openldap/ldap.conf:

suffix      "dc=crazy,dc=lan"
rootdn      "cn=Manager,dc=crazy,dc=lan"
rootpw      {SSHA}fFVfwsztHn+xuQPjN/q/urdNC+V0G+dW

# ACL's to use
loglevel acl stats
access to dn.base="" by * read
access to dn.base="cn=Subschema" by * read
access to *
        by self write
        by * read
        by anonymous auth

That's all you need there for a basic configuration. This will not be using TLS/SSL, I will cover that in a later post. It's always good to start with the basics so that you can spend more time learning the protocol. Once you've gotten that down hardening a service is easy.

Now we just need to finish up and start the service. Since slapd.conf is deprecated in favor of the slapd.d/ directory, we can use the configuration file to generate the appropriate directory structure. The huge advantage of this style is dynamic backend configuration changes. Yes, that means persistent dynamic changes. No more restarting your directory server to make changes to things such as your logging level.

# cd /etc/openldap
# rm -rf slapd.d
# mkdir slapd.d
# slaptest -f slapd.conf -F slapd.d
# chown -R ldap:ldap slapd.d
# chkconfig slapd on
# service slapd start

You can test for functionality by running the following:

ldapsearch -x -h localhost

If all went well you will see some output and no errors.

Configure rsyslog for logging

If you're not using Fedora, or rsyslog, you'll want to lookup the specifics for your logging daemon's configuration. OpenLDAP sends it's logging output to the local4 log facility. For rsyslog, just add the following line to /etc/rsyslog.conf:

local4.*                                                -/var/log/ldap

Then restart your logging service.

service rsyslog restart

Importing some data to LDAP

The LDIF formatted file is where things start to get a little bit tricky. Don't worry about it so much as there are plenty of good GUI clients for LDAP, but we do need to get a few things imported.

# crazy.lan
dn: dc=crazy,dc=lan
objectClass: dcObject  
objectClass: organization
o: Crazy Organization  
dc: crazy

# Manager, crazy.lan   
dn: cn=Manager,dc=crazy,dc=lan
objectClass: organizationalRole
cn: manager

# People, crazy.lan
dn: ou=People,dc=crazy,dc=lan
objectClass: organizationalUnit
ou: People

# Group, crazy.lan
dn: ou=Group,dc=crazy,dc=lan
objectClass: organizationalUnit
ou: Group

# users, Group, crazy.lan
dn: cn=users,ou=Group,dc=crazy,dc=lan
objectClass: posixGroup
objectClass: top
cn: users
gidNumber: 2000

# myaccount, People, crazy.lan
dn: uid=myaccount,ou=People,dc=crazy,dc=lan
uid: myaccount
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
gidNumber: 2000
uidNumber: 3000
homeDirectory: /home/myaccount
loginShell: /bin/bash
userPassword: {SSHA}cc696ZrGCPmgx0/CD2zGJrMqR3tb8iMk
cn: Allan Feid

I saved the above as import.ldif. You will probably want to generate a new password with slappaswd to use with the userPassword section above. Now to import, it all I have to do is this:

ldapadd -f import.ldif -D "cn=Manager,dc=crazy,dc=lan" -W

This will prompt you for your rootdn password that you created earlier using slappasswd. Now you have the following basic hierarchy configured:

Base Domain: crazy.lan
- Organizational Unit: People
  - User: myaccount
- Organizational Unit: Group
  - Group: users

User accounts will go into the People OU, and group accounts will be assigned the Group OU. This is a pretty standard configuration as the pam_ldap plugin will look in these OU's later for user/group information.

Client configuration

Fedora 12

Redhat has really made this easier than it used to be. Their authconfig command is pretty straight forward, the following commands will get you the nss_ldap, pam_ldap and ldap utility commands on your client:

# yum install -y openldap-clients
# authconfig --enableldap \
    --enableldapauth \
    --disablenis \
    --enablecache \
    --ldapserver=ldap1.crazy.lan \
    --ldapbasedn=dc=crazy,dc=lan \
    --updateall

That's it. You will now see the myaccount user if you run getent.

# getent passwd |grep myaccount
myaccount:x:3000:2000:Allan Feid:/home/myaccount:/bin/bash

OpenSolaris

This should also work for plain Solaris, but I prefer OpenSolaris and haven't tested on Solaris 10. This, just like Redhat, is very easy to configure, but does require one extra step. The ldapclient command by default copies /etc/nsswitch.ldap to /etc/nsswitch.conf. Which is fine, but the ldap nsswich configuration by, by default, uses ldap for hostname lookups. This is probably not desirable in most cases as that's what we use DNS for. So just change the following lines in /etc/nsswitch.ldap:

hosts:      files dns
ipnodes:    files dns

Then just a simple command will get you up and running:

# ldapclient -v manual -a defaultSearchBase=dc=crazy,dc=lan \
    -a domainName=crazy.lan \
    -a defaultServerList=ldap1.crazy.lan
# getent passwd |grep myaccount
myaccount:x:3000:2000::/home/myaccount:/bin/bas

General Configuration for client's using pam_ldap/nss_ldap

If you aren't using Fedora or OpenSolaris, anything that uses pam_ldap/nss_ldap can be configured pretty easily. The main configuration file for these two is /etc/ldap.conf. All you need is to make sure you have the following lines:

base dc=crazy,dc=lan
uri ldap://ldap1.crazy.lan/

Then you will have to configure nss by changing the following in /etc/nsswitch.conf:

passwd:     files ldap
shadow:     files ldap
group:      files ldap

You will also need to modify your PAM configuration. Generally you will want to edit /etc/pam.d/system-auth but this may very from OS to OS. Always check with the documentation!

auth        sufficient    pam_ldap.so use_first_pass

Use ADS, make your life easier!

I highly recommend Apache Directory Studio as a great LDAP client. It is cross platform (java based) and having a GUI for ldap sometimes just makes more sense as you get to visualize your hierarchy easily. Setup a new connection and login as cn=Manager,dc=crazy,dc=lan using your rootdn password. You can now easily create a template user to be copied for new users, add new groups, add OU's, etc all from a comfortable GUI.

In upcoming posts, I plan on covering enabling TLS encryption, and eventually working on Kerberos integration with LDAP for a true Single SIgn-On (SSO) environment. If all goes well I'll share how I got it all working.

Kernel Virtual Machines using libvirt

Allan Feid — Mon, 01 Mar 2010 19:45:04 +0000

I use iSCSI from a giant ZFS storage pool in order to setup virtual machines. There's a couple advantages to this, as you get the power of ZFS snapshots and the ability to grow your virtual machine's disks fairly easily. To make administrating things a bit easier, I use one large iSCSI target and put LVM on top of this. So really this article will apply to anyone that wants to use logical volumes for their disks, which is much preferred over using files.

I run my servers all in a headless manner, so I will be using virsh and virt-install over virt-manager. Also, I will be using Fedora 12 for the virtual host as this is where KVM is actively developed and generally where you will find the best performance / stability.

Installing the required software

Fedora has a nice "Virtualization" group package which will get you setup to run KVM based virtual machines.

# yum groupinstall "Virtualization" -y
# chkconfig libvirt on

Since I want my virtual machines to have direct access to the network, I will put one of my NICs into a bridge. You will want to do the same unless you prefer the default NAT'd mode.

Network configuration

To configure your network you need to create /etc/sysconfig/network-scripts/ifcfg-br0 for a bridged interface. Use the following as an example:

# cat ifcfg-br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
IPADDR=10.10.0.3
NETMASK=255.255.0.0
ONBOOT=YES

Then same directory, you need to edit the file for the interface you want attached to the bridge. For my configuration I'm using eth3.

# cat ifcfg-eth3
DEVICE=eth3
HWADDR=00:15:17:D6:35:69
ONBOOT=YES
TYPE=Ethernet
IPV6INIT=no
USERCTL=no
BRIDGE=br0

That's it. I highly recommend removing NetworkManager. Don't just disable it, it's a waste of space and complicates things.

# yum remove -y NetworkManager
# chkconfig network on
# service network restart
# ifconfig br0
br0       Link encap:Ethernet  HWaddr 00:15:17:D6:35:69  
          inet addr:10.10.0.3  Bcast:10.10.255.255  Mask:255.255.0.0
          inet6 addr: fe80::215:17ff:fed6:3569/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:68994 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25910 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:59178792 (56.4 MiB)  TX bytes:1746255 (1.6 MiB)

Now you have br0 to put virtual NICs (those created by KVM) on and access your network directly.

Disk configuration

Since LVM is useful in this situation, we're going to use it. I'm starting with a freshly partitioned disk so I will go over creating a proper LVM configuration. You need to know which partition and disk you will be creating this on, so if you don't know, do not just blindly run these commands. Consult with fdisk -l and the man pages.

# fdisk -l /dev/sdc

Disk /dev/sdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xae668fde

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       26108   209712478+  8e  Linux LVM

# pvcreate /dev/sdc1
# vgcreate zfs_lun /dev/sdc1
# lvcreate -L 20G zfs_lun -n guest1
# lvdisplay /dev/zfs_lun/guest1 
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  --- Logical volume ---
  LV Name                /dev/zfs_lun/guest1
  VG Name                zfs_target
  LV UUID                pVt1tv-OZSr-3joc-xaO1-iklH-z2eS-FEAbRf
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                20.00 GB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

You now have a logical volume of 20GB that you can use for installing virtual machines. To briefly go over the commands above, first we needed to create the physical volume on top of /dev/sdc1. From there you create a volume group, which is a collection of logical volumes. You can have more than one of these on a disk, but for my purposes I just let the volume group consume the entire physical volume. Finally, we setup a logical volume with 20GB of space. The last line is useful for double checking your work, which is always a good idea.

libvirt Configuration

Since I'm on an isolated trusted network, I like to have full access to the VNC serial connections made available. By default these only listen on localhost, and ssh port forwarding becomes tedious when you are trying to manage more than one at a time. So to open up VNC to the world, you need to uncomment this line in /etc/libvirt/qemu.conf:

# VNC is configured to listen on 127.0.0.1 by default.
# To make it listen on all public interfaces, uncomment
# this next option.
#
# NB, strong recommendation to enable TLS + x509 certificate
# verification when allowing public access
#
vnc_listen = "0.0.0.0"

Restart libvirt after, also we'll make sure it starts on boot at this time:

# service libvirtd restart
# chkconfig libvirtd on

virt-install, it's easy!

This command line utility is great. I strongly recommend reading the man page for this one as you will find useful information for installing guests the way you would like. I find it much more powerful than virt-manager, as last I checked virt-manager leaves out a few useful features from the libvirt library. (Disclaimer: I haven't used virt-manager in a long long time, at least in development years)

# virt-install --name guest1 \
    --ram 1024 \
    --location http://mirror.crazy.lan/Fedora/12/Fedora/x86_64/os \
    --file /dev/zfs_lun/guest1 \
    --vnc \
    --bridge=br0

The long names of each option should make this fairly easy to understand. Location refers to the location of your installation media. This can be a variety of options (check the man page!) but http is my personal choice. File of course specifies the disk that you will be using, vnc tells qemu-kvm to allow VNC access, and bridge specifies the network bridge to attach to.

An interesting option that I once played around with was --pxe. This allows your VM to boot over the network and install from there. This is a great little option if you've already got the infrastructure in place for netbooting, and possibly well worth setting up if you're tired of carrying around install CDs.

virsh

Your virtual shell. This is where you can modify your VMs and again, well worth looking at the man page for. If you're too lazy, just type virsh help and you'll be presented with a list of options. So earlier we started an install, how can we check if it's running and connect to the serial console?

# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list
 Id Name                 State
----------------------------------
  1 guest1               running

virsh # vncdisplay guest1
:0

This tells you what VMs are running and the state they are in. The vncdisplay command lets you find out which display to connect to when using vnc. So from your management station you can run the following to get in:

vncviewer virt02.crazy.lan:0

Now you have a VNC window open and you will be taken through a normal installation process. That's it. There's a ton of things you can do with libvirt and really it just requires some reading. At some point I may consider looking more into virt-manager, but honestly it leaves some to be desired compared to the already powerful command line tools.

Connect to iSCSI targets on Linux

Allan Feid — Mon, 01 Mar 2010 18:59:56 +0000

I'm going to be describing this process using Fedora, but any Linux that is using iscsiadm should use the same sort of configurations. To start off make sure you have the proper packages installed.

# yum install -y iscsi-initiator-utils

Now you can easily see what iscsi targets are available on your portal (server):

# iscsiadm -m discovery -t st -p 10.20.0.2
10.20.0.2:3260,1 iqn.1986-03.com.sun:02:bf1479ba-a71d-c9d7-ab45-d51a1dca3416
# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.1986-03.com.sun:02:bf1479ba-a71d-c9d7-ab45-d51a1dca3416, portal: 10.20.0.2,3260]
Login to [iface: default, target: iqn.1986-03.com.sun:02:bf1479ba-a71d-c9d7-ab45-d51a1dca3416, portal: 10.20.0.2,3260]: successful

The first command will search for all targets available, the second will then login (-l) to all the found targets. If you only need to login to a specific target you can use the following:

# iscsiadm -m node --target "iqn.1986-03.com.sun:02:bf1479ba-a71d-c9d7-ab45-d51a1dca3416" --portal "10.20.0.2:3260" -l
Logging in to [iface: default, target: iqn.1986-03.com.sun:02:bf1479ba-a71d-c9d7-ab45-d51a1dca3416, portal: 10.20.0.2,3260]
Login to [iface: default, target: iqn.1986-03.com.sun:02:bf1479ba-a71d-c9d7-ab45-d51a1dca3416, portal: 10.20.0.2,3260]: successful

Once you have logged in to your target(s) you'll see them as a disk if you run fdisk -l:

# fdisk -l

Disk /dev/sdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xae668fde

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       26108   209712478+  8e  Linux LVM

I've already partitioned my target with fdisk but you may want to do the same. My primary use for iSCSI is to serve up a single large volume (zvol) and then I use LVM to give me volumes to use with my virtual machines.

Creating a Chroot Jail for SSH Access

Allan Feid — Wed, 01 Apr 2009 03:57:00 +0000

Update 5/30/2018:

@CodingFree points out that the l2chroot script used in this blog post fails to handle /lib64, and @el_shoo confirms everything in this post still works as long as you create /lib64. I have updated the post to reflect this, great to see it still working almost 10 years later.

I wanted to setup a way to allow SSH access to my machine but limit their abilities heavily. To do that I figured a chroot jail was the best way. In this example I'm using ArchLinux and OpenSSH 5.1p1. It should be a very similar process on any *nix operating system.

Setup your test user

The way I'm setting this up, is that all my chrooted users will be added to the sshusers group. So we must setup the group, then add the user.

$ groupadd sshusers
$ adduser -g sshusers user

Setup the jail directories

The next step is to setup all the directories needed. This needs to emulate the / directory to a bare minimum. That is we need a dev, etc, lib, usr, and bin directory as well as usr/bin/. The base directory has to be owned by root.

$ mkdir -p /var/jail/{dev,etc,lib,lib64,usr,bin}
$ mkdir -p /var/jail/usr/bin
$ chown root.root /var/jail

You also need the /dev/null file:

$ mknod -m 666 /var/jail/dev/null c 1 3

You need to fill up the etc directory with a few minimum files:

$ cd /var/jail/etc
$ cp /etc/ld.so.cache .
$ cp /etc/ld.so.conf .
$ cp /etc/nsswitch.conf .
$ cp /etc/hosts .

Once this is done you need to figure out what commands you want accessible by your limited users. In this example I only want the users to be able to get into bash and use the ls command. So you must copy the binaries to the jail.

$ cd /var/jail/usr/bin
$ cp /usr/bin/ls .
$ cp /usr/bin/bash .

Now that you've got all the binaries in place, you need to add the proper shared libraries. To find out what libraries are need you can run ldd /path/to/bin. The output looks similar to this:

$ ldd /bin/ls
         linux-gate.so.1 =>    (0xb7f2b000)
         librt.so.1 => /lib/librt.so.1 (0xb7f1d000)
         libacl.so.1 => /lib/libacl.so.1 (0xb7f16000)
         libc.so.6 => /lib/libc.so.6 (0xb7dcf000)
         libpthread.so.0 => /lib/libpthread.so.0 (0xb7db7000)
         /lib/ld-linux.so.2 (0xb7f2c000)
         libattr.so.1 => /lib/libattr.so.1 (0xb7db2000)

Then you have to manually copy each file to the lib directory in your jail. That is a pain. Especially if there is a lot of shared libraries for a binary you want. I came across a useful script called l2chroot which automatically finds the libraries and copies them to your chroot jail.

cd /sbin
wget -O l2chroot http://www.cyberciti.biz/files/lighttpd/l2chroot.txt
chmod +x l2chroot

Edit the l2chroot file and change BASE=”/webroot” to BASE=”/var/jail”. This tells l2chroot where your jail is located so it copies everything to the right place. Now go ahead and run the command on the binaries you want.

l2chroot ls
l2chroot bash

Configure SSHd to Chroot your users

All that is left is to set a few things in your sshd configuration file. You need to make sure you have at least OpenSSH 4.8p1, because before that they didn't have this nice ChrootDirectory() function. Previously there was a few extra steps you had to take to get it working, but really you should have a newer version anyway. To configure ChrootDirectory add the following to /etc/ssh/sshd_config:

Match group sshusers
          ChrootDirectory /var/jail/
          X11Forwarding no
          AllowTcpForwarding no

Note that this also disables X11Forwarding and does not allow port forwarding. If you want to setup a box to allow secure tunneling for your friends, you may want to change this.

Optional Steps

When you login to your test user, you'll notice a prompt as such:

bash-3-2$

That is not a very useful bash prompt. So if you want something a little better I recommend simply copying the contents of /etc/skel to /var/jail/home/user. This gives you a .bashrc file which sets the PS1 variable to a much nicer looking prompt. Here's what mine looks like:

phrygian:~> echo $PS1
\h:\w>
phrygian:~>

Securing LAMP: Apache

Allan Feid — Fri, 30 Jan 2009 02:25:49 +0000

I've decided to start a little mini-series here on how to secure a LAMP environment. For those that don't know this is a Linux + Apache + MySQL + PHP environment. This week I'll look at Apache specifically and continue with the other parts of this environment early next month. Click the link below to check it out.

Hardening an Apache Server

Allan Feid — Fri, 30 Jan 2009 02:18:46 +0000

Apache is a very popular web server for a Linux environment. Just as most services visible on the Internet, you should implement ways of protecting yourself from malicious users. Here I discuss how I harden my Apache installations as installed through the apt-get package management system on Debian (should be the same on Ubuntu as well). This should easily be applied to other distributions but just note that the locations of files are probably different.

1: Disabling unnecessary modules

When you install apache through apt-get on a Debian based server, it comes with a few modules pre-enabled that give an attacker too much information about your system. These are the autoindex and status modules. Here's what they do:

autoindex: provides a nice list of all the files in a directory where no index file is given
status: gives you a nice little server monitoring web page to visit

There's more than one way to disable these modules. You can either remove the symlink from /etc/apache/mods-enabled/ or use a built in command that does the exact same thing.

a2dismod status
a2dismod autoindex

2: Change some of the default settings

By default, most applications give you settings that let them just work. To secure things up you usually have to edit those config files. Well, apache is no different. Open up /etc/apache2/apache2.conf and change the following:

TIP: use the find function of your editor to quickly find these options

ServerSignature Off
- Stops your error pages from showing your Apache version number
ServerTokens Prod
- Stops HTTP header information from including Apache version information
Remove or comment out entire ... "); ?> section
- You already disabled this module, no need to have it here
Remove the Alias /icons/ line along with the directory listing below
- These icons are only used by the autoindex module, no point in having these here

3: Add some custom configurations

Add these to /etc/apache2/httpd.conf. I'll explain what they do in the comments above each line.

"); ?>

# This is used to prevent the TRACE http method from 
# gathering path information from proxy or cache servers

  "); ?>

     deny from all
  ") ?>


# The '-' disables the following options
#  FollowSymLinks - could allow an attacker to browse
#  outise the doc tree
#
#  Includes - deals with .shtml; prevents server-side includes
#
#  AllowOverride None - prevents developers from changing
#  options with .htaccess files other places in the doc tree

  Options -FollowSymLinks -Includes -Indexes  -MultiViews
  AllowOverride None
  Order allow,deny
  Allow from all
"); ?>

Also, add this to the same file to prevent access to your root directory (/):

"); ?>

    Order deny,allow
    deny from all
"); ?>

4: Change some file permissions

Your public html folder (in Debian by default this is /var/www) should be only writable by root. It does still need to be world readable however so apply the following permissions:

chown -R root /var/www
chmod -R 775 /var/www

Summary

This will give you a pretty secure Apache server, but it's by no means complete and depends entirely on what you need to do on your server. This is just a list of my own personal recommendations and is what I use to setup a web server from scratch. The key to staying secure, however, is to keep your systems up-to-date and watch bug tracking mailing lists (Bugtraq for example) for any reports as they often supply workarounds and fixes for these issues.

Installing an Atheros Based Wireless Card on the Eee 901

Allan Feid — Thu, 25 Dec 2008 17:28:14 +0000

So, I got pretty fed up with the Ralink RA2860 wireless card that comes with my ASUS Eee 901. It does not play well with Backtrack 3 nor does it do packet injection. I want to turn this thing into a nice little security machine that I can bring with me and easily start testing some networks. In order to do that I got a new Atheros based card from Gigabyte. This card is known to work well with Backtrack and of course is highly supported in the Linux world. For those that care, it is an Atheros AR5007 chipset.

Once I got the part in the mail I opened it up to install:

Very simple to open, all you have to do is remove two screws and take off the plate. To remove the wireless card, take off the two pigtail connectors, which I have circled in red, then remove the two screws circled in blue. Finally move the two pigtail wires and you should see the card pop itself up.

Remove the MiniPCI Express card and insert your new wireless card. Then screw it back in and pop on the pigtail connectors. You are now good to go. If you're using Windows I'll assume you have to install the drivers included but on Ubuntu and Backtrack my card was recognized instantly.

Enjoy your new Atheros based wifi card!

Pandora Desktop on Adobe AIR

Allan Feid — Thu, 18 Dec 2008 19:54:41 +0000

So today Adobe AIR came out of beta for Linux. That is great news because I also came across the Pandora Desktop (currently in beta) in the same day. These two complement each other very well. It is great for removing that one Firefox window or tab that I always have open to Pandora.

Take a look at Pandora Desktop in action:

And here you can see the slick menu accessible from a nice icon in your tray:

Being that this is a beta it is still very useful and I highly recommend you give it a try. The only problem I had was getting the full view Pandora open every now and then. That's not really needed though since you can do pretty much everything through the menu on the icon.

Allan Feid - Linux

Create an inventory tool using MCollective's registration feature

Setting up a registration plugin

What do you do with this data?

Improving performance with a queue

The end result

Tags:

Capistrano for the sysadmin

Have a centralized ssh-key

Install the software

Configuring Capistrano

Tags:

Central login management with OpenLDAP

Some information about the network

Basic configurations

Install the needed software!

Configure rsyslog for logging

Importing some data to LDAP

Client configuration

Fedora 12

OpenSolaris

General Configuration for client's using pam_ldap/nss_ldap

Use ADS, make your life easier!

Tags:

Kernel Virtual Machines using libvirt

Installing the required software

Network configuration

Disk configuration

libvirt Configuration

virt-install, it's easy!

virsh

Tags:

Connect to iSCSI targets on Linux

Tags:

Creating a Chroot Jail for SSH Access

Setup your test user

Setup the jail directories

Configure SSHd to Chroot your users

Optional Steps

Tags:

Securing LAMP: Apache

Tags:

Hardening an Apache Server

1: Disabling unnecessary modules

2: Change some of the default settings

3: Add some custom configurations

4: Change some file permissions

Summary

Tags:

Installing an Atheros Based Wireless Card on the Eee 901

Tags:

Pandora Desktop on Adobe AIR

Tags: