The module – at /srv/www/html/test_auth/perl/Danny/loweruser.pm – looks like:

package Danny::loweruser;

1;


And the Apache config looks vaguely like this:

PerlSwitches -I/srv/www/html/test_auth/perl

AuthType basic
AuthName "Auth test"
AuthBasicProvider file
AuthUserFile "/srv/www/html/test_auth/userdb"
Require valid-user



Yeah, everything is in one directory. That’s not a secure way to do things; it’s a way to quickly test and easily clean up later. :)

I was testing with a PHP page that printes $_SERVER[“PHP_AUTH_USER”], so I’ve consequently pulled out most of my hair, because that stupid variable pulls from the HTTP headers instead of what the web server actually provides; it stays upper-case. The REMOTE_USER key is what I really wanted from the array. What I get from that is that PHP documentation is evil and anyone who uses PHP_AUTH_USER should be banned from writing code.

Actually, I’m not entirely clear on how PHP_AUTH_USER is getting set, and it’s late enough that I’m not going to dig in to it. This will work properly with anything that actually uses REMOTE_USER, per “everything since the 90’s except for PHP”, so it’ll be fine for what I actually needed.

I was in the process of migrating data in a replicated volume from one machine to another when the destination machine was interrupted (it was actually rebooted by an automated process kicked off by another admin; that’s what poor communication gets you). Then the destination machine wouldn’t boot. This machine mounted several gluster volumes from localhost, but glusterd wouldn’t start, which caused the boot process to hang on mounting.

So, one lesson is that mounting from the load balancer address might be better than localhost. :)

Anyway, the error I was getting was that the directory didn’t exist. Specifically, I saw this at the end of /var/log/glusterfs/etc-glusterfs-glusterd.vol.log:

[2016-03-02 15:34:16.093779] I [MSGID: 106513] [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706
[2016-03-02 15:34:16.500257] E [MSGID: 101032] [store.c:434:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/vols/sec_backup/bricks/newhostname:-srv-gluster-bricks-sec_backup-brick1-data. [No such file or directory]
[2016-03-02 15:34:16.500312] E [MSGID: 106201] [glusterd-store.c:3042:glusterd_store_retrieve_volumes] 0-management: Unable to restore volume: sec_backup
[2016-03-02 15:34:16.500357] E [MSGID: 101019] [xlator.c:428:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2016-03-02 15:34:16.500374] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed
[2016-03-02 15:34:16.500383] E [graph.c:661:glusterfs_graph_activate] 0-graph: init failed
[2016-03-02 15:34:16.500979] W [glusterfsd.c:1236:cleanup_and_exit] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xda) [0x405cba] -->/usr/sbin/glusterd(glusterfs_process_volfp+0x116) [0x405b96] -->/usr/sbin/glusterd(cleanup_and_exit+0x65) [0x4059d5] ) 0-: received signum (0), shutting down

Sorry about the formatting of that. It should probably be in a scrollable div, but I’ll have to fix that later.

Apparently the volume file on the new machine had been updated, but the brick directory hadn’t been created yet. The rest of the nodes in the pool still reflected the old location, and the volume still worked fine. But glusterd just refused to start.

As it turns out, this was really simple to fix. The volume in question was sec_backup, so I just did:

sudo rm -fr /var/lib/glusterd/vols/sec_backup

When glusterd started back up, it recreated the directory from the copy on one of the other nodes, and all was fine.

I have a manager_map table which maps a person’s assigned alias to a manager. If the person is new, I want to create a new entry. If the person is not new, though, I just want to update their manager to match the new value. Note that This query takes two parameters (manager’s alias is $1, subordinate’s alias is $2). Using the pg_placeholder_dollaronly parameter in the prepare allows me to put the parameters wherever in the query without worrying about the order they appear in the query string, and allows them to be duplicated in the query without having to specify them 2-3 times in the execute line later. This isn’t present in all DBI modules or even remotely portable, but you don’t need to do this on all RDBMS types, anyway. :)

$q_manager_set = $secdbh->prepare(q{
  WITH upsert AS (
    UPDATE public.manager_map
      SET manager = $1
      WHERE alias = $2
      RETURNING *
  )
  INSERT INTO public.manager_map
    (manager, alias)
    SELECT $1, $2
    WHERE NOT EXISTS (
      SELECT 1
      FROM upsert
      WHERE alias = $2
      )
}, {pg_placeholder_dollaronly => 1} );

The update query can be set to return the rows which were affected by the update. If no rows were updated, no rows will be returned. So, we take advantage of that by saving the results using a with clause (it could be done in the insert directly, but this is easier to read, IMHO). Then I construct a new temporary table consisting of the values I want to insert – which is just one set (one row) in this case. I subtract all of the rows where the primary key (alias, in this example) is in the list of rows affected by the update. Anything left gets inserted.

This works really nicely, because the update doesn’t complain if no rows are affected, and the insert doesn’t complain if no rows need to be inserted. It also works out well because it tries the update first, then the insert. The only race condition here is in the situation where rows might be deleted between update and insert. In my situation, this database is never deleted from. If yours is, though, all you have to do is lock the relevant table around this query, which is pretty straightforward.

To put this into use, I have a data structure defined earlier on which is an associative array where the key is the supervisor and the value is an array of direct reports. I also have autocommit turned off, as I want to make all of my updates in one transaction (for reasons beyond the scope of this post). So, the rest of the code (aside from populating the data structure from the input source, defining variables, etc) looks like this:

foreach $supervisor ( keys(%supervisors) ){
  foreach $alias ( @{ $supervisors{$supervisor} } ){
    $q_manager_set->execute( $supervisor, $alias )
        or print qq{manager_map:  Unable to add mapping '$alias:$supervisor': }
           . $q_manager_set->errstr . qq{\n};
  }
}
$secdbh->commit unless( $secdbh->{AutoCommit} );

Say you have four systems, and you want to distribute traffic evenly among them in a round-robin fashion. New connection 1 goes to server 1, new connection 2 goes to server 2, and so on. You can use the “nth” module for that. I want to do this with CFEngine, so I’ll use incoming port 5308, and will create a new chain to keep my rules somewhat clean:


sudo iptables -t nat -N CFENGINE
sudo iptables -t nat -F CFENGINE
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-m statistic --mode nth --every 4 \
-j DNAT --to-destination server1:5308
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-m statistic --mode nth --every 3 \
-j DNAT --to-destination server2:5308
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-m statistic --mode nth --every 2 \
-j DNAT --to-destination server3:5308
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-m statistic --mode nth --every 1 \
-j DNAT --to-destination server4:5308
sudo iptables -A PREROUTING -p tcp \
-m tcp --dport 5308 -j CFENGINE


How does that work? Connections coming in with port 5308 all go to the CFEngine chain. Every fourth packet (25%) will get redirected to server1. The other three packets will pass through that rule. So, you have 3/4 of the traffic hitting the next rule. Of those, every third packet will go to server 2, and the other two will pass on through. Of those two packets going through, one out of every two goes to server 3, and the other passes through. Then, finally, 1 out of every one packets (or “all of them”) goes to server 4. That last rule doesn’t need the “nth” thing, but I like to leave it in there for consistency.

You’ll see some documentation online showing “–every 4” on all four lines, but that doesn’t work; each rule is processing the remainder of packets which pass through the rules above, so you would end up with 25% of the traffic going to the first server, then 25% of 75% (about 19%) going to the second rule, etc. In current Linux kernels, the rules don’t share a counter – and they haven’t for years.

Those rules work in round-robin fashion. If you want to do something more complex with an uneven distribution, you can either find a common demoninator and alternate hosts, or use the probability module. With that one, the rule uses a random number for each packet which will match some percentage of the time. So, say you have the same four servers. You want servers 1, 2, and 3 to get about 30% of the incoming new traffic each, and server 4 should only get about 10%:

sudo iptables -t nat -N CFENGINE
sudo iptables -t nat -F CFENGINE
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-m statistic --mode random --probability 0.3000 \
-j DNAT --to-destination server1:5308
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-m statistic --mode random --probability 0.4286 \
-j DNAT --to-destination server2:5308
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-m statistic --mode random --probability 0.7500 \
-j DNAT --to-destination server3:5308
sudo iptables -t nat -A CFENGINE -p tcp \
-m conntrack --ctstate NEW \
-j DNAT --to-destination server4:5308
sudo iptables -A PREROUTING -p tcp \
-m tcp --dport 5308 -j CFENGINE


So, what’s the deal with those numbers? Well, it’s the same thing as the other one. Server 1 gets 30% of the traffic, so the probability should be 0.3 – which is 30/100. Then 70% passes through. So, server 2 only sees 70% of the traffic. So, it needs to match 30% of 70% – which is 30/70, or 0.428571. It’s matching about 43% of the remainder, which is 30% of the total. For server three, 60% of the traffic is gone now, so it gets 30/40, which is 0.75; in other words, 75% of the remainder is 30% of the total after 60% is gone. After that, we should have about 10% left, and we want to match 100% of the remainder. Even though I like consistency, I don’t want to allow any traffic to pass this chain (which would mean processing would go back to the prerouting chain, surprising everyone). So, I didn’t use a probability module on the last rule; it just always matches if traffic gets that far.

Basically, the value used is “percentage of whole / (100 – sum of previously matched percentages)”, which should be fairly easy to implement in any kind of automation script.

Even distribution would be easily attained by counting the target servers, and either doing “nth” rules while decrementing the number of servers as rules are added, or by dividing 100 by the total number of servers and using that as the target percentage.

Oh, you’ll also want to enable ip forwarding with sysctl, and you’ll likely need to use something like this
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
in order to make sure that the servers send responses back through you rather than directly connecting to the client.

To start with, here’s a query to just find the lowest unused value which is greater than 1000 (reserved for system IDs):

SELECT uid+1 AS uid FROM account.agtab WHERE uid > 1000
EXCEPT
SELECT uid FROM account.agtab WHERE uid > 1000
ORDER BY uid ASC
LIMIT 1

Here, the EXCEPT clause returns all of the entries from the first query which are not in the second query.  Pretty straightforward.  But, this only works when there are values in the database, and it only takes into account the uid field.  Going forward, I want every new account to have the same uid and gid value, which means I need to find the lowest value which is unused in both columns.  So, I’ll calculate a union of the uid and gid columns, and do the same thing with subqueries:

(
  SELECT uid+1 AS id FROM account.agtab WHERE uid > 1000
  UNION
  SELECT gid+1 AS id FROM account.agtab WHERE gid > 1000
)
EXCEPT
(
  SELECT uid AS id FROM account.agtab WHERE uid > 1000
  UNION
  SELECT gid AS id FROM account.agtab WHERE gid > 1000
)
ORDER BY id ASC
LIMIT 1

Almost there.  This solution requires that the tables have at least one value, and I don’t like writing solutions which fail in edge cases.  Initially I thought I’d use COALESCE, but then it occurred to me that it’d make more sense to just add another UNION statement with the bottom of the allowable range:

(
 SELECT uid+1 AS id FROM account.agtab WHERE uid > 1000
 UNION
 SELECT gid+1 AS id FROM account.agtab WHERE gid > 1000
 UNION
 SELECT 1001 AS id
)
EXCEPT
(
 SELECT uid AS id FROM account.agtab
 UNION
 SELECT gid AS id FROM account.agtab
)
ORDER BY id ASC
LIMIT 1

Hooray; the function returns the first value – 1001 – if the table is empty.  One could argue that the sane alternative would be to just use a sequence if the table was empty, but this would still be useful in a situation where people could manually insert arbitrary IDs into the table.  For example, perhaps you’re supporting a large variety of users, some of whom occasionally purchase software from incredibly bad vendors who write code which relies upon an account having a specific UID.  Not that such a thing has ever happened to me several times or anything… :/

But wait, there’s one more wrinkle.  The range 10,000 to 9,999,999 is reserved for a different purpose.  When my IDs get to 9,999, I need to jump to 10 million as the next available ID.  Thanks to using the union solution rather than a coalesce, I just add that second alternative lower bound value as well, and modify the where clause:

(
 SELECT uid+1 AS id FROM account.agtab WHERE (uid > 1000 AND uid < 9999) OR (uid >= 10000000)
 UNION
 SELECT gid+1 AS id FROM account.agtab WHERE (uid > 1000 AND uid < 9999) OR (uid >= 10000000)
 UNION
 SELECT 1001 AS id
 UNION
 SELECT 10000000 AS id
)
EXCEPT
(
 SELECT uid AS id FROM account.agtab
 UNION
 SELECT gid AS id FROM account.agtab
)
ORDER BY id ASC
LIMIT 1

It should be pretty visible from this how one could extend the approach to exclude multiple ranges.  It should also be pretty visible how this is gonna be a whole bunch of repeated typing.  So, let’s move it to a WITH statement:

WITH ids AS (
 SELECT uid AS id FROM account.agtab
 UNION
 SELECT gid AS id FROM account.agtab 
),
ids_plus_one AS (
 SELECT id+1 AS id
 FROM ids
 WHERE (id > 1000 AND id < 9999)
    OR (id >= 1000000)
)
(
  SELECT id FROM ids_plus_one
  UNION SELECT 1001 AS id
  UNION SELECT 1000000 AS id
)
EXCEPT ( SELECT id FROM ids )
ORDER BY id ASC
LIMIT 1

I think that’s pretty readable, which translates to being more maintainable.  I had a hard time finding an easy solution online (people were using integer tables and all sorts of weird stuff, and no one had anything which excluded ranges), so hopefully this helps someone. :)

Originally, I wrote the function roughly like this:

function append_managers {
  typeset -l user=${1:?}
  typeset -n arr=${2:?}
  typeset manager

  lookup "$user"
  manager="${manager[$user]}"
  lookup "$manager"
  arr[${email[$manager]}]=1
  is_vp "$manager" && return
  append_manager "$manager" "${!arr}"
}

I think this is pretty self-explanatory, but the way this should work is that the function takes a user name and an array name as mandatory arguments.  It uses a function (lookup()) to populate a few global arrays, including “manager”, “email”, and one with the title (so is_vp() can later determine if the person is a VP or below).  The manager’s email address is inserted into the associative array which was passed in, and then if the manager is not a VP or above, the function calls itself to continue climbing up the management chain.  The things in here which might be unfamilar are

• typeset -l, which forces the variable’s value to lowercase (which works better with my directory)
• typeset -n, which makes the variable a “nameref”, so using the variable affects a variable with the name of the nameref’s value.  It’s kind of like a pointer.  In this case, referencing $arr will actually reference the variable with the same name as what was passed in as the second argument.  If you do “a=’hello’;typeset -n b=’a’;print $b”; you should get “hello” in ksh93.
• ${!var}, which gets the name of the variable pointed to by a nameref.  In the case above of ${!arr} and a function call like “append_manager ‘danny’ ‘cc'”, that would expand to ‘cc’.

This last bullet point is where things get hung up.  The variable scoping in ksh93 with functions defined like “function f{}” (as opposed to “f(){}”) is lexical.  In general, lexical scoping means that the scope is like the code would read; if you fully expand the code by copying the whole functions into the places where they’re called, copy includes in place, etc so that you have one long procedure, then variables’ scope would be where they appear.  In this situation, that essentially translates to “variables defined in a function using typset are local to that function and to any functions called by that function.”  So, recursive calls to “append_manager” with the same array name should all be able to see the same original array.

Unfortunately, in the version of ksh93 on the machine where I’m developing this – a RHEL6 system with “AT&T ksh 93u+” – this does not work as expected.  On the second iteration through the recursion, I get an “arithmatic syntax error” when I try to use the nameref.  I tried a number of variations on the approach; I tried hard-coding the array name, I tried putting the array name into a variable and then using that variable to hold the nameref name (instead of $2), and I tried making argument $2 optional so recursive calls just provided a new name while using the nameref scoped within the first call to the function.  In all cases, I got either the same error or an empty array at the end.  I suspect that the recursion messes up the internal variable tracking.  Regardless of the cause, wWhat I finally ended up with was replacing the recursive call with this:

append_manager "$manager" "arr"

So, each call gets a new locally-scoped nameref variable pointing to the nameref array in its parent.  It makes me feel slightly ill to have a call where I’m basically doing “typeset -n a=a”, but since the new “arr” is in a different, lower scope than the first, they’re technically two separate variables.  It actually works, though, and even with various combinations of things I threw at it, it works properly.  So, that’s what’s in my final script – along with some comments hopefully saving a future programmer the time.

One final note: I mentioned that I used an associative array to avoid duplicates.  But that means I need to pull the indices (AKA “keys”) from the array in order to generate my final list of email addresses.  Considering an array named “cc”, the code to generate an Outlook-friendly (separated by semicolons – ugh) list of email address looks like this:

    OLDIFS=$IFS; IFS=";"
    print "cc: " "${!cc[*]}"
    IFS=$OLDIFS

Or like this:

    list=$( printf '; %s' "${!cc[@]}" )
    print "cc: ${list#??}"

The first option uses IFS to separate the arguments, which is limited to a single character.  The second option (the one I actually used), prints each element preceded by a semicolon and a space, then prunes the first two characters off of that list.  One could also use “${list:2}”, but I use the “prune” modifier more often than “substring” because I still have some older POSIX shell habits where the substring option wasn’t an option. :)

In order for this to integrate with CFEngine, I need to have a filesystem available that the policy masters can read from and distribute to clients.  I used to just check the repo out and have a scheduled job do an “svn update” every 5 minutes.  But I hate polling; event-driven solutions are almost always better.  In this case, that 5 minute polling not only was checking in to the subversion server way more often than changes actually happened, it also introduces another 5 minute maximum delay into policy propagation.  I want as close to real-time as I can get.  There’s also the problem of .svn directories occasionally getting corrupted, which requires another full checkout.

So, I could use one of the WebDAV filesystems out there.  The problems with those are first performance, as I’ve got over 110K files involved in the different policy branches and stuff.  Assuming the performance goes away, the other problem is that I use some SVN externals to move directories around; the policy isn’t laid out on the clients just like it’s laid out in Subversion.  The DAVfs implementations I could find don’t really deal well with externals.

Thus, I need to roll my own solution.  The idea is basically to have something in the post-commit and post-revprop-change hooks which will update the filesystem to match subversion whenever anything changes in the repo.  The second part is to have something watching the filesystem for unauthorized changes (i.e. anything not done through the hook script) and revert any such changes.

I’m teaching myself Python, so both solutions will be implemented in Python.  I’l host the code on github, since that seems to be what all the cool kids are doing today.  And there will eventually be a couple more blog posts here about how it works.  The links should show up in the comments below as trackbacks, or they’ll be in the subversion + python tag sections here.

This is also an excuse for me to learn Python, so bear with me if the code below is horrible.  Actually, don’t bear with me – leave a comment letting me know how it should have been done. :)

I’ll put the general code up here when it’s all done.  For now, though, one of the things I wanted to do was to run the trac-admin commands directly rather than having to make a system call to launch another python binary from within python.  I couldn’t find any documentation, but reading through the code, I ended up with two ways to do it.

One way is to use the console command tool, which is what runs when you type “trac-admin /path/to/trac command arg”:

import trac.admin.console
adm = trac.admin.console.TracAdmin('/srv/trac/instances/abc/')
adm.onecmd('permission list')

The other way is to skip over some of the line parsing and run what that eventually runs:

from trac.env import Environment
from trac.admin import AdminCommandManager
mgr = AdminCommandManager(
        Environment('/srv/trac/instances/abc/') )
mgr.execute_command('permission', 'list')

I’m not sure which is more portable to future versions, or if either one is.  But the AdminCommandManager method is ever so slightly faster, probably because there’s slightly less overhead.  I ended up putting that into a simple class like this:

from trac.env import Environment
from trac.admin import AdminCommandManager
class TracWrap:
    def __init__(self, env):
        self.mgr = AdminCommandManager(Environment(env))

    def trac_admin(self, *cmd):
        self.mgr.execute_command(*cmd)

And then in my post_commit hook script, I do this, which I think is more readable:

trac = TracWrap('/srv/trac/instances/abc/')
trac.trac_admin('changeset', 'added', repo, rev)

I couldn’t find anything when I searched Google for this, so maybe this will help someone else who’s using the same keywords I was using. :)

While it’s possible to use LDAP or define some variables on the central master, the main way configuration is done is by putting the policy into some files on the master and then allowing individual systems to copy those files down; the central master is basically just a fairly efficient file server.

There are basically two ways to manage the differences between individual systems in this “everything’s configured in a file” scenario.  One way is to encode all of the logic into the site policy, and have individual machines determine which parts of the policy apply to them, ignoring the rest.  This is probably the more common approach.  However, as a security person, it bothers me to have every system on my network contain the blueprints to the complete configuration of the entire network.  There are situations where this is not ok.  For example, defining which users have access to a system, defining account passwords (root or otherwise), defining network ACLs, etc. are all things that should only be known to specific machines.  We have several machines where someone has root access to a couple of machines but has no access to others.

So, the other way to manage this is to have separate configuration files for separate machines.  I accomplish this by having part of the policy in a “perhost” directory.  On the central master, the structure looks like “policy/perhost/hostname/users.cf”  Each system pulls down “policy/perhost/hostname” locally as “policy/perhost”.  Then, the common portions of the policy can simply refer to “perhost/users.cf” to get the details about which users to build on that particular machine.  It works well, and scales fairly well – as long as there aren’t too many hosts to keep in a single directory.  Modern filesystems like XFS have little issue with enormous directories, but it’d be pretty trivial to split the directory up like “perhost/h/hostname” as appropriate if need be.

This introduces a new problem, however.  With thousands of machines, there are thousands of versions of each config file under perhosts.  Managing such a structure is potentially quite a challenge.  If I want to define the same user on 500 hosts, then I may have to edit 500 files.  If I then want to change the user’s shell, now I need to make the same edit to 500 files.  That gets old really fast.  One solution to this issue would be to break everything down into the smallest reasonable parts, and use links.  So, maybe I have users/joe.cf, and perhost/host1/users/ contains a link to joe.cf.  That takes care of the data duplication, but is still an enormous filesystem to manually manage – and now I have links that also need maintained.  What I’m really trying to do is to implement a configuration management database here.  So, wait, why not actually use a database to begin with?

As I mentioned earlier, CFEngine can’t speak directly to a database for its configuration.  But using a database would be way easier than writing a zillion scripts to parse files and edit them.  All we need is a way to get the database to create files.  It’s not terribly difficult to implement a script which polls a database for changes and generates files, but then you have polling delays and quite a bit of time spent generating files which may not need regenerated.  I don’t like polling; it’s my view that things should be event-driven when possible.  So, one could implement triggers on update in the database which cause a filesystem to be updated.

That’s what I did initially.  However, because my CFEngine “master” is actually made up of a clustered trio of first-tier master servers and then several geographically-distributed secondary servers, I have the issue of file replication to contend with.  My goal is to have changes made into the database be essentially made available to the client systems as close to real time as possible. In order to reach this goal, it would be convenient if the filesystem could basically be a view into the database.

After pondering the problem a bit, I decided that I could write a FUSE implementation which took data from my database and presented it as a regular filesystem.  As I’m partial to Perl and the database is presently Postgres, I poked around CPAN a bit and found a module which had already done much of the work for me.  That module was written to allow editing databases through a filesystem interface, though, so it wasn’t exactly what I wanted in terms of performance and functionality.  But It was close. It works basically by having two SELECT staments defined.  One statement returns rows with values like “path/to/file” along with attributes like the size and a unique identifier (the equivalent to an inode).  The other statement takes a unique identifier and returns the file’s content.

I set up a view on my database to lump a few SELECTs together with a union, allowing management of the virtual filesystem structure by manipulating the view in the database rather than having to modify the FUSE client every time I wanted to add a new file.  The content SELECT statements either use a .cf file template and fill in some values from the database, or they populate a file with lines of data which will be read into an slist later.  With the new version of CFEngine, JSON data is supported, so I’m particularly looking forward to that.

Once this was all working, I found that performance wasn’t great.  FUSE is already not exactly lightening-fast, but when throwing in the overhead of querying the database repeatedly, it’s even more not-great.  So, I implemented caching in the perl script.  The directory contents and file contents were stored in a hash, and the hash was referenced when data was needed.  This works better, but now I have an issue with cache invalidation.  I have to time-out data in my cache, and that means I’m basically back to polling for changes.  I’m also duplicating the same cached data across every policy server.  That’s inefficient. The next step is to implement a smarter caching system.

The memcached system is pretty widely accepted, and has an easy-to-use API.  Most of my data’s relatively small.  So, now I’m populating data in memcached spread out across the servers.  I only need to duplicate caching across geographic regions, rather than across all machines in a region.  But what about invalidation?  Part of the memcached API is the ability to invalidate data.  So, I can set up a trigger in the database to invalidate the things which relate to changed data, and I can even have the same trigger go ahead and preload the changed data into the cache.  Most of the time, this results in the clients hitting just the cache and not even needing to see the database.  If a trigger fails for whatever reason, though, only one client has to hit the database to populate.  I still set a timeout on data in the cache as well, so in the worst case there’s just some minor delay before data propagates; it doesn’t stay broken forever.

Right now, this is somewhat theory.  I’m working on implementing it (the memcached part) over the next couple of weeks, and applying some real stress to it.  I think it’s a solid plan, though.  Let me know if you disagree. :)

For the baseline, I use “find | xargs chgrp”, which is slightly slower than “chgrp -R”, but not much slower (and, in my mind, slightly more fair). I then use a simple CFEngine policy and a simple Puppet policy to do the same thing.  The summary?  Puppet is dog slow at file recursion, while CFEngine is nearly as fast as pure find.  CFEngine actually uses less memory than the shell when you get to many files (probably due to the pipe to xargs), and Puppet wastes memory like it’s been surfing the web for weeks using an old version of Firefox.

Here’s the CFEngine policy file (using local bodies rather than the standard library, to be fair):

bundle agent do_perms {
files: 
  "/srv/gluster/sec/tmp"
    depth_search => recurse( "inf" ),
    perms => g( "security" );
}

body depth_search recurse(d) {
  depth => "$(d)";
}

body perms g(G) {
  groups => { "$(G)" };
}

Here’s the Puppet manifest (yeah, it’s slightly simpler):

class{ "test": }

class test {
  file { "/srv/gluster/sec/tmp/test":
    recurse => "inf",
    group => "security",
  }
}

and here’s the test script:

#!/usr/bin/ksh
################################################################################
# compare CFEngine, Puppet, and find
################################################################################

# note that $D will get deleted between runs
D=/srv/gluster/sec/tmp/test
#S=$(basename "$0")
S="test_file"
CF=$( rpm -q --qf 'CFEngine %{VERSION}\n' cfengine-community )
PU=$( rpm -q --qf 'Puppet %{VERSION}\n' puppet )

function clean {
  rm -fr "$D"
}
function create {
  mkdir -p "$D"
  seq 1 $1 | sed "s?^?$D/?; s?\$?.$S?" | xargs touch
}

alias t="/usr/bin/time -f '%E, %MKB, %U user, %S kernel'"

for N in 1 10 100 1000 2000 5000 10000 20000 30000 50000 100000 200000 500000
do
  clean
  create $N
  # system test
  print -n "$N, system set, "
  t \
  sh -c "find '$D' -type f -not -group security -print0 | xargs -0 chgrp security"
  print -n "$N, system check, "
  t \
  sh -c "find '$D' -type f -not -group security -print0 | xargs -0 chgrp security 2>/dev/null"

  # CFEngine test
  clean
  create $N
  print -n "$N, $CF set, "
  t /var/cfengine/bin/cf-agent -K -b do_perms -f ./cfengine_v_puppet.cf
  print -n "$N, $CF check, "
  t /var/cfengine/bin/cf-agent -K -b do_perms -f ./cfengine_v_puppet.cf

  if [[ $N -le 10000 ]] # it's just too slow
  then
    # Puppet test
    clean
    create $N
    print -n "$N, $PU set, "
    t /usr/bin/puppet apply --logdest /dev/null cfengine_v_puppet.pp
    print -n "$N, $PU check, "
    t /usr/bin/puppet apply --logdest /dev/null cfengine_v_puppet.pp
  fi
done
clean

The results are clear.  CFEngine can verify permissions on a half million files in about the amount of time it takes for Puppet to start up, and can set perms on a quarter million files before Puppet can even start up to check one.  Further CFEngine’s memory consumption remains constant while it’s doing this.

$ ./cfengine_v_puppet.ksh
1, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel
1, system check, 0:00.00, 5488KB, 0.00 user, 0.00 kernel
1, CFEngine 3.3.4 set, 0:00.02, 17536KB, 0.00 user, 0.01 kernel
1, CFEngine 3.3.4 check, 0:00.02, 17520KB, 0.00 user, 0.01 kernel
1, Puppet 2.7.16 set, 0:06.20, 374752KB, 5.14 user, 0.91 kernel
1, Puppet 2.7.16 check, 0:06.22, 374528KB, 5.11 user, 0.97 kernel
10, system set, 0:00.00, 5504KB, 0.00 user, 0.00 kernel
10, system check, 0:00.00, 5488KB, 0.00 user, 0.00 kernel
10, CFEngine 3.3.4 set, 0:00.02, 17568KB, 0.00 user, 0.01 kernel
10, CFEngine 3.3.4 check, 0:00.02, 17520KB, 0.01 user, 0.00 kernel
10, Puppet 2.7.16 set, 0:06.28, 378336KB, 5.14 user, 0.96 kernel
10, Puppet 2.7.16 check, 0:06.19, 378192KB, 5.09 user, 0.94 kernel
100, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel
100, system check, 0:00.00, 5520KB, 0.00 user, 0.00 kernel
100, CFEngine 3.3.4 set, 0:00.03, 17568KB, 0.01 user, 0.01 kernel
100, CFEngine 3.3.4 check, 0:00.02, 17552KB, 0.01 user, 0.01 kernel
100, Puppet 2.7.16 set, 0:07.77, 379184KB, 6.53 user, 1.10 kernel
100, Puppet 2.7.16 check, 0:07.18, 382912KB, 5.97 user, 1.05 kernel
1000, system set, 0:00.01, 5504KB, 0.00 user, 0.01 kernel
1000, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel
1000, CFEngine 3.3.4 set, 0:00.05, 17680KB, 0.02 user, 0.02 kernel
1000, CFEngine 3.3.4 check, 0:00.03, 17664KB, 0.01 user, 0.01 kernel
1000, Puppet 2.7.16 set, 0:23.47, 610880KB, 20.60 user, 2.65 kernel
1000, Puppet 2.7.16 check, 0:17.62, 515184KB, 15.31 user, 2.11 kernel
2000, system set, 0:00.02, 6592KB, 0.00 user, 0.02 kernel
2000, system check, 0:00.00, 6560KB, 0.00 user, 0.00 kernel
2000, CFEngine 3.3.4 set, 0:00.07, 17664KB, 0.03 user, 0.03 kernel
2000, CFEngine 3.3.4 check, 0:00.04, 17648KB, 0.01 user, 0.02 kernel
2000, Puppet 2.7.16 set, 0:45.38, 668560KB, 40.77 user, 4.29 kernel
2000, Puppet 2.7.16 check, 0:32.01, 626832KB, 28.34 user, 3.46 kernel
5000, system set, 0:00.06, 9968KB, 0.01 user, 0.05 kernel
5000, system check, 0:00.01, 9936KB, 0.00 user, 0.01 kernel
5000, CFEngine 3.3.4 set, 0:00.14, 17664KB, 0.05 user, 0.07 kernel
5000, CFEngine 3.3.4 check, 0:00.07, 17648KB, 0.04 user, 0.02 kernel
5000, Puppet 2.7.16 set, 2:08.18, 1337632KB, 118.14 user, 9.51 kernel
5000, Puppet 2.7.16 check, 1:31.71, 1191152KB, 84.26 user, 6.95 kernel
10000, system set, 0:00.12, 15600KB, 0.02 user, 0.12 kernel
10000, system check, 0:00.03, 15552KB, 0.00 user, 0.02 kernel
10000, CFEngine 3.3.4 set, 0:00.24, 17664KB, 0.11 user, 0.11 kernel
10000, CFEngine 3.3.4 check, 0:00.12, 17648KB, 0.07 user, 0.04 kernel
10000, Puppet 2.7.16 set, 5:36.38, 2457280KB, 316.83 user, 18.17 kernel
10000, Puppet 2.7.16 check, 4:20.40, 1766560KB, 246.25 user, 12.97 kernel
20000, system set, 0:00.25, 26832KB, 0.03 user, 0.27 kernel
20000, system check, 0:00.06, 26800KB, 0.00 user, 0.05 kernel
20000, CFEngine 3.3.4 set, 0:00.45, 17664KB, 0.22 user, 0.22 kernel
20000, CFEngine 3.3.4 check, 0:00.22, 17648KB, 0.14 user, 0.07 kernel
30000, system set, 0:00.43, 38080KB, 0.06 user, 0.41 kernel
30000, system check, 0:00.08, 38048KB, 0.02 user, 0.06 kernel
30000, CFEngine 3.3.4 set, 0:00.71, 17664KB, 0.28 user, 0.38 kernel
30000, CFEngine 3.3.4 check, 0:00.31, 17648KB, 0.17 user, 0.13 kernel
50000, system set, 0:00.75, 60592KB, 0.12 user, 0.73 kernel
50000, system check, 0:00.14, 60560KB, 0.02 user, 0.11 kernel
50000, CFEngine 3.3.4 set, 0:01.17, 17680KB, 0.51 user, 0.61 kernel
50000, CFEngine 3.3.4 check, 0:00.52, 17648KB, 0.32 user, 0.19 kernel
100000, system set, 0:01.30, 116832KB, 0.22 user, 1.36 kernel
100000, system check, 0:00.28, 116800KB, 0.05 user, 0.22 kernel
100000, CFEngine 3.3.4 set, 0:02.25, 17680KB, 0.98 user, 1.21 kernel
100000, CFEngine 3.3.4 check, 0:01.01, 17648KB, 0.62 user, 0.35 kernel
200000, system set, 0:02.87, 229328KB, 0.45 user, 2.86 kernel
200000, system check, 0:00.58, 229296KB, 0.09 user, 0.47 kernel
200000, CFEngine 3.3.4 set, 0:04.64, 17664KB, 1.95 user, 2.46 kernel
200000, CFEngine 3.3.4 check, 0:02.01, 17648KB, 1.25 user, 0.74 kernel
500000, system set, 0:07.48, 566832KB, 1.09 user, 7.45 kernel
500000, system check, 0:01.44, 566800KB, 0.26 user, 1.16 kernel
500000, CFEngine 3.3.4 set, 0:11.75, 17664KB, 4.97 user, 6.24 kernel
500000, CFEngine 3.3.4 check, 0:05.01, 17648KB, 3.19 user, 1.80 kernel

And again with CFEngine 3.5.1p3, the most current release as of this writing:

1, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel
1, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel
1, CFEngine 3.5.1 set, 0:00.02, 17920KB, 0.01 user, 0.01 kernel
1, CFEngine 3.5.1 check, 0:00.02, 17936KB, 0.01 user, 0.00 kernel
10, system set, 0:00.00, 5504KB, 0.00 user, 0.00 kernel
10, system check, 0:00.00, 5488KB, 0.00 user, 0.00 kernel
10, CFEngine 3.5.1 set, 0:00.03, 17952KB, 0.01 user, 0.01 kernel
10, CFEngine 3.5.1 check, 0:00.02, 17920KB, 0.01 user, 0.01 kernel
100, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel
100, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel
100, CFEngine 3.5.1 set, 0:00.03, 17936KB, 0.01 user, 0.01 kernel
100, CFEngine 3.5.1 check, 0:00.03, 17920KB, 0.01 user, 0.01 kernel
1000, system set, 0:00.01, 5520KB, 0.00 user, 0.01 kernel
1000, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel
1000, CFEngine 3.5.1 set, 0:00.08, 18016KB, 0.04 user, 0.03 kernel
1000, CFEngine 3.5.1 check, 0:00.05, 17984KB, 0.03 user, 0.01 kernel
2000, system set, 0:00.03, 6592KB, 0.00 user, 0.02 kernel
2000, system check, 0:00.00, 6560KB, 0.00 user, 0.00 kernel
2000, CFEngine 3.5.1 set, 0:00.11, 18000KB, 0.06 user, 0.04 kernel
2000, CFEngine 3.5.1 check, 0:00.07, 18000KB, 0.04 user, 0.02 kernel
5000, system set, 0:00.06, 9968KB, 0.01 user, 0.05 kernel
5000, system check, 0:00.01, 9952KB, 0.00 user, 0.00 kernel
5000, CFEngine 3.5.1 set, 0:00.22, 18016KB, 0.13 user, 0.08 kernel
5000, CFEngine 3.5.1 check, 0:00.13, 17984KB, 0.09 user, 0.03 kernel
10000, system set, 0:00.12, 15584KB, 0.01 user, 0.12 kernel
10000, system check, 0:00.03, 15552KB, 0.00 user, 0.01 kernel
10000, CFEngine 3.5.1 set, 0:00.40, 18000KB, 0.24 user, 0.15 kernel
10000, CFEngine 3.5.1 check, 0:00.24, 18000KB, 0.17 user, 0.06 kernel
20000, system set, 0:00.26, 26832KB, 0.04 user, 0.26 kernel
20000, system check, 0:00.05, 26800KB, 0.01 user, 0.04 kernel
20000, CFEngine 3.5.1 set, 0:00.79, 18016KB, 0.50 user, 0.27 kernel
20000, CFEngine 3.5.1 check, 0:00.47, 18000KB, 0.33 user, 0.13 kernel
30000, system set, 0:00.38, 38080KB, 0.06 user, 0.39 kernel
30000, system check, 0:00.08, 38048KB, 0.01 user, 0.07 kernel
30000, CFEngine 3.5.1 set, 0:01.21, 18016KB, 0.74 user, 0.44 kernel
30000, CFEngine 3.5.1 check, 0:00.69, 17984KB, 0.53 user, 0.14 kernel
50000, system set, 0:00.70, 60592KB, 0.09 user, 0.70 kernel
50000, system check, 0:00.14, 60560KB, 0.02 user, 0.11 kernel
50000, CFEngine 3.5.1 set, 0:01.94, 18000KB, 1.21 user, 0.70 kernel
50000, CFEngine 3.5.1 check, 0:01.15, 18000KB, 0.86 user, 0.25 kernel
100000, system set, 0:01.32, 116832KB, 0.22 user, 1.35 kernel
100000, system check, 0:00.28, 116816KB, 0.04 user, 0.23 kernel
100000, CFEngine 3.5.1 set, 0:03.93, 18016KB, 2.45 user, 1.38 kernel
100000, CFEngine 3.5.1 check, 0:02.23, 18000KB, 1.62 user, 0.59 kernel
200000, system set, 0:02.70, 229328KB, 0.42 user, 2.71 kernel
200000, system check, 0:00.56, 229296KB, 0.10 user, 0.44 kernel
200000, CFEngine 3.5.1 set, 0:07.96, 18016KB, 5.00 user, 2.77 kernel
200000, CFEngine 3.5.1 check, 0:04.56, 17984KB, 3.38 user, 1.14 kernel
500000, system set, 0:07.17, 566832KB, 1.14 user, 7.07 kernel
500000, system check, 0:01.38, 566800KB, 0.26 user, 1.11 kernel
500000, CFEngine 3.5.1 set, 0:20.63, 18016KB, 12.95 user, 7.11 kernel
500000, CFEngine 3.5.1 check, 0:11.09, 18000KB, 8.41 user, 2.63 kernel
1000000, system set, 0:15.29, 1129328KB, 2.29 user, 15.13 kernel
1000000, system check, 0:02.96, 1129296KB, 0.56 user, 2.38 kernel
1000000, CFEngine 3.5.1 set, 0:42.48, 18016KB, 25.03 user, 16.32 kernel
1000000, CFEngine 3.5.1 check, 0:22.29, 17984KB, 16.60 user, 5.62 kernel