Create an inventory tool using MCollective's registration feature

Allan Feid — Sat, 08 Oct 2011 22:05:09 +0000

I started working at a company (Shutterstock) that had MCollective implemented a few months ago. We use it to perform various functions across a wide range of servers. Things such as rolling the application servers to pick up new code, clearing caches, checking replication lag on databases, telling our puppetmaster to sign certificates, and even doing code deployment. There are endless possibilites with this system, and it is so simple to write your own agent/client utilities, that I'm not sure why more people aren't using MCollective. One of the things we were lacking was a central inventory of our systems in an easy to view manner. Thankfully, R.I. Pienaar's MCollective has a great feature called registration. Essentially, it causes your nodes to blast out to an ActiveMQ/rabbitmq topic at a set interval, and any node configured to listen can read that data and perform actions based on that data. With that in mind, I set out to create a database of information.

Setting up a registration plugin

Taking a look at this registration plugin, you can begin to see how easy generating your data set is. From this example, you get MCollective's agentlist, facts, and classes sent during registration. I wanted to also report on virtual machines running on hypervisors with an in-house libvirtd agent. To achieve that, your agent needs to have a simple method that can be called without the request/reply data structures. An example of that would be this:

module MCollective
  module Agent
    class Custom<RPC::Agent

      def some_method
        # gather some data and return it
      end

    end
  end
end

Once your agent has a method that can simply be called, you can call this method inside your registration plugin with something like this:

module MCollective
  module Registration
    class Meta<Base
      def body
        result = {
          :agentlist => [],
          :facts => [],
          :classes => []
        }

        cfile = Config.instance.classesfile
        if File.exists?(cfile)
          result[:classes] = File.readlines(cfile).map {|i| i.chomp}
        end

        result[:agentlist] = Agents.agentlist
        result[:facts] = PluginManager["facts_plugin"].get_facts

        if Agents.agentlist.include?("custom")
          # because of how PluginManager works, you need to 
          # append _agent to the agent's name
          result[:custom] = PluginManager["custom_agent"].some_method
        end

        result
      end
    end
  end
end

This way your nodes can report back with literally any information that you'd like to include. I've had great success reporting on my hypervisor's available disk, memory, and CPU utilization as well as what virtual machines are running.

What do you do with this data?

With inspiration from this mongodb registration agent, I decided MongoDB was a pretty good way to store data that was dynamicly generated. I wanted to make it easy to store/retrieve data as I saw fit, so using Mongoid seemed like the way to go. Something as simple as this is all you really need:

class Node
  include Mongoid::Document
  include Mongoid::Timestamps
  field :hostname, :unique => true, :type => String
  index :hostname
  validates_uniqueness_of :hostname
end

Improving performance with a queue

After running registration for a bit, memory usage by mcollective began to skyrocket on the machine processing the data, and eventually I started seeing timeouts or pool limits being hit on my mongo connection. I was able to talk a bit with R.I. Pienaar on IRC, and he informed me that every request that a node recieves spawns a new thread for an agent to run in. This is not scalable for the registration process, and once you hit over about 100 nodes, you run into problems since your machine is constantly opening up new threads to handle registration data. He did offer a solution that he uses personally and has suggested to others. That is to use a queue rather than a topic, and to have a simple daemon dedicated to processing registration data. There's some

Basically the mcollective user that you use from the client side needs to be able to read/write to the new queue. This also requires that your registration plugin sends the result out to the queue rather than a topic. Another simple change:

module MCollective
  module Registration
    class Meta<Base
      def body
        ...
        PluginManager["connector_plugin"].connection.publish("/queue/mcollective.registration", result.to_json)
        nil
      end
    end
  end
end

By having the body method return nil, you won't send anything out to the standard registration topic, which would be redundant in this case.

The end result

With a bit of tweaking, I was able to get my registration daemon/agent saving data into mongo in less than a second per request recieved. Once I had all that data in place, I was able to implement a slick interface using Sinatra and Bootstrap that gives me good insight into our inventory. I'll probably go more into the frontend interface when it becomes more mature, but once you have all this data, it's up to you to decide what you want to do with it.

Allan Feid - MCollective

Create an inventory tool using MCollective's registration feature

Setting up a registration plugin

What do you do with this data?

Improving performance with a queue

The end result

Tags: