Code Available - A python module for monitoring a beanstalkd instance with ganglia (opensource)
As part of my FidoFetch project, I’m using beanstalkd to do a lot of background work. It’s really awesome and is super simple. I’m using it as a worker queue for a lot of the feed catching, processing and recommending. Without it fidofetch would be super slow and would not be able to use some of the cool algorithms to recommend articles.
There is a problem with an asynchronous model though. It can be a lot trickier to monitor. Workers can be in funky states or defunct. Right now I do have the workers (I hope to one day opensource my worker module) all logging to syslog.
From there I can do the tail -f, but that only gets you so far. What that doesn’t tell you is how many jobs are waiting to be run, how many failed, how many workers are active etc.
##Enter Ganglia## Ganglia is also pretty cool. It’s not quite as simple as beanstalk but it’s pretty minimalist for what it does. It’s a distributed monitoring solution. Ilya Grigorik wrote up a nice article on ganglia (albeit with ruby). Apparently they use it over at postrank.
So it turns out ganglia has a neat system where you can write a module for it in python! Which is awesome since I don’t need to learn another DSL for monitoring.
##Show Me The Code## I wrote a simple plugin for ganglia on github, check it out and let me know what you think…