For awhile now, we’ve been in a foggy area with our monitoring and alerting. I’ve been aware for months that we should be doing more than we currently are. We’ve been running Nagios for years and do an okay job with it in basic form. We also have Ganglia metrics being collected for all of our servers, but they are largely ignored until needed for debugging. And finally, I’ve also just started seriously evaluating Graphite + various collectors as of 2 weeks ago. We do not currently perform any alerting based on the data in Ganglia. It is all a slowly moving work in progress.

Today, I just so happened to have a look at Ganglia and noticed 2 servers with elevated status (showing as yellow instead of green). They were both OpenAFS “database” servers (we have 4), and for the sake of not sidetracking this blog post, are largely depended on for providing OpenAFS storage volume location information.

It’s worth pointing out that there was no noticeable OpenAFS service degradation to the clients of the affected servers.

We’ll only look at one because they both suffered the same problem. Below, you can see to the left side of the graphs the scene I was presented at the time I noticed the problem. We have completely flat load that is elevated, completely flat low CPU idle time, and a completely flat stream of ~360KB/sec total network traffic:

Though the graphs above show the last 4 hours, digging further back in time (not shown here) showed that this situation started at around 9AM on April 9th. The flat graph shape seen above at the left of all 3 graphs was found in our data from that day and time all the way to today! Hmmm. What changed around 9AM on April 9th? A little digging through my email and our revision control history showed absolutely nothing, so it was time to dive in with no hints.

I found the OpenAFS Volume Location server process, vlserver, eating a steady ~60% of the CPU on each of the 2 servers in question. Additionally, looking into the flat ~360KB/sec network data with snoop showed a solid stream of OpenAFS UDP packets coming from port 7001 on a client node to port 7003 (vlserver!) on this server:

# OpenAFS client request to vlserver then response back
rogle.ourdomain -> shiva.ourdomain UDP D=7003 S=7001 LEN=56
shiva.ourdomain -> rogle.ourdomain UDP D=7001 S=7003 LEN=40

Very rough calculations indicate at least 2500 of these exchanges were happening per second.

Was this legit traffic from some long-running job created by one of the engineers? Looking more closely with snoop -x showed that the request and response were identical every time. Something was clearly broken somewhere on host rogle.

Once logged into rogle, I found 2 bash processes with high CPU utilization time. One is shown below.

jfivale  19137     1  0 Mar07 ?        03:43:03 bash

That’s not so interesting on its own, but adding in the detail that bash for our users is actually /afs/rcf.ourdomain/some/path/bin/bash changes things quite a bit.

Brain says:

Wait a second. Some number of days ago we had a weird situation where we had to kill off 5-10 bash processes on several hosts due to them eating loads of CPU time. They had eaten far more CPU time than these, but this is getting familiar.

Sure enough, when I attached to the processes with strace -f, they showed the same behavior as the broken processes a week or so ago:

...
rt_sigreturn(0x7)                       = 0
--- SIGBUS (Bus error) @ 0 (0) ---
rt_sigreturn(0x7)                       = 0
--- SIGBUS (Bus error) @ 0 (0) ---
rt_sigreturn(0x7)                       = 0
--- SIGBUS (Bus error) @ 0 (0) ---
rt_sigreturn(0x7)                       = 0
--- SIGBUS (Bus error) @ 0 (0) ---
rt_sigreturn(0x7)                       = 0
--- SIGBUS (Bus error) @ 0 (0) ---
rt_sigreturn(0x7)                       = 0
--- SIGBUS (Bus error) @ 0 (0) ---
...

or alternatively:

...
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0xb)                       = 316804680
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0xb)                       = 316804680
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0xb)                       = 316804680
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0xb)                       = 316804680
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
...

Once I had killed off the offending bash processes, both of the affected servers returned to the normalcy you see in the right-hand portion of the graphs above.

The question then was: What was going on with the bash processes? Unfortunately, I have no answer. The bash processes that spiraled out of control all belonged to a small subset of a certain department’s users and no other problematic instances of the same binary on the same hosts were reported.

Alert on things that are abnormalities in your particular environment. Over 2500 volume location lookups per second is not normal for us. A flat and consistent load on our OpenAFS servers is also not normal. This should have been noticed and fixed within a few hours on April 9th.

Who did what now?

Apparently first showing its face as part of Eric Allman’s sendmail, syslog is available from the ubiquitous logger(1) UNIX/Linux command to the Log4j Java library and everywhere in-between. Syslog provides simplistic fire-and-forget UDP[1] messages with a timestamp.

We’ve had a centralized syslog setup for a good 13 years now. Why not use syslog over UDP to deliver metrics due to its nearly zero barrier to entry for developers, systems, and sysadmins?

We’ve been very passively using Ganglia for what amounts to our largely “get something going” metric gathering effort. Having heard about Graphite (technically the home is here but… ew) many times now and having played with it briefly a few months ago, I decided to see what all the hubbub was about.

After about 2 hours of fussing around and reading[2], I had Graphite up and functioning and… there it was. Oh, right. I have to somehow feed it data.

There’s this StatsD thing I keep hearing in the same breath as Graphite. Turns out it is for (drum roll) allowing special UDP fire-and-forget metric packets to be sent to it where the daemon aggregates data before sending calculated statistics off to Graphite’s carbon backend. We don’t have a use for that right now and it’s yet another thing to run and maintain, so I’m passing it over for the time-being.

Right. I like using, when possible, what is widely available instead of distributing new package or library dependencies to all nodes in our infrastructure. And so, the goal is: Figure out how to get arbitrary metrics (zero configuration per new metric desired) sent to a central location via syslog and get graphs.

Enter Logster

It became clear from the generous replies to my request for comments that, at least for this proof of concept effort, Logster was going to play a role.

Logster is typically instantiated via cron with command-line options indicating a “parser” to use and a log file to watch. It determines (via its logtail dependency) what was last seen in the log file from the previous cron run, and feeds all new lines through the specified parser. The parser (essentially a custom plugin you write in Python) is responsible for returning instances of MetricObject. If you’re curious about the parsers right now, have a look at one of the samples from the source: parsers/SampleLogster.py

syslogmaster% sudo yum install -y logcheck
syslogmaster% cd /tmp
syslogmaster% sudo wget -q --no-check-certificate -O logster.tar.gz https://github.com/etsy/logster/tarball/master
syslogmaster% sudo tar xzf logster.tar.gz
syslogmaster% cd logster*
syslogmaster% vi logster # look around for crap to change and change it
syslogmaster% make install
syslogmaster% cd parsers && echo "Now write a parser..."

On our syslog master where this was installed, I cut my teeth on a simple parser for our Cyrus IMAP server, copied it to /usr/share/logster where it could be picked up properly, and made a per-minute cron job:

0-59 * * * * /usr/sbin/logster -o graphite --graphite-host=rcf-metrics:2003 CyrusIMAPSyslog /speaker/log/auth.log

Rad. A graph. I think this server is a little underutilized:

The MetricLogster Parser

Since I had Logster successfully feeding working data into Graphite’s mouth, I set out to the real task at hand and crafted my “MetricLogster” parser, which can be found in my fork of the Logster repo on github (etsy/logster pull request 12 has been filed).

Interlude: Syslog snag!

If you syslog frequently enough, you get the dreaded last message repeated N times lines in your log file and not actual data. I worked around this by switching our syslog master server from RHEL 5′s syslogd to a hand-build of rsyslog (which comes as default in RHEL 6). The rsyslog package has a $RepeatedMsgReduction global directive which one can set to off to always log data and never log those annoying messages.

Back to The MetricLogster Parser

Let’s test this thing out.

Generate syslog data with random numbers from bash’s builtin $RANDOM:

client% while :; do \
logger -t whatever -p local1.info metric=test.bash.random value=$RANDOM; \
sleep 2; \
done

On the syslog master server, instead of waiting for cron every minute (since we’re testing), run Logster every 3 seconds and feed our metric data into Graphite/carbon:

syslogmaster% while :; do \
sudo /usr/sbin/logster -o graphite --graphite-host=rcf-metrics:2003 \
                       MetricLogster /speaker/log/local1.log ; \
sleep 3; \
done

Generates this kooky but successful graph:

Notes and Ideas

• rsyslog supports input and output modules. If there was an output module for rsyslog that wrote directly to Graphite (technically carbon-cache or carbon-relay), we could remove Logster, its logcheck dependency (for logtail) and the custom Logster parser module (MetricLogster) from the the whole picture.
• If StatsD accepted well-formed basic syslog data, that would be slick too.
• I have no idea how well Logster scales.
• The keys in the metric syslog lines could really be shortened to m and v (from metric and value) to save a 10 bytes per log line.
• I am an infant in this arena.
• My statistics knowledge is 1/100th what it needs to be.
• There needs to be a Graphite and Friends resource hub/wiki
• Sure, this isn’t a gigantic achievement of mine, but words here are free and I’d be remiss to not say: Thanks to the Etsy and Orbitz engineers who have made Logster and Graphite freely available, the rsyslog developers, all of the generous people who offered comments to my previous post on this topic, and all of the noisy people I follow on Twitter who share info and/or provide me with ideas or answers weekly.

Footnotes

1. Unless TCP is specifically configured as the transport where supported. Use whatever suits your needs – the information above still applies.
2. If the graphite-dev team at Launchpad would be so kind as to let me join the club, I would happily submit patches to a lot of things I found wrong, even if I do have to learn stupid Bazaar. It’s been a week now or so since my request to join. Do you guys want help or not? ;)

Idea

Collect Graphite (or Graphite-style) centralized metrics (anything you can think of) via syslog. We don’t need a new way to send and collect small UDP messages.

Why

“Why not?” is the question really.

“Why?” is easy:

1. Uses as much omnipresent tooling as possible.
2. Uses established system library calls instead of requiring developers to write, install, or copy/paste new ones.
3. Because I found myself asking the following this past week: “Why am I about to build and package GNU netcat, for distribution to all of our Solaris 10 boxes, to get remarkably simple frigging UDP packets sent from shell commands?”

How: Client (Ideas)

Standard syslog configuration:

well-known-facility.well-known-severity        @metricshost

Developers use standard syslog calls specifying the well-known-facility and well-known-severity

For non-real-time metrics from “shell land”, just use logger(1)

How: Server (Ideas)

Tweak rsyslog or other well-known syslog “collector” product to:

1. Parse basic Graphite or Graphite-like metric messages and perform RRD and/or Whisper writes. Additionally could implement built-in stats aggregation stuff like StatsD quite easily.
2. Ignore all non-metric-conforming syslog data

Thanks for any thoughts below or via the original thread that links here.

All I can say is: This worked for me.

I’ve turned off comments on this post, because I don’t want to even begin to suggest that I will have any answers to any questions you may ask.

# .bash_profile
#
# I recursively copied /usr/share/lib/terminfo on a modern
# Linux box to $HOME/.terminfo so that I could have modern
# stuff with me wherever I go, like Solaris 10 which has
# no modern 256 color crap.  Make sure you have the screen-256color
# terminfo stuff.  Then...
TERMINFO=$HOME/.terminfo
# xterm-256color should work below as well, but since I am
# always connecting from PuTTY, I use this which is technically
# more correct.
TERM=screen-256color

# .screenrc portion for GNU screen which MUST BE compiled with 256
# color support.
#
term "screen-256color"

" .vimrc portion
"
" "People" say this should never be required if your terminfo crap is
" correct, but it is required for me *when running vim under GNU screen*
set t_Co=256

" Enable syntax highlighting
syntax enable

" Use whatever you like here
colorscheme lucius

" Dunno, this seems to be the only thing that leaves my terminal in a
" proper state once I exit vim in 256-color mode under GNU screen
" when using either TERM=putty-256color or TERM=xterm-256color.  Found it
" mentioned in some IRC log after digging through Google results for an
" hour or more.
set t_ti= t_te=

Those reading the article, please be sure to also read any comments to the article, as others may have additional ideas, solutions, or helpful information.

As you can see from the documentation links in this post, I am working with version 1.3. I am not aware of any differences in 1.3.1 or 1.4.0 regarding the topic covered in this post.

Scenario

You have a Django model. That model has a field which is a ManyToManyField. When an instance of your model is modified via any mechanism (custom web form, Django admin interface, code using the ORM, etc), you want to execute some custom code based on what happened to the model instance’s fields (including the ManyToManyField).

Specifically, what if you need to keep a non-Django source in sync with a Django model’s data? In my case, I have a “simple” need to push changes to a Django model’s data to an LDAP server.

class Interface(models.Model):
    fqdn = models.CharField(primary=True,
                            verbose_name="Fully Qualified Domain Name",
                            max_length=80)

class Netgroup(models.Model):
    name = models.CharField(primary=True,
                            max_length=80)

    interfaces = models.ManyToManyField(model=Interface,
                                        null=True,
                                        blank=True)

Did NetgroupInstance.interfaces change? What changed? Was 1 interface removed from 500? What was that 1 interface? Were 9 interfaces added to the existing 15 interfaces? What were the 9 interfaces? Was this a brand new Netgroup creation with interfaces added at creation-time? What were the interfaces?

Solution 1?

Just override the Netgroup save() method! Call the superclass’s save first, then run your custom code against the saved data.

FAILS: The ManyToManyField data is not reliable at this point in the code flow for some reason. See the first few posts in my thread here.

Solution 2?

Maybe try Django’s signals? Register your custom code as a callback function for the Django post_save signal?

FAILS: Does not work properly in the ‘Admin’ interface at a minimum. Creating a new Netgroup with some Interface relations selected does not show the Interface relations inside the post_save callback function. Another similar case to above. See the final post here (as of 1/18/2012).

Solution 3?

Try solution 2 in combination with another signal callback function for the m2m_changed Django signal. This was recommended to me in #django-users IRC as the way to do it.

FAILS: Does not work properly in the ‘Admin’ interface. Removing Interface relationships does not present itself as data associated with the pre_remove and/or post_remove “actions”. Instead, it appears the ‘Admin’ interface clears all interfaces from the field and then adds the correct interfaces back in. See bug 16073, bug 14482, and this thread.

Sad Conclusion

If there wasn’t a bug, we would obviously use solution 3, even though I find it goofy to have to catch an extra signal (m2m_changed) when a model happens to have a ManyToManyField on it.

As it is now, we just use “Solution 2″ and tell users of the ‘Admin’ interface, via extra help_text on the Netgroup.name field, to enter the name, save, then add items to the Netgroup instance.

class Interface(models.Model):
    fqdn = models.CharField(primary=True,
                            verbose_name="Fully Qualified Domain Name",
                            max_length=80)

class Netgroup(models.Model):
    name = models.CharField(primary=True,
                            max_length=80,
                            help_text="WARNING: save new netgroups with just the name, then add interfaces and save again.  Sorry!")

    interfaces = models.ManyToManyField(model=Interface,
                                        null=True,
                                        blank=True)

from django.db.models.signals import post_delete, post_save, m2m_changed

def netgroup_delete(sender, **kwargs):
    pass # DO STUFF HERE

def netgroup_save(sender, **kwargs):
    pass # DO STUFF HERE

post_delete.connect(netgroup_delete, sender=Netgroup,
                    dispatch_uid="netgroup_delete")
post_save.connect(netgroup_save, sender=Netgroup,
                  dispatch_uid="netgroup_save")

I sent this message (in blue below) with the subject shown. I received it too, as I was a recipient.

I then selected it (as you see above) and replied to it (replying to my own message, to add new info). I didn’t touch the subject line. I sent the new reply out. I receieved it too, as I was a recipient. Here is how I received it. Take note of the changed subject.

Update: Testing shows that the subject is being altered by Thunderbird as soon as I click on the original message and hit reply. The reply window appears with the subject altered.

As an aside, in an unrelated ticket, a customer mentioned slow page load times for one of the pages served by our GForge Advanced Server instance. He indicated that this particular page always takes nearly 20 seconds to load and offered that it might be tied to the fact that he is a member of roughly 20 projects hosted via that GForge server.

This aside from the customer was a remarkable coincidence, as I had created an internal task ticket 2-3 weeks ago for us to implement metrics gathering for this service because “we should”. We already were performing the most basic of up/down monitoring for the host and service via Nagios. Now we get to have a problematic baseline of metrics and watch things improve from here.

After tweaking our Apache logging to log request service time (%D via mod_log_config, we noticed some (too many) problematic pages for certain users and projects. One of those pages was, of course, the page the customer had reported.

So far, we’ve instrumented metric gathering for each block of PHP (…) code in the most commonly accessed problematic page and tracked down the specific section where the slowness happens. The metric gathering is simplistic: for each major block of execution in the PHP file, store a start time, store an end time, and calculate total seconds to execute that block. Finally, syslog() the accumulated metrics as one line.

Anyway, that’s not the point of this post. The point is: Never stand up a service that works overall and assume your users will complain of slowness. Turns out 10-20 users have been quietly suffering through seriously long page load times for years now without saying a word to us. According to other admins in other departments (who responded to our nice-but-essentially “WTF people?” message to all customers of the GForge service), they’ve experienced the same phenomenom.

Belittling those who productively scrutinize and debate similar software offerings is LAME.

I’m so sick of hearing people dismiss “Chef vs. Puppet” discussions (and hundreds of other very valid comparison-style blog posts, etc) as worthless, un-enlightened, and several other holier-than-thou attributions, as if the content is a simplistic bantering about Coke vs. Pepsi.

My camel’s back broke when John Allspaw (@allspaw) recently tweeted:

Sometimes I imagine web engineers as carpenters; too busy being angry about each other’s brand of hammers to actually build the damn house.

Greg Fodor (@gfodor) replied with:

amateurs argue over tools. journeymen argue over techniques. experts argue if we should be building the house in the first place.

Both of these statements are nothing more than horriffic broccoli farts with the hive-minded in the DevOps world apparently smelling them as roses. I don’t know Greg Fodor, but I’m pretty positive John Allspaw is a lot more intelligent than that bullshit tweet of his, and he should be concerned about saying what he actually means, in detail. Because, you know, people listen to him.

Decisions are all based on something. Just because you don’t need to bother yourself with making the “small” decisions anymore doesn’t give you a new license to be an asshole toward those involved in discussions about those “small” decisions. Those people making the “small” decisions now, with debates and careful evaluations surrounding them, MAKE SHIT ULTIMATELY HAPPEN. I understand that everyone, more and more, just wants stuff to magically happen, but the reality still is that things need to be planned, discussed, scrutinized, evaluated, and finally decided on. Don’t get pissy because that part still has to happen and your Idea-to-Realization time can’t be measured in minutes.

God damn.

Official Graphite documentation is at http://graphite.readthedocs.org/

export PATH=/usr/bin:/bin:/usr/sbin:/sbin
mkdir -p /graphite/src
cd /graphite/src

# Download Python 2.7.x source from python.org
# Unpackage

cd Python-2.7.2
./configure --prefix=/graphite
make
make install
cd ..

# Now make sure you get your python before RHEL's

export PATH=/graphite/bin:$PATH

# Download Django 1.x source from django.org
# Unpackage

cd Django-1.3
python setup.py install
cd ..

# Download django-tagging from http://code.google.com/p/django-tagging/
# Unpackage

cd django-tagging-0.3.1
python setup.py install
cd ..

# Download Twisted 11.x source here from http://twistedmatrix.com/
# Unpackage

cd Twisted-11.0.0
python setup.py install
cd ..

# Download setuptools from http://pypi.python.org/pypi/setuptools
# Unpackage

cd setuptools-0.6c11
python setup.py install
cd ..

# Download pip from http://pypi.python.org/pypi/pip
# Unpackage

cd pip-1.0.2
python setup.py install
cd ..

# Now you have to build the Cairo graphics library, as RHEL's
# is too old for py2cairo's needs.

# Start with pixman, a dependency for building cairo!

# Download pixman from http://www.cairographics.org/releases/
# Unpackage

cd pixman-0.22.2
./configure --prefix=/graphite
make
make install

# Download cairo from http://www.cairographics.org/releases/
# Unpackage

cd cairo-1.10.2
# Make sure we find our shit and don't have runtime linker problems
export LDFLAGS="-L/graphite/lib -Xlinker -rpath -Xlinker /graphite/lib"
# PKG_CONFIG_PATH is set below so pixman is found
export PKG_CONFIG_PATH=/graphite/lib/pkgconfig
# Turn off all the X shit and gobject since RHEL glib2 is too old
./configure --prefix=/graphite --without-x --enable-xlib=no \
    --enable-xlib-xrender=no --enable-xcb-shm=no --enable-qt=no \
    --enable-gl=no --enable-gobject=no
make
make install

# Download Pycairo (aka py2cairo) from http://www.cairographics.org/pycairo/
# Unpackage

# Had to add these to LDFLAGS or ./waf configure bombs on Python.h test
# due to not including them when it should be in its test
export LDFLAGS="$LDFLAGS -lm -ldl -lutil"
cd py2cairo-1.10.0
./waf configure --prefix=/graphite
./waf build
./waf install
cd ..

# Download carbon, whisper, and graphite-web from
# https://launchpad.net/graphite
# Unpackage

cd whisper-0.9.9
python setup.py install
cd ..

cd carbon-0.9.9
python setup.py install
cd ..

cd graphite-web-0.9.9
./check-dependencies.py # Ignore WARNINGs as you see fit
python setup.py install
cd ..