{"id":546,"date":"2010-12-10T14:43:07","date_gmt":"2010-12-10T19:43:07","guid":{"rendered":"http:\/\/www.kickflop.net\/blog\/?p=546"},"modified":"2011-02-25T14:14:52","modified_gmt":"2011-02-25T19:14:52","slug":"any-metric-graphing-with-cron-some-code-syslog-and-splunk","status":"publish","type":"post","link":"https:\/\/www.kickflop.net\/blog\/2010\/12\/10\/any-metric-graphing-with-cron-some-code-syslog-and-splunk\/","title":{"rendered":"Any Metric Graphing with cron, some code, syslog, and Splunk"},"content":{"rendered":"

Sometimes (rarely), I get what I consider to be a clever idea.<\/p>\n

Today, while tying up an evaluation of Zenoss Core<\/a> it occurred to me that one could get nice system performance graphs by simply syslogging the performance data to Splunk<\/a>, which provides for time series-based graphing.<\/p>\n

Monitoring agent, schmonitoring schmagent. We’ve got syslog, cron, and bash\/perl\/python\/ruby on the system already, and we’re syslogging to Splunk already.<\/p>\n

The selling feature here is that you can turn any metric’s data into a chart, and that data can be anything you can gather from a UNIX shell<\/em> (in our case, spawned by cron).<\/p>\n

As a proof of concept, I spent 5 minutes and whipped up the following test script which runs out of cron repeatedly (choose your own interval).<\/p>\n

#!\/bin\/sh\r\n\r\nVMSTAT=`vmstat 1 2 | tail -1 | awk '{print \"runqueue=\" $1 \" scanrate=\" $12 \" blockedprocs=\" $2}'`\r\nLOAD1=`uptime | sed 's\/.*load average: \\(.*\\), .*, .*\/load1=\\1\/g'`\r\nlogger -t stats -p user.info $VMSTAT $LOAD1\r\n<\/pre>\n
This syslogs a line like the following one at the chosen cron interval:<\/p>\n
Dec  1 00:24:19 ourhost stats: [ID 702911 user.info] runqueue=1 scanrate=0 blockedprocs=0 load1=0.78<\/pre>\n
Now, since you’re syslogging all of your host data to Splunk (you are, right?), it’s just a matter of graphing the data against the event’s timestamp in Splunk.<\/p>\n
Our Splunk 4.1.6 search query was as follows for our proof of concept data:<\/p>\n
\"ourhost stats:\" | multikv | table _time load1 runqueue blockedprocs scanrate<\/pre>\n
Clicking “Show Report”, setting Chart Type to “Area”, Multi-Series Mode to “Split”, and Null Values to “Treat as zero”, we get the following:<\/p>\n
\"\"<\/a><\/p>\n
We’d love to hear your comments.  Jeff Blaine with Splunk search brainstorming assistance from Jeremy Maziarz<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
Sometimes (rarely), I get what I consider to be a clever idea. Today, while tying up an evaluation of Zenoss…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"class_list":["post-546","post","type-post","status-publish","format-standard","hentry","category-sysadmin"],"_links":{"self":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts\/546","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/comments?post=546"}],"version-history":[{"count":9,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts\/546\/revisions"}],"predecessor-version":[{"id":1198,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/posts\/546\/revisions\/1198"}],"wp:attachment":[{"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/media?parent=546"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/categories?post=546"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kickflop.net\/blog\/wp-json\/wp\/v2\/tags?post=546"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}