Syslog Output for Chef Runs

A new blog post category was added to my blog for this post: Yak Shave. It was a small yak shave, but a table-flip inducing yak shave nonetheless.

UPDATE 6/3/2015: Per Lamont Granquist of Chef, As of Chef 12.4, the following works in client.rb without any other cookbooks needed (negating this whole blog post finally!):

log_location Chef::Log::Syslog.new("chef-client", ::Syslog::LOG_DAEMON)

THE FOLLOWING BLOG POST IS KEPT FOR HISTORICAL PURPOSES ONLY.

If you don’t want to read a pissy rant, don’t read any further (you can jump to the answer). If you’re going to comment to suggest Puppet, CFEngine, Ansible, or SaltStack, save your typing ;)

Begin

There you are with your configuration management tool. You say to yourself:

Self, surely there is a way to configure this damn-near-Linux-centric systems management tool to syslog its output so that, based on the common sane IT pattern of syslogging to a central server, all Chef run output can be stored in a centralized location and perhaps queried via Splunk or whatever other OSS tool. Because, you know, syslog has been around for 20+ years and is the de-facto means of logging information on UNIX/Linux hosts. And CFengine 2, dating back at least 15 years, is a CM tool that has offered such incredibly basic functionality. Surely the default of logging to stdout is just the absolute safe case that the product must default to, but there is definitely an option to turn on syslog as the log output location…

Knowing that this sort of thing would be defined as a “Chef Report Handler”, and that Chef has been alive for 4 years now, you start with Google: “chef syslog handler”

Handler – chef-syslog-handler

Bingo! The first search result is chef-syslog-handler, a “Chef handler to send syslog messages”.

You visit the link which takes you to rubygems.org, click on “Homepage” and get a 404 error from github.

Maybe the author renamed the repository? Viewing all of the author’s repositories shows that to be untrue. It’s simply gone. Returning to the rubygems page, you see “August 19, 2011”

Welp.

The Old Magically Worked Ticket

Returning to our Google results, we scroll down past all of the chef-syslog-handler results and end up at an old Opscode JIRA ticket.

Yes! This must be good news. The ticket is marked as Fixed!

17/Mar/11 5:49 PM
This is done... configure chef like this:

    log_location SyslogLogger.new("chef-client")

My follow-up comment to the ticket says it all:

I went to implement this today per snippet above.

It does not work with Chef 11.4.0

Looking through the mixlib-log 1.3.0+ versions on github, and looking at 10-stable, I don’t understand how it would have ever worked there either, but surely that’s just my ignorance.

WELP.

Revise Search

This is going nowhere past 2011, fast. Let’s feed Google something more generic: “chef syslog”

Cookbook – chef-client_syslog

Described as “Send chef-client log to syslog”, this too seemed promising.

And then I see that it does its own management of /etc/chef/client.rb via a template. As I said above, any sane person is handling via the chef-client cookbook.

SyslogLogger log_location

And finally, if you’re “lucky” (I forget how I got there), you end up at CHEF-2560 from August 11th 2011 which is still open!

Oh. It’s someone reporting, like I did, that the old Magically Worked Ticket’s solution does not work at all. It even includes code to fix it (unchecked), yet the ticket remains just dangling in limbo for the last 18 months.

Let’s follow the link to the pull request!

Here we find our old friend “pmorton” (the author of the old chef-syslog-handler gem in the first section of this blog post):

@guillermo – This is some great stuff. I seem to have located the ticket for this pull request (http://tickets.opscode.com/browse/CHEF-2560) but noticed two issues, the ticket has not been resolved and you are not part of the approved contributors list. I hope that you will take the time to get a CLA signed and sent to opscode. See http://wiki.opscode.com/display/chef/How+to+Contribute for more details.

The pull request is then closed out by Opscode due to lack of Contributor License Agreement. The original JIRA ticket CHEF-2560 remains open.

Fitna Cut Someone

This is where, per @jordansissel, I entered Hate-Driven Development mode.

I vowed to fix this once and for all, submit the code, and set aflame anyone at Opscode who rejected it.

Then I stumbled across one more thing.

Bleeding Edge chef-client Cookbook

UPDATE: THE BLOG POST IS KEPT FOR HISTORICAL PURPOSES ONLY. The solution here no longer works and I give up chasing this elusive dream.

Again, I don’t recall how I got there, but the current master branch of the chef-client community cookbook has a syslog example! That looks promising. Heh, cute, it references yet another JIRA ticket about syslog support, COOK-2326 (which links to almost everything listed above).

AND SO … if you are willing to make the leap to the heavily refactored latest chef-client cookbook that was released around June 11, 2013, you have a clean method of syslogging your output!

# our-chef-client-wrapper/files/default/syslog.rb

require 'rubygems'
require 'syslog-logger'
require 'syslog'

Logger::Syslog.class_eval do
  attr_accessor :sync, :formatter
end

log_location Logger::Syslog.new('chef-client', Syslog::LOG_DAEMON)
# our-chef-client-wrapper/recipes/whatever.rb

chef_gem 'syslog-logger'

# Drop off our chef-client customization file that
# allows for syslogging
cookbook_file '/etc/chef/client.d/syslog.rb' do
  source 'syslog.rb'
  mode 00644
  notifies :create, 'ruby_block[reload_client_config]', :immediately
end

include_recipe 'chef-client::config'

And this, my friends, is why you better know how to navigate the world of Ruby, Ruby gems, OSS, if you ever intend to have a remote chance with Chef. You will note that there was not a SINGLE THING in the experience above that was even close to being “sysadmin” friendly.

And the solution requires installing a gem at a critical point in what you want to be rock solid — the reporting of your 2000 machines’ chef runs. Behind an HTTP proxy? Hopefully you’ve run the chef-client cookbook already so that ENV['http_proxy'] is set so that the gem can fetch the right stuff.

You’d think Ruby didn’t have native syslog support in it’s standard library since version 1.8.6 or something.

Cause, you know… syslog is obscure. Who would ever want to use the logging mechanism used by the OS daemons since 1990… when running a daemon… that manages the OS and its daemons.

fuck-this-thursday

Filed under Quality Control, Sysadmin, UNIX/Linux, Yak Shave

  • In a show of yak shave solidarity: I am familiar with this state of affairs. I ran a similar gauntlet trying to get rabbitmq because I was trying to get redis because I was trying to get sensu going. I hacked the ever living crap out of the problem and submitted a pull request but also went back to the source of my problem and asked them to just fix it which they said they would do and it’s about time for me to go back and see if they have. https://twitter.com/sascha_d/statuses/337714679344410625 and https://twitter.com/sascha_d/status/338144736865038336 and for good measure: https://twitter.com/sascha_d/status/337700595878215680

  • Brendan Germain

    this is broken with the 11.6 client

    monologger.rb:69:in `exist?’: can’t convert Logger::Syslog into String (TypeError)

  • Oh goody. *sigh*

  • Brendan Germain

    forgot to say awesome writeup though – I wish i had found this before doing all the same work on my own… If you happen to tackle this newest problem please share. I haven’t had any luck yet.

    thanks.

  • You’re welcome, and know that I feel your pain obviously. I will be sure to update the post if I uncover the solution with 11.6.

  • Zed

    as an admin, this is just one of the reasons I prefer other automation systems (specifically a popular table seasoning)

  • FWIW, I got back to this today and it works fine with my 11.8.2 install. I can’t say that I am going to bother to install an older release to see/solve the issue if it is not appearing in modern releases.

  • Brendan Germain

    interesting, i just tried it again on 11.12.4 with no luck.

    [2014-06-02T16:40:46-04:00] FATAL: Failed to open or create log file at #: TypeError (can’t convert Logger::Syslog into String)

    [2014-06-02T16:40:46-04:00] FATAL: Aborting due to invalid ‘log_location’ configuration

  • My stupid error, Brendan. I conflated 2 separate syslog/chef-related projects. I’ve not gotten back to THIS one.

  • I looked into this (really this time) today. With monologger in use now, the log_location setting must either be a file pathname or an object that responds to :write and :close. Since there is no :write method on any of the ruby Syslog stuff, syslogging from Chef is a dead-end until someone more Ruby-clued than I determines how to patch :write into Syslog at runtime.

    Reference: https://github.com/opscode/chef/blob/11.12.8/lib/chef/monologger.rb#L47

    For now I have updated the blog post to at least indicate that the post’s information no longer works.

  • Chris Sibbitt

    Here’s what I use and it works in 11.14.2. The chef-client cookbook manages my client.rb and I set these two attributes:

    normal[‘chef_client’][‘load_gems’][‘SyslogLogger’][‘require_name’] = “syslog/logger”

    normal[‘chef_client’][‘verbose_logging’] = “true

    class Syslog::Logger

    attr_accessor :sync

    def initialize

    @level = ::Logger::DEBUG

    @formatter = Formatter.new

    @@syslog ||= Syslog.open(‘chef-client’,nil,Syslog::LOG_LOCAL0)

    end

    def <<(msg)

    info(msg)

    end

    #This is added for Chef 11 compatibility, should not affect Chef 10

    def write(msg)

    info(msg)

    end

    def close(msg)

    #This method intentionally left blank. Syslog::Logger has no close equivalent

    end

    end

    log_location Syslog::Logger.new"

  • Saw the following in the UsingChef Issue 27. Maybe this is a better path for people:

    https://supermarket.getchef.com/cookbooks/chef-client_syslog

  • Lamont Granquist

    So with the upcoming 12.4.0 release this should work in client.rb:

    log_location Chef::Log::Syslog.new(“chef-client”, ::Syslog::LOG_DAEMON)

    No other cookbooks necessary.

  • hi5 – Updated top of blog post.