I didn’t want to leave you hanging without another way to get email alerts from syslog messages! I’ve been messing around with this awesome application called Riemann, and it can pretty much do anything you need it to in the world of real time event monitoring. Although it can do awesome things like show you events in real time and forward events to other places like Graphite, I’m just going to show you a quick and easy way to under-utilize the crap out of it and just send you email alerts when a certain condition is met.

So, go here and follow the installation instructions. If you are on CentOS or Debian just grab the rpm/deb file from the home page

Now, if you read my last post (or you already know what you’re doing), then you already have Logstash running with the Grok filter splitting out your logs to make them queryable. If not… then go do it!!!

All set? Alright edit /etc/logstash.conf and add something like this to the output section:

All this does is tell Logstash to output your stuff to Riemann. Logstash already knows the default port and assumes Riemann is running on the same machine. All we do is stick the status code from your logs into the state event field and set the description of the event equal to the request portion of your log line. Again, I’m basing this off my last post that shows you how to do this if its not already done!

Ok, Logstash is all set. Now edit /etc/riemann/riemann.config (Or wherever you put it). Add this to the bottom:

Restart / reload logstash and Riemann:

OK so all we did here was:

  1. Setup our mailer, and configure outbound settings.
  2. Define our rollup called “tell-ops” where we set thresholds and who gets the message
  3. Create a new stream that looks for “404” inside the state event field and runs tell-ops when it finds a match.

This is different from the Logstash email output because it has the ability to be configured with more options, and thanks to rollup it won’t send you an email every single time it finds a match. IN my example, it will send 3 emails, then wait 3600 seconds (an hour) before sending a full summary of everything that happened since the last email. You can create as many streams as you want to have more control over what emails you get.

The email will come as soon as the event occurs. The subject will contain the host(s), service name and match (404). The body will contain whatever you put in “description” in the Logstash config.

Some quick reminders:

  1. If you are having trouble, tail the Riemann log for clues
  2. Check and make sure you have a running mail transfer agent (like Sendmail)
  3. Make sure Logstash and Riemann are running (e.g. service riemann status)

And there you have it.

How to get email alerts from syslog via Logstash

If you reached this post, you are probably either trying to figure out how to get that dang email output in Logstash to work, which means you already have Logstash up and running. If not, you should probably go over to the Logstash documentation and hang out there for a bit, or maybe you were trying to figure out how to stash logs or something…

I’m posting this because it took me forever to figure it out in part due to missing / incomplete documentation on the email output page. (Just a quick note, you need an MTA in order to send mail, so i suggest you install sendmail or something else and tail the mail log if you want to troubleshoot outbound mail issues.)

So lets say for example you want to send an email alert when there is a particular string or integer such as an error code in your web logs.

You should already have your logs split up in a query-able format with Grok. This is necessary so that you can easily select the fields you want to match in your alerts. Here’s a simple Grok pattern example just in case you have no clue how to get this going.

Lets say you have a log line that looks like this:

Then you could use a Grok pattern like this:

Now, each item in the log line is accessible with the identifier specified above. So, if you want to access the status, anywhere in Logstash, you simply use %{status} and boom! Obviously your log line won’t look exactly like the example, but you can use this awesome Grok debugger to get yours up to snuff.

OK now onto the actual email alert, from here on its easy.

Say you want to get an email alert when a 504 or 404 error shows up in your web log. In the output section of your logstash.conf you would do something like this:

That should do it. Just swap out “status” for whatever you set the name (semantic) to in your Grok filter.