I didn’t want to leave you hanging without another way to get email alerts from syslog messages! I’ve been messing around with this awesome application called Riemann, and it can pretty much do anything you need it to in the world of real time event monitoring. Although it can do awesome things like show you events in real time and forward events to other places like Graphite, I’m just going to show you a quick and easy way to under-utilize the crap out of it and just send you email alerts when a certain condition is met.

So, go here and follow the installation instructions. If you are on CentOS or Debian just grab the rpm/deb file from the home page

Now, if you read my last post (or you already know what you’re doing), then you already have Logstash running with the Grok filter splitting out your logs to make them queryable. If not… then go do it!!!

All set? Alright edit /etc/logstash.conf and add something like this to the output section:

All this does is tell Logstash to output your stuff to Riemann. Logstash already knows the default port and assumes Riemann is running on the same machine. All we do is stick the status code from your logs into the state event field and set the description of the event equal to the request portion of your log line. Again, I’m basing this off my last post that shows you how to do this if its not already done!

Ok, Logstash is all set. Now edit /etc/riemann/riemann.config (Or wherever you put it). Add this to the bottom:

Restart / reload logstash and Riemann:

OK so all we did here was:

1. Setup our mailer, and configure outbound settings.
2. Define our rollup called “tell-ops” where we set thresholds and who gets the message
3. Create a new stream that looks for “404” inside the state event field and runs tell-ops when it finds a match.

This is different from the Logstash email output because it has the ability to be configured with more options, and thanks to rollup it won’t send you an email every single time it finds a match. IN my example, it will send 3 emails, then wait 3600 seconds (an hour) before sending a full summary of everything that happened since the last email. You can create as many streams as you want to have more control over what emails you get.

The email will come as soon as the event occurs. The subject will contain the host(s), service name and match (404). The body will contain whatever you put in “description” in the Logstash config.

Some quick reminders:

1. If you are having trouble, tail the Riemann log for clues
2. Check and make sure you have a running mail transfer agent (like Sendmail)
3. Make sure Logstash and Riemann are running (e.g. service riemann status)

And there you have it.