How to get email alerts from syslog via Logstash
If you reached this post, you are probably either trying to figure out how to get that dang email output in Logstash to work, which means you already have Logstash up and running. If not, you should probably go over to the Logstash documentation and hang out there for a bit, or maybe you were trying to figure out how to stash logs or something…
I’m posting this because it took me forever to figure it out in part due to missing / incomplete documentation on the email output page. (Just a quick note, you need an MTA in order to send mail, so i suggest you install sendmail or something else and tail the mail log if you want to troubleshoot outbound mail issues.)
So lets say for example you want to send an email alert when there is a particular string or integer such as an error code in your web logs.
You should already have your logs split up in a query-able format with Grok. This is necessary so that you can easily select the fields you want to match in your alerts. Here’s a simple Grok pattern example just in case you have no clue how to get this going.
Lets say you have a log line that looks like this:
|
1 |
71.59.153.254 Thu, 27 Jun 2013 00:09:52 GMT GET /images/somefolder/image.jpg 200 |
Then you could use a Grok pattern like this:
|
1 2 3 |
grok { pattern => "{IP:ip} %{DAY:day}, %{MONTHDAY:date} %{MONTH:month} %{YEAR:year} %{TIME:time} %{WORD:tz} %{WORD:method} %{URIPATHPARAM:request} %{INT:status}" } |
Now, each item in the log line is accessible with the identifier specified above. So, if you want to access the status, anywhere in Logstash, you simply use %{status} and boom! Obviously your log line won’t look exactly like the example, but you can use this awesome Grok debugger to get yours up to snuff.
OK now onto the actual email alert, from here on its easy.
Say you want to get an email alert when a 504 or 404 error shows up in your web log. In the output section of your logstash.conf you would do something like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
email { match => [ "Error 504 Gateway Timeout", "status,504", "Error 404 Not Found", "status,404" ] subject => "%{matchName}" via => "smtp" body => "Here is the event line that occured: %{@message}" htmlbody => "<h2>%{matchName}</h2><br/><br/><h3>Full Event</h3><br/><br/><div align='center'>%{@message}</div>" } } |
That should do it. Just swap out “status” for whatever you set the name (semantic) to in your Grok filter.
Hi,
I tried to get the email alerts working with the following config . I found that the matched patterns do get printed on stdout but I could never get the email working.
Am I missing something in my config
input {
file {
type => “run_loop”
path => “/home/data/apprunner_test17”
start_position => “beginning”
}
}
filter {
grep {
type => “run_loop”
add_field => [ “verification”, “%{@message}” ]
match => [“@message”,”Unable to verify”]
add_tag => “error”
drop => “false”
}
}
output {
stdout {
type => “run_loop”
debug => true
tags =>[“error”]
debug_format => “json”
}
email {
type => “run_loop”
from => “[email protected]”
match => [ “Error” ,”verification,verify”]
tags =>[“error”]
subject => “ALERTS”
to => “mymailid”
via => “sendmail”
body => “Here is the event line that occured: %{@message}”
}
}
Thank you for this article ! But I have an issue when I try to send email via logstash with the following message :
Exception in thread “LogStash::Runner” org.jruby.exceptions.RaiseException: (ECONNREFUSED) Connection refused – Connection refused
at org.jruby.ext.socket.RubyTCPSocket.initialize(org/jruby/ext/socket/RubyTCPSocket.java:126)
at org.jruby.RubyIO.open(org/jruby/RubyIO.java:1181)
at RUBY.tcp_socket(jar:file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/META-INF/jruby.home/lib/ruby/1.9/net/smtp.rb:540)
at RUBY.do_start(jar:file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/META-INF/jruby.home/lib/ruby/1.9/net/smtp.rb:549)
at org.jruby.ext.timeout.Timeout.timeout(org/jruby/ext/timeout/Timeout.java:127)
at RUBY.do_start(jar:file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/META-INF/jruby.home/lib/ruby/1.9/net/smtp.rb:549)
at RUBY.start(jar:file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/META-INF/jruby.home/lib/ruby/1.9/net/smtp.rb:519)
at RUBY.deliver!(file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/mail/network/delivery_methods/smtp.rb:112)
at RUBY.deliver!(file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/mail/message.rb:248)
at RUBY.receive(file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/logstash/outputs/email.rb:246)
at RUBY.handle(file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/logstash/outputs/base.rb:85)
at RUBY.initialize((eval):35)
at org.jruby.RubyProc.call(org/jruby/RubyProc.java:271)
at RUBY.output(file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:254)
at RUBY.outputworker(file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:213)
at RUBY.start_outputs(file:/mnt/data/var/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:140)
Have you any idea to help me ?
Hi.. I configured email alerts. But instead of receiving log message i receive “@message” word in the email. What should I do to get message value?
Oh my goodness! an amazing article dude. Thanks Nonetheless I am experiencing issue with ur rss . Dont know why Unable to subscribe to it. Is there anybody getting similar rss drawback? Anybody who is aware of kindly respond. Thnkx dadbgedkckaf