Monitor OpenSMTPD using Logstash and Grafana
491 words, 3 minutes
Logs are usefull. Graphs are sexy. Here’s a way to get a view on what happens to your OpenSMTPD traffic, using Web v2.0 tools ; namely Logstash & Grafana.
For those who would not be aware of those tools, logstash is some kind of log-parser that can eat syslog formatted logs and write them into elasticsearch ; in “document” format. Grafana is a Web frontend that can dig into various databases and render graphics from requests.
I won’t go into the whole “how to install” process here. Installation is quite straight forward and online documentation is quite clear.
What you need
OpenSMTPD deals with emails and logs its activity via Syslog.
Syslog is configured to send the logs to Logstash.
Logstash has a set of rules configured to transform the text-oriented information into searchable document-oriented data. The transformed data is stored into Elasticsearch.
Elasticsearch provides Web API to search and find stuff.
Grafana connects to ELS to get data and draw the graphs.
Configuration snippets
There is nothing special to do with opensmtpd.
You have to tell syslog to forward it’s log to the logstash instance. In my case, all my OpenBSD’s logs are sent to the logstash host on port 80601:
# cat /etc/syslog.conf
(...)
*.* @tls4://remote.logstash.io:80601
Of course, you have to configure TLS on the Logstash side. I used nginx as a stream-proxy to do that. But Logstash may be able to expose a TLS port on its own.
The logs are parsed by Logstash using Grok expressions. Mine looks like:
# cat conf.d/carnat_net.conf
(...)
# If OpenSMTPD log
if [program] == "smtpd" {
grok {
match => {
"message" => [
# subsystem "mda"
"%{WORD:envelope-id} %{WORD:subsystem} event=%{DATA:event} evpid=%{DATA:evpid} from=%{DATA:from} to=%{DATA:to} user=%{USERNAME:user} method=%{DATA:method} delay=%{DATA:delay} result=%{WORD:result} stat=%{WORD:stat}",
# subsystem "smtp"
"%{WORD:envelope-id} %{WORD:subsystem} event=%{USERNAME:event} address=%{IP:address} host=%{IPORHOST:host}(?: command=\"%{DATA:command}\")?(?: reason=%{WORD:reason})?(?: result=\"%{DATA:result}\")?",
# subsystem "mta"
"%{WORD:envelope-id} %{WORD:subsystem} event=%{USERNAME:event}(?: evpid=%{WORD:evpid})?(?: address=%{IP:address})?(?: address=smtp\+tls\:\/\/%{IP:address}\:%{POSINT})?(?: host=%{IPORHOST:host})?(?: ciphers=\"%{DATA:ciphers}\")?(?: reason=%{WORD:reason})?(?: msgid=%{WORD:msgid})?(?: from=%{NOTSPACE:from})?(?: to=%{NOTSPACE:to})?(?: rcpt=%{NOTSPACE:rcpt})?(?: source=\"%{IP:source}\")?(?: size=%{INT:size})?(?: ndest=%{INT:ndest})?(?: proto=%{WORD:proto})?(?: relay=\"%{DATA:relay}\")?(?: delay=%{WORD:delay})?(?: result=\"%{WORD:result}\")?(?: stat=\"%{DATA:stat}\")?(?: messages=%{INT:messages})?",
# other unmatched messages
"%{GREEDYDATA:opensmtpd-unmatched}"
]
}
remove_field => ["message"]
}
There may be missing things as I didn’t found the opensmtpd log-format documentation and I have a simple configuration. In a more complex configuration, it may be required to modify those rules.
From here, the logs should be stored in ELS and can be queried and rendered by Grafana.
Beauty and the Graph
For the record, this is what I got in 5 minutes using Collectd and Facette.
Design is slick. No distraction. It only requires a bit of training to browse history. But it worked!
Now, here’s what you get with Grafana and simple SMTP facts.
Tastes and colour may choose for you. But Grafana is really easy to use ; although it requires a bit of training. And it is able to query various databases while displaying a single dashboard.
In case you don’t speak Elasticsearch, here are some under the hood details :
Now your turn. Let us know what you achieved ;-)