Honeycomb.io's honeytail ♡ systemd's instantiated services

Update: For those using Puppet, I've now relased a module to the forge which implements what's set out in this blog post: https://forge.puppet.com/seanhood/honeytail

Second post in a row about Honeycomb.io. Anyway, a question popped up in the Honeycomb Slack channel asking:

A question on honey tail, can you have multiple parsers and log files, in one config file and run a single honeycomb instance, or should you have seperate instances for each type of parser?

I wondered this a few months back and came up with a solution using a feature of systemd known as Instantiated Services I had learned about a few years ago. Which allows you to template a variable in a systemd unit file.

On to how we can allow us to run multiple instances of Honeytail on a server. In this ficticious world we have a server running Nginx and MySQL and we want to parse their logs and send them off to Honeycomb.io.

This guide assumes you're have already installed honeytail, if not here's that guide: https://docs.honeycomb.io/getting-data-in/integrations/honeytail/

Step 1

Create a directory for us to store our honeytail files in. conf.d made sense to me, like how many other bits of software keep their config files separate.

mkdir /etc/honeytail/conf.d

Step 2

Create the config file for parsing the MySQL slow logs:

[Required Options]
 ParserName = mysql
 WriteKey = REDACTED
 LogFiles = /var/lib/mysql/slow-query.log
 Dataset = mysql

Copy this into: /etc/honeytail/conf.d/mysql.conf

N.B. There's more to parsing these logs than just this file, see docs here: https://docs.honeycomb.io/getting-data-in/integrations/databases/mysql/logs/

Create the config files for parsing Nginx access logs:

[Required Options]
 ParserName = nginx
 WriteKey = REDACTED
 LogFiles = /var/log/nginx/*.log
 Dataset = traces

[Nginx Parser Options]
ConfigFile = /etc/nginx/conf.d/honeytail-logformat.conf
LogFormatName = combined-honey

As above information about the actual configuration is here: https://docs.honeycomb.io/getting-data-in/integrations/webservers/nginx/

Step 3

Now we need to drop in the new systemd unit file. It’s not very different than the original except for the modification to line 6. The magic is all in the %i. This goes in: /etc/systemd/system/[email protected]

[Unit]
Description=Honeycomb log tailer honeytail - %i
After=network.target

[Service]
ExecStart=/usr/bin/honeytail -c /etc/honeytail/conf.d/%i.conf
KillMode=process
Restart=on-failure
User=honeycomb
Group=honeycomb

[Install]
WantedBy=multi-user.target

Step 4

Since we have added a new unit file, we need to let systemd know about this:

systemctl daemon-reload

Step 5

All going well we can now start our instantiated services like so:

systemctl start honeytail@nginx
systemctl start honeytail@mysql

Assuming we want these to start on boot, let’s also enable them:

systemctl enable honeytail@nginx
systemctl enable honeytail@mysql

Check that all is well:

root@myserver:~# systemctl status honeytail@nginx
● [email protected] - Honeycomb log tailer honeytail - nginx
   Loaded: loaded (/etc/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-01-16 16:26:39 UTC; 2s ago
 Main PID: 32052 (honeytail)
   CGroup: /system.slice/system-honeytail.slice/[email protected]
           └─32052 /usr/bin/honeytail -c /etc/honeytail/conf.d/nginx.conf

Jan 16 16:26:39 myserver systemd[1]: Started Honeycomb log tailer honeytail - nginx.
Jan 16 16:26:40 myserver honeytail[32052]: time="2020-01-16T16:26:40Z" level=info msg="Starting honeytail"

I hope you enjoyed this useful tidbit of information about systemd. And if you haven't heard of Honeycomb, go check them out.