Honeycomb.io's honeytail ♡ systemd's instantiated services
Update: For those using Puppet, I've now relased a module to the forge which implements what's set out in this blog post: https://forge.puppet.com/seanhood/honeytail
Second post in a row about Honeycomb.io. Anyway, a question popped up in the Honeycomb Slack channel asking:
A question on honey tail, can you have multiple parsers and log files, in one config file and run a single honeycomb instance, or should you have seperate instances for each type of parser?
I wondered this a few months back and came up with a solution using a feature of systemd known as Instantiated Services I had learned about a few years ago. Which allows you to template a variable in a systemd unit file.
On to how we can allow us to run multiple instances of Honeytail on a server. In this ficticious world we have a server running Nginx and MySQL and we want to parse their logs and send them off to Honeycomb.io.
This guide assumes you're have already installed
honeytail, if not here's that guide: https://docs.honeycomb.io/getting-data-in/integrations/honeytail/
Create a directory for us to store our
honeytail files in.
conf.d made sense to me, like how many other bits of software keep their config files separate.
Create the config file for parsing the MySQL slow logs:
[Required Options] ParserName = mysql WriteKey = REDACTED LogFiles = /var/lib/mysql/slow-query.log Dataset = mysql
Copy this into:
N.B. There's more to parsing these logs than just this file, see docs here: https://docs.honeycomb.io/getting-data-in/integrations/databases/mysql/logs/
Create the config files for parsing Nginx access logs:
[Required Options] ParserName = nginx WriteKey = REDACTED LogFiles = /var/log/nginx/*.log Dataset = traces [Nginx Parser Options] ConfigFile = /etc/nginx/conf.d/honeytail-logformat.conf LogFormatName = combined-honey
As above information about the actual configuration is here: https://docs.honeycomb.io/getting-data-in/integrations/webservers/nginx/
Now we need to drop in the new systemd unit file. It’s not very different than the original except for the modification to line 6. The magic is all in the
%i. This goes in:
[Unit] Description=Honeycomb log tailer honeytail - %i After=network.target [Service] ExecStart=/usr/bin/honeytail -c /etc/honeytail/conf.d/%i.conf KillMode=process Restart=on-failure User=honeycomb Group=honeycomb [Install] WantedBy=multi-user.target
Since we have added a new unit file, we need to let systemd know about this:
All going well we can now start our instantiated services like so:
systemctl start [email protected] systemctl start [email protected]
Assuming we want these to start on boot, let’s also enable them:
systemctl enable [email protected] systemctl enable [email protected]
Check that all is well:
[email protected]:~# systemctl status [email protected] ● [email protected] - Honeycomb log tailer honeytail - nginx Loaded: loaded (/etc/systemd/system/[email protected]; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-01-16 16:26:39 UTC; 2s ago Main PID: 32052 (honeytail) CGroup: /system.slice/system-honeytail.slice/[email protected] └─32052 /usr/bin/honeytail -c /etc/honeytail/conf.d/nginx.conf Jan 16 16:26:39 myserver systemd: Started Honeycomb log tailer honeytail - nginx. Jan 16 16:26:40 myserver honeytail: time="2020-01-16T16:26:40Z" level=info msg="Starting honeytail"
I hope you enjoyed this useful tidbit of information about systemd. And if you haven't heard of Honeycomb, go check them out.