Getting Data from HashiCorp Vault into Splunk

HashiCorp has a great lab tutorial on how to configure your Vault/Splunk environment to gather all this data here; what follows is an educational guide based on that lab, and is not a step by step guide for installing a complete, enterprise solution.

Configure your Splunk Indexes and Splunk HECs as laid out in the document (note:- see the SSL gotchas below) and record the “tokens” for each HEC; you’ll need them soon.

So, what do we need to know? Well, the solution depends on two open-source applications, TD-Agent (or FluentD) for audit data, and Telegraf for metrics data.

• Install TD-Agent as per the instructions on the FluentD website. This is the .rpm link, however builds for other distros are available and work well (of note, Ubuntu Server 19.10 and 20.04)
• Now install the fluent-plugin-splunk-hec and fluent-splunk-splunk-enterprise plugins using the “td-agent-gem install” command (For CentOS, the HEC plugin needs to have  -v ‘>=1.0’ appended to get the correct, working plugin – see the “gotchas” section at the end)
• Edit the td-agent.conf file as described in the HashiCorp document, noting the next item
• The HEC plugin 1.2.10 needs the following renames for the <match> section in the td-agent.conf file: host, port and token are now hec_host, hec_port and hec_token
• Ensure the td-agent user has read access to the Vault audit logs (744) and read/write to its own .pos file
• Review the td-agent log file. After about 5 minutes of running and enqueuing, you should see JSON data sent to your HECs. You can trigger an audit file update with a simple “vault status” command

Now you can install Telegraf from the InfluxDB website. Once installed, edit the config file as laid out in the HashiCorp document, but comment out the [[outputs.influxdb]] stanza if you are not using InfluxDB, as it will throw errors.

Still not working? (or Gotchas)-

• The HashiCorp guide suggests turning SSL off for the lab. We wouldn’t recommend that, but if you are using self-signed certificates for the HECs add the:-“insecure_ssl true” setting to td-agent.conf file within the <match> stanza, and “insecure_skip_verify = true” in telgraf.conf file in the [[outputs.http]] stanza
• If TD-Agent just enqueues and no other errors appear in the logs, remove the HEC plugin td-agent-gem remove fluent-plugin-splunk-hec and re-install it with the version command on the end. (This is particular to CentOS.)
• These file permissions are quite fussy. Ensure that the Vault audit log permissions are set as 744 and that td-agent has read/write access to both the pos file and the directory where it is stored
• TD-Agent sometimes just doesn’t run. Use systemctl to stop it and run it with the “td-agent –vv &” command for your (or sudo) context and view the output. Ctrl-C kills this process and you can restart it with systemctl once any errors have been fixed
• If Telegraf is not running, check the conf file to ensure you have entered the correct index for your metrics data. Also check your vault.hcl file for the inputs.statsd entry and validate the ports assigned are not in use elsewhere
• Keep your cluster names correct for the cluster. The Vault App for Splunk correlates data per cluster, so setting these incorrectly can leave gaps in your Splunk views

Now, unfortunately, CentOS has reached EOL and the repos have been archived recently, so some of these steps may be a lot more complicated. We are looking at testing CentOS Stream and other distros soon. In the meantime, if you are looking at getting some valuable insight into how Vault is operating and being used, and have some concerns about some of these “gotchas” – including which operating system you should use – feel free to get in contact and speak to the team via the form at the bottom of the page.