Protecting HashiCorp Vault against Insider Threats
Author: John Jarvis
Release Date: 20/09/21
We’ve talked before about HashiCorp Vault’s many features, and how it can help organisations manage their secrets. To securely do so, however, requires much from many teams across your organisation. This includes from your Vault deployment’s inception, through design to Day 0, and beyond to Day 2 through N. This is true of many of the human and technological processes in any modern organisation, and, so, as we look at protecting Vault from insider threats, it’s important to keep in mind that, ultimately, Vault is just another system in your broader risk management processes.
That said, however, you could most certainly consider your Vault deployment to be your crown jewels.
Harden Vault
And so, how do we protect them? How do we ensure that we meet even the most carefully-calibrated, millisecond-round-time service level agreements in an environment where those very requests may represent an insider threat? The good news is that much time and effort has already been expended in answering this question, for systems more generally, and for Vault, specifically.
HashiCorp has Production Hardening recommendations which should absolutely be your first port of call. Therein, you’ll find links to HashiCorp’s official Linux repositories, official Docker images, and official SELinux policy for Red-Hat-based Linux distributions (and a similar AppArmor policy for Debian-based Linux distributions is in the pipeline). Use of these tools, and risks associated with doing otherwise, are presupposed from this point forward. “A custom build of Vault can do anything from leak keys and secrets, to corrupt and hold your data hostage,” is one of my favourite quotes from HashiCorp’s John Boero in his excellent article on attacking Vault; I would recommend reading it to security novices and veterans alike.
Insiders; One of Many Threats
And so, thinking again about the topic of this post, what do we mean by an insider threat? One size does not fit all. As with all threats, they need to be assessed, systematically, in the context of your organisation: what does business critical mean for you, for example?
- Is five nines, or class five, uptime paramount in your organisation?
- Threats targeting your availability will probably translate as high risk.
- Is a rock-solid audit trail for your regulators something that keeps you up at night?
- Threats targeting your data integrity, and processes around non-repudiation, will be key areas of consideration.
Ultimately, all the CIA — i.e., confidentiality, integrity and availability — threats apply to your Vault deployment too, and insider threats need to be considered in that context. And so, managing risks while preparing for the worst looks very much the same if you’re the Vault system owner, or another:
- Make sure your more generic break-glass procedures keep pace with your Vault deployment.
- In the context of Vault itself, this also means testing your token generation ceremony — for both root and disaster recovery operation — regularly, and weighing the costs and benefits of running unplanned tests as well.
- Ensure that your incident management and incident response teams are in at the ground floor with your Vault deployment, and continue to be involved in the processes and procedures around it as it grows.
- Similarly, ensure that business continuity plans, and those who design and test them, are similarly integrated with your Vault deployment.
- Test cluster replication, disaster recovery, and backups regularly, obviously ensuring that their configurations, and the processes around them, meet your Recovery Point Objective (RPO) and Recovery Time Objective (RTO).
Practical Considerations
And, staying with Vault itself, beyond system hardening, what should you consider when looking at insider threats to Vault?
- Well, in terms of leading up to Day 0, make sure you validate your deployment. Obviously the design will need to go through this process, but also the following:
- The implementation, including pentesting, and more generally,
- Consider how you manage Vault itself, its configuration, and its place in any CI/CD pipeline; also, particularly for larger deployments,
- Consider how responsibilities are delegated, with namespaces — a feature of Vault Enterprise — and the training of all the teams involved in that process.
- And then, on Day 2 through Day N:
- Monitor your deployment; there are many options available, but Vault Enterprise can take advantage of the Vault App for Splunk, as an example.
- Audit that deployment regularly as well, as you would as part of a wider systems certification process.
- Consider how new applications and teams are on-boarded — and, importantly, off-boarded! — with Vault, and how to do this securely.
- And this applies to upgrading Vault itself, of course; an absolutely crucial step in your bid to fight off insiders who mean you harm.
Further Reading
I’ve obviously only scratched the surface of this huge topic. I’ve collected the material I linked to above, and included some additional resources as well.
- Running HashiCorp Vault in Production by Dan McTeer and Brian Krausen (book)
- Production Hardening | Vault
- Hardening HashiCorp Vault with SELinux
- How I’d attack your HashiCorp Vault (and how you can prevent me): System Hardening by John Boero
- Onboarding Applications to Vault Using Terraform: A Practical Guide
- Vault Configuration as Code via Terraform: Stories From the Trenches at Hippo Technologies
- Codify Management of Vault Using Terraform | Vault
- Secret Zero Problem Solved for HashiCorp Vault
Thanks for reading, and, remember, Somerford Associates are a Hyper-Specialised Partner of HashiCorp, with many professional certified consultants in both Terraform and Vault. Please don’t hesitate to get in touch if you have any questions about what you’ve read today, or Vault deployments you have or are planning for.
