Skip to content

HashiCorp Vault

The secret store at vault.448.global. Where every password, API key, certificate, and token should live. Tier-0 — losing access here is catastrophic.

Field Value
Public URL https://vault.448.global
Admin URL https://vault.448.global/ui
Audience engineers + automated systems
Criticality critical — stores the keys to everything
Maturity [INFO NEEDED]
Owner [INFO NEEDED]
Last reviewed 2026-05-05

1. At a glance

Vault is the company's secret vault. Every other app, every script, and every operator should read passwords and keys from here, never from a .env file or someone's laptop. If Vault is unavailable, automated systems that read secrets at startup may fail to come up. If Vault's master keys are lost, recovery is impossible — encrypted secrets are unrecoverable.

2. Business purpose

  • One place to store secrets so they aren't scattered through code, sticky notes, and Slack DMs.
  • Audit trail of who read what.
  • Rotation of credentials.
  • Per-app access control (an app can read only its own secrets).

3. Audience

Engineers (UI / CLI), automated systems (API).

4. Hosting & cloud infrastructure

Infrastructure map

Item Value Notes
Public hostname vault.448.global publicly reachable via Caddy; should be Wireguard-only (RM-025)
Backend host O1 shared with ~13 other apps
Open ports 443 (Caddy), 8200 (internal only via docker network) port 8200 NOT bound to host
TLS cert Caddy auto-LE
Reverse proxy Caddy on O1
Container image hashicorp/vault:1.18.5 (pinned 2026-05-06) was :latest until incident KI-033
Container user non-root vault (uid 100, set by Dockerfile) SKIP_SETCAP=true therefore required
Storage backend Raft data at /home/ubuntu/docker/vault/data (bind mount)
Auto-unseal none — manual (RB-003 Path A) OCI KMS auto-unseal is a candidate Phase-3
Cluster size single node (Raft cluster of 1)
Watchtower auto-update opted out (com.centurylinklabs.watchtower.enable=false label) per KI-033 lessons
Memory locking (mlockall) disabled (SKIP_SETCAP=true) trade-off — memory could swap

Credentials and recovery materials

The unseal keys and root token are themselves Tier-0 secrets — by definition they cannot live inside Vault.

Unseal-key custody (4 holders):

Holder Role
Vishnu Kant Solutions Architect (day-to-day Vault operator)
Stacy Carpenter Company owner
Adam Pitt-Stanley Company owner
Bradley Leggett DBA

The Vault was initialised with a 5-share / 3-threshold Shamir configuration. Each of the four holders currently holds all 5 shares, rather than one share each — see KI-039 for the security implication and the proposed re-issuance pattern.

Item Storage location Notes
Unseal key shares (5 of 5) held by each of the four custodians above offline / personal-store; not centralised
Root token [INFO NEEDED] (recommend: revoked after recovery) recreate on demand via vault operator generate-root
Auto-unseal KMS key n/a — manual unseal today OCI KMS auto-unseal is a Phase-3 candidate
Operator admin creds (e.g. akadmin-style) n/a for Vault — uses OIDC via Authentik the akadmin is in Authentik, not Vault

5. Technology behind it

  • Type: off-the-shelf
  • Product: HashiCorp Vault (open-source / Community Edition)
  • Stack: Go binary, runs as a single process; storage backend separate

6. Data it handles

Every secret in the company that's been migrated in: database passwords, API keys, OIDC client secrets, signing keys, TLS private keys, registrar tokens, registry credentials. The single most sensitive system in the estate.

7. External dependencies

  • Storage backend (Raft local disk / cloud KMS for auto-unseal / Postgres if used).
  • Authentik or local user/pass for operator login.
  • Certificate authority for its own TLS cert (Let's Encrypt typical).

8. Authentication & access

  • Human auth: OIDC via Authentik (auth.448.global) → which itself federates upstream to Microsoft Entra (Azure AD). So the full chain is: M365 → Authentik → Vault. [CONFIRM] whether userpass auth is also enabled as a fallback for break-glass.
  • App auth: [INFO NEEDED] — likely AppRole or token-based for n8n CI/CD workflows.
  • MFA: inherited from Microsoft Entra at the upstream IdP — strongest signal in the chain. Authentik also enforces MFA when configured. [CONFIRM] end-to-end test.
  • Audit log destination: /vault/logs/ bind mount on O1 host. [INFO NEEDED] whether logs are also externalized (syslog, OCI Logging).
  • Access policies: [INFO NEEDED] — least-privilege per-app paths under ur/, plus per-engineer human policies. Document under RM-024.

9. Maturity assessment

Dimension Status Evidence
Backups [INFO NEEDED] snapshot Raft regularly + offsite
Restore testing [INFO NEEDED] a Vault you can't restore is no Vault at all
Monitoring [INFO NEEDED] seal-status, audit-log volume, leader changes
Alerting [INFO NEEDED] unseal/seal events, repeated auth failures
Redundancy [INFO NEEDED] single node = critical SPOF
Patching cadence [INFO NEEDED]
Audit logging enabled [INFO NEEDED] must be on
Unseal procedure documented [INFO NEEDED] runbook tested?

10. Known risks & vulnerabilities

  • [CONFIRM] Single point of failure if running as a single node — restart requires unsealing; if storage is corrupted, all secrets are lost.
  • [INFO NEEDED] Unseal key custody — if all key holders are unreachable, Vault stays sealed forever. If too many keys are concentrated in one place, the threshold can be defeated.
  • [INFO NEEDED] No tested restore — most Vault outages are storage-backend failures; if a snapshot has never been restored, it's not a backup.
  • [INFO NEEDED] Public exposure — Vault's UI/API on the public internet means brute-force surface; private/Wireguard-only is safer.
  • [INFO NEEDED] Audit log not externalized — if logs only live on the same host, an attacker can erase them.
  • [INFO NEEDED] Root token still exists — should be revoked after setup; recreate on demand.
  • [INFO NEEDED] Certificate / key expiry of Vault's own TLS — if the cert lapses, applications using strict TLS validation can't connect.

11. Impact if it goes down

  • Apps that read secrets at startup may fail to (re)start.
  • New deployments stall.
  • If sealed and key holders are unavailable: indefinite outage of secret-dependent systems.
  • If storage is destroyed without restorable backup: all secrets must be considered compromised and rotated.

12. Owner & on-call

  • Primary owner: [INFO NEEDED]
  • Unseal key holders: [INFO NEEDED] (recommend ≥3 distinct people)

13. References & links