Skip to content

Backups

What gets backed up, where it goes, and — most importantly — whether it has ever been restored.

Posture, 2026-05-05: backups exist for some surfaces and are missing for others. No backup, anywhere in the estate, has ever been restored as a test. Until tested, every backup is a hypothesis.

Summary

Surface Backup mechanism Frequency Retention Off-host? Restore-tested?
E1 (Caddy proxy VPS) none (KI-038) n/a n/a n/a n/a
E2 (Dokploy VPS) none (KI-038) n/a n/a n/a n/a
O1 (ORA448 all-in-one) OCI block-volume incremental snapshot policy weekly + monthly + yearly (see policy below) 4 weeks / 12 months / 5 years OCI snapshot store (same region)
E3 (Free ADB — TnE Eidos) Oracle automated continuous 60 days OCI internal cannot be restored on Free Tier
E4 (Free ADB — Fourway) Oracle automated continuous 60 days OCI internal cannot be restored on Free Tier paying customer!
E5 (Paid ADB — Parallax) Oracle automated + ad-hoc manual at prod release continuous 60 days OCI internal
O2, O3 (Free ADBs — internal dev) Oracle automated continuous 60 days OCI internal n/a (cannot restore on Free)
Dokploy-hosted apps (CRMs, WordPress, GitLab, Teams Bot) Dokploy automatic backup → OCI bucket (EIDOSDev1) [INFO NEEDED] [INFO NEEDED] yes (OCI bucket) no DR instructions, untested
Authentik (on O1) none n/a n/a n/a n/a
Vault (on O1) first ad-hoc tarball + Raft snapshot 2026-05-06; recurring backup TBD one-off indefinite (manual) tarball in PECommon/infra/vault.448.global/; Raft snapshot still on O1
n8n (on O1) none n/a n/a n/a n/a
MinIO (on O1) none explicit n/a n/a host snapshots only n/a
WireGuard config / WG-Easy DB (on O1) host snapshot only inherits O1 schedule inherits OCI snapshot
n8n workflow definitions none — workflows live only in n8n on O1 n/a n/a n/a n/a
Caddyfiles none — host filesystem only n/a n/a n/a n/a
GitLab (gitlab-backup) not yet configured n/a n/a n/a n/a

O1 backup policy (detailed)

Tier Schedule Retention Notes
Weekly incremental midnight Sunday 4 weeks the most-frequent on-rolling backup
Monthly incremental midnight on the 1st of each month 12 months medium-term recovery
Yearly incremental first part of January 5 years long-term archival

Estimated monthly cost on OCI: ~£15/month for the storage of these snapshots. Restore-test status: never performed. This is otherwise the strongest backup posture in the estate.

OCI buckets (EIDOSDev1)

Object-storage buckets in EIDOSDev1 — the off-host backup destination:

Bucket Purpose Used by
PECommon General infra backups, with subdirectories per service Manual + scripted uploads. As of 2026-05-06: holds the first off-host Vault data tarball at infra/vault.448.global/vault-data-backup-2026-05-06.tar.gz
[INFO NEEDED] (CI/CD artifacts bucket name) Build artifacts from pipelines n8n CI/CD outputs
[INFO NEEDED] (Dokploy backups bucket name) Automatic Dokploy app backups E2 Dokploy → bucket

PECommon directory layout (proposed convention):

PECommon/
└── infra/
    ├── vault.448.global/        ← Vault tarballs + raft snapshots
    ├── auth.448.global/         ← Authentik DB dumps + media (when RM-014 lands)
    ├── git.projecteidos.com/    ← GitLab backups + gitlab-secrets.json (when RM-015 lands)
    ├── n8n.448.global/          ← n8n DB dump + workflow exports (when RM-016 lands)
    └── e1-caddy-state/          ← E1 Caddy data dir snapshots (optional)

No buckets in ORA448Global. No backups crossing tenancies. No off-OCI off-site copy yet (Phase-2: RM-018).

What's missing (the Phase-2 backup gap list)

Tier-0 (untouchable production secrets / identity)

  1. Vault — no backup. If O1 is lost, every secret in vault.448.global is unrecoverable. Highest priority.
  2. Authentik — no backup. If O1 is lost, the SSO IDP state (users, OIDC clients, signing keys) is gone.

Tier-1 (operational state)

  1. GitLab — no scheduled gitlab-backup. Source code on E2 is at risk if the host snapshot is corrupted or restore fails.
  2. n8n workflow definitions + DB — production CI/CD pipelines live here, only on O1.
  3. MinIO objects — if MinIO is being used as backup storage for other apps, it must itself be backed up to a separate destination.

Tier-2 (backups exist but DR untested)

  1. Dokploy-hosted apps — backups land in OCI bucket but no DR procedure exists; untested.
  2. Parallax (E5) Paid ADB — Oracle's 60-day retention works in theory; restore has never been performed.
  3. All host snapshots — same story.

Free Tier limitation

  1. Free ADBs (E3, E4, O2, O3) — Oracle keeps the 60-day backup but does not allow restore on the Free Tier. For E4 (paying-customer Fourway tenant) this means a corruption event is unrecoverable. See KI-019.

Restore-test priorities (proposed)

A backup has not been proven to work until it has been restored. Ordered by impact:

  1. Vault — once Vault backups exist, run a quarterly cold-restore drill in a parallel container.
  2. Parallax E5 paid ADB — Oracle restore is standard; do at least one "restore to a clone" drill annually.
  3. GitLab — once gitlab-backup is scheduled, restore to a parallel test instance and verify CI variables decrypt (validates gitlab-secrets.json capture).
  4. Dokploy app DBs (CRMs, WordPress) — restore to a sandbox and verify boot.
  5. OCI block-volume snapshot for each of E1, E2, O1 — restore once to confirm bootable.

Phase-2 actions (the "fix the backup gaps" plan)

  1. Vault snapshot job — periodic Raft snapshot or filesystem backup; ship to OCI bucket and an off-OCI destination (e.g. cross-tenancy or external S3).
  2. Authentik backup — Postgres dump + media volume; ship to OCI bucket.
  3. GitLab backup schedule — gitlab-backup create daily, including gitlab-secrets.json; ship to OCI bucket.
  4. n8n — export workflows + Postgres backup; encryption key (N8N_ENCRYPTION_KEY) into Vault.
  5. Cross-region copy — at minimum, mirror the EIDOSDev1 backup bucket to another OCI region (or to MinIO on O1, which is itself in another tenancy).
  6. Off-OCI backup — long-term, add an external storage provider (Backblaze B2, AWS S3 cheapest tier) for ransomware-resilient copies.
  7. Document and rehearse the restore procedure for each Tier-0 / Tier-1 surface.
  8. Upgrade E4 (Fourway tenant ADB) off Free Tier so its 60-day backups become restorable.
  9. Bring gitlab-backup files into a known retention policy with off-host copy and gitlab-secrets.json included.

The principle

A backup that has never been restored is a hope, not a recovery plan. Until every backup in the table above has a "" in the restore-tested column, the company is not actually backed up.