OCI ADB Failure & Low-Cost DR Brief¶
Parallax + sister-app fleet. Audience: technical owner deciding how much to spend on DR for OCI Autonomous Database (APEX workload).
Last reviewed: 19 May 2026. All factual claims below are linked inline to the official source.
TL;DR¶
- Today: paid OCI ADB in the UK on APEX Service workload. UR-Prod (E5) now runs Local Autonomous Data Guard (live since 2026-05-19) — RTO ~2 min, RPO 0 (zero data loss), automatic failover, no URL change. Cost roughly doubled (+100% compute +100% storage). The previous Local Backup-Based DR peer (live 15 Jan 2026 → 19 May 2026) was replaced — tenancy quota is 1 local peer per database. Other apps in the fleet need to be checked individually; any still on daily-backup-only have a 24h RPO and hours-of-RTO. Remaining E5 gap: no cross-region peer — a London regional event still takes Parallax down. Next step is a cross-region BBDR peer (cheap, uses the available cross-region slot) under RM-047.
- Oracle's published SLA is 99.95% (no standby) or 99.995% (with Autonomous Data Guard) — but these only entitle you to service credits, not data protection.
- The OCI London region had an unacknowledged ~10-minute incident on 24 Jan 2026 (The Register, 28 Jan 2026) — proof that regional events do happen and Oracle's own status page can't be trusted as the alarm.
- Major change since Nov 2025 — verified live on our tenancy: APEX Service workload now supports Autonomous Data Guard, and since 8 May 2026 local ADG performs automatic failover with zero data loss. The OCI Console Edit disaster recovery dialog for UR-Prod offers both Autonomous Data Guard (RTO 2 min, RPO 10 sec) and Backup-based DR as choices. No need to switch to ATP.
- Recommendation: for a fleet of small APEX apps, do not pay for Oracle's premium DR fleet-wide. Use a DIY pattern (hourly Data Pump + APEXExport → Object Storage, optional shared warm-standby VPS) that gives RPO ≈ 1 hour and RTO 1–2 hours at near-zero per-app cost. Reserve Oracle's Autonomous Data Guard for the one or two apps (likely UR-Prod itself) that genuinely need sub-2-minute RTO with automatic failover.
1. How reliable is OCI ADB in practice?¶
Oracle's published SLAs:
| Configuration | Availability SLA | Implied monthly downtime |
|---|---|---|
| ADB with no standby (current) | 99.95% | ~21.9 min/month |
| ADB with Autonomous Data Guard | 99.995% | ~2.2 min/month |
The SLA only entitles you to service credits — it does not protect your data.
Reality — recent UK incidents:
- 24 Jan 2026 — OCI London region wobble. Fusion Apps users in the UK reported a ~10-minute production blackout around 12:30 local, followed by a separate 13:45 incident where test environments returned 502 Bad Gateway. Oracle's status page did not acknowledge it; Oracle declined public comment. (The Register, 28 Jan 2026.)
- This is consistent with a broader 2025–2026 pattern of OCI European incidents missing from Oracle's official status page. Third-party trackers (IsDown, StatusGator) surface them first.
- OCI London (UK-LONDON-1) has 3 ADs × 3 fault domains — an AD-level fault is contained, but a regional control-plane fault (Jan 2026 style) affects everything.
Net for the fleet:
- 99.95% sounds high but allows ~22 min/month of outage, and that excludes the daily-backup data-loss risk.
- Daily backup only = worst-case RPO up to 24h, RTO measured in hours.
- A regional event would take down a single-region setup completely; cross-region cover is the only protection against it.
2. Two paths to consider¶
| Path | Spirit | Best for |
|---|---|---|
| A. Oracle native DR (premium) | Pay Oracle to handle replication & SLA | One critical app where sub-minute RTO matters and downtime is expensive |
| B. DIY (recommended for the fleet) | Build a reusable cheap pattern that protects many apps at once | Small/medium apps, low traffic, hours-of-RTO is acceptable |
The two paths are not exclusive — you can run DIY for most apps and elevate one or two to Oracle native DR if the business case demands.
3. Path A — Oracle native DR options (reference)¶
| # | Option | RTO | RPO | Approx. cost vs. primary | Notes |
|---|---|---|---|---|---|
| A0 | Daily backup only (status quo) | Hours | Up to 24h | £0 extra | No protection beyond Oracle's automatic daily backups |
| A1 | Local Backup-Based DR (same region, different AD) | 1h + 1h per 5TB | 10 sec | £0 extra | Free upgrade. AD-fault protection. Manual switchover. |
| A2 | Local Autonomous Data Guard | < 2 min, auto | 0 (sync) | +100% compute, +100% storage | Unlocks 99.995% SLA. APEX supported since Nov 2025. |
| A3 | Cross-Region Backup-Based DR | 1h + 1h per 5TB | 1 min | Small extra (backup replication + remote storage) | Cheapest regional protection. Manual failover. |
| A4 | Cross-Region Autonomous Data Guard | 15 min | 1 min | +100% compute + 2× storage | Best regional protection. Manual by design — cross-region failover is never auto. |
Key nuance on "auto" failover: only local ADG is genuinely automatic (instance-level faults handled inside Oracle's 99.995% SLA). Cross-region failover is always a manual decision — Oracle deliberately won't auto-fail-over across regions because the mid-tier (APEX URL, DNS, traffic) must move too. "Fully automatic cross-region zero-touch failover" is not an out-of-the-box ADB feature at any price.
3.1 Does the "APEX Service" workload type support Autonomous Data Guard? (Verified against official Oracle docs, May 2026)¶
Short answer: yes — and as of 8 May 2026, automatic failover with zero data loss is now built in. No workload-type switch required.
Terminology check — per Oracle's official About Autonomous AI Database Workload Types page, Autonomous AI Database has four workload types: Lakehouse, Transaction Processing, JSON Database, and APEX Service. "APEX Service" IS a workload type (not a separate SKU). So when the release notes say "APEX workload," they mean exactly the workload type you're on now.
Two relevant official release notes (from the Autonomous Database Serverless release notes index):
| Date | Release note (verbatim) | Impact |
|---|---|---|
| 11 Nov 2025 | "You can enable Autonomous Data Guard on Autonomous AI Databases with the JSON and APEX workloads for both local and cross-region standby databases." | ADG can be enabled on APEX-workload ADBs — both local & cross-region. No paid-only / ECPU-only / size caveats. |
| 8 May 2026 (release notes index) | "For Autonomous Data Guard, zero data loss protection (RPO = 0) is provided for a local standby database. Autonomous Data Guard performs automatic failover to a local standby database when a standby is available and the system guarantees zero data loss. If a data loss limit is specified (0 to 3600 seconds), failover occurs within the defined limit." | Local ADG failover is now genuinely automatic with RPO=0. You can also set a tolerance (0–3600s) if you'd rather fail over even when the standby is slightly behind. |
The Oracle documentation also confirms (in the Automatic Failover with a Standby Database guide): after an automatic failover completes, a new local standby database is automatically created — i.e. you stay protected for the next event without manual intervention.
⚠️ Contradictory Oracle page to be aware of: the "Autonomous Database Limitations in APEX Service" docs page is still up and still says "No ability to set up Autonomous Data Guard standbys." This page is stale — superseded by the Nov 2025 and May 2026 release notes. If anyone on the team cites it, point them at the release notes index. (It's safe to assume Oracle will eventually update the limitations page; for now it's a documentation defect.)
Verified live on our tenancy (19 May 2026): the OCI Console Edit disaster recovery dialog on UR-Prod (APEX Service workload, UK South / London) presents both Autonomous Data Guard and Backup-based disaster recovery as selectable DR types. Oracle's own dialog quotes:
- Autonomous Data Guard — RTO 2 mins, RPO 10 secs
- Backup-based disaster recovery — RTO 1 hour + 1 hour per 5 TB, RPO 10 secs
So no ATP upgrade is needed — switching DR type happens entirely through Edit disaster recovery. The console also warns that "Backup-based disaster recovery... should be used for non-production databases" — Oracle's own positioning is that production should run ADG. Note the tenancy quota: 1 local peer and 1 cross-region peer per database — switching from BBDR to ADG replaces the existing peer rather than running alongside it.
3.2 What if you upgrade APEX Service → full ATP anyway?¶
The upgrade is online and in-place (no downtime, no dropped connections) via Workload type → Edit → Convert. But the pricing model changes — you lose the APEX Service discount tier.
Indicative per-ECPU rates (USD list, PAYG — see Oracle Autonomous AI Database Pricing UK and the OCI Price List):
| SKU | $/ECPU/hr | Monthly for 2 ECPU (730 hr) baseline | Notes |
|---|---|---|---|
| APEX Service (your current tier) | ~$0.082 effective | ~$122/mo for 2 ECPU + 20 GB | Heavily discounted, locked to APEX workload |
| ATP (Transaction Processing) PAYG | $0.336 | ~$490/mo for 2 ECPU before storage | Full feature set |
| ATP BYOL | $0.1058 | ~$155/mo + license cost | Only if you already own DB EE / ULA licences |
So upgrading to ATP just to get ADG would roughly 4× your base bill, then doubling again for ADG itself — a ~8× jump from ~$120/mo to ~$1,000/mo per app. This is why the current state of play matters: since Nov 2025 you can stay on APEX Service and still get ADG.
3.3 When upgrading to ATP would actually be worth it¶
Independent of DR, ATP unlocks features APEX Service blocks: - Database links (cross-database queries / federation) - Refreshable clones (cheap read-only copies for reporting) - Customer-managed ORDS deployment - MongoDB API, DataSafe, Graph Studio, OML Notebooks - Read-only mode toggle - BYOL licensing (drops the per-ECPU rate from $0.336 to $0.106 if you already own DB EE licences — only worth it at scale)
If none of these are in scope, stay on APEX Service.
Indicative cost per app (~$120/mo APEX-Service baseline, partial usage):
| Option | Workload type needed | Extra/mo | New total |
|---|---|---|---|
| A0 status quo | APEX Service (current) | $0 | ~$120 |
| A1 Local BBDR | APEX Service | $0 | ~$120 |
| A2 Local ADG | APEX Service (since Nov 2025) | ~$80–120 | ~$200–240 |
| A3 Cross-Region BBDR | APEX Service | ~$10–25 | ~$130–145 |
| A4 Cross-Region ADG | APEX Service (since Nov 2025) | ~$100–150 | ~$220–270 |
| A5 Upgrade to ATP + Local ADG (for non-DR reasons) | ATP | adds ~$370 to switch + ~$490 for ADG | ~$980/mo |
Multiply by N apps to see why this gets expensive fast: 5 small apps on Local ADG = ~$1,000+/mo just for DR. This is where DIY wins.
4. Path B — DIY pattern (recommended for the fleet)¶
Goal: RPO of 15–60 minutes and RTO of 1–2 hours for any app in the fleet, at near-zero per-app cost, using only tools Oracle already ships and infrastructure you already pay for.
Implementation runbooks: the concrete how-to for every operation in this section lives in two companion runbooks: - RB-004 — OCI ADB Data Pump runbook: copy-pasteable PL/SQL recipes (full / schema export, scheduled hourly backup, restore-to-fresh-ADB), the
temp/permanentbucket-prefix convention, lifecycle rules, troubleshooting. - RB-005 — APEX app + schema clone runbook: the prod → pre-prod clone procedure that pairs Data Pump withapex export/apex import— verified end-to-end on 19 May 2026 against UR-Prod → UR-PreProd. Together these implement the "Data Pump + APEXExport + metadata script" pattern this section recommends.
4.1 What gets backed up¶
For every app, three things must be captured together to make a full restore possible:
- Database content — schema + data, via Oracle Data Pump (
DBMS_DATAPUMP) exported directly to OCI Object Storage. Supported by ADB out-of-the-box. No agents, no extra licensing. - APEX applications + workspace — via the
APEXExportJava utility (or SQLclapex export). Captures every app, page, theme, shared component, supporting object. - Environment metadata — ORDS modules, REST data sources, scheduler jobs, ACLs, credential names (not values), wallet references. A small SQL script can dump these into a versioned text file.
All three artefacts get pushed to a single OCI Object Storage bucket per environment (versioned, lifecycle-managed). One bucket can hold the artefacts for all apps in the fleet.
4.2 Backup schedule¶
Use DBMS_SCHEDULER inside ADB itself to run the export jobs — no external compute needed.
- Every 1 hour (recommended starting point): full Data Pump export of all user schemas → Object Storage. Small DBs (< 5 GB) finish in seconds.
- On every APEX deploy (and once daily as a safety net):
APEXExportof every app, written to the same bucket. - Weekly: full environment metadata snapshot (ORDS config, jobs, ACLs).
- Lifecycle rule on bucket: keep last 7 days hourly, last 30 days daily, last 12 months monthly. Object Storage handles this natively.
This gives RPO ≈ 1 hour at essentially zero infra cost. You can tighten to 15 min by running the job more often (still fine — Data Pump on a small DB is fast).
4.3 Recovery runbook (same for every app in the fleet)¶
- Provision a fresh ADB instance from the OCI console (~5–10 min) — or keep a pre-provisioned Always Free ADB warm in a second region as a hot landing zone (2 free per tenancy, 20 GB each).
- Run a one-line
impdpagainst the latest dump in Object Storage (~5–30 min depending on size). - Run
APEXImportfor each app's latest export (~1 min per app). - Apply the environment metadata script (ORDS, ACLs, scheduler jobs).
- Re-point DNS / app config to the new ADB endpoint.
Realistic RTO: 60–120 minutes for a small APEX app, mostly limited by ADB provisioning time. Same runbook works for every app in the fleet — write it once, reuse across all apps.
4.4 Optional: warm standby for tighter RTO¶
If 60–120 minutes RTO is too slow, add a single shared warm-standby VPS running Oracle Database 23c Free (formerly XE — 12 GB user data limit, free for production, includes APEX). Hetzner / Contabo / OVH VPSes start around £4–8/month.
- Single VPS hosts multiple app schemas (one DB instance, many schemas) — cost is shared across the fleet.
- A nightly
impdpfrom the latest Object Storage dump keeps it within ~24 hours of primary. - On disaster: re-point app config to the standby VPS — RTO drops to ~5–10 minutes.
- Trade-offs: 23c Free has feature limits (12 GB per database total, no enterprise features); fine for small APEX apps but not suitable for any app that uses partitioning, advanced compression, Spatial/Graph, etc.
4.5 What this costs¶
| Component | Approx. cost |
|---|---|
| OCI Object Storage for entire fleet of backups (well under 20 GB total for most APEX shops) | $0 (covered by Always Free 20 GB) |
| Always Free ADB as warm landing zone (optional) | $0 |
| Shared warm-standby VPS (optional) | £4–8 / month total, not per app |
| One-off scripting effort | ~1–2 days |
| Per-app marginal cost | ≈ $0 |
For a fleet of (say) 5 apps, total DR spend is at most £8/month for everything, vs. ~£500–1000/mo if you ran Local ADG on each.
4.6 Honest limitations of the DIY approach¶
- No 99.995% SLA — Oracle won't credit you for outages of a setup they don't manage.
- RPO floor ≈ 15 min (export frequency). Sub-minute RPO needs real replication (Data Guard / GoldenGate), which means paying Oracle.
- RTO depends on script discipline — if the runbook isn't drilled, recovery will be slower than estimated.
- APEX session state and in-flight transactions are lost — anything committed since the last export.
- 23c Free standby has feature limits; check each app's feature use before relying on it.
- No automatic failover trigger — a human decides when to fail over (same as cross-region ADG, actually).
5. Recommended path for the Parallax fleet¶
Given the multi-app context and minimum-cost preference:
Phase 1 — Free wins (this week, no extra spend)
- ✅ UR-Prod has Local Backup-Based DR enabled (since 15 Jan 2026) — done. RPO 10 sec, RTO ~1 h, no extra cost.
- Audit every other app in the fleet in OCI Console → Disaster Recovery tab. For any still on daily-backup-only, click Add peer database → Backup-based, Local → save. One click, zero cost per app.
- Build the DIY export pipeline (Path B): hourly Data Pump + APEXExport + metadata script, pushing to a single Object Storage bucket. Belt-and-braces on top of Oracle's Backup-Based DR, and gives you portable artefacts (independent of Oracle's tooling and tenancy).
- Document the recovery runbook once, used by every app.
Phase 2 — If a regional event must be survived (next month, ~£5/mo)
- Add cross-region replication on the Object Storage bucket (replicates dumps to Frankfurt or Amsterdam automatically). Object Storage cross-region replication is much cheaper than Cross-Region BBDR. Tracked as RM-047 — also covers subscribing both tenancies to the second region (the prerequisite step) and enabling cross-region BBDR peers on the customer-facing ADBs.
Phase 3 — Production needs sub-2-minute RTO (decision point for UR-Prod)
- ✅ UR-Prod (E5) — Local Autonomous Data Guard enabled on 2026-05-19. RTO ~2 min, RPO 0, automatic failover, no URL change. Cost approximately doubled (paid APEX Service + ADG = +100% compute + 100% storage). Done.
- Next E5 step — add a cross-region Backup-Based DR peer to cover the single-region failure mode. The cross-region peer slot is independent of the local ADG peer (tenancy quota = 1 local + 1 cross-region per database), so this stacks on top of the local ADG without disturbing it. Low cost (~$10–25/mo backup-storage). Tracked under RM-047. Cross-region ADG remains not recommended for E5 — the cost (+2× storage + 100% compute again) is hard to justify versus a cheap cross-region BBDR peer when cross-region failover is a manual decision in either case.
- For other apps, only repeat step 6 if the business case justifies it. Don't pay for ADG fleet-wide.
- (Optional) Stand up the shared warm-standby VPS (Oracle 23c Free, ~£5/mo for the fleet) as a cheap secondary recovery target for non-production apps.
Total fleet DR spend with this approach: ~£0–10/mo for the fleet, plus the ~$120/mo for UR-Prod elevated to ADG (now live as of 2026-05-19). Compared to ADG-everywhere (~£500+/mo), this protects the production app well while keeping the long tail cheap.
6. Migration plan — concrete steps¶
6.1 Phase 1 implementation (½ day per environment)¶
- Create an Object Storage bucket in your home region (London). Enable versioning + a lifecycle rule (7d hourly / 30d daily / 12m monthly).
- Create an OCI credential inside each ADB (
DBMS_CLOUD.CREATE_CREDENTIALwith an auth-token). - Write the export PL/SQL block that calls
DBMS_DATAPUMPwith the bucket URL as the dumpfile target (Oracle supports direct-to-Object-Storage dumps). - Schedule it with
DBMS_SCHEDULERto run hourly. - Stand up an
APEXExportrunner — either a small GitHub Actions job (free) or a cron on any existing server. It connects via SQLcl and exports each app to the same bucket. - Capture environment metadata — a SQL script that queries
USER_ORDS_*,USER_SCHEDULER_JOBS,DBA_NETWORK_ACL_PRIVILEGES, then writes the result to Object Storage. Run weekly. - Enable Local Backup-Based DR in the OCI Console for each ADB (zero cost, two clicks).
- Write the recovery runbook — one Markdown doc, parameterised by app name. Drill it once against a throwaway ADB.
6.2 Phase 2 — Cross-region replication (1 hour)¶
- In OCI Console → Object Storage → bucket → Replication Policy → target a bucket in eu-frankfurt-1 or eu-amsterdam-1. Asynchronous, cheap.
- Validate by triggering a manual export and confirming the object appears in the remote bucket within minutes.
6.3 Phase 3 — Warm standby (optional, 1 day)¶
- Provision a VPS (Hetzner CX22 ~£4/mo is plenty for a small APEX standby).
- Install Oracle Database 23c Free (Docker image:
container-registry.oracle.com/database/free). APEX is bundled. - Write a nightly job that pulls the latest dump from Object Storage and
impdps into a per-app schema. - Update the recovery runbook with the "fail to standby" path.
6.4 Validate¶
- Twice a year, run a real DR drill: pick one app, simulate a regional outage, recover into a new ADB / the warm standby, time it, compare to the documented RTO/RPO. Adjust the runbook with anything you learned.
7. Risks & things to verify before committing¶
- Object Storage egress — minimal at the volumes implied (dumps measured in MB for small APEX apps), but worth measuring once the fleet's real size is known.
- APEX session state and saved interactive reports — live in flashback-managed APEX tables; exports do capture them, but worth confirming on a real recovery test.
- Data residency — if the fleet stores UK-personal-data, confirm Frankfurt/Amsterdam replication targets are acceptable to legal. Object Storage replication is no different from Oracle's own cross-region DR in this respect.
- DBMS_SCHEDULER quotas on ADB — minimum-shape instances have job-slave limits; an hourly export job is well within them, but check before scheduling 5-minute exports.
- 23c Free production-use clause — free including production, but Oracle does not patch it. Treat the standby VPS as ephemeral; rebuild on a quarterly cadence to pick up the latest Free release.
- Multi-app tenancy hygiene — keep each app's schema and bucket prefix clearly separated so recovery never touches the wrong app.
Sources¶
Autonomous Data Guard & failover (the core "is auto-failover possible on APEX?" question)¶
- Oracle release note, 11 Nov 2025 — ADG support for JSON and APEX workloads (authoritative for the APEX-workload change)
- Oracle release notes index — Autonomous Database Serverless — 8 May 2026 entry: zero data loss protection with local ADG
- Oracle docs — Automatic Failover with a Standby Database
- Oracle docs — Use Standby Databases with Autonomous Data Guard for Disaster Recovery
- Oracle docs — Autonomous Data Guard Switchover and Failover Operations
- Oracle docs — Enable Autonomous Data Guard
- Oracle docs — About Autonomous Data Guard with Cross-Region Standby
- Oracle blog — Cross-Region Autonomous Data Guard overview
- Oracle blog — Configure a data loss limit for your local standby's automatic failover
- ⚠️ Oracle docs — Autonomous Database Limitations in APEX Service (STALE — still says ADG not supported; superseded by the Nov 2025 release note)
Backup-Based Disaster Recovery (the free / low-cost Oracle option)¶
- Oracle docs — About Backup-Based Disaster Recovery
- Oracle docs — Backup-Based DR — Switchover and Failover Operations
- Oracle docs — Backup-Based DR — Cross-Region Operations
- Oracle blog — Introducing Backup-Based Disaster Recovery: low-cost DR option
Workload types, APEX-vs-ATP upgrade, pricing¶
- Oracle docs — About Autonomous AI Database Workload Types
- Oracle docs — Upgrade APEX Service to Autonomous Transaction Processing
- Oracle — Autonomous AI Database Pricing (UK)
- Oracle — OCI Price List
- Oracle — APEX Application Development Service pricing
- Oracle — Autonomous Database ECPU FAQ (PDF)
SLA & uptime¶
- Oracle — OCI Service Level Agreement
- Oracle blog — 99.995% SLA with Autonomous Data Guard announcement
- Oracle — Cloud Status and Availability (UK Trust Center)
- Oracle — Official OCI Status page (known to under-report — see Jan 2026 incident below)
- Third-party status trackers — IsDown · StatusGator
Recent incidents¶
- The Register, 28 Jan 2026 — UK users say Oracle Cloud Infrastructure wobbled last week
DIY backup pipeline (Path B references)¶
- Oracle docs — Use Oracle Data Pump to Export Data from Autonomous Database (to Object Storage)
- Oracle docs — Export Data to Object Store — DEFAULT_CREDENTIAL setup
- Oracle docs — Exporting Data from Autonomous AI Database to Object Store or Other Oracle Databases
- Oracle docs — APEX — Managing Application Backups (APEXExport)
- Oracle docs — Always Free Autonomous AI Database
- Oracle — Database 23c Free product page · Licensing restrictions (12 GB user data)
OCI region structure¶
- Oracle docs — Regions and Availability Domains