KernforceBook a call

E-commerce · 18 RDS PostgreSQL instances on AWS · under NDA

Consolidating 18 production databases into 5

Eighteen RDS PostgreSQL instances, one per service, most of them idling at 2–10% CPU, costing roughly $2,000 a month in instance hours and allocated storage while doing the work of a fraction of that hardware. The real constraint was not technical: two windows of fifteen to twenty minutes, on separate days, for a payment-adjacent platform.

The situation

The guiding decision was to separate two things that are usually conflated: how long a migration takes, and how long the service is down. The first ran for weeks. The second had to fit inside twenty minutes.

Solution

01Move everything that can happen outside the window, outside it

For each database, native PostgreSQL logical replication — a publication on the source, a subscription on the target — with the initial copy running under live traffic for as long as it needed. The largest pair (800 GB) took about eleven hours to seed. By the time the window opened, replication lag was zero and had been for days.

02Choose the tool by its failure mode

We chose logical replication over AWS DMS deliberately. A prior benchmark on the same data had shown DMS reporting a green, successful full load while silently dropping 1.5 million rows. Native replication is slower to set up and less forgiving of schema drift, but its failure modes are loud and its guarantees are the database's own.

03Pre-stage everything replication does not carry

Logical replication carries rows, not structure. We loaded the schema ahead of time, generated primary keys from the source catalog — a table without a primary key stalls the apply worker for the entire subscription, not just itself — and transferred object ownership explicitly, a detail schema dumps drop by default and which surfaces later as permission denied on the first application query.

Secondary indexes were built online on the running target: 26 indexes totalling 81 GB, created with CREATE INDEX CONCURRENTLY, which costs a second table scan but never blocks the apply worker. Replication kept flowing, no write-ahead log piled up on the sources, and it finished in 1 hour 42 minutes with zero downtime on the existing instance class.

04Reduce the cutover to reversible steps

Scale the applications to zero, confirm replication lag is zero, synchronise sequences — logical replication does not carry them, and without this step the first insert collides with existing keys — repoint the applications by rewriting their entries in Secrets Manager, force the secret operator to refresh, refresh planner statistics, scale back up, verify.

05Verify every step independently, rather than trusting it

This was the highest-value discipline of the project. The sequence-sync step once reported “10 sequences transferred” while applying none of them: mixed-case identifiers had been silently folded to lower case, and the errors were formatted in a way the script's own filter missed. A comparison of sequence values between source and target caught it. Had it gone unnoticed, two services would have begun inserting from ID 1 on top of nine million existing rows.

Outcome

18 → 5
instances, 12 databases migrated
25 min
total downtime across both windows
$24k
recurring saving per year
Downtime, window 1
17 min 48 s — 10 databases, 13 deployments
Downtime, window 2
7 min 20 s — 2 databases, 800 GB
Index build
26 indexes, 81 GB, 1 h 42 min, no downtime
Data loss
None
Rollbacks used
None

Both windows came in under the allotted time. The rollback path was written, rehearsed and never needed. Retired instances were stopped first and deleted only after a settling period, with their backup retention extended to one year so the old state stays recoverable.

Core tech

  • PostgreSQL
  • Amazon RDS
  • AWS Secrets Manager
  • AWS Backup
  • AWS Systems Manager
  • Amazon EKS
  • Argo CD
  • External Secrets Operator
  • Bash

Method

What transfers to other migrations

Nothing here is proprietary. This is the part that transfers to your estate whether or not you ever call us.

Design downtime as a separate budget from migration time

Almost every step can be moved outside the window if you are willing to let it run long. What must stay inside is small: catch up the lag, fix sequences, repoint the clients.

Verify the result, not the tool's report

Three separate tools here reported success while doing nothing — a migration script, a GitOps controller showing Synced against a stale revision, and a REVOKE that ran without error but blocked no one. Every step needs a check that looks at the world, not at the exit code.

Know what your replication does not carry

Sequences, tables without primary keys, ownership, large objects, and anything outside the publication's schema list. Each of those is a silent gap, not an error.

Build indexes concurrently, on the target, before the window

CREATE INDEX CONCURRENTLY costs a second scan and buys the one thing that matters: the apply worker never blocks, so no write-ahead log accumulates on a production source.

More

Other work

All case studies

E-commerce · two EKS clusters on AWS · under NDA

Making a savings number smaller

Detectors summed to a headline figure. Reconciling the overlaps between them removed roughly three quarters of it — and that is what made the remainder usable.

E-commerce · production EKS on AWS · under NDA

A storage bill grew 20× on a filesystem holding 8 GB

Daily spend started climbing in the same week a dozen database instances were shut down — so the bill should have been falling. The billing line said EFS. The cause was in node provisioning, two layers away.

Gambling · managed Kubernetes on AWS · under NDA

Three EKS versions upgraded without a maintenance window

A cluster two versions past its support window, moved forward across three minor releases with no scheduled downtime — rehearsed first, with the disruption budgets fixed before anything moved.

B2B SaaS · CI on a hosted runner fleet · under NDA

A CI pipeline from 38 minutes to 9

Build and test times measured per step before anything was touched, then caches that actually hit, parallelism where the dependency graph allowed it, and the work that had been running on every commit for no reason.