Skip to content

Gambling · several production databases on one server · under NDA

Published · 3 min read

Splitting one database server into three

Several of the platform's production databases shared one PostgreSQL server. That was a sensible shape when it was chosen; by the time it was a problem, every database on the machine was exposed to whatever the busiest one happened to be doing.

The situation

The databases competed for the same CPU, memory and disk I/O. A noisy neighbour is not a figure of speech here: one query plan going wrong on one database showed up as latency on all of them.

Capacity could only be bought for the whole machine. One database under load meant upgrading a server that every other database also sat on — paying for headroom that only one of them needed.

And maintenance was shared whether or not the work was. A restart, a minor-version upgrade or an unplanned fault on that host reached every database at once, which turned routine work into something that had to be scheduled like an incident.

Solution

01Group by what contends, not by what is biggest

The grouping came from what actually competed: which databases, and which of the heaviest tables, were driving CPU, memory and I/O in the same hours. A split by storage footprint produces neat numbers and leaves the contention exactly where it was.

That gave three dedicated RDS instances rather than one per database. Isolation is bought per instance, and an instance per database would have been paying for separation nobody had asked for.

02Stand each environment up before moving anything into it

Each target instance was configured as an independent environment first — parameters, backup schedule and monitoring belonging to that database rather than inherited from a shared host.

The data then moved into the environment it was going to live in, instead of into a copy of the old one that would be adjusted afterwards. Adjusting afterwards is how a shared configuration quietly reappears on three machines.

03Treat the blast radius as the thing being bought

The purpose was not performance in the abstract. It was that a fault, a tuning change or a backup on one database stops being an event for the others — which is a reliability property, and it is what the three instances were chosen to buy.

Outcome

3
dedicated instances, from one shared server
0
maintenance windows shared between databases
Shape before
Several production databases on one server
Shape after
Three dedicated RDS instances
Split decided by
What contended in the same hours, not by size
Scaling after
One instance at a time, where the load is

Response times became predictable, because they stopped depending on what a neighbouring database was doing. That is the outcome the client asked for, and it is as far as we will put it: there was no controlled before-and-after benchmark across the move, so there is no percentage in this paragraph.

Scaling became granular. The instance carrying the load can be scaled up on its own, rather than upgrading a machine that every other database also sits on and paying for headroom none of them use.

The single point of failure is reduced, not removed. Three instances is three blast radii instead of one; it is not high availability, and the case for that is a separate conversation with a separate price.

Core tech

  • AWS
  • Amazon RDS
  • PostgreSQL

Method

What makes this routine rather than risky

Nothing here is proprietary. This is the part that transfers to your estate whether or not you ever call us.

Group by contention, not by size

The databases worth separating are the ones competing for the same resource in the same hour. A split by storage footprint looks tidy on a diagram and changes nothing about the hour that hurts.

One instance per database is usually over-buying

Isolation is purchased per instance. Three was the number at which every pair that actually contended ended up apart; a fourth would have bought a boundary nobody needed and a bill that never stops.

Fewer instances is not the goal, and nor is more

We have published the opposite move as well — eighteen RDS instances consolidated into five, for a client paying for idle hardware. They are the same decision taken from different starting points: count what contends and what idles, then buy exactly that many boundaries. A rule that says consolidate, or a rule that says isolate, will be wrong on one of these two estates.

Make maintenance boring before you try to make it rare

The change that is hardest to put a number on is that routine work stopped needing a window large enough to cover every database at once. Upgrades that can be done one at a time get done.

More

Other work

All case studies

Gambling · 500+ applications across several data centres · under NDA

Moving more than 500 applications to AWS, one at a time

An on-premise estate spread across several data centres, grown past what it could carry. More than 500 applications moved to AWS one at a time, least critical first, with the strategy chosen per application — and about one in seven retired instead of moved.

B2B SaaS · two EKS clusters on AWS · under NDA

Making a savings number smaller

Detectors summed to a headline figure. Reconciling the overlaps between them removed roughly three quarters of it — and that is what made the remainder usable.

E-commerce · production EKS on AWS · under NDA

A storage bill grew 20× on a filesystem holding 8 GB

Daily spend started climbing in the same week a dozen database instances were shut down — so the bill should have been falling. The billing line said EFS. The cause was in node provisioning, two layers away.

E-commerce · 18 RDS PostgreSQL instances on AWS · under NDA

Consolidating 18 production databases into 5

A legacy account with one RDS instance per service, most idling at 2–10% CPU. The business would grant two maintenance windows of fifteen to twenty minutes, on separate days, for a payment-adjacent platform. Everything else had to happen under live traffic.

Illustrative · managed Kubernetes on AWS

Three EKS versions upgraded without a maintenance window

A cluster two versions past its support window, moved forward across three minor releases with no scheduled downtime — rehearsed first, with the disruption budgets fixed before anything moved.

Illustrative · CI on a hosted runner fleet

A CI pipeline from 38 minutes to 9

Build and test times measured per step before anything was touched, then caches that actually hit, parallelism where the dependency graph allowed it, and the work that had been running on every commit for no reason.