Zero Downtime Database Migration: Billions of Rows, No Drama

We had billions of nodes in Neo4j. The system was slowing down. Our access patterns had nothing to do with graph traversals anymore. And we couldn't afford a single second of downtime.

So we migrated everything to MongoDB. Live. While serving millions of requests per day.

Here's how we did it -- and what almost went wrong.

Why We Had to Move

Neo4j was picked early in Richpanel's life when the data model genuinely looked like a graph. Customer conversations, agent assignments, ticket routing -- it all had relationships. Graph made sense.

But as the product evolved, 90% of our queries became simple lookups and aggregations. We were paying the overhead of a graph database for operations that a document store handles natively. Query performance degraded as the dataset grew. What used to take 20ms started taking 200ms, then 2 seconds. At 10 million orders ingested per day across Shopify, Magento, and WooCommerce, this wasn't sustainable.

The data model had drifted from its original design. We weren't traversing relationships. We were fighting the database to do things it wasn't built for.

The decision wasn't "Neo4j is bad." It was "Neo4j is wrong for what this system became."

The Approach: No Big Bang

The worst thing you can do with a migration this size is a big bang cutover. Freeze writes, dump everything, import, flip the switch, pray. With billions of nodes and a system that can't go down, that's not an option.

Instead, we used a phased approach:

1. Dual-write -- every new write goes to both databases
2. Backfill -- migrate historical data in the background
3. Validate -- compare results between old and new
4. Cut over -- switch reads to the new database
5. Clean up -- remove old code paths

Each phase was independently deployable and reversible. No phase required downtime. Every phase had a kill switch.

Phase 1: Dual-Write

The first step was making every write operation hit both Neo4j and MongoDB. This is simpler in concept than in practice.

The key decision: MongoDB writes were secondary. If a MongoDB write failed, we logged the failure and moved on. Neo4j remained the source of truth. If Neo4j failed, the whole operation failed -- same as before.

async function createOrder(data) {
  const result = await neo4j.createOrder(data)

  mongoWrite(data).catch(err => {
    failureQueue.push({ operation: 'createOrder', data, error: err.message })
  })

  return result
}

We wrapped every write path. There were about 40 of them. Each one got the same treatment: write to Neo4j synchronously, fire-and-forget to MongoDB, capture failures for retry.

The failure queue was critical. We processed it every 5 minutes, retrying failed MongoDB writes. If a write failed three times, it got flagged for manual inspection. In practice, less than 0.01% of writes ever hit the failure queue, and most of those were transient network issues.

This phase ran for two weeks before we moved to backfill. We wanted confidence that the dual-write layer was stable and that MongoDB's schema could handle everything being thrown at it.

Phase 2: Backfill

New data was flowing into both databases. But we had billions of existing nodes in Neo4j that MongoDB didn't know about.

Backfilling at this scale requires patience. You can't just run a migration script that reads everything and writes it all at once. You'll crush both databases.

We processed records in batches of 5,000. Each batch had a deliberate pause between iterations. We rate-limited ourselves to ensure production traffic wasn't affected.

async function backfill(entityType, lastProcessedId) {
  const batch = await neo4j.query(
    `MATCH (n:${entityType}) WHERE id(n) > $lastId RETURN n ORDER BY id(n) LIMIT 5000`,
    { lastId: lastProcessedId }
  )

  if (batch.length === 0) return

  await mongodb.insertMany(entityType, batch.map(transform))
  await sleep(200)
  return backfill(entityType, batch[batch.length - 1].id)
}

The backfill ran for about three weeks. We processed different entity types in parallel but throttled the total throughput. During peak hours, we slowed the backfill down. During off-peak, we cranked it up.

One thing we learned early: don't trust the transform function on day one. We ran the backfill for a single entity type first, validated the output manually, then expanded to others. The first run revealed three data type mismatches that would have silently corrupted data if we'd gone all-in immediately.

Phase 3: Validate

This is the phase most people skip. It's also the phase that saves you.

We built a shadow-read system. For a percentage of read queries, we'd hit both databases and compare results. The comparison ran asynchronously -- it never affected response times for users.

async function getOrder(orderId) {
  const result = await neo4j.getOrder(orderId)

  validateAgainstMongo(orderId, result).catch(logMismatch)

  return result
}

The mismatch rate started at about 2%. That sounds small, but at our scale it meant millions of inconsistent records. We traced the mismatches to three root causes:

Timing issues. A record gets updated in Neo4j but the MongoDB write hasn't been processed yet. By the time validation runs, the two databases are momentarily out of sync. We fixed this by adding a 30-second grace period -- if a record was updated in the last 30 seconds, we skipped validation for it.

Data type differences. Neo4j stores everything as native types. Some of our transform functions were converting integers to strings, or dates to timestamps in different formats. These weren't bugs in the data -- they were bugs in the comparison logic. We normalized both sides before comparing.

Edge cases with deletes. When a record was deleted from Neo4j, the dual-write layer deleted it from MongoDB too. But the backfill script didn't know about deletes -- it only copied what existed. If a record was deleted after the backfill read it but before the backfill wrote it, we'd have a ghost record in MongoDB. We added a reconciliation pass that cleaned these up.

After fixing these, the mismatch rate dropped to 0.001%. We ran validation for another two weeks at that level before proceeding.

Phase 4: The Cutover

This is the moment that keeps you up at night. Switching reads from Neo4j to MongoDB.

We didn't do it all at once. We used feature flags to route a percentage of read traffic to MongoDB.

• Day 1: 1% of reads go to MongoDB
• Day 3: 10%
• Day 5: 25%
• Day 7: 50%
• Day 10: 100%

At each step, we monitored latency, error rates, and business metrics. If anything looked off, we could flip the flag back in seconds.

The 1% stage caught a query that performed well in testing but fell apart under production load. One of our aggregation queries wasn't using the right index. It worked fine at low traffic but started timing out at higher percentages. We added the missing compound index, verified in staging, and resumed the rollout.

By day 10, all reads were on MongoDB. P95 latency for our core APIs dropped by about 40%. Queries that were fighting Neo4j's architecture now had a database built for exactly those access patterns.

Phase 5: Cleanup

Once MongoDB was handling 100% of reads and writes, we kept Neo4j running in receive-only mode for another two weeks. This was our safety net. If something catastrophic showed up in MongoDB, we could switch back.

Nothing did.

We removed the dual-write code, the validation layer, the feature flags, and finally the Neo4j connection itself. Each removal was its own deployment. We didn't batch them because each one carried risk, and we wanted to isolate failures.

The cleanup took about a week of engineering time. It's tempting to skip this and leave the old code in place "just in case." Don't. Dead code paths are a maintenance burden and a source of confusion for every engineer who touches the codebase after you.

What Almost Went Wrong

Concurrent write conflicts. During the backfill, a record could be modified by a user while the backfill was copying it. The backfill would write the old version to MongoDB, then the dual-write layer would write the new version. But if the backfill write happened after the dual-write, it would overwrite the newer data with stale data.

We solved this with a simple rule: backfill writes use insertIfNotExists. If a record already exists in MongoDB (put there by the dual-write layer), the backfill skips it. The dual-write version is always more recent than whatever the backfill would copy.

Connection pool exhaustion. Writing to two databases means twice the connections. We didn't account for this initially. MongoDB's connection pool hit its limit during a traffic spike, causing a cascade of timeouts. We bumped the pool size, but the real fix was making the MongoDB writes truly async with a dedicated connection pool separate from any future read pool.

Schema evolution during migration. Halfway through the backfill, we shipped a feature that added a new field to one of our core entities. The backfill script didn't know about this field. Records backfilled before the feature shipped were missing it. We had to add a patch migration to fill in defaults for those records.

Lesson: freeze schema changes during migration if you can. If you can't, make sure every schema change also updates the backfill transform.

What to Monitor

During a migration like this, your dashboards are your lifeline. Here's what we watched:

• Dual-write failure rate -- should stay under 0.1%. Spikes mean connectivity issues or schema problems.
• Backfill throughput -- records per second. Drops mean you're hitting rate limits or the source database is under pressure.
• Validation mismatch rate -- should trend toward zero. If it goes up, stop and investigate.
• Read latency on new database -- compare against the old. It should be equal or better. If it's worse, your indexes or schema are wrong.
• Error rates on both databases -- any spike on either side during any phase means pause and investigate.
• Application-level business metrics -- orders processed, messages sent, tickets created. These are your ground truth. If the numbers don't match historical patterns, something is wrong regardless of what your technical metrics say.

When NOT to Do This

Zero-downtime migration is expensive. Not in money -- in engineering time, complexity, and cognitive load. It took us about six weeks end to end. A maintenance-window migration of the same data might have taken a weekend.

Do the maintenance window when:

• Your system can tolerate a few hours of downtime (many B2B products can, late on a Saturday night)
• The dataset is small enough to migrate in one shot (under a few hundred million rows)
• You don't have the engineering bandwidth to build and maintain the dual-write infrastructure
• The migration is a one-time event, not something you'll need to repeat

Do the zero-downtime approach when:

• Downtime directly costs you revenue or trust (e-commerce, messaging, real-time systems)
• The dataset is too large to migrate in any reasonable maintenance window
• You need the ability to roll back instantly
• You're migrating a system that other systems depend on

We didn't have a choice. With billions of nodes and a system handling 10 million orders per day, even an hour of downtime was unacceptable.

The Takeaway

Zero-downtime migration isn't about clever engineering tricks. It's about discipline. Each phase is simple. The hard part is having the patience to run each one long enough to trust it, and the restraint to not skip validation because things "look fine."

The migration took six weeks. The system never went down. Not once.

Build the kill switches. Watch the dashboards. Don't rush the validation. And when it's done, clean up your mess.