Parallel Migration Pipeline
Scaling a distributed financial application from sequential processing to parallel execution.
Legacy state
35K
group records, about 14 hours
Sequential processing, Manual recovery, Limited scaling
Migration capability
3.26M+
records migrated, about 5.5 hours
Parallel processing, Automatic recovery, Horizontal scaling
Figures reflect the legacy state and the resulting migration capability, not a controlled apples-to-apples benchmark.
The problem
The distributed financial application at the centre of this project originally processed its data sequentially. Records moved through the pipeline one after another, which made the system simple to reason about but slow to run at scale.
In practice, processing 35,000 group records took nearly 14 hours. Recovery from a failure was manual, and scaling meant running the same sequential path harder rather than running more of it in parallel.
Legacy architecture
The legacy design was a single, linear processing path: records were pulled from the source, processed in order, and written out. It was correct, but every record waited for the one before it.
Note
The bottleneck wasn’t simply CPU. The source system’s data transmission rate also limited ingestion speed: the application could process faster than data was being delivered.
The engineering challenge
The redesign had to introduce concurrency without sacrificing correctness: no lost records, no double-processing, and it had to recover from failures without an operator watching over it.
New architecture
Records are now ingested and distributed across a pool of parallel workers, processed concurrently, and persisted to PostgreSQL. The worker count is the scaling lever: more workers, more throughput.
Scroll to see the full diagram.
- Parallel execution
- Automatic recovery
- Horizontal scaling
- Monitoring
Parallel processing
With records distributed across workers, throughput scales with the number of active workers. Peak Loan Account processing reached about 300 records per second, and Savings Account processing reached about 170 records per second.
Automatic recovery
Stuck jobs are detected and recovered automatically. A job that stalls no longer blocks the queue or requires a manual restart: the system returns it to a healthy state on its own.
Database optimization
Database efficiency was improved as part of the redesign, reducing the contention that a concurrent workload would otherwise introduce.
Monitoring
Live monitoring and visibility were added, so operators can see what the pipeline is doing in real time rather than guessing after the fact.
Load testing
Note
The migration was deliberately tested at about half of server capacity. That left meaningful headroom for production.
Testing at roughly half of available capacity validated the redesign while keeping a comfortable safety margin for real workloads.
Results
The redesigned pipeline migrated more than 3.26 million records in about 5.5 hours. The sequential legacy state, 35,000 group records in nearly 14 hours, illustrates the capability gap the redesign closed.
Production scale
In production, the system has processed more than 5.7 million Celery task executions with a 99.7% task success rate, operating across 79 branches.
Lessons learned
- The bottleneck is rarely where you first look: measure before you optimize.
- Concurrency is only valuable when the constraint isn’t upstream of you.
- Recovery should be part of the design, not a runbook written after the outage.
- Test with headroom: production behaves differently than a saturated bench.
Related
- DRIVER, thirty years of data, one migration, zero loss.
- Why parallel processing isn’t automatically faster