When more concurrency broke my bulk workflow
I increased concurrency to speed up a high-volume bulk workflow.
At first, it looked like the right move. Smaller runs got faster, throughput improved, and the pipeline seemed healthier.
Then larger runs started failing with PrematureCloseException.
That was the moment I realized the problem was no longer just performance. It had become a system pressure problem.
A few lessons from the debugging journey:
- more parallelism does not always mean more throughput
- chunk size is not just a batch setting — it becomes a stability boundary
- retries only help after the concurrency model is sane
- connection pool behavior matters a lot more under load
- partial-failure handling makes bulk workflows much more trustworthy
What finally helped was not one magic fix. It was a combination of:
- reducing unsafe parallelism
- tuning chunk size more carefully
- adding retry with backoff
- stabilizing connection pool behavior
- treating concurrency as a budget instead of a goal
I wrote the full debugging story here:
https://medium.com/p/758f87e312d5
Curious how others handle this kind of issue in bulk or async workflows.
