The Concurrency Bug That Only Appears Under Load: A Django Race Condition Story

Section 1: The Bug That Never Shows Up in Development

There's a category of bugs that haunts every backend engineer: the ones that pass every test you write, work perfectly on your laptop, and then — one day — silently break in production.

I found one of those bugs last week. Not in a library, not in a framework. In my own code.

It started with a simple feature: a user adds another user to a workspace. The service layer looked clean:

with transaction.atomic():
    if Membership.objects.filter(workspace=workspace, user=user).exists():
        raise ValueError(f"User {user} is already a member of workspace {workspace.name}")

    membership = Membership.objects.create(
        workspace=workspace,
        user=user,
        role=MembershipRole.MEMBER,
    )

    NotificationService.create(
        recipient=user,
        notification_type=Notification.Type.WORKSPACE_JOINED,
        actor=actor,
        workspace=workspace,
    )
    return membership

Read it once. Read it twice. It looks correct. There's a check, there's an insert, there's a notification. All inside an atomic block. What could possibly go wrong?

That pattern — check-then-create — is one of the most natural things a developer can write. And it has a flaw that only reveals itself when two requests arrive at the exact same time.

I wrote a concurrency test to prove the code was safe. Instead, the test proved it wasn't.

Section 2: The Journey — From a Green Test to a Hidden 500

Step 1: Writing the test that would expose everything

To test concurrency properly, I couldn't just call the service twice in a row. Sequential calls aren't concurrency — they're just two isolated operations. I needed two real requests racing against each other.

That meant three things:

1. Real transactions — not the wrapped ones pytest uses by default
2. Two independent threads — each with its own database connection
3. A synchronization barrier — so both threads start their critical section at the same moment

The setup looked like this:

@pytest.mark.django_db(transaction=True)
def test_concurrent_add_member_creates_one_membership_and_one_notification():
    workspace = WorkspaceFactory()
    actor_a = UserFactory()
    actor_b = UserFactory()
    target = UserFactory()

    MembershipFactory(user=actor_a, workspace=workspace, role=MembershipRole.OWNER)
    MembershipFactory(user=actor_b, workspace=workspace, role=MembershipRole.ADMIN)

    barrier = threading.Barrier(2)
    successes = []
    errors = []

    def worker(actor):
        try:
            barrier.wait()  # ← both threads stop here until both arrive
            membership = WorkspaceService.add_member(
                workspace=workspace, user=target, actor=actor,
            )
            successes.append(membership.id)
        except Exception as exc:
            errors.append(exc)
        finally:
            connection.close()

    threads = [
        threading.Thread(target=worker, args=(actor_a,)),
        threading.Thread(target=worker, args=(actor_b,)),
    ]
    for t in threads: t.start()
    for t in threads: t.join()

    assert len(successes) == 1
    assert len(errors) == 1
    assert Membership.objects.filter(workspace=workspace, user=target).count() == 1

The barrier is the key. Without it, one thread always finishes before the other even starts, and you're not testing concurrency — you're testing race conditions that don't exist.

Step 2: When green isn't good enough

The test passed. Two threads, one success, one failure. The final state was correct: exactly one membership, exactly one notification.

But something felt off. I added a diagnostic line to inspect the error:

print(type(errors[0]).__name__)

The output was:

IntegrityError

Not ValueError. IntegrityError.

For those unfamiliar with Django: ValueError is a domain error. You raise it intentionally. It's what a ViewSet catches and converts into a clean 400 Bad Request.

IntegrityError is a database error. It means the SQL layer rejected your query. Left uncaught, it becomes a raw 500 Internal Server Error — the kind of error that pages you at 3 AM and makes users think your product is broken.

The race condition wasn't in the data — the database's UNIQUE constraint had protected the data perfectly. The race was in the error semantics.

Step 3: Understanding the actual race

I traced through the timeline, and it became obvious:

Thread 1: exists()?  → NO
Thread 2: exists()?  → NO       ← both threads pass the check
Thread 1: create()   → ✅ commit
Thread 2: create()   → ❌ IntegrityError

Both threads run inside transaction.atomic(). But PostgreSQL's default isolation level is READ COMMITTED. That means each transaction can only see data that has already been committed by other transactions.

Neither thread has committed when the other runs exists(). So both see "no existing membership" and both proceed to insert.

One wins. One loses. And the loser surfaces a raw database error to the user.

This is the textbook example of why check-then-create is dangerous. And it's exactly the kind of bug that never shows up in development — because nobody clicks "add member" twice in the same millisecond on their laptop.

Step 4: Choosing between three fixes

I considered three approaches:

Option A: get_or_create
Replace the check-then-create with Django's get_or_create:

membership, created = Membership.objects.get_or_create(
    workspace=workspace, user=user,
    defaults={'role': MembershipRole.MEMBER},
)
if not created:
    raise ValueError("already a member")

This looks elegant. It isn't. get_or_create is itself check-then-create underneath — it just hides the pattern behind a helper. Under the same race, both threads can fail the get, both attempt the create, and one still hits IntegrityError. Same bug, more subtly disguised.

Option B: select_for_update
Lock the workspace row before checking:

Workspace.objects.select_for_update().get(pk=workspace.pk)
# then check-then-create

This eliminates the race entirely — but at a cost. Every add_member on the same workspace now serializes behind a database lock. Under real load, that's a throughput bottleneck. Two users adding members to the same workspace would queue behind each other, even though there was never any actual conflict.

Option C: Catch IntegrityError, convert to ValueError
Let the database do its job, then translate its error into a domain error:

try:
    with transaction.atomic():
        membership = Membership.objects.create(
            workspace=workspace, user=user,
            role=MembershipRole.MEMBER,
        )
except IntegrityError as exc:
    raise ValueError(
        f"User {user} is already a member of workspace {workspace.name}"
    ) from exc

This is the option I chose. Here's why.

Step 5: The nested atomic — and why it matters

The natural first attempt is to catch IntegrityError at the outer level:

with transaction.atomic():
    if Membership.objects.filter(...).exists():
        raise ValueError(...)
    try:
        Membership.objects.create(...)   # ← IntegrityError happens here
    except IntegrityError:
        raise ValueError(...)

This doesn't work. Django detects that an error occurred inside an atomic block, marks the transaction as broken, and raises TransactionManagementError on the next query. You can catch the IntegrityError, but you can't use the database connection afterward.

The fix is to wrap only the insert in a nested atomic block — a savepoint:

with transaction.atomic():                    # outer transaction
    if Membership.objects.filter(...).exists():
        raise ValueError(...)

    try:
        with transaction.atomic():            # ← savepoint
            membership = Membership.objects.create(...)
    except IntegrityError as exc:
        raise ValueError(...) from exc

    NotificationService.create(...)           # outer transaction still healthy
    return membership

The nested atomic() creates a SAVEPOINT in PostgreSQL. When the IntegrityError fires, only the savepoint rolls back. The outer transaction survives and remains usable.

I placed the savepoint only around the insert, not around the notification. Two reasons:

1. If the notification ever raised IntegrityError (a bug, not a race), converting it to "already a member" would be misleading.
2. The savepoint should be as small as possible — the tighter the scope, the more precise the semantics.

Step 6: The exception chaining detail

One subtle line matters more than it looks:

raise ValueError(...) from exc

That from exc preserves the exception chain. In production logs, the developer sees both:

ValueError: User user2 is already a member of workspace workspace-0
    ↑ caused by
IntegrityError: duplicate key value violates unique constraint "unique_membership"

The user gets a clean 400 Bad Request with a meaningful message. The developer gets a full stack trace explaining exactly what happened. Both win.

Step 7: Proving the fix at the API level

A service-level fix isn't enough. I also needed to prove that the HTTP contract held under concurrency. Two simultaneous POST requests to the members endpoint should produce exactly:

• One 201 Created
• One 400 Bad Request
• Zero 500 Internal Server Error

The test:

@pytest.mark.django_db(transaction=True)
def test_concurrent_add_member_returns_201_and_400_not_500():
    # ... same threading setup ...

    def worker(user):
        try:
            client = APIClient()
            client.force_authenticate(user=user)
            barrier.wait()
            response = client.post(url, {'user': target.id})
            status_codes.append(response.status_code)
        finally:
            connection.close()

    # ... start and join threads ...

    assert set(status_codes) == {201, 400}, (
        f"Expected {{201, 400}}, got {status_codes}"
    )

Note the set() comparison. During development, I initially wrote:

assert sorted(status_codes) == [400, 201]

And got this beautifully confusing error:

AssertionError: Expected [201, 400], got [201, 400]

sorted() sorts ascending, so sorted([201, 400]) is [201, 400], not [400, 201]. Order-independent assertions need set, not sorted. Small mistake, big lesson: in concurrency tests, never assume order.

The result

Before the fix:

service test: ❌ Expected ValueError, got IntegrityError
API test:     ❌ Expected [201, 400], got [201, 500]

After the fix:

service test: ✅ exactly one ValueError, no IntegrityError
API test:     ✅ exactly {201, 400}, never 500
full suite:   ✅ all green

The full architecture came out cleaner too:

DB Layer      → unique constraint: data integrity protected
Service Layer → IntegrityError → ValueError: domain semantics enforced
API Layer     → ValueError → 400: HTTP contract preserved

No layer is overstepping its role. The view doesn't need to know about database constraints. The service doesn't need to know about HTTP status codes. Each layer converts errors into the language of the layer above it.

Section 3: Five Lessons I Took Away

Debugging a race condition in a single service method taught me more about backend architecture than any framework tutorial ever could. Here are the five lessons that stuck.

1. If your code uses check-then-create, you have a race condition

Every single time.

if not exists():
    create()

This pattern is everywhere in real codebases — and it's always wrong under concurrency. The window between check and act is small, but it's real, and under load it will be hit.

The fix isn't a smarter check. The fix is either a database constraint, a lock, or a caught exception. But it's never "just add another check."

2. The database constraint is not the bug — the error handling is

I initially thought the database had let me down because it threw an IntegrityError under load. It hadn't. The database was doing its job perfectly — protecting data integrity under concurrent access.

The bug was in my service layer: I hadn't accounted for the fact that the constraint could fire. The database is a safeguard, not a substitute for correct error handling.

Once I framed it that way, the fix became obvious: convert the database's error into the language of the domain.

3. Savepoints are not optional — they're the mechanism

When I first tried to catch IntegrityError inside a transaction, Django raised TransactionManagementError. I didn't understand why until I read the docs carefully.

The rule: once a transaction is broken, it's broken. You can't "undo" the break. But you can isolate the broken operation inside a savepoint so that the outer transaction never sees the damage.

This is what nested transaction.atomic() does. It's not a syntax trick — it's the way Django exposes PostgreSQL's savepoint semantics. Any error recovery inside a transaction requires it.

4. The right fix has zero cost in the normal path

Of the three solutions I considered, select_for_update was the most obviously safe. Lock the workspace, serialize the requests, done.

But it would serialize every add_member call, even the ones that never raced. In the common case — one user adding one member, no concurrency — I'd be paying for a lock I didn't need.

The exception-conversion approach has zero overhead in the normal path. Membership.objects.create() runs, succeeds, and no exception handler executes. Only in the rare race does the code take the except branch.

The best fix is often the one that does nothing in the common case.

5. Concurrency tests are the ones you'll never see fail — until production does

Every test in this project passed before I wrote the concurrency tests. Every single one. Unit tests, integration tests, API tests — green across the board.

They were green because they were testing the code the way developers write it: one call, one result. The race condition only exists when two calls happen at the same time.

If I hadn't written a test with threading.Barrier and transaction=True, this bug would have shipped. And it would have been invisible for months. And then one day, in production, with real users on real networks hitting real servers, some poor engineer would have seen a spike in 500s and had no idea where to start.

If a bug can only exist under concurrency, only a concurrency test can find it.

Final Thought

The pattern that caused this bug — if not exists(): create() — is the first thing most developers reach for. It's intuitive. It reads like English. It passes code review because it looks correct.

But "looks correct" and "is correct under load" are different things. The gap between them is where production bugs live.

What changed my approach wasn't learning a new library or memorizing a design pattern. It was accepting that any code touching shared state can be raced — and being willing to write the tests that prove it can't.

The fix itself was four lines. The savepoint around one .create(). The conversion from IntegrityError to ValueError. The preservation of the exception chain.

But those four lines are the difference between a 400 Bad Request and a 500 Internal Server Error. Between a system that degrades gracefully under load and one that collapses.

The next time you write if not exists(), pause. Ask yourself what happens if two users hit that line at the same time. If the answer is "I don't know" — that's the test to write.

Not every race condition has a race at its heart. But every one has a lesson.

About the author:

Majid Khazaei is a backend engineer specializing in Django, PostgreSQL, and production-grade API design. He writes about concurrency, data integrity, and the engineering discipline behind reliable systems.

🔗 GitHub: https://github.com/majidkhazaei
🔗 LinkedIn: https://www.linkedin.com/in/majid-khazaei-dev