If your third-party OAuth integration works for days and then dies with {"error":"invalid_grant"}, the cause is usually not clock skew, a wrong client secret, or an expired consent. It is two of your own processes calling the token endpoint with the same refresh token at the same time. When the provider rotates refresh tokens, the second call presents a token that was already consumed, and many providers treat that as replay and revoke the whole token family — which is why the user has to click "Reconnect" instead of just retrying.
The fix is to make refresh a single-flight operation per integration, and to stop treating "the access token expired" as something every worker discovers independently.
What does invalid_grant actually mean?
invalid_grant is the most overloaded error in OAuth 2.0. The spec assigns it to any grant that is "invalid, expired, revoked, does not match the redirect URI, or was issued to another client," so providers pile several unrelated conditions into one string. In practice, on a refresh call it means one of:
- The refresh token expired (idle-expiry, common on providers that expire unused tokens).
- The user or an admin revoked the app's access.
- The client credentials do not match the token.
- The refresh token was already used once and the provider rotates.
Only the last one comes and goes. If your error rate is bursty, correlates with traffic spikes or with cron minute boundaries, and affects healthy accounts that just worked, you are looking at rotation plus concurrency. Refresh-token rotation is a recommended practice in the OAuth 2.0 Security Best Current Practice and it is the default in OAuth 2.1 drafts, so assume any provider you integrate with today may rotate — check the response body rather than the docs, because the behavior sometimes differs per app registration.
The tell is simple: if the token response includes a refresh_token field with a value different from the one you sent, that provider rotates, and every refresh is a state mutation you have to serialize.
Why does this only blow up in production?
Locally you run one process. In production you run a web dyno, three queue workers, and a scheduled job — all sharing one row in your integrations table.
The race is boring and unavoidable without a lock:
- Worker A reads the token row, sees
expires_atis in the past, POSTs to the token endpoint. - Worker B reads the same row 40 ms later, sees the same stale
expires_at, POSTs the same refresh token. - The provider rotates: A gets a new pair, B's request presents an already-redeemed token.
- B gets
invalid_grant. Depending on the provider, the family is now revoked and A's brand-new token is dead too.
That last step is what makes this worth fixing properly instead of retrying. A retry loop makes it worse: it turns one replayed token into five, which looks exactly like the attack that reuse detection exists to catch.
The window is not "when the token expires" — it is "when several workers first notice."
How do you serialize refresh across processes?
If you already run Postgres, advisory locks are the cheapest correct answer: no extra infrastructure, no lease expiry to reason about, and the lock is released automatically when the transaction ends, including on a crashed connection.
create table oauth_tokens (
integration_id text primary key,
access_token text not null,
refresh_token text not null,
prev_refresh_token text,
expires_at timestamptz not null,
rotated_at timestamptz not null default now()
);
import hashlib
from datetime import datetime, timedelta, timezone
import httpx
REFRESH_SKEW = timedelta(seconds=120)
TOKEN_URL = "https://provider.example.com/oauth/token"
def _lock_key(integration_id: str) -> int:
"""Stable 64-bit signed key for pg_advisory_xact_lock."""
digest = hashlib.sha256(f"oauth:{integration_id}".encode()).digest()
return int.from_bytes(digest[:8], "big", signed=True)
def _read(conn, integration_id):
return conn.execute(
"SELECT access_token, refresh_token, expires_at "
"FROM oauth_tokens WHERE integration_id = %s",
(integration_id,),
).fetchone()
def get_access_token(conn, integration_id: str) -> str:
now = datetime.now(timezone.utc)
access, refresh, expires_at = _read(conn, integration_id)
if expires_at - REFRESH_SKEW > now:
return access # fast path: no lock, no network call
with conn.transaction():
conn.execute("SET LOCAL lock_timeout = '15s'")
conn.execute("SELECT pg_advisory_xact_lock(%s)", (_lock_key(integration_id),))
# Re-read inside the lock: someone may have refreshed while we waited.
access, refresh, expires_at = _read(conn, integration_id)
if expires_at - REFRESH_SKEW > now:
return access
resp = httpx.post(
TOKEN_URL,
data={
"grant_type": "refresh_token",
"refresh_token": refresh,
"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
},
timeout=10.0,
)
resp.raise_for_status()
body = resp.json()
conn.execute(
"""
UPDATE oauth_tokens
SET prev_refresh_token = refresh_token,
access_token = %s,
refresh_token = %s,
expires_at = %s,
rotated_at = now()
WHERE integration_id = %s
""",
(
body["access_token"],
body.get("refresh_token", refresh),
now + timedelta(seconds=body["expires_in"]),
integration_id,
),
)
return body["access_token"]
Three details carry most of the value. The fast path avoids taking a lock on every API call, so the hot path stays a single indexed read. The re-read inside the lock is the part people skip — without it, every worker that queued behind the lock still performs its own refresh the moment it acquires one. And body.get("refresh_token", refresh) keeps working on providers that do not rotate, so the same code path handles both.
The honest drawback: this holds a database transaction open across an HTTP call. With a 10-second client timeout and a 15-second lock_timeout, worst case you pin one pooled connection for ~25 seconds. If your pool is small or you refresh thousands of integrations concurrently, move the refresh into a dedicated worker instead of doing it inline.
Should you refresh on 401 or before expiry?
Refresh proactively. Reacting to a 401 means every worker discovers expiry at the same moment — you have built a thundering herd on purpose, and you now need the lock to be perfect because it is on the critical path of every request.
The REFRESH_SKEW above (two minutes) means the first worker to come within the skew window refreshes while the current token is still valid, and everyone else keeps using the still-good token instead of blocking. You still keep 401 handling as a fallback for revocation and for providers that expire tokens earlier than advertised, but it should be rare enough that a 401 in your logs is a signal rather than routine noise.
Treat a 401 on a token you refreshed 30 seconds ago as an alert, not a retry.
Which locking strategy fits your stack?
| Approach | Safe across processes | Extra infra | Main weakness |
|---|---|---|---|
In-process mutex / asyncio.Lock
|
No | None | Only protects one process; useless with multiple dynos |
Redis SET key val NX EX 30 lease |
Mostly | Redis | Lease can expire mid-refresh; no fencing, so two refreshes are still possible |
Postgres pg_advisory_xact_lock
|
Yes | None if you have Postgres | Holds a connection across the HTTP call |
SELECT ... FOR UPDATE on the token row |
Yes | None | Same connection cost, plus it blocks unrelated readers of that row |
| Dedicated refresh worker (one consumer per integration) | Yes | Queue | More moving parts; needs its own liveness monitoring |
If you would rather not own token storage at all, Nango handles OAuth token refresh and rotation for third-party integrations as a managed service, and Merge does the same as part of a unified API layer — both cost real money per connected account and both put a vendor between you and the provider's raw API, which matters the day you need an endpoint they have not mapped.
For most teams running a single Postgres, the advisory lock is the correct amount of machinery.
How do you tell a race apart from a real revocation?
Log the discriminating fields, not the error string. On every refresh attempt, record the integration id, the worker/process id, a hash prefix of the refresh token you sent (never the token), and the rotated_at you read. Then:
- Two refreshes for the same integration within seconds, from different workers → race.
- One refresh,
invalid_grant, and the token was last rotated hours ago → genuine expiry or revocation; re-consent is the only fix. -
invalid_granton a token whose hash prefix matchesprev_refresh_token→ you replayed a rotated token, usually from a retry or a stale cached copy in process memory.
That last case is why the prev_refresh_token column exists. It costs one text field and turns an unexplainable error into a one-query diagnosis.
FAQ
What causes invalid_grant on refresh token requests?
Most often the refresh token was already used and the provider rotates refresh tokens, so the second use is rejected as replay. The other common causes are user revocation, idle expiry of an unused refresh token, and client credentials that do not match the token.
Does every OAuth provider rotate refresh tokens?
No, but you cannot assume either way. Check whether the token response contains a refresh_token different from the one you sent, and write your storage code so it persists a new one whenever it appears.
Can I just retry after invalid_grant?
No. If rotation caused it, retrying replays an already-redeemed token and can trigger the provider's reuse detection, which revokes the entire token family and forces the user to reconnect. Re-read your stored token first; if it changed, another worker already refreshed and you should use the new one.
Bottom line
If you integrate with any provider that returns a new refresh_token on refresh, treat token refresh as a mutation that must happen exactly once per integration and put a cross-process lock around it — a Postgres advisory lock is enough for most teams and needs no new infrastructure. Refresh proactively inside a skew window rather than reacting to 401s, so workers never discover expiry simultaneously. Keep the previous refresh token in a column purely for diagnosis; it is the difference between "random OAuth errors" and a five-minute root cause. And never retry invalid_grant blindly — that is the one response where a retry can turn a recoverable error into a forced reconnect.
