Before we asked anyone to trust us with their logins, we went over our own auth server the way an attacker would: not by running it, but by reading it. What came out was a written review of the whole surface: sixteen numbered sections, 261 lines, from SAML through OIDC federation, SCIM, the admin API, MFA and the backup tooling, every finding carrying a severity and a source location. Eight were flagged fix first. We fixed those eight in one evening and the rest of the list half an hour later. Two and a half weeks after that we ran the same exercise over the multi-tenant control plane and got five more, closed the evening before we went to production.
Here is the number that stuck. The test suite reported 367 of 367 passing before that evening, and 367 of 367 after it. Nothing went red, because nothing in the suite was pointed at any of it. Tests prove what you thought to test, and nobody had thought to test whether stripping an XML attribute the signature does not cover would route a replayed assertion down a friendlier branch.
Read as eight unrelated bugs, a list like this is useless to anyone else. Read as shapes, it is three habits, repeated across subsystems, in code written by people who had read the specs.
The client_id, the forwarded header, the InResponseTo and the email claim
The first habit is treating a value the caller writes as a fact about the caller. The two the review marked Critical were each a single input: a query parameter, and an email claim.
MFA policy was derived from the client_id parsed out of the returnUrl on the login form, and /connect/authorize never re-checked. Aim the login at a client with MFA disabled and the session came out without a second factor, then spent perfectly well at the client that required one. That one has its own write-up.
Per-IP limits keyed on the remote address, and forwarded headers were trusted from 0.0.0.0/0. One X-Forwarded-For value per request made every per-IP control advisory, and rewrote the IP column in the audit log as a bonus. The details are here.
The SAML assertion consumer chose between the SP-initiated path (consume the request id, single use) and the IdP-initiated path (check an assertion-id cache) purely on whether the response carried an InResponseTo attribute. That attribute sits on the <Response> element, which is not covered when only the assertion is signed, which is the common case. Strip it and a captured response took the branch whose replay cache the original login had never populated, every signature still verifying. Why the signature was never the problem.
And returning federated users were resolved by the email in the assertion, so any connection could assert somebody else's address and land on their account. That one is written up too.
The admin API was a factory for its own credential
Admin authorization keyed off exactly one thing: a token whose scope claim carried the admin scope. Two admin endpoints would put that scope into a fresh token.
POST /api/v1/token, the impersonation endpoint support tooling uses, split its scopes query parameter and copied the result verbatim into an access token and a refresh token, with no check against the client's registered AllowedScopes. One admin credential, however short lived, therefore minted a long-lived admin refresh token for any user and any client: privilege that outlives rotating the credential you were worried about. Client create, meanwhile, bound a raw client object straight off the request body, ClientSecretHashes included, so an admin could register a client_credentials client holding the admin scope and a secret hash they already knew. List and get echoed every stored hash back.
The impersonation endpoint now refuses the admin scope and bounds the rest by the client's own AllowedScopes, create and update bind a DTO with no secret-hash field, and no response carries a hash. A permission that can issue itself is not a permission.
A hyphen, an unowned group, and an email doing an identity's job
The second habit is letting something with a name in it stand in for an identity.
Email was the join key for federated sign-in: global, present at many providers, and whatever the asserting connection says it is. The fix was to resolve on the pair of provider and subject, which no other connection can mint.
In the control plane the tenant slug was the name. The router that turns a slug into a storage table prefix stripped hyphens, so acme-corp and acmecorp produced the same prefix, which meant the same table set and the same Vault signing key. Two separately registerable tenants sharing one table set and one signing key. Register the second and you are looking at the first one's users. The fix was not a cleverer prefix function. It was forbidding hyphens in slugs so the mapping is injective, and it is worth the longer version.
SCIM was the one that stung, because half of it was right. Every user read and write re-checked that the resource had been provisioned by the calling client. Groups, in the file next door, had no owner at all: create never stamped one, list passed null as the owner filter and returned every group in the environment, and get, replace, patch and delete accepted any id. Knowing a group id was the entire authorization, and where several provisioning clients share an environment that is cross-client read and write of group membership, which is how roles get granted. Groups now carry the creating client, and every read and write is scoped to it.
The cookie was signed before the webhook answered
The third habit is doing the irreversible thing before running the check.
The clearest instance is a feature customers buy: a tenant registers a webhook that is asked "allow this login?" at authentication time, and can set it to enforce the answer. The hook ran on every session-establishing path, correctly. It just ran after SignInAsync. So a rejection threw, the request came back 500, and the browser was already holding a signed session cookie. Any client that ignored the error was logged in, blocked or not. The login-blocking webhook did not block.
The fix is one line moved, six times: run the hook before SignInAsync on password, OIDC, SAML, MFA verify and MFA setup, and before minting tokens at the token endpoint. The library change and the control-plane change landed two minutes apart. A rejection is now a clean 403 with no Set-Cookie, and an end-to-end test asserts that a denied login issues no .AspNetCore.Cookies.
The same ordering turned up in three quieter places. Authorization codes were read and then deleted as two operations, so two concurrent redemptions of one code could both pass the read and both get tokens; the grant store now exposes TryConsumeAsync, a conditional delete that returns true only to the caller that actually removed the row. Restore upserted entities before verifying anything: the FileHashes field was documented as verified during restore while never being populated or checked, which makes a tampered backup a write primitive against production. And the Stripe webhook acted on events before recording that it had seen them, so at-least-once redelivery could flap a tenant's plan and status or re-send dunning. All three check first now.
Validating the URL when it is saved proves nothing when it is dialled
Customers hand us URLs: auth webhooks, backup targets, retention callbacks. Rejecting one that resolves to a private address at the moment it is saved is the obvious guard, and on its own it is close to worthless, because DNS is not a promise. A name that resolves publicly when you validate it can resolve to 169.254.169.254 when you connect.
So the load-bearing check does not live in the settings handler. It is a connect callback on the HTTP handler itself. Every outbound call to a customer-supplied URL resolves the host, refuses loopback, RFC1918, unique-local and the whole 169.254.0.0/16 link-local range the cloud metadata service lives in, then opens the socket to that exact validated address instead of to the hostname, so there is no second resolution left to rebind. Automatic redirects are off, because a 302 is one more chance to name an internal target. The save-time check stays as an error message for the customer, which is all it was ever good for.
Two endpoints and a backup file that only the network protected
/_internal/cluster/gossip was mapped anonymous, and its shared-secret check existed as a comment rather than as code. /_internal/backchannel-logout was mapped anonymous with antiforgery disabled on the public listener, took a body of {"SubjectId": "..."}, and revoked every grant held by that subject. Neither had any in-process guard. The only thing in front of them was ingress configuration, and given the forwarded-header finding above, any IP-based defence was spoofable anyway. Both now require a shared secret compared in constant time, or an internal source address when no secret is configured.
Backups had the same character. On hosts using the local key source, the signing-key table holds the EC private scalar, and backup serialized every column verbatim into plaintext gzipped JSONL. Anyone who could read a backup file could forge tokens for the issuer it came from. That table is now excluded by default with an explicit opt-in for encrypted targets, and every data file is SHA-256 hashed into the manifest and verified before restore writes a single row.
Where a check runs is most of what the check is worth
None of these was a missing check in the sense of nobody having thought about it. MFA was implemented and tested. The webhook hook ran on every path. The SSRF validation existed. SCIM users were scoped correctly on every verb, in the file beside the one where groups were not. In nearly every case the logic was written, correct, and installed somewhere it could not do its job: on a value the caller supplied, at a step the caller could skip, after the state it was supposed to prevent, or on the resource next door.
So the useful question at each endpoint is not "do we validate this". It is three others. What does this request actually prove, as opposed to assert? Is this identifier something a caller can name, or something only the right caller can hold? And does anything irreversible happen before the answer arrives? A written review, worked section by section with a severity on every finding, is mostly a device for forcing those three past the code you are certain about. Ours was certain. It was sixteen sections of certain, plus eight things that would have become somebody's incident.
None of it was reachable by a customer, because there were no customers yet. That is the argument for doing this before launch rather than after the first report: the same list, written six months later, is a disclosure timeline instead of an evening's work. If you want the current version of the answers, our security page sets out how tenants are isolated, what is encrypted at rest, and what a backup of your data actually contains.
