The Railway Env-Var Breach: What Happened and How to Be Ready Next Time

On October 21, 2026, Railway told every customer to rotate every environment variable they had stored on the platform. If you deploy anywhere that injects env vars at runtime, the same thing can happen to you, and the only thing you control is how fast you recover.

TL;DR

• Railway says an attacker reached encrypted environment variables and, for some, the decrypted values.
• No tool on the developer side could have prevented a platform-side breach like this. Good tooling decides whether rotation takes 20 minutes or two days. Managing env files before deployment, with one versioned source the whole team pulls from, is what makes the difference.
• Below: what happened, a post-incident rotation runbook, how to prepare before the next one, and an honest comparison of the tools that help.

Disclosure: I build evnx, an open-source .env security CLI with an end-to-end encrypted cloud vault. I use it as the worked example here, but I also cover where other tools are the better fit. My team does not run anything on Railway, so this is written from the outside.

What happened

Railway detected unusual activity in the systems that store customer environment variables on the morning of October 21. Its official disclosure says an unauthorized party may have accessed encrypted variables and, for some of them, obtained the decrypted contents. Those variables typically hold API keys, database passwords and private registry credentials.

Key facts from the disclosure and follow-up coverage:

• Who was hit. Railway's initial analysis suggests a subset of users, but it asked all users to rotate as a precaution. It will email confirmed affected accounts directly.
• Why. Railway believes this is part of a series of attacks on platform companies aimed at compromising AI inference providers. It notified those upstream providers so they could revoke keys proactively.
• Deployments paused. All builds were paused during the investigation, so customers could rotate a key at the provider but could not ship the new value. Enterprise builds came back first, on October 22 at 03:32 UTC, with other plans restored in stages.
• Follow-up guides. Railway published guides on checking for anomalous GCP access and AWS access.
• Postmortem. Railway has not said how the attacker got in or how many accounts were affected. A formal postmortem is expected after service is restored.

Scale matters here: Railway said in January 2026 that more than 2 million developers build on it. Follow live updates on status.railway.com.

The paused deployments are the detail most write-ups skip. They exposed a hard trade-off: revoke a leaked key now and take an outage, or leave it live until you can deploy. Teams that knew exactly which secrets lived where could make that call per secret. Teams that didn't had to guess.

Why this kind of breach is hard to defend against

A platform that injects environment variables into your app at runtime has to be able to decrypt them. Encryption at rest protects against a stolen disk. It does not help when the attacker reaches the system that holds the keys. That is a property of the env-var model, not a Railway-specific failure.

It also creates the secret-zero problem. Say you move secrets into an external vault and have your app fetch them at boot. The app still needs a credential to reach that vault, and that credential is... an environment variable on the same platform. If the platform leaks it, the attacker can read the vault too. You have moved the problem, not removed it.

So the honest goal is not "make a platform breach impossible". It is:

1. Small blast radius: each leaked credential reaches as little as possible.
2. Fast, complete rotation: you know every secret that lived on the platform and can replace them all quickly.
3. Evidence: you can show what was rotated, when and by whom, for customers, auditors and your own postmortem.

Reaction time: why env work done before deployment pays off

When a platform says "rotate everything", revoking keys is the easy part. What eats the hours is everything around it: which secrets were where, how the team gets the new values, whether anyone missed one, and what broke. evnx is built for the stage before deployment, so most of those answers already exist when the incident starts. Each question maps to one command.

Team rotation in practice. Share the vault once, before anything goes wrong:

evnx vault share api/production --with teammate@example.com --role developer
evnx vault members api/production

Then, on incident day, one person rotates and everyone else follows without a single secret crossing chat:

# The person rotating
evnx validate --env-name production && evnx cloud push

# Everyone else
evnx cloud watch --vault api/production   # tells you a new version landed
evnx cloud pull                           # fetch and decrypt the new values locally

cloud watch only notifies; it never pulls on its own, so nobody's local edits get overwritten. The team works from one versioned source instead of a chain of messages, and that is where most of the saved time comes from.

Post-incident: a rotation runbook with evnx

Here is how the recovery looks if your secrets live in evnx vaults, one vault per service and environment. Commands are from evnx 0.10.15; every flag is in the command reference.

1. Get the list of what to rotate. The vault listing shows which key names each vault holds (DATABASE_URL, STRIPE_KEY and so on). The server stores names in the clear on purpose so a listing is useful, and that listing becomes your rotation checklist. No digging through dashboards or old Slack threads.

2. Revoke at the source, by risk. Database credentials, cloud and IAM keys, payment keys, AI provider keys and registry tokens go first, even if it means downtime while deploys are paused. Read-only or low-value tokens can wait for builds to return.

3. Update locally and validate before anything ships.

evnx validate --env-name production                  # placeholders, weak secrets, duplicate keys, DEBUG="False"
evnx diff --env-name production --fail-on missing    # did the rotation drop a variable the template declares?

A panicked rotation is exactly when someone pastes CHANGE_ME into production or forgets a variable.

4. Store the new values as a new version, and check who touched the vault.

evnx cloud push                            # encrypts locally, uploads ciphertext as a new version
evnx cloud history                         # what changed, when, and who pushed it
evnx cloud audit --vault api/production    # who pushed, pulled, shared or revoked

If a CI credential for the vault lived on the breached platform, cloud audit shows whether anyone pulled after the leak. Every push is a new version and nothing is overwritten. If a rotated key breaks something, evnx cloud pull --version <n> restores the previous one while you investigate.

5. Load the new values into the platform when deploys resume.

evnx migrate --env-name production --to railway \
  --railway-project <project> --railway-environment production --print-only

evnx prints a single railway variable set command for you to read and run; it contacts nothing itself. The same works for Vercel, Heroku, Doppler, Infisical and the AWS, GCP and Azure secret stores.

6. Make sure the old or new keys didn't leak somewhere else.

evnx scan --severity high
evnx scan --format sarif > scan.sarif

It detects AWS (including STS), GitHub, Slack, Google, Stripe, OpenAI and Anthropic keys, JWTs, private keys and passwords in connection strings. It reads every .env variant it finds, such as .env.production. In a git repo it reports what git would commit, so your gitignored .env doesn't fail the run, but a key pasted into a tracked script does.

7. Keep the evidence. cloud history and cloud audit give you a dated record of every rotation and every access. If you're working toward SOC 2, ISO 27001 or Korea's ISMS-P, that is the evidence an auditor asks for after an incident.

Then do the checks Railway recommends: review usage and billing for every AI provider key, grouped per key, and look at database and cloud logs for access you don't recognize.

Pre-incident: set up today, recover fast tomorrow

The runbook above only works if the groundwork is already in place. This is what to set up now.

A source of truth that isn't the platform. If your hosting dashboard is the only place a secret lives, a platform incident leaves you rotating blind. Keep the canonical copy in a vault the platform can't read:

brew install urwithajit9/evnx/evnx      # or: npm install -g @evnx/cli, pipx install evnx
evnx commands                           # should report: built with net, migrate, backup, cloud
evnx vault create api --env production
evnx cloud link api/production
evnx cloud push

The prebuilt binaries include the cloud commands; PyPI has carried them since 0.10.0. Encryption happens on your machine (AES-256-GCM), and the server stores only ciphertext, so a database dump of the evnx server yields no plaintext. Cloud sync is still in beta; the next section says exactly what that means.

One vault per service and environment. That gives clean boundaries: when one platform is compromised, you know exactly which vaults were deployed there.

Write down which variables are secrets.

evnx spec init     # infers a contract for every variable into .evnx.toml

Mark a credential secret = true and scan reports it even when its name gives nothing away. validate enforces which variables each environment requires. That contract is your rotation checklist, written before you need it.

Keep plaintext off disk.

evnx cloud run --vault api/production -- ./deploy.sh

The values are decrypted in memory and handed to the child process: no .env file, nothing in shell history, nothing visible in ps.

Scoped, expiring tokens for CI.

evnx auth token create ci --scope read --vault api/production --expires-in-days 90

A leaked CI token reaches one vault, read-only, for at most 90 days.

Clean offboarding. evnx vault revoke re-keys the vault, so a removed member cannot open versions pushed after they left.

Stop the other leak path: git. Platform breaches get the headlines, but committed secrets are still the most common leak. Run evnx as pre-commit hooks:

repos:
  - repo: https://github.com/urwithajit9/evnx
    rev: v0.10.15          # pin 0.10.0 or newer
    hooks:
      - id: evnx-scan        # blocks commits containing secrets
      - id: evnx-validate    # blocks a misconfigured .env
      - id: evnx-diff        # blocks if .env lacks a key the template declares
      - id: evnx-sync        # blocks if the template lacks a variable from .env
      - id: evnx-scan-push   # strict scan on push

Add evnx validate --format github-actions and evnx scan --format sarif to CI, so findings show up in GitHub's Security tab. evnx doctor --fix makes sure every env file is gitignored, untracked and readable only by you.

Turn on 2FA, and do a rotation drill. evnx auth totp enable protects the vault account. Then pick a quiet afternoon and rotate one non-critical service end to end, using the runbook above. The first rotation is always the slowest; do it before an incident forces you to.

What evnx would not have done

I'd rather say this plainly than have you find it out mid-incident.

• It would not have stopped this breach. If your app reads secrets from Railway env vars, Railway holds them in usable form. evnx protects its own copy, not the platform's.
• Pulling from evnx at boot doesn't escape secret zero. A pipeline pull, or cloud run, needs an EVNX_TOKEN and the master password. Store both as env vars on a breached platform and the attacker can open the vault. The token only authenticates; the password is what decrypts, and it is the more sensitive of the two. Scoping limits what the token can reach. It is not a substitute for rotation.
• Cloud sync is in beta, and it does not yet survive a server takeover. What is already stored stays closed: a database dump or stolen backup yields no plaintext. But an attacker with write access to the evnx server could present a vault key of their choosing and read what you push next. Signed key wraps and locally pinned identities would fix that, and they aren't built yet. Our own README used to say the vault was safe "even after a breach". That was wrong, and we corrected it. Until this is fixed, don't keep secrets there that you couldn't rotate.
• It does not rotate keys for you. You still revoke at each provider by hand. Infisical and Doppler can automate rotation for some credential types (below).
• It does not verify whether a leaked key is live. TruffleHog does that.
• It doesn't call Railway's API. evnx migrate --to railway prints the command for you to run; evnx convert --to railway exists but is marked experimental.

Other tools worth knowing

No single tool covers this whole problem. These are the ones I'd reach for, and what each does best.

Infisical: an open-source secrets platform. It offers syncing to GitHub Actions, GitLab CI, Vercel, AWS and Kubernetes, dynamic secrets, and automated rotation for PostgreSQL, MySQL, AWS IAM and more. It can be self-hosted, including on Railway itself (Postgres + Redis + the app). Best for teams that want automated rotation and short-lived database credentials, and can run a heavier service.

Doppler: hosted secrets management with a native Railway integration. It syncs a Doppler config to a Railway project and environment continuously and can redeploy on change. Railway's own docs describe syncing from Doppler, Infisical or Vault as a supported pattern. Best for teams who want rotation to propagate to Railway automatically and are happy with a hosted SaaS. One caveat: the sync needs a Railway API token, which is a powerful credential in its own right.

dotenvx: encrypted .env files in git. It encrypts values with a public key so the encrypted file can be committed, while the private key in .env.keys stays out of git. Best for small teams who want secrets versioned alongside code. On a platform it still needs the private key as an env var, so the same secret-zero caveat applies.

Gitleaks and TruffleHog: secret scanners. Gitleaks is fast, regex-based and MIT-licensed, which makes it great as a pre-commit hook. TruffleHog's standout feature is verification: it tests whether a detected credential is still live. After an incident, that tells you which keys are still dangerous.

Railway's native options. Railway itself offers service, shared, reference and sealed variables. Use them for least privilege inside the platform, but remember they live on the platform that was breached.

Where evnx fits, and where it excels

evnx is built for the developer side of secrets: the .env on your laptop, onboarding, pre-commit checks, CI, and the path from local config to whatever platform you deploy to.

Where evnx stands out:

• Encrypted on your machine. The server stores ciphertext and never holds a usable key, so a stolen database is useless. Team sharing wraps vault keys with hybrid X25519 + ML-KEM-768, and revoking a member re-keys the vault.
• One tool for the whole recovery. Validate, diff, scan, version, audit and roll back, then hand the values to 14 conversion targets or print migration commands for Railway, Vercel, Heroku, Doppler, Infisical, AWS, GCP and Azure. In the middle of an incident, one tool beats four.
• It catches the mistakes rotation creates. Placeholder values, the DEBUG="False" truthy trap, duplicate or missing variables and .env / .env.example drift are what break production right after a hurried rotation.
• Plaintext stays off disk. cloud run injects secrets straight into a process, with no file and nothing in ps.
• Clear answers in CI. Exit code 0 means nothing found, 1 means a problem was found, and 2 means evnx could not finish. "I found nothing" and "I couldn't look" are never confused.
• Lightweight and portable. A single Rust binary, MIT-licensed, installable from Homebrew, npm, pipx, Cargo, Scoop or winget. The server is open source and self-hostable.

If you need automated rotation for databases today, use Infisical or Doppler. evnx can still sit alongside them on the developer side.

Get started with evnx

Installing takes one line, and scanning or validating needs no account. Pick a channel:

curl -fsSL https://evnx.dev/install.sh | bash   # macOS and Linux, prebuilt binary
brew install urwithajit9/evnx/evnx              # Homebrew
npm install -g @evnx/cli                        # npm
pipx install evnx                               # PyPI
cargo install evnx --all-features               # from source, every feature
winget install urwithajit9.evnx                 # Windows

Then run a first check on a project you already have:

evnx doctor    # env files gitignored, untracked, not world-readable
evnx scan      # secrets in any file git would commit

For encrypted vaults and team sharing, create a free account. The free tier includes 3 vaults, 5 versions each and 2 API tokens, with no card required.

• evnx.dev: the project site
• Quick start: your first audit in five minutes
• Installation guide: all nine install channels
• Command reference: every command, flag and exit code
• Cloud architecture: how the encryption works, including what it does not protect against
• GitHub: source, issues and discussions

Your checklist

If you're on Railway right now:

• [ ] Revoke database, cloud, payment, AI provider and registry credentials at each provider
• [ ] Check AI provider usage and billing per key, plus database and cloud access logs
• [ ] Follow Railway's GCP and AWS guides if you use either
• [ ] Redeploy with new values once builds return, then confirm the old values fail
• [ ] Watch your inbox for Railway's direct notice to affected accounts

Wherever you deploy, before the next incident:

• [ ] Keep a source of truth for secrets that your hosting platform can't read
• [ ] Split secrets per service and environment, and know which platform each set is deployed to
• [ ] Use scoped, expiring tokens in CI
• [ ] Scan and validate in pre-commit and CI
• [ ] Run one rotation drill end to end

Platform breaches will keep happening, especially while AI inference keys are worth stealing. You can't stop the next one. You can make sure it costs you an afternoon, not a week.

Try evnx with the install line above, read the docs, and star or open an issue on GitHub. I'd love feedback from anyone who rotated through this incident: what slowed you down?

Sources

• Railway: Security Disclosure, Wednesday October 21st
• Railway: Check for Anomalous GCP Access
• Railway: Check for Anomalous AWS Access
• Railway Status
• Railway docs: Sync secrets from an external secrets manager
• Cyber Updates 365: Railway security incident timeline
• OpenClassActions: Railway hack coverage
• evnx on GitHub
• evnx 0.10.15 on PyPI (README, including the cloud sync threat model)
• evnx documentation
• Doppler: Railway integration docs
• Infisical on Railway
• dotenvx quickstart
• AppSec Santa: Gitleaks vs TruffleHog