The LiteLLM Supply Chain Backdoor: What 3 Hours Means for Your AI Agent Gateway

title: "The LiteLLM Supply Chain Backdoor: What 3 Hours Means for Your AI Agent Gateway"
published: true

tags: security, ai, llm, infosec

On March 24, 2026, two backdoored versions of litellm sat on PyPI. Version 1.82.7 went up at 10:39 UTC. Version 1.82.8 followed 13 minutes later. According to an NHS England cyber alert, PyPI quarantined the packages at 13:38 UTC that same day. That is roughly 3 hours where a pip install in the wrong window could hand an attacker your SSH keys, your cloud tokens, your Kubernetes secrets, and every LLM API key your agents use.

LiteLLM's own incident blog describes a shorter window, about 40 minutes. I could not fully reconcile the two timelines, so treat the exact duration as uncertain. The point survives either way. A package with over 95 million monthly downloads shipped attacker code from its official PyPI project, and most teams running agent stacks had no process that would have caught it.

I build security tools for AI agents. mcpscan scans MCP servers for dangerous patterns. secops-toolkit-mcp wraps security operations for agent workflows. agent-memory-protocol is my attempt to make agent memory inspectable. Most of my time goes into one question: who controls the front door of an agent system. This incident is the front door story I keep warning about. Here is what happened, then what changes for anyone running a gateway.

What happened, as far as I can verify

Multiple security teams published analyses, and the core facts line up:

• Threat actor: a group calling itself TeamPCP.
• Initial access: they never touched LiteLLM's code. They stole PyPI publishing credentials through a compromised Trivy GitHub Action in LiteLLM's CI pipeline. Trivy is a vulnerability scanner. A security tool was the way in.
• 1.82.7, 10:39 UTC: malicious code injected into proxy_server.py, triggered when the LiteLLM proxy module gets imported.
• 1.82.8, 10:52 UTC: same injection, plus a file named litellm_init.pth. A .pth file runs on every Python interpreter startup in that environment. It does not wait for you to import litellm. Install it, and every Python process on that box executes it.
• The payload collected SSH keys, cloud credentials, Kubernetes tokens, database credentials, crypto wallets, and LLM API keys, and it installed persistence designed to survive reboots.
• Sonatype tracked a three-stage stealer, advisory sonatype-2026-001357. The Python advisory database lists it as PYSEC-2026-2.
• PyPI eventually quarantined the entire litellm project, all versions, according to JFrog. LiteLLM later confirmed the Trivy compromise was contained and the affected releases deleted.

Why gateways hurt more than apps

If a random library in your app gets backdoored, that is bad. If your LLM gateway gets backdoored, it is worse, for boring structural reasons.

A gateway is a choke point by design. Every agent call to every model flows through it. It holds API keys for every provider you route to, plus database URLs for rate limiting and logs. Backdoor the gateway and you inherit the whole tenant, not one feature.

Agent stacks compound this. Agents run long-lived processes that restart on their own schedule. They pull dependencies from the same unpinned requirements files everyone copies between projects. The .pth trick fits that perfectly. The malware does not need your agent to call litellm. It runs the moment any Python process starts in that environment. Your nightly restart becomes the exfiltration trigger.

And the credentials involved have the worst blast radius there is. Cloud tokens, Kubernetes service accounts, SSH keys. Not one chat API key. All of them.

What I checked on my own machines

The first thing to do on any machine that runs agent stacks: check whether anything pulled the bad versions inside the window. The minimal check takes two minutes:

pip show litellm

If the version shows 1.82.7 or 1.82.8 and it was installed on or after March 24, treat the machine as compromised. Not "reinstall the package". Compromised. Sonatype and the NHS alert both say removal is not enough, because the stealer writes persistence and your secrets may already be gone.

Version pinning that actually holds

Most requirements files I see in agent projects look like this:

litellm
openai
anthropic

That is not pinning. That is "give me whatever is newest at build time", the exact behavior that turns a fast PyPI takedown into your problem. Unpinned means the attacker only needs minutes, because your next CI run pulls the poison for you.

What I now do for anything gateway adjacent:

Pin exact versions:

litellm==1.82.6

Better, pin hashes. I generate a constraints file with pip-compile from pip-tools in hash mode, then install with require-hashes enabled. With hash pinning, a malicious new release on PyPI does nothing to you. Your build refuses anything that does not match the hash you recorded. This is the single change I would push on every team running agents this week.

The official LiteLLM Proxy Docker image survived the same way: it pins dependencies inside the image. A built image is a reviewable, immutable artifact. If your gateway matters, ship it as an image you built, not as pip install on a VM someone forgot about.

What should actually change

Three things, none of them fancy.

1. Treat your LLM gateway as production infrastructure with a change process. It holds every key you own, and .pth files execute with the same privileges as your shell. Anything that touches interpreter startup deserves the same suspicion.
2. Stop letting CI decide your gateway version. Unpinned deps in agent projects are a delay bomb. Pin, hash, bump on purpose, with a human reading the changelog.
3. Assume the supply chain will fail again. This campaign did not stop at LiteLLM. Reporting connects TeamPCP to Trivy itself, Checkmarx KICS, TanStack, and the Telnyx SDK. I have not verified each of those in depth, so read them before repeating them. The pattern is the lesson: security tooling and AI infrastructure are the same attack surface now, because they run with the same privileges on the same machines.

If you did install 1.82.7 or 1.82.8

Short version, in order:

1. Pull the box off networks you care about before anything else.
2. Rotate every credential that machine could see: LLM provider keys, cloud tokens, SSH keys, Kubernetes secrets, database passwords. Assume exfiltration, because the payload collected and shipped files out.
3. Rebuild the environment. Do not clean in place. The persistence was designed to survive a reboot.
4. Check egress logs for the days after March 24 for anything odd.

If you run the official proxy Docker image, you were in the lucky group. Still rotate if you share machines or CI runners with anyone who ran pip install inside that window.

I keep coming back to one number: 13 minutes. That is the gap between 1.82.7 and 1.82.8 hitting PyPI. The attacker iterated on a live compromise of a 95 million downloads a month package faster than most incident reviews get scheduled. Your defense cannot be "I will notice". It has to be "my build refuses unknown code".

Here is my question for you: when did you last read the changelog and diff the versions you pin, instead of running pip install -U and hoping? Genuinely curious, because my honest answer before March 24 was embarrassing.

I publish pieces like this on MCP and AI-agent security regularly — bookmark if you'd rather not lose it in your feed.