TL;DR
- LLMs can reproduce real leaked keys: Using only API output access, researchers recovered DeepSeek and Google keys from Qwen3-27B, GPT-Codex 5.3, and Claude Haiku 4.5, and every key had a matching public GitHub occurrence.
- Agents pull secrets in from the laptop: Developer endpoints hold credentials in .env files, shell history, and MCP configs that never reach Git. Claude Code-assisted commits leaked secrets at 3.2%, versus a 1.5% baseline.
- Stop secrets before the model sees them: AI Hooks block secrets in prompts, tool calls, and output, while Developer Endpoint Protection reduces the plaintext credentials on managed fleets that agents can find.
More AI, more places to leak secrets
More and more teams are adopting AI agents, allowing them to create new tools, apps, and software at unprecedented speed. At the core of every connection these apps require is a credential, often in the form of an API key.
Those keys get leaked every time they are touched by the AI, and in ways that might not be obvious at first glance.
A coding agent can read a local configuration file, run a command, or call a Model Context Protocol (MCP) server. Any of those actions can place a credential into the context sent to a large language model (LLM).
Recent research shows that exposed credentials can also survive much farther downstream. Researchers behind "Practical Secrets Extraction against Black-box LLMs" recovered strings matching real API keys from Qwen3-27B, GPT-Codex 5.3, and Claude Haiku 4.5 using output-only access.
Very importantly, they had no access to model weights or token probabilities. This was done completely through "API output-only access."
LLM API key leakage now reaches across the credential lifecycle. Existing credentials are now part of training data and can and do appear in model output. AI agents can also move fresh credentials into model context during normal development work.
What the researchers found from LLM output
The researchers recovered real credential strings from three black-box LLMs. All three exposed a secret.
Qwen3-27B returned two DeepSeek keys and one Google key. GPT-Codex 5.3 returned a DeepSeek key. Claude Haiku 4.5 returned two DeepSeek keys.
They then searched GitHub for each complete credential and found a public match every time. That supports the conclusion that the models reproduced real leaked secrets rather than generating strings that only looked valid.
The researchers stopped there. They masked the credentials and did not test whether the keys still worked.
An attacker or rogue agent would have no reason to stop. A recovered key could be tested, used to access downstream systems, or abused until it is revoked, as we have seen across many attacks.
What caught our attention about this paper is that black-box, read only access alone can be enough to recover memorized credentials from model output. The next obvious question is, how did those keys get there?
Where are LLMs getting keys?
LLMs can find API keys in public training data, and we know there are a lot there, but AI agents create a more immediate source: the developer laptop.
Public repositories can become part of datasets used to train or evaluate models. Once a credential appears in public code, it can also spread through forks, copied examples, logs, and other development artifacts. This happens a lot.
GitGuardian detected 28.65 million new hardcoded secrets in public GitHub commits in 2025. Credentials tied to AI services reached 1,275,105, an 81% increase year over year.
The black-box LLM research shows the downstream result. Researchers recovered credential strings from deployed models and found matching public occurrences on GitHub for every recovered key.
AI agents find secrets closer to the developer
Developer laptops have quietly become credential stores. Secrets live in .env files, cloud configuration files, and shell history. They also appear in IDE settings, CLI data, local logs, and temporary debugging files.
Many of those credentials never reach Git. This is especially true on 'citizen coders' laptops where people are only ever deploying to localhost.
AI coding agents can still encounter them. Agents can read files, run commands, and inspect application output. They can also call MCP servers that return credentials as part of a response.
GitGuardian's research really reveals just the tip of this iceberg, looking just at committed code: Claude Code-assisted public commits had a 3.2% secret leak rate, compared with a 1.5% baseline across public GitHub commits. MCP adds another source: 24,008 unique secrets were found in public MCP configuration files, including 2,117 valid credentials.
Public repositories show the part of credential exposure that becomes visible. The developer endpoint holds a much larger pool of potential secrets. Local configurations, command history, application data, and AI tooling can all contain credentials that never appear in a repository.
LLMs can therefore encounter credentials from both ends of the development lifecycle. Public data can carry old exposures into model training. AI agents can pull fresh secrets directly from the environments where developers work.
How AI agents expose secrets to LLMs
AI coding agents all have a model at their core, taking prompts and data in as tokens and returning generated replies. The tool chain around this LLM defines how credentials can reach a model.
The developer never needs to copy the API key into the prompt. The credential can enter the interaction as the agent gathers context needed to solve the problem.
A developer might ask an agent to troubleshoot an API connection. The agent could read an .env file, inspect a cloud configuration, or search application logs. It could also run a shell command or call an MCP server.
Traditional approaches to secret scanning focused on repository and CI scanning may never see this exposure because there may be no commit or pull request. The secret can move directly from the developer endpoint into an AI workflow.
AI agents are entering environments where credentials are already scattered across developer systems. More autonomy gives those agents more opportunities to encounter them.
Model context is becoming part of the credential boundary
We do need to acknowledge here that model exposure and model training are separate concerns. A credential can reach an LLM without becoming training data. Enterprise AI providers may limit whether prompts and outputs are used for training.
Credential exposure happens earlier, when an agent sends a secret outside the developer environment. Once that happens, security teams need to know where the credential went, how long it may be retained, and whether it should still be trusted.
Training is one possible destination. Model context is the immediate concern. What will it do with that secret it finds?
The strongest control point sits before transmission. AI agents can encounter secrets while reading files, running commands, or calling tools. Security controls need to inspect those interactions before credentials become part of model context.
That creates a new credential boundary at the developer endpoint and inside the agent workflow.
Keep secrets away from the AI agent
The best place to stop LLM credential exposure is before the secret reaches the model. The best way to do that is with deterministic tools around the agent's workflow.
GitGuardian AI Hooks put controls directly around the actions AI coding agents take. The GitGuardian CLI, ggshield, can scan a developer's prompt before submission and inspect file reads, shell commands, and MCP calls before execution. When a secret is detected, the action can be blocked before the credential reaches the model. Scanning also happens at any tool output, withholding secret-bearing responses from being passed to the model entirely.
Anything you ask an agent to do is a suggestion; it might or might not follow it, as we have seen in countless rogue agent stories. Unlike skills, Markdown, or system prompts, AI hooks fire at predictable points along the agent's defined workflow. They are outside the reach of the agent, working at the system level where the agent interacts with the environment. Calling a deterministic tool in a deterministic way is how you go from suggesting the agent not use a secret to halting an operation that would have exposed a secret to the LLM or elsewhere.
GitGuardian's Developer Endpoint Protection tackles the much larger pool of credentials the agent might discover
GitGuardian is working to secure the entire credential layer. That is why GitGuardian introduced Developer Endpoint Protection to scan developer machines for credentials sitting in local files and local storage. We are giving security teams visibility into which tools are running and what resources the agents can reach, all from one unified secrets security interface. These capabilities can be deployed across managed fleets through Jamf, Iru, Intune, and other MDM solutions, rather than relying on individual developers to configure protections themselves.
This empowers security teams to work with developers, no matter what department they work in, to reduce how many dangerous plaintext credentials are available for an agent to find.
Secrets needed by applications and developers belong in managed vaults. GitGuardian's secrets manager integrations can identify whether exposed credentials are already vaulted, connect incidents with vault context, and help move unvaulted secrets into secure storage. That context also makes remediation faster by showing teams where the managed secret lives, the full context around it, and who needs to act.
The goal is bigger than just catching a secret in a prompt
Reducing the credentials sitting on developer endpoints should be a stated goal of a security program. Any secrets that do need to remain in the hands of the developer should be properly maintained and scoped as tightly as possible to only the smallest needed access the developer can still use.
Teams need to work to place controls around AI agents before they can carry credentials into another system.
GitGuardian can put those controls in place today and scale them across the organization. Every secret stopped before it reaches an LLM is one less credential an AI agent can expose, one less incident to remediate, and one less secret that can enter the downstream data lifecycle that may eventually include model training.
Keep API keys out of LLM context
The black-box LLM research shows how far a leaked credential can travel, all the way into training data. Plaintext secrets bring danger wherever they appear because they require no additional proof of identity to use. If you have the string, you can try to log in.
At the same time, the actions an AI agent takes are non-deterministic and therefore unpredictable. Trying to control every possible agent behavior is a non-starter because that rule set becomes infinitely large, even if AI helps write it.
The most practical approach is to keep fewer plaintext secrets within reach, limit what agents can access, and stop credentials before they travel any farther.
GitGuardian is here to help developers put those controls in place with AI Hooks. You can start using this control today for free as an individual or open-source project.
For organizations that need the same protection across an entire fleet of developer endpoints, GitGuardian can help roll those controls out at scale while helping you evolve your secrets security program.
FAQ
What is LLM API key leakage?
LLM API key leakage is the exposure of credentials through large language models. It can happen downstream, when a key from public code is memorized and later reproduced in model output, and it can happen immediately, when an AI agent moves a fresh credential into model context while doing normal development work.
What did researchers find when they tested LLMs for leaked API keys?
The researchers behind "Practical Secrets Extraction against Black-box LLMs" recovered real credential strings from all three models they tested, using only API output access and no model weights or token probabilities. Qwen3-27B returned two DeepSeek keys and one Google key, GPT-Codex 5.3 returned a DeepSeek key, and Claude Haiku 4.5 returned two DeepSeek keys. Every recovered key had a matching public occurrence on GitHub.
Where do LLMs get API keys from?
Public repositories are one source, since code with exposed credentials can become part of datasets used to train or evaluate models. GitGuardian detected 28.65 million new hardcoded secrets in public GitHub commits in 2025. AI agents add a more immediate source, the developer laptop, where .env files, cloud configurations, shell history, and MCP settings hold credentials that never reach Git.
How do AI coding agents expose secrets to LLMs?
An agent troubleshooting an API connection might read an .env file, inspect a cloud configuration, search application logs, run a shell command, or call an MCP server. Any of those actions can put a credential into the context sent to the model, and the developer never has to paste the key into a prompt.
Does a secret sent to an LLM become training data?
Not automatically. Model exposure and model training are separate concerns, and enterprise AI providers may limit whether prompts and outputs are used for training. The exposure still happens earlier, when an agent sends a secret outside the developer environment, so teams need to know where the credential went and whether it should still be trusted.
How can I stop AI agents from sending secrets to an LLM?
Use deterministic controls around the agent's workflow. GitGuardian AI Hooks use ggshield to scan prompts before submission, inspect file reads, shell commands, and MCP calls before execution, and withhold secret-bearing tool output from the model. Developer Endpoint Protection then reduces how many plaintext credentials are available for an agent to find, and can be deployed across managed fleets through MDM solutions like Jamf, Iru, and Intune.
