LiteLLM traces its malicious 1.107.0 release to a leaked master key left unrevoked for 136 days
LiteLLM has published what it calls the canonical record of how a credential-stealing build of its Python package, version 1.107.0, reached PyPI, Docker Hub and the GitHub Container Registry on October 21, 2026. The attack-chain post, released by the project on October 23 at 21:00 UTC and signed by its director of security and an AI engineer, replaces the company's first explanation of the incident. The starting point, according to LiteLLM, was not an internal log leak, as it had initially assumed, but a master key for an experimental agent service that had been committed to a public GitHub repository on May 27 and never revoked.
The incident matters well beyond one project. LiteLLM is a widely used open-source gateway that gives applications a single, OpenAI-compatible interface to many model providers, and its proxy commonly stores the API keys an organization uses for Anthropic, OpenAI, AWS Bedrock and other services. A poisoned release of such a component sits, by design, next to the most valuable secrets in an AI deployment. The malicious wheels were live on PyPI for roughly an hour and forty minutes and the container images for up to about two and a half hours, but LiteLLM's guidance treats any environment that started Python with the bad version installed as compromised.
Microsoft published its own technical analysis of the payload on October 23 at 18:12 UTC, describing a collector that runs through Python startup hooks, encrypts what it gathers with AES-256 and an RSA-4096 key, and ships it to a lookalike domain. Taken together, the two documents give an unusually complete picture of a modern software supply-chain attack aimed at AI infrastructure: one exposed secret, one over-privileged token, a release process that had been loosened days earlier, and a payload built to harvest cloud, developer and AI-tool credentials.
··········
FROM A FORCE-PUSHED CONFIG FILE TO OWNER-LEVEL GITHUB ACCESS.
According to LiteLLM, the root cause dates to May 27, 2026, at 18:53:14 UTC, when an `opencode.json` configuration file containing the master key for a LiteLLM Agent Platform deployment was committed to the public `LiteLLM-Labs/lite-harness` repository. The commit was later force-pushed over and the exposed keys were identified for revocation, but the master key itself was never revoked. Rewriting Git history removes a secret from the visible branch; it does not remove it from copies already taken, which is why revocation, not deletion, is the control that counts.
The Agent Platform was, in LiteLLM's description, an experimental internal alpha service for self-hosted agents that open pull requests on GitHub. To do that work it held a GitHub token, and that token is where the exposure became severe. It was not bound to a dedicated service account and was not scoped down. It was a GitHub CLI OAuth token belonging to an owner of LiteLLM's GitHub organization. The company also notes that GitHub CLI tokens are not subject to GitHub's two-factor re-authentication flows for sensitive actions, so possession of the token was enough.
On October 10, between 21:13 and 21:28 UTC, the attacker used the leaked master key to call the Agent Platform API from the IP address 69.48.229.143. At 21:27:10 a request listing managed agents returned the configuration of every agent on the platform, including the owner's GitHub CLI token. From that moment, LiteLLM says, the attacker had owner-level access to all of the organization's repositories. The interval between the leak and its use was 136 days.
··········
HOW THE ATTACKER COLLECTED MORE TOKENS BEFORE TOUCHING A RELEASE.
The attacker did not move straight to publishing a package. About forty-five minutes after obtaining the owner token, at 22:11:45 UTC on October 10, a workflow on a short-lived branch named `ci-tmp-a351cc` in `BerriAI/litellm` posted six repository secrets to a webhook.site endpoint. Among them was `GH_TOKEN`, the GitHub CLI token of a second maintainer. Twenty-four seconds later a similar workflow in `BerriAI/litellm-auto` sent `RELEASE_PLEASE_TOKEN`, belonging to a third maintainer, to the same endpoint. Both workflow runs were deleted within about three minutes, at 22:14:14 and 22:14:31.
On October 12, between 12:08 and 12:30 UTC, further workflows on several private repositories posted all of their repository secrets to the same destination. LiteLLM states that these secrets were not part of the path that led to the poisoned artifacts and describes them as minimally sensitive secrets and minimally funded provider keys. The value of the October 10 step was different: it gave the attacker three separate maintainer identities, which turned out to be exactly what the release process required.
........
Time (UTC) | Event reported by LiteLLM |
May 27, 18:53 | Master key committed in `opencode.json` to a public repo |
Oct 10, 21:13–21:28 | Agent Platform API used with the key; owner token exposed |
Oct 10, 22:11–22:14 | Two maintainer tokens exfiltrated; runs deleted |
Oct 12, 12:08–12:30 | Secrets from private repos posted to webhook.site |
Oct 21, 10:08 | Branch protection rejects change, then is overridden |
Oct 21, 10:46 | Staging run built from lookalike `Berri-AI/litellm` |
Oct 21, 11:11 | Production gate approved with a second maintainer token |
Oct 21, 11:39–11:40 | Seven 1.107.0 wheels uploaded to PyPI |
Oct 21, 11:46 | Docker Hub and GHCR images published |
Oct 21, 11:53–11:54 | Third token deletes production and staging run records |
Oct 21, 12:06 | External researcher opens issue #47826 |
Oct 21, 13:23 | PyPI quarantines the release |
Oct 21, 14:20 | Docker Hub tags removed or repointed |
........
··········
THE RELEASE PIPELINE WAS REPOINTED, APPROVED AND THEN SCRUBBED.
The publication itself took place on October 21 and relied on LiteLLM's private `BerriAI/project-releaser` repository. Commit `9cf97e9` changed the release workflow so that it could build from a source other than `BerriAI/litellm`. At 10:08:29 UTC branch protection rejected a push made with the second maintainer's token. Five seconds later the attacker disabled the protections and force-pushed using the owner token, which had the authority to override them.
At 10:46:39 the staging workflow was dispatched against the lookalike repository `Berri-AI/litellm`, note the hyphen, at commit `20535d8f`, a commit that exists only on the attacker's fork. The production pipeline started at 11:08:53, and at 11:11:18 the second maintainer's token satisfied the production environment approval gate, from a different IP address, 199.91.221.39. Seven `litellm==1.107.0` wheels were uploaded to PyPI between 11:39:51 and 11:40:13, and container images followed at 11:46:56.
LiteLLM is candid about why the approval gate did not stop this. A rule requiring an approver account locked to a physical device had been relaxed earlier that week so that any two engineers could trigger and approve a release. The attacker held tokens for several engineers and used one to start the release and another to approve it. The company adds that the owner token could have removed the restriction entirely, and suggests the attacker chose separate tokens to avoid a noisy entry in the audit log.
The clean-up was equally deliberate. Between 11:53:49 and 11:54:01 the third maintainer's token deleted the production and staging run records, erasing workflow logs. From 12:09 to 12:24 the owner token pushed a revert commit and then force-pushed `main` back to an earlier commit, again overriding branch protection and the force-push block. LiteLLM points out that the official `v1.107.0` source tag does not contain the malicious files; the poisoned artifacts were built from the attacker's fork.
··········
WHAT THE 1.107.0 WHEELS DID ON INSTALLED SYSTEMS.
Microsoft's analysis explains why simply having the package installed was enough. The malicious wheels placed files under the `purelib` data directory that end up directly in site-packages, outside the `litellm/` folder. One of them, `cook.pth`, contains the single line `import cook`. Python processes `.pth` files every time the interpreter starts, so the code in `cook.py` runs whenever Python launches in that environment, whether or not the application ever imports LiteLLM. Even a later `pip` command can re-trigger it, which is why LiteLLM tells users not to run `pip install` as their first remediation step.
`cook.py` decodes a Base64- and zlib-encoded program that identifies itself as PRIVCOLLECT and runs its collection in a daemon thread. Socket, which flagged the wheels about seven minutes after publication, counts 16 collection modules, each with its own error handling. The targets span cloud credentials for AWS, Azure and Google Cloud, Kubernetes service-account files, Docker configuration, Git and package-registry credentials, `.env` files, SSH, GPG and TLS private keys, browser credential and history databases, shell histories, and configuration and authentication files for AI coding tools and assistants. On Linux the collector also reads other processes' environments through procfs.
A dedicated routine is aimed at LiteLLM itself. When `DATABASE_URL` is set, a Prisma-based Python child process reads the `credential_values` field of `LiteLLM_CredentialsTable` and the `litellm_params` field of `LiteLLM_ProxyModelTable`, up to 500 rows per query. Those tables are where a LiteLLM proxy keeps the provider credentials and model settings it routes traffic with. Microsoft and Socket both stress that the presence of a target in the code shows intent, not confirmed access in any particular environment.
Each run generates a 32-byte AES key and a 16-byte IV, encrypts the collected data with AES-256-CBC and wraps the AES key with an embedded 4,096-bit RSA public key using OAEP. The result is packed as a gzip tar archive and sent by HTTP POST to `www.litellm.dev/healthcheck`, a domain chosen to look like part of the project. After a successful upload the collector writes a marker file in `/tmp` to avoid running twice.
A second path, `sitecustomize.py`, targets Unix systems. It takes a lock on `/tmp/remote_code_loader.lock`, reads a pointer file named `domain.txt` from a GitHub repository, requests `/codepy` from whatever server that file names, and executes the response without any signature check. Socket reports that the pointer returned an empty response at 13:01 UTC on October 21, and Microsoft says it observed the file empty, so no second-stage execution has been confirmed. Both analyses describe the loader as a mechanism that could be re-armed at any time by editing that one file.
........
Artifact | Window live (UTC, Oct 21) | Status |
PyPI `litellm==1.107.0`, seven wheels | 11:39:51 to 13:23 | Quarantined |
GHCR tags in three image repositories | 11:46:56 to 13:51 | Unavailable |
Docker Hub `v1.107.0`, `latest`, `main-latest`, `main-stable` | 11:46:56 to 14:20 | Removed or repointed |
Official `v1.107.0` source tag | Not affected | Clean |
Private enterprise release | Not affected | Clean |
........
··········
DETECTION, CLEANUP AND THE CREDENTIALS LITELLM SAYS TO ROTATE.
LiteLLM's incident page gives concrete checks. `pip show litellm` reveals the installed version, and any environment that started Python with 1.107.0 should be treated as compromised, as should any container that ran the image index `sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e`. Searching for `litellm-1.107.0.dist-info`, `cook.pth` and `sitecustomize.py` can confirm exposure, but the company warns that not finding them does not prove an environment is clean.
The cleanup order is important because of the startup hooks. LiteLLM advises deleting `cook.pth`, `cook.py`, `sitecustomize.py` and the matching `__pycache__` files from every affected site-packages directory, killing the detached Python daemon that holds the loader lock, and only then reinstalling a pinned earlier release such as `litellm==1.106.2`. Rebuilding virtual environments and images from scratch is the preferred route. Microsoft goes further, recommending that investigation happen from a known-clean machine, because starting Python inside a suspect environment can run the payload again, and that teams check pip and uv caches, internal mirrors, lock files and build outputs for stray copies.
If 1.107.0 ran on a host with outbound internet access, LiteLLM says to assume data was exfiltrated. The rotation list covers the instance or task IAM role, every provider key stored in the proxy database, the proxy's master and virtual keys, the password in `DATABASE_URL` and any secret present in the environment of any process on the host. On AWS, the company suggests reviewing CloudTrail for `GetCallerIdentity` calls from the instance role around install time and for use of its credentials from outside the organization's infrastructure.
Microsoft lists Defender Antivirus signatures, including `Trojan:PyPI/LiteLlmSupplyChain.A` and `.B` and `Backdoor:PyPI/LiteLlmSupplyChain.C`, a vulnerability-management identifier, `TVM-2026-0003`, and hunting queries that look for connections to the collector domain, the known hashes of the startup files, the inline database-collection command and OpenSSL encryption staged in `.pc_e_` temporary directories. It also states plainly that removing the package or the release tag is not sufficient recovery.
On its own side, LiteLLM says it removed every GitHub organization member except one account temporarily promoted to owner, stripped all GitHub permissions from the release workflow at 14:19 UTC, and is revoking every token tied to its internal systems and vendor platforms on the assumption that all of them were exfiltrated. Customers running the private enterprise release were not affected, according to the company, and a separate embargoed security fix for enterprise is scheduled for disclosure on October 27. Process changes are promised for the following week.
··········
WHAT IS ESTABLISHED AND WHAT REMAINS OPEN.
The sequence of events is now documented to the second by the maintainers themselves, and the payload has been dissected independently by Microsoft and Socket. Several points are firm: the root cause was an exposed master key that was never revoked; that key unlocked a service holding an unscoped organization-owner token; a release safeguard had been weakened in the same week; the malicious code ran at interpreter startup and targeted cloud, developer, AI-tool and LiteLLM proxy credentials; and the public registries were cleaned within roughly three hours of the first upload.
Other questions are not answered. LiteLLM has not published how many environments installed 1.107.0, and Socket's figure of about 23 million weekly downloads describes the project's overall popularity, not the reach of the bad release. No attacker has been identified. Microsoft did not independently confirm the initial access path, did not confirm second-stage execution, and did not confirm in its telemetry that cloud metadata credentials were actually returned to the attacker. LiteLLM's own first account of the incident was revised once already, which is a reason to watch for further updates.
The technical lesson is less about Python packaging than about the identities that AI tooling accumulates. An internal agent service was given a human owner's GitHub token for convenience, its key outlived a history rewrite, and its compromise flowed directly into the release path of infrastructure that guards model-provider secrets for many organizations. For teams running LiteLLM, the immediate work is to verify versions, rebuild affected environments and rotate credentials; for anyone operating agent platforms, the incident is a documented case of what an unscoped, long-lived token can open.
··········
FOLLOW US FOR MORE.
·····
DATA STUDIOS
·····
datastudios.org
