SANDCLOCK: LiteLLM Supply Chain Attack Siphons CI/CD Secrets

Published on 18 August 2026

A single manipulated software component was enough to siphon off cloud keys, SSH access, database passwords and the credentials of entire build processes: on 24 March 2026, two malicious versions of the widely used AI gateway LiteLLM made their way onto the official Python package repository PyPI. Anyone who installed or ran versions 1.82.7 or 1.82.8 during this window could have unwittingly triggered a data theft at the heart of their IT infrastructure – especially in automated build and deployment pipelines (so-called CI/CD environments, in which software is built and published).

This incident is not an ordinary programming error that can be fixed with a quick update. It is a supply-chain attack – that is, an attack on the software supply chain: not a fake copycat package, but the genuine, trusted project was laced with malicious code. That is what makes it so dangerous. In this article we explain, in plain terms, what happened, whether you might be affected, how to check and what you need to do right now.

What happened?

LiteLLM is an open-source tool that lets applications interact with various AI models through a single unified interface – for example for chatbots, internal tools or AI agents. The project is hugely popular: according to PyPI it was typically installed around 15 to 20 million times per week.

According to LiteLLM's official statement, an attacker published the manipulated version 1.82.7 at 10:39 UTC on 24 March 2026, shortly followed by 1.82.8 at 10:52 UTC. Based on the state of the investigation at the time, the cause was a compromised maintainer or publishing account. The attacker bypassed the regular, secured release paths via GitHub and uploaded the malicious packages directly to PyPI.

„The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8.“ – LiteLLM, official security statement

The two versions contained a so-called credential stealer – malicious code that specifically collects access credentials and sends them out. According to analyses by OSV, Datadog and Snyk, the following could be affected: API keys, environment variables, SSH keys, Git and registry tokens, .env files, cloud credentials for AWS, GCP and Azure, Kubernetes service account tokens, database and LDAP configurations, and shell histories. The stolen data was encrypted locally and transmitted via HTTPS to the attacker domain models.litellm.cloud.

The malicious versions were quickly discovered and pulled from circulation – LiteLLM cites an exposure window of around 40 minutes, while PyPI calculated 2 hours and 32 minutes from upload to quarantine. In that short time, however, the malicious packages had already been downloaded more than 119,000 times.

The technical background – explained simply

The second version, 1.82.8, was particularly insidious. It contained a file named litellm_init.pth. Such .pth files are processed by Python as soon as the interpreter starts – that is, every time Python is launched anywhere. The crucial point: the malicious code could trigger without LiteLLM even being actively used or imported. For automated build processes this is dangerous, because just any build step or any tool that starts Python could activate the malicious code.

In the first version, 1.82.7, the malicious code instead sat in the file litellm/proxy/proxy_server.py and was only triggered when the proxy module was imported.

Security researchers at Datadog and Snyk trace an entire attack chain: earlier, on 19 March 2026, the security scanning tool Trivy had already been compromised. Via stolen CI/CD and PyPI credentials, the path then led on to the LiteLLM releases. This attribution – the campaign is credited with the names "TeamPCP" and "SANDCLOCK" – should be understood as a public technical assessment, not as a legally proven attribution of the perpetrators.

Important for context: there is no classic CVE number for this incident, because it is not a programming error. The formal identifier is PYSEC-2026-2 (alias MAL-2026-2144); Snyk tracks the case as SNYK-PYTHON-LITELLM-15762713 and explicitly classifies it as "Attacked" – that is, actually exploited, not a theoretical vulnerability.

Who is affected?

For German SMEs the risk becomes concrete if your website, an AI chatbot, an AI-agent application, an internal tool or a build pipeline obtained LiteLLM via pip – directly or as a transitive dependency (that is, as a sub-component of another package). Particularly at risk are systems with far-reaching privileges: build runners with publishing rights, cloud administration access, access to secrets managers, Kubernetes permissions, registries, or customer and production data.

There is good news according to LiteLLM: „Customers running the official LiteLLM Proxy Docker image were not impacted.“ So anyone who used the official LiteLLM Proxy Docker image was, according to the vendor, not affected.

Regarding scale, large figures are circulating, but they need to be put in the right context. CloudSEK reconstructed a potential exposure of more than 2,500 companies and 434,000 CI/CD pipelines. Resecurity reports a 152.5 GiB data archive with 415,427 secret-capture files, plus manifests covering 898 GitHub owners and 2,038 repositories. Both providers expressly emphasise that these figures are not confirmed victim counts:

„The 2,500+ company and 434,000 pipeline figures describe reconstructed exposure. They should not be read as proof that every listed organization was successfully compromised or that every credential was stolen.“ – CloudSEK

The 119,000 downloads are likewise not a count of individual companies or personal data records. Anyone who can be certain they never obtained or ran the affected versions faces no immediate technical risk from this incident.

How to check whether you are affected

Important safety note first: With version 1.82.8, merely starting Python in the suspicious environment can trigger the malicious code. Therefore, disconnect suspicious hosts, containers and CI runners from the network (especially from outbound traffic) before you analyse them, and do not carelessly run pip or python within them.

1. Identify all systems on which LiteLLM was used directly or as a sub-dependency. Check: requirements.txt, constraints.txt, pyproject.toml, poetry.lock, uv.lock, Pipfile.lock, Dockerfiles, Helm charts, CI/CD workflows and package caches.
2. Search the logs: Search build, deployment and registry logs for 24 March 2026 for the exact strings litellm==1.82.7 and litellm==1.82.8. Also check whether an unpinned pip install litellm ran during that window.
3. Look for the tell-tale file: On an isolated or offline-mounted file system, search the site-packages directory for litellm_init.pth. Finding it is a strong indication of version 1.82.8.
4. Check network traffic: Search DNS, proxy and EDR logs for models.litellm.cloud and checkmarx.zone, as well as for outbound HTTPS POSTs from a Python or build context. No hit, however, does not reliably rule out an earlier exfiltration.
5. Check for traces in the file system (IoCs): ~/.config/sysmon/sysmon.py, ~/.config/systemd/user/sysmon.service, /tmp/tpcp.tar.gz, /tmp/session.key, /tmp/payload.enc, /tmp/session.key.enc, /tmp/.pg_state and /tmp/pglog. Do not delete prematurely – secure the evidence first.
6. Check Kubernetes: Search the audit logs for unusual secret access, privileged pods and pod names matching the pattern node-setup-*.
7. Document what secrets were reachable: For each suspicious system, note which credentials the process could read. This analysis determines the scope of rotation and the data-protection risk scope.

What you need to do now

Datadog sums up the central message:

„Do not treat reverting the compromised packages as a complete remediation.“ – Datadog Security Research

Simply reverting or upgrading is therefore not enough. Proceed systematically:

1. Isolate and pause: Disconnect affected runners, hosts and containers, or those that cannot be reliably ruled out. Pause new deployments from these environments. Secure logs, images and system artefacts for the investigation.
2. Rotate all reachable credentials – not just the LiteLLM or OpenAI key: GitHub tokens and app private keys, PyPI/registry tokens, cloud credentials and roles, SSH/deploy keys, database passwords, Kubernetes service account tokens, CI/CD secrets, webhooks and LLM provider keys. Where possible, revoke sessions and refresh tokens.
3. Rebuild cleanly: Rebuild and redeploy from known-clean build inputs. Remove the compromised versions from requirements, constraint and lock files as well as from all caches.
4. Analyse audit logs: Review cloud, Git and registry logs from the exposure period onwards for unexpected logins, token usage, IAM changes, retrievals from secrets managers, new access and suspicious commit or workflow changes.
5. Secure Kubernetes: Where there is suspicion, review and rotate service account tokens and RBAC permissions, and isolate privileged or unknown pods.
6. Harden for the future: Use exact versions along with hashes in lock files, do not adopt new package versions immediately and unchecked (cooldown strategy), use short-lived, minimally privileged credentials as well as OIDC/workload identity instead of long-lived keys, and pin build dependencies to immutable commit SHAs. For publishing, PyPI recommends "Trusted Publishers" and phishing-resistant two-factor authentication.

On the patch status: the compromised versions 1.82.7 and 1.82.8 have been removed. On 30 March 2026, LiteLLM released the clean version v1.83.0 from a reworked CI/CD pipeline and, after checking against the known indicators, confirmed releases v1.78.0 to v1.82.6 as clean. But bear in mind: in an environment that may be permanently compromised, an upgrade alone is not a complete clean-up.

Context and GDPR: when does it become reportable?

A compromised Python package is not automatically a reportable data breach. The incident only becomes GDPR-relevant if the affected environment could process or access personal data and a breach of confidentiality, integrity or availability has occurred or is sufficiently likely. Under Art. 4(12) GDPR, the term expressly covers unauthorised access to and unauthorised disclosure of personal data. In the case of a website, this could affect, for example, customer databases, CRM and shop access, ticketing and newsletter systems, backups, log data, employee data or cloud storage.

The European Data Protection Board (EDPB) formulates the principle as follows:

„all data breaches should be notified to the relevant DPA, except for those unlikely to present any risk to individuals.“ – EDPB, SME guide

Your obligations in a genuine incident:

• Documentation: Every incident must be documented – even if you ultimately submit no notification.
• Notification of the supervisory authority (Art. 33 GDPR): If a risk to the rights and freedoms of natural persons is likely, the competent data protection authority must be informed without undue delay and, where feasible, within 72 hours of becoming aware. A phased notification is possible if information is initially lacking.
• Notification of data subjects (Art. 34 GDPR): Where a high risk is likely, the affected individuals must be informed without undue delay and in clear language – subject to the statutory exceptions.
• Processors (Art. 33(2) GDPR): Anyone affected as a service provider must inform the controller without undue delay.

On liability: Art. 32 GDPR requires technical and organisational measures appropriate to the risk. Infringements of Arts. 32 to 34 can, under Art. 83(4) GDPR, be penalised with fines of up to 10 million euros or up to 2% of worldwide annual turnover – whichever is higher. That is a maximum framework, not a forecast for this specific case. At the time of research, no fine imposed by an authority is known for LiteLLM. What matters is a documented, rapid risk assessment on a case-by-case basis – ideally together with your data protection officer and, where appropriate, legal advice.

Not an isolated case: the bigger picture

The LiteLLM incident is part of a series of similar attacks. The preceding compromise of Trivy in March 2026 already shows how a manipulated security tool serves as a springboard into further supply chains. And the prominent case of XZ Utils in 2024 (CVE-2024-3094), in which malicious code was smuggled into the upstream packages of a widely used compression library, makes the underlying pattern clear: manipulated components in a trusted supply chain can reach systems even though the package itself appears entirely legitimate.

Conclusion

The LiteLLM attack is a case study in modern software supply-chain risks: not a programming error, but a genuine, trusted package that was laced with malicious code and downloaded more than 119,000 times in a short window. Anyone who installed or ran versions 1.82.7 or 1.82.8 in a production environment must assume that all credentials reachable by the affected process may be compromised – and act accordingly.

The decisive steps are: identify and isolate affected systems, rotate all reachable credentials, rebuild cleanly, analyse audit logs and – if personal data was involved – carefully and promptly assess the GDPR notification obligation. Those who are certain they never obtained the versions can breathe a sigh of relief, but should use the case as an opportunity to harden their own build and deployment processes. Because the next supply-chain attack is not a question of if, but of when.