Defences against supply-chain attacks: lessons from LiteLLM

Every piece of software you run is mostly made of other people's software. A typical Python service pulls in a few dozen packages directly and a few hundred more transitively. Your CI pipeline downloads actions, scanners and base images from registries you do not control. Your Kubernetes cluster runs Helm charts written by strangers. None of this is a problem in itself. It is how modern software gets built, and a large part of why it gets built so quickly.

A supply-chain attack is what happens when someone abuses that trust. Instead of attacking you directly, the attacker compromises something you already depend on (a package, a build tool, a CI action, a maintainer's account) and then lets your own automation deliver the payload for them. This is not what most people picture when they think of a hack, where a software vulnerability or a misconfigured firewall gets someone through the door. It is more insidious, closer to the Trojan Horse the Greeks used to get past the walls of Troy.

I wrote about the mechanics of signing and verifying images in Hardening the Software Supply Chain as one of the best practices we should all follow. This post is the more practical follow-up: why this matters more now than it did a year ago, what a real attack looked like from the inside, and where to start if you have not done much about it yet.

Why do supply-chain attacks matter more in 2026?

For years supply-chain attacks were treated as exotic. SolarWinds and xz-utils were nation-state level operations that took months or years of patient work. That is no longer the shape of the problem. In 2026 these attacks are cheap, automated and chained together, and the payload is almost always the same thing: your credentials.

Build systems and platform services are where the keys live, which makes them excellent targets. They also tend to run with more relaxed rules so that builds can work. If you run Terraform from a GitLab pipeline to build your cloud infrastructure, for example, you have a runner with egress access that holds your cloud provider keys.

That chaining is exactly what happened in March 2026.

Case study: how a security scanner compromised LiteLLM

We use LiteLLM as a gateway in front of our local and public AI models. It is a popular project: Wiz found it in 36% of the cloud environments they analysed. That popularity is also what made it a target.

Trend Micro have published an excellent breakdown of the incident. The short version is this:

1. It started somewhere else. In late February an attacker abused a misconfigured pull_request_target workflow in the CI of Trivy, the popular vulnerability scanner, and stole a bot token. Aqua rotated credentials, but not atomically, and on 19 March the attackers force-pushed 76 of the 77 release tags of the trivy-action GitHub Action to malicious commits.
2. The scanner became the attacker. LiteLLM's CI pipeline ran Trivy as part of its build, and the poisoned code harvested the PyPI publishing token from the runner. A security tool, added to make the build safer, was the way in.
3. Two malicious releases. On 24 March, LiteLLM 1.82.7 and then 1.82.8 were published to PyPI, thirteen minutes apart. The first injected code into the proxy server module. The second went further and dropped a .pth file into site-packages. Python runs .pth files on every interpreter start, so installing the package was enough. You did not even need to import it.
4. It went for everything. The payload collected SSH keys, cloud credentials (including calls to AWS Secrets Manager and the instance metadata service), Kubernetes service account tokens and secrets across all namespaces, .env files, Terraform state, Vault and database configs, CI secrets, crypto wallets and, of course, every LLM provider API key in the environment. It encrypted the lot and posted it to models.litellm.cloud, a domain registered just before the attack and chosen to look like normal LiteLLM traffic.
5. It tried to stay. On ordinary hosts it installed a systemd user service, sysmon.service, disguised as a "System Telemetry Service", which polled for new commands. On Kubernetes the code could create a privileged pod on each node in kube-system to write the same backdoor onto the host, although Trend Micro found no confirmation this was used in the wild.

PyPI quarantined both versions about three hours after publication. What gave the game away was not a scanner or an EDR alert. It was a bug in the malware itself that fork-bombed machines until production systems fell over, and a researcher who noticed because LiteLLM had arrived as a transitive dependency of a Cursor MCP plugin they were testing.

It is worth sitting with that for a moment. If the attackers had written slightly better code, the main detection mechanism would not have fired. And plenty of people who were affected had never knowingly installed LiteLLM at all.

The lesson is not "avoid LiteLLM". The same campaign, attributed to a group known as TeamPCP, hit Trivy, Checkmarx KICS, npm packages, Docker Hub images and VS Code extensions, each one using credentials stolen from the last. The lesson is that any dependency can turn, and your defences have to assume it will.

Where should you start defending against supply-chain attacks?

You do not need to fix everything at once, and you should not try. These are the steps that give the most protection for the least effort, roughly in the order I would tackle them.

1. Stop installing whatever is newest

Most poisoned releases are spotted and pulled within hours or days. LiteLLM's lasted about three hours. If your builds always grab the latest version the moment it appears, you are volunteering to be in the first wave.

Pin versions, commit your lockfiles, and add a short cooling-off period before upgrades are picked up. Renovate and Dependabot both support this:

{
  "minimumReleaseAge": "7 days"
}

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"
    cooldown:
      default-days: 7

A week is not a guarantee, but it would have kept you well clear of this one. GitHub has since made a three-day cooldown the default for Dependabot version updates, which is a good start; setting it explicitly lets you choose a longer window.

2. Pin by hash, not by name

A version number is a request. A hash is a promise. For Python, generate a lockfile with hashes and make the installer refuse anything that does not match:

uv pip compile requirements.in --generate-hashes -o requirements.txt
pip install --require-hashes -r requirements.txt

The same principle applies to containers (reference images by digest, as covered in the earlier post) and, critically, to CI actions. A tag such as @v0.28.0 can be moved, which is exactly what happened to trivy-action. A full commit SHA cannot:

# Pinned to a commit, not a tag. Let Renovate or Dependabot propose updates.
- uses: aquasecurity/trivy-action@<full-40-character-commit-sha> # v0.x.y

3. Treat CI runners as production

Your build pipeline holds the keys to everything you publish. Give it the same care you give production:

• Scope tokens to the job that needs them. A publish token should only exist in the publish step, not in every job that runs tests and scanners.
• Prefer short-lived credentials. PyPI and most cloud providers support OIDC trusted publishing, which removes the long-lived token entirely.
• Audit third-party actions and avoid pull_request_target unless you understand exactly what it exposes. That one setting was where this whole chain began.

4. Assume a dependency will run malicious code, and limit what it can reach

This is the step most teams skip, and it is the one that turns a breach into a non-event. If the LiteLLM payload had landed in a pod that could not talk to the internet and held nothing worth stealing, it would have been an annoyance.

Egress control. Your AI gateway needs to reach a handful of known provider endpoints. It does not need to reach a domain registered this morning. Start with default-deny egress and allow only what is needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: ai
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    # DNS only; add explicit rules for each provider endpoint you use
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Plain NetworkPolicy works on IP ranges, so to allow named domains you will want a CNI that supports FQDN policies, such as Cilium, or an egress proxy.

No privileged pods. The Kubernetes persistence trick relied on creating privileged pods with host access. Pod Security Admission blocks that with one label:

kubectl label namespace ai pod-security.kubernetes.io/enforce=restricted

Equally, most application pods do not need a service account token at all. Set automountServiceAccountToken: false and give RBAC only to the workloads that really talk to the API.

Keep secrets out of the filesystem and environment where you can. The payload read every environment variable and every .env file it could find, searching up to six directories deep. Pull secrets from Vault or OpenBao at runtime with short leases (we covered the pattern in dynamic credentials with OpenBao and Vault), and in LiteLLM itself hand teams scoped virtual keys rather than the real provider keys, so a leak from one service does not expose the lot. Where a .env file is unavoidable, at least do not leave it lying around in plain text. I covered one way of doing that in Encrypting .env files with OpenBao Transit: a stolen file is useless without access to the key, and that access can be revoked the moment you suspect a compromise.

5. Know what you run, and answer "are we affected?" fast

When an advisory lands, the first question is always whether you are exposed. If answering it means grepping fifty repositories and asking around on Slack, you will be slow when it matters.

Generate SBOMs for your images, keep an inventory of what is deployed where, and include transitive dependencies. uv pip tree and pipdeptree will show you what pulled what in. For this particular incident, the check was as simple as:

find / -iname 'litellm_init.pth' 2>/dev/null

If that returns anything, assume every credential on that machine is compromised and rotate it. Then go and check your CI runners too.

6. Watch for the strange, not just the known

Signature-based tools did not catch this. Behaviour did, accidentally. Some cheap signals worth alerting on:

• New .pth files appearing in site-packages after install.
• A single process reading SSH keys, cloud credential files and kubeconfigs in quick succession.
• Outbound connections from pods to domains or IP ranges you have never seen before.
• Unexplained resource exhaustion. Fork-bombing turned out to be an indicator of compromise.

And have a credential rotation runbook that you have actually tested. When you need it, you will need it in an hour, not in a week.

Stop trusting by default

Supply-chain attacks work because they turn your good habits against you. Automated updates, CI pipelines and security scanners all exist to make you safer and faster, and in March 2026 every one of them was used as a delivery mechanism. The answer is not to stop using open source or to freeze every dependency forever. It is to stop trusting by default: pin what you run, slow down how quickly you adopt new releases, and design your platforms so that a poisoned package finds nothing worth stealing and nowhere to send it.

None of this is glamorous, and most of it never appears in a demo. It is also exactly the work we do at Digitalis.