My minute-by-minute response to the LiteLLM malware attack

The LiteLLM Malware Attack: When Another Supply Chain Incident Hits Home

You've probably seen the headlines by now: LiteLLM, the universal library for LLM providers, got hit by a sophisticated malware attack. Another supply chain attack, another credential stealer.

For those of us who've been watching the threat landscape, especially after the Trivy and Checkmarx incidents just days before, it felt less like a surprise and more like a sickening sense of déjà vu. This isn't just a news story; it's a stark reminder of how fragile our software dependencies are, and frankly, how quickly things can go sideways.

I was tracking the TeamPCP activity closely after the Trivy compromise on March 19, and then the Checkmarx incident on March 23. The pattern was clear: they were targeting critical open-source tools, especially those integrated into CI/CD pipelines. So when the alerts started firing on March 24, indicating suspicious activity around PyPI packages, my gut immediately went to "here we go again."

The Incident: A Five-Hour Window of Opportunity

Here's what actually happened: On March 24, 2026, between 10:39 UTC and 16:00 UTC, malicious versions of LiteLLM—specifically 1.82.7 and 1.82.8—were uploaded to the PyPI repository. That's about five hours where anyone installing litellm without pinning a specific, known-good version could have pulled down malware. This specific LiteLLM malware attack was swift and targeted.

The LiteLLM team acted fast, removing the malicious packages from PyPI within three hours of discovery. They suspended new releases, rotated credentials, and brought in external incident response. That's the kind of rapid, decisive action you want to see from a project maintainer. But even a few hours is a lifetime in a supply chain attack, especially for a library with millions of daily downloads.

The compromise vector? A hacked maintainer's account. This wasn't some zero-day exploit against PyPI itself; it was a stolen publishing token, likely obtained through a vulnerable dependency in LiteLLM's own CI/CD pipeline, echoing the earlier Trivy compromise where TeamPCP used credentials stolen in a previous incident to generate new access tokens before the old ones were revoked. It's a classic chain reaction, leading to the LiteLLM malware attack.

The Mechanism: How TeamPCP Got Their Hands Dirty

This wasn't a subtle attack. TeamPCP's malware was an infostealer, designed to grab anything valuable it could find. The mechanism of the LiteLLM malware attack involved:

1. Initial Execution: The malware was embedded in proxy_server.py in version 1.82.7. Version 1.82.8 also contained a malicious litellm_init file, which is a Python .pth configuration file. These .pth files are designed to extend Python's module search path, but they can also execute arbitrary code on import. This is a common persistence mechanism in Python supply chain attacks.
2. Payload Unpacking: The litellm_init.pth file executed multiple nested base64 encoded payloads. This obfuscation makes initial detection harder and gives the attackers a multi-stage execution capability.
3. Reconnaissance: The malware started by collecting network data and environment variables. It was looking for a lay of the land, understanding the compromised system's context.
4. Credential Hunting: This is where it got serious. The malware systematically searched for:
   • AWS and GCP cloud access tokens and credentials.
   • Kubernetes secrets, including /run/secrets/kubernetes.io/serviceaccount paths.
   • Database connection data (MySQL, PostgreSQL, MongoDB, Redis, Vault).
   • API keys and secrets from .env, .json, and .yml files.
   • SSH keys (~/.ssh/id_rsa, id_ed25519, authorized_keys, etc.).
   • Slack and Discord webhooks.
   • Cryptocurrency wallet data (Solana, Bitcoin, Ethereum, etc.).
   • It even scanned memory (/proc/*/mem) to extract secrets from Runner.Worker and Runner.Listener processes, which is a particularly nasty trick for CI/CD environments.
5. Exfiltration: All this collected data was encrypted with a hard-coded key using OpenSSL. Then, it was uploaded to typosquatting domains. For LiteLLM, the primary C2 was models.litellm[.]cloud. This is a common tactic to blend in with legitimate traffic. As a backup, data could also be uploaded to a GitHub repository named docs-tpcp.
6. Third Payload (Dormant but Dangerous): The malware also attempted to download and run an executable from checkmarx[.]zone. At the time of analysis, this payload wasn't fully activated, but it shows the attackers' intent for further compromise and persistence. The inclusion of a hardcoded YouTube URL (leading to "Bad Apple!!") is a classic anti-analysis trick, designed to waste a reverse engineer's time.

The destructive capabilities seen in the broader TeamPCP campaign—wiping Kubernetes clusters if specific time zones or languages were detected—weren't explicitly confirmed for the LiteLLM variant, but the potential for severe damage from this LiteLLM malware attack was certainly there.

The Blast Radius: Who's Actually Hit?

The practical impact here is significant. Anyone who installed LiteLLM versions 1.82.7 or 1.82.8, especially with unpinned dependencies (e.g., pip install litellm or litellm^=1.82.6), during that five-hour window is compromised. The chatter on Hacker News and Reddit was immediate and intense, and for good reason. Developers are frustrated, and I get it. This isn't just an isolated incident; it's part of a wave of supply chain attacks on AI tooling, leading many to call it the "new normal." The scale of the LiteLLM malware attack underscores this.

LiteLLM's high download rate—over 3 million per day—means a potentially large number of compromised users. Attackers claim to have stolen hundreds of gigabytes of data and over 500,000 accounts across their campaign.

Who was safe? Customers using LiteLLM Cloud or the official LiteLLM Proxy Docker image were unaffected. Why? Because they use strict version locking. This is a critical distinction and a testament to why dependency pinning is non-negotiable.

A particular concern I've seen raised, and one I share, is for AI agent teams. LiteLLM's role as a universal gateway means it often manages API keys for multiple LLM providers. A compromise here isn't just a data leak; it's a potential catastrophic failure for entire AI agent workflows, giving attackers keys to multiple kingdoms.

Our Response: What We Did and What You Need To Do Now

The LiteLLM team's response was swift and transparent, which is essential for rebuilding trust. They removed the malicious packages, suspended new releases, rotated their own credentials, and engaged external incident response. This is the playbook for containing a breach.

For everyone else, especially if you were running LiteLLM, here's what you need to do, immediately:

1. Assume Compromise: If you installed litellm between 10:39 UTC and 16:00 UTC on March 24, 2026, assume your environment is compromised by the LiteLLM malware attack.
2. Check for Indicators: Look for the litellm_init.pth file in your Python environment. This is a strong indicator.
3. Rotate Everything: This is the big one. Rotate all session tokens and API keys that could have been exposed. This includes cloud credentials (AWS, GCP), Kubernetes secrets, database credentials, SSH keys, Slack/Discord webhooks, and any other API keys. Don't wait. Some users on social media questioned the timeliness of credential rotation following earlier related compromises, and they have a point. This needs to be a continuous process.
4. Network Log Review: Check your network logs for any outbound traffic to models.litellm[.]cloud or checkmarx[.]zone. Also, look for connections to scan.aquasecurtiy[.]org if you're concerned about the broader TeamPCP campaign.
5. GitHub Audit: Look for the appearance of a repository named docs-tpcp in your GitHub organizations. This was a backup exfiltration mechanism.
6. Threat Hunt: Conduct a proactive threat hunt across your systems. Don't just look for the known IOCs; look for anomalous behavior.
7. Restore from Backups: If possible, restore affected environments from verified, clean backups.

Beyond the immediate fire drill, we need to talk about long-term strategy. The debates about dependency pinning on forums are valid, but the conclusion is clear:

• Pin Dependencies: Always pin exact dependency versions using cryptographic hashes in all your pipelines and Dockerfiles. pip install litellm==1.82.6 is good; pip install litellm@sha256:abcdef... is better.
• Short-Lived Credentials: Move away from long-lived tokens. Use short-lived credentials with a secrets manager and OIDC integrations. Minimize secret injection into the runtime environment.
• Least Privilege: Configure GITHUB_TOKEN and other access keys with the principle of least privilege. Don't grant write permissions unless absolutely necessary.
• Behavioral Monitoring: Existing signature-based checks and dependency scanning are not enough. You need behavioral monitoring in your CI/CD pipelines. They are the new perimeter.
• Approved Actions: Allow only GitHub Actions from an organization-approved list.
• Security Tools: Tools like zizmor for static analysis, gato and Gato-X for identifying vulnerable pipelines, and OpenSSF's allstar for enforcing security policies are essential.

The Hard Truth

This LiteLLM incident, coming on the heels of Trivy and Checkmarx, isn't just a series of unfortunate events. It's a clear signal that our software supply chain is under sustained, sophisticated attack. The LiteLLM malware attack is a stark example. TeamPCP is methodical, and they're going after the foundational tools we all rely on.

The idea that we can simply trust upstream packages without rigorous verification is dead. We have to assume compromise and build our defenses accordingly. That means a shift from reactive patching to proactive, continuous security validation, especially in our CI/CD environments. This isn't optional anymore; it's the cost of doing business in a world where every dependency is a potential attack vector.