How to Audit Your AI Toolchain for Supply Chain Risks

The Check You’re Probably Not Running

Every morning I open my laptop and pull in a dozen dependencies before I’ve written a single line of my own code. npm install. pip install. A Hugging Face model download. Some of those packages were written by people I’ll never meet, maintained by volunteers, or — increasingly — generated by an AI coding assistant that may have pulled in a library I’ve never heard of.

That’s the supply chain. And if you’re not auditing it, you’re trusting a lot of unverified code to run in your environment.

The numbers back this up. According to Veracode’s 2026 state of software supply chain security, 66% of the most critical long-lived vulnerabilities in enterprise applications come from third-party code — not what your team wrote. That’s what’s driving the $2.8 billion AI security market we’re seeing this year. Their GenAI Code Security Report found that nearly 45% of AI-generated code contains known security vulnerabilities when no security guidance is provided. And Cloudsmith’s 2026 supply chain guide notes that a modern enterprise app is on average 80% third-party code — and that figure keeps climbing.

The Shai-Hulud worm in September 2025 compromised over 500 npm packages by hijacking developer credentials and spreading through CI/CD pipelines. In July 2026, an attacker hijacked AsyncAPI’s own CI/CD to publish trojanized packages reaching 2.9 million weekly downloads. In August 2026, keyv and cacheable were hit by a self-propagating worm that’s still developing as Cloudsmith writes about it.

These aren’t edge cases. They’re the new normal — and you can see it in the funding: HiddenLayer just raised $100M to build AI deployment security tools — and the funding numbers show it, with companies like HiddenLayer raising $100M to build AI deployment security tools.

Here’s a practical audit you can run this afternoon. No special tools required — just your package manager, a few commands, and a willingness to look at what you’ve been quietly trusting.

Step 1: Generate a Real SBOM (Not a Compliance Snapshot)

A Software Bill of Materials sounds boring until you need to answer “are we affected by this CVE?” at 2 AM and realize your last SBOM was generated six months ago and filed away.

For Node projects, npm sbom (built into npm 9+) outputs a CycloneDX JSON bill of materials:

npm sbom --output-file sbom.json

For Python, pip-licenses combined with a requirements freeze gives you a starting point:

pip install pip-licenses
pip freeze > requirements.txt
pip-licenses --format=json --output-file pip-licenses.json

The key insight from Cloudsmith’s guide: a static SBOM is a compliance paperweight. What matters is whether you can query it against a newly disclosed CVE right now and get an answer in minutes, not days. If your SBOM lives in a drawer, it’s not a security control — it’s a document.

Audit question: When was your last SBOM generated? Can you search it for a specific package name in under 30 seconds? If either answer is “I don’t know,” start here.

Step 2: Map Your Dependency Tree — Especially the Transitive Parts

The vulnerability that takes you down is usually three levels deep in a dependency you didn’t choose and don’t recognize. Direct dependencies are visible. Transitive ones are where the surprise lives.

Check what’s actually installed, not what your manifest says:

# Node — see the full tree with versions
npm list --all --depth=0

# Python — see what's actually resolved
pip list --format=columns

# Check for deprecated or unmaintained packages
npm outdated
pip list --outdated

Look for packages with no recent releases, single maintainers, or names you can’t immediately trace to a GitHub org or PyPI project page. If a package was last updated in 2019 and has 50 dependents, that’s not automatically a problem — but it’s worth a 30-second Google to confirm the project is alive.

Audit question: Can you name every package in your top-level dependencies? If not, pick one you can’t name and trace it this afternoon.

Step 3: Scan for Known Vulnerabilities — But Don’t Trust the “Clean” Result Blindly

Run your scanner. Read the output. Then understand what it’s not telling you.

# npm audit — note what it checks and what it misses
npm audit --json > audit.json

# pip safety check
pip install safety
safety check --json > safety.json

A clean audit result is not a clean bill of health. The Veracode report notes that 66% of critical vulnerabilities come from third-party code — but a scanner can only flag what’s in its vulnerability database. A package with no published CVE today may have an undisclosed issue. And a new vulnerability in a transitive dependency you didn’t know you had won’t show up until the next scan runs.

The August 2026 keyv worm is a case in point — the malicious packages had no published advisory when they were spreading. npm audit returned zero vulnerabilities for packages that were actively compromised. That’s not a flaw in npm audit; it’s a limitation you need to know about.

Audit question: When was the last time your scanner’s vulnerability database was updated? Are you running scans continuously, or just at release time?

Step 4: Audit Your AI Package Imports (This Is the New Part)

This is where the 2026 threat model diverges from 2020. AI coding assistants accelerate the pace at which dependencies enter your project — and they can hallucinate package names that attackers then typosquat.

Cloudsmith calls this “slopsquatting” — attackers register malicious packages matching the names AI models tend to invent. If your AI assistant suggests import torch-optimizer and a malicious package by that name exists on PyPI, you may install it without ever checking.

Here’s how to audit this specifically:

• Check every new dependency an AI tool introduced. Go through your recent commit history. For each dependency added in the last 90 days, open its PyPI/npm page and verify: (a) the publisher is who you think it is, (b) the package has a reasonable download count and version history, (c) the repository link actually works and points to a real project.
• Watch for one-release packages. A package with version 0.1.0, published last week, with 47 downloads and no linked source repository is a red flag — regardless of whether an AI suggested it.
• Verify model provenance if you’re loading weights. If your project downloads model files from Hugging Face or similar, check: who uploaded it, when, and whether there’s a linked paper or repo. A model with no documentation and few downloads is a risk — pickle-format models can execute arbitrary code on load.

As Cloudsmith’s guide puts it: “Developers using AI coding tools pull in OSS packages rapidly and at scale, often without manual risk review. This bypasses traditional governance, meaning attack vectors reach production faster than ever.”

Audit question: Look at your most recent package.json or requirements.txt commit. How many of the added dependencies did you personally verify before merging?

Step 5: Check Your CI/CD Pipeline for Credential Leakage

The Shai-Hulud worm spread by scraping CI/CD pipeline credentials from compromised developer environments and using them to publish poisoned package versions. The AsyncAPI attack in July 2026 worked the same way — hijack the pipeline, publish under a trusted namespace, reach millions of downloads before anyone notices.

Two checks:

• Secrets in environment variables. If your build pipeline has long-lived tokens or registry credentials in environment variables that a compromised build step can read, those credentials are a worm’s entry point. Rotate them. Prefer short-lived tokens with scoped permissions over static ones.
• GitHub Actions and similar workflows. Review your workflow files for pull_request triggers that run unchecked code from forks. A malicious PR that adds a post-install script to your build can exfiltrate credentials before you merge. The Cloudsmith guide on the AsyncAPI attack covers this in detail — the attacker didn’t break in; they were invited.

Audit question: If someone opened a PR to your repo today that added a post-install hook, would your CI pipeline run it with access to your registry credentials?

Step 6: Evaluate What Your AI Tools Themselves Bring In

The AI tools are part of your supply chain too. An IDE extension, a CLI coding assistant, or a model-serving endpoint — each one introduces dependencies, network calls, and potentially model weights that run in your environment.

For IDE extensions and marketplace tools:

• Check who published it and whether the publisher owns the namespace (the Manifold Security disclosure on Open VSX “evil twin” extensions showed how easy namespace confusion is)
• Look at what permissions the extension requests — file system access, network calls, terminal execution — and ask whether those permissions match the tool’s stated purpose
• If the tool downloads models or packages on your behalf, verify those downloads are going to the right source

For local model serving (Ollama, LM Studio, etc.):

• Know where your models come from — a model pulled from a public registry with no verification is a supply chain risk, especially in pickle or other executable formats
• Check whether the model-serving tool itself has dependencies you haven’t audited — Ollama, for instance, pulls in container images and network services on install

I covered IDE extension vetting in depth in my VS Code extension vetting tutorial — the Open VSX API metadata checks and VSIX inspection workflow apply to any marketplace tool you’re evaluating — I covered this in my VS Code extension vetting tutorial. For broader supply chain scanning, I wrote about detecting compromised npm packages after the Shai-Hulud incident — the detection commands there still work — I wrote a full guide to detecting compromised npm packages after the Shai-Hulud incident with the exact commands to run.

Step 7: Build a One-Page Audit Checklist You Can Run Monthly

Identity the highest-impact checks and make them repeatable. Here’s a starter checklist you can put in a markdown file and run every sprint:

## Monthly Supply Chain Audit Checklist

- [ ] Regenerate SBOM and verify it's searchable
- [ ] Run npm audit / pip safety / SCA scan
- [ ] Check for new dependencies added in the last 30 days — verify each one
- [ ] Review CI/CD secrets — rotate anything older than 90 days
- [ ] Review GitHub Actions / CI workflow files for untrusted trigger risks
- [ ] Check IDE extensions and AI tool permissions — remove anything unused
- [ ] Verify model sources if loading weights from registries
- [ ] Check for packages with no recent releases or single maintainers in your tree

This isn’t a compliance exercise. It’s the difference between noticing a compromised package on day one and discovering it six months later when something odd starts happening in production.

What This Doesn’t Cover (Yet)

A proper supply chain security program goes further — SLSA build provenance, signed artifacts, policy engines that block vulnerable packages at the registry level, continuous monitoring with EPSS-based prioritization. The Cloudsmith 2026 guide covers the full spectrum from static SBOMs to agentic governance, and Veracode’s blueprint walks through building a program from scratch.

But those are program-level commitments. The audit above is something you can start today with the tools you already have — and it will catch real risks that a clean npm audit result would miss.

As an ICT manager, I’ve learned that the supply chain is only as strong as the weakest link your team didn’t know existed. Start with visibility. Build from there.

If you found this useful, you might also want to read How to Get Reliable JSON From Local LLMs With Ollama — because getting structured output from a local model you control is one way to reduce your dependency on external AI services you can’t audit — like in my guide to getting reliable JSON from local LLMs with Ollama.