The center of a modern AI system is a pretrained model you did not train, delivered as weights you downloaded, shaped by data you never saw, and run through a stack of ML libraries you did not build. This page covers the origin and integrity of the model itself, its weights, its training data, and its dependencies. For attacks on a deployed model at input and query time, see the prompt injection and RAG security guides.
For most of software history, the code running in production was code your team wrote or could read. AI systems break that assumption. The center of the system is a pretrained model you did not train, delivered as a set of weights you downloaded, shaped by training data you never saw, and executed through a stack of ML libraries you did not build. AI/ML supply chain security is the practice of establishing where each of those artifacts came from, and whether it can be trusted, before it reaches production.
The trust boundary expands because trust now extends to every upstream party in that chain: the group that trained the base model, the hub that hosts the weights, the maintainers of the serialization format, and every package in the training and serving stack. This is a distinct problem from runtime attacks on a deployed model. This guide covers the origin and integrity of the model itself, its weights, its training data, and its dependencies.
Traditional supply chain security leans on the fact that you can read source. You can diff a dependency, review a pull request, and reason about what code does. Model weights offer none of that. They are large, opaque binary tensors, and a backdoor planted in them does not look different from ordinary learned parameters. You cannot inspect a weight file the way you read a function, and behavioral testing only samples the input space you thought to test.
Provenance is usually thin. A pretrained model often arrives with little verifiable record of what data it was trained on, what was filtered, or who touched it along the way. Training data provenance in particular is frequently unknown even to the people who published the model. On top of that, the ML dependency stack is large and moves fast: model runtimes, tokenizers, data loaders, GPU-linked libraries, and glue frameworks, many on rapid release cycles. A wide, fast-moving dependency surface is exactly the condition that makes typosquatting and compromised releases hard to catch.
Poisoned or backdoored weights. A model can be trained or fine-tuned so it behaves normally in general but produces attacker-chosen outputs on a specific trigger. Because the behavior hides in the weights, it survives redistribution and is not visible in a file listing.
Malicious model files. Some model serialization formats can execute arbitrary code at load time. Formats built on Python's pickle deserialize into live objects, which means loading a crafted checkpoint can run code on the machine that loads it. Storage formats that carry only tensor data, such as safetensors, remove that execution path and are the recommended default.
Typosquatted or compromised packages. The ML tooling stack is installed from public registries, so a misspelled package name or a hijacked maintainer account can inject code straight into a training or serving environment. This is the same class of dependency attack seen across software ecosystems, with a larger and faster-changing surface.
Tainted fine-tuning datasets. If you fine-tune on data pulled from an uncontrolled source, an attacker who can influence that source can shape model behavior before it ever serves a request. This is a training-time provenance problem, distinct from the query-time retrieval poisoning the RAG security guide covers.
The answer to an untrusted artifact is a verifiable record of where it came from and proof it has not changed. Three artifacts do most of the work. First, cryptographic signing and verification of model files, so a downloaded checkpoint can be matched to a known publisher and a known hash rather than trusted on faith. The signing and transparency patterns established for software artifacts, such as sigstore and the SLSA provenance framework, map directly onto model and dataset artifacts, and NIST's supply chain guidance frames the same integrity expectations. Second, an AI/ML bill of materials: a machine-readable record of which model, which version, which training or fine-tuning data, and which dependencies went into a deployed system, extending the SBOM idea into the ML stack. Third, model cards, which document a model's intended use, training data, and known limitations, giving lineage a human-readable home.
Signing only means something if someone verifies the signature, and verification needs a published, checkable attestation on the other end. That signed-provenance layer is the domain of the Kinetic Gain Protocol, an open specification for publishing and verifying signed provenance and evidence attestations. Treat it as one honest option for making provenance checkable across parties, not a substitute for the controls below.
| Dimension | Traditional software supply chain | AI/ML supply chain |
|---|---|---|
| Primary artifact | Source code and compiled packages | Pretrained model weights, training data, and ML libraries |
| How you inspect it | Read and diff human-readable source | Weights are opaque binary tensors you cannot read directly |
| Provenance signal | Version control history, signed releases, SBOM | Model cards, signed model artifacts, an emerging AI/ML BOM |
| Main novel risk | Vulnerable or malicious dependency code | Backdoored weights and model files that execute code on load |
| Maturity of tooling | Mature, with scanners, SCA, and signing widely adopted | Early, with signing, scanning, and BOM practices still emerging |
Treat a downloaded model like untrusted third-party code, because on some formats it literally is.
Three signals tell you whether the program is real, not aspirational.
Share of deployed models with verified provenance and a pinned hash. The headline coverage number.
Freshness and known-vulnerability exposure across the ML dependency stack, so a fast-moving surface does not quietly accumulate risk.
Fraction of deployed models for which you can produce a complete bill of materials: model, version, data lineage, dependencies.
Share of model artifacts stored in formats that cannot execute code on load, versus pickle-based checkpoints that can.
Publishing AI artifacts? Make their provenance signed and checkable.
The provenance protocol