AI Model Security Assessment

Where MCP Security Assessment covers the tools and integrations an agent can reach, this guide covers the model itself — its weights, its training and fine-tuning data, its resistance to adversarial manipulation, and the infrastructure it runs on. Whether you're evaluating a third-party foundation model API, an open-weight model you're self-hosting, or a model your own team fine-tuned on internal data, the model is a distinct asset with its own attack surface: it can be stolen, poisoned, extracted, or manipulated into behaving in ways its own safety training was supposed to prevent. This guide gives security teams a structured assessment framework aligned to the NIST AI Risk Management Framework (AI RMF 1.0), its Generative AI Profile (NIST AI 600-1), NIST SP 800-218A (Secure Software Development Practices for Generative AI and Dual-Use Foundation Models), NIST SP 800-53 Rev. 5, and MITRE ATLAS.

Scope: Assessing a specific model or model deployment before/during onboarding — foundation model APIs, self-hosted open-weight models, and internally fine-tuned models. Pairs with MCP Security Assessment for the tool/integration layer built around the model.

Why Models Are a Distinct Asset Class

A model isn't just software — it's a compressed, queryable artifact partly derived from its training data, and its behavior can't be fully specified or verified by reading code the way a traditional application can. That produces risks with no direct conventional-software equivalent:

Risk	Description
Model extraction / theft	Systematic querying of an inference API can reconstruct a functionally equivalent copy of a proprietary model — the API itself becomes the exfiltration path, no traditional data breach required.
Membership inference / model inversion	Carefully crafted queries can reveal whether specific records were in the training set, or reconstruct approximations of training data itself — a privacy failure even when the model's outputs look benign.
Training/fine-tuning data poisoning	An attacker who can influence training or fine-tuning data (including RLHF feedback data or user-submitted correction data if it's fed back into retraining) can implant behavior that activates only under specific trigger conditions.
Backdoored / "sleeper" models	A model can behave normally on all standard evaluation and red-team inputs, then exhibit malicious behavior only when a specific, hard-to-guess trigger pattern appears — evaluation results alone don't rule this out.
Adversarial evasion	Inputs crafted (sometimes imperceptibly, for non-text modalities) specifically to cause misclassification or bypass safety/content filters, distinct from a general-purpose jailbreak prompt.
Weight exfiltration	For self-hosted models, the weight files themselves are a high-value target — theft doesn't require any inference access at all, just filesystem/storage access.
Supply chain risk in pretrained checkpoints	A downloaded pretrained or fine-tuned checkpoint from a public hub can carry the same backdoor/poisoning risk as a malicious software dependency, with far less tooling available to inspect it than a code SCA scan.

Assessment Domains

1. Model Provenance and Supply Chain

Check	What You're Looking For
Publisher/source verification	Is the model from a known, accountable organization, or an unverified community upload with no traceable authorship?
Model card review	Does a model card document training data sources, intended use, known limitations, and evaluation results — or is provenance undocumented?
Checksum/signature verification	Are downloaded weights verified against a published checksum or signature, not trusted implicitly from the download source?
License and usage rights	Does the license actually permit your intended use case (commercial, fine-tuning, redistribution)?
Version pinning	Is a specific, reviewed model version/checkpoint pinned, or does the pipeline auto-pull "latest" from a public hub?
Fine-tuning lineage	If fine-tuned internally or by a vendor, is the base model and the fine-tuning dataset both known and reviewed, not just the final checkpoint?

2. Training and Fine-Tuning Data Integrity

Check	What You're Looking For
Data source review	Where did training/fine-tuning data originate — licensed, scraped, internally generated, user-submitted?
PII/sensitive data exposure	Was the dataset screened for PII, secrets, or regulated data before training — see Data Security Posture Management for the discovery approach
Poisoning controls	Was the dataset from any untrusted/crowdsourced/scraped source validated for anomalous or intentionally mislabeled samples before use?
Feedback loop integrity	If user interactions or corrections feed back into future fine-tuning (RLHF, continual learning), is that pipeline itself protected against being gamed by adversarial submissions?
Data retention and deletion	Can specific records be identified and removed from future training runs if required (right-to-erasure, license revocation, discovered contamination)?

3. Model Weight and Artifact Protection

Check	What You're Looking For
Encryption at rest	Are weight files encrypted at rest, with access restricted to the specific service/role that needs to load them — see Encryption
Access control	Is access to raw weights separated from access to the inference API — can most engineers query the model without ever being able to download it?
Storage location	Are weights stored in an access-controlled artifact registry/bucket, not a general-purpose shared drive or unrestricted object storage bucket?
Transfer protection	Is TLS enforced for any transfer of weights between training, storage, and serving environments?
Audit logging on artifact access	Is every read/download of the weight files logged and attributable to an identity, the same bar applied to source code repositories?

This directly reflects the second pillar of CISA/NSA's joint guidance: model weights are described there as "the most valuable and vulnerable component" of a deployed AI system and warrant protection on par with your most sensitive source code or cryptographic key material.

4. Adversarial Robustness and Extraction Resistance

Check	What You're Looking For
Adversarial input testing	Has the model been tested against crafted adversarial inputs designed to cause misclassification, bypass content filters, or induce unsafe output — not just standard accuracy benchmarks?
Jailbreak resistance testing	For generative/LLM models specifically, has it been red-teamed against known jailbreak technique families, not just a fixed prompt-injection test suite that goes stale quickly?
Extraction attack resistance	Are there rate limits, query-pattern anomaly detection, or output perturbation defenses that make systematic model-extraction querying impractical?
Membership inference resistance	For models trained on sensitive data, has differential privacy or comparable technique been evaluated/applied during training, and has membership-inference testing been performed?
Backdoor/trojan scanning	For third-party or fine-tuned checkpoints, has any trigger-pattern/backdoor scanning been performed (e.g., activation clustering, trigger-search tooling), rather than relying solely on clean-input evaluation?

5. Input/Output Guardrails

Check	What You're Looking For
Input validation	Are inputs constrained (length, format, encoding) before reaching the model, reducing malformed/adversarial input surface?
Output filtering	Is a moderation/filtering layer applied to outputs before they reach a user or downstream system, independent of the base model's own safety training?
Defense in depth	Are guardrails implemented as a separate, independently auditable layer, rather than relying entirely on the model's own alignment holding under adversarial pressure?
Prompt injection handling	If the model consumes untrusted content (documents, web content, tool output), is that content flow reviewed the same way as in MCP Security Assessment?

6. Deployment and Inference Infrastructure Security

Check	What You're Looking For
Inference API authentication	Is the inference endpoint authenticated and rate-limited per caller, not an open, unauthenticated endpoint?
Network exposure	Is the serving infrastructure on an internal/segmented network unless a public-facing use case specifically requires exposure?
Isolation	Does the serving environment isolate the model process appropriately from other workloads (container/VM boundaries, dedicated service accounts)?
Resource limits	Are compute/token quotas enforced per caller to prevent cost-harvesting or denial-of-service against the serving infrastructure?
Environment parity	Is the production serving environment's configuration reviewed/hardened the same way any production system would be, not left at development defaults?

7. Monitoring, Logging, and Drift Detection

Check	What You're Looking For
Inference logging	Are inference requests and responses logged (with appropriate data-sensitivity handling) for audit and incident reconstruction?
Anomalous query detection	Is there monitoring for query patterns consistent with extraction attempts (high-volume systematic querying, boundary-probing inputs)?
Output monitoring	Is a sample of outputs monitored for policy-violating or anomalous content in production, not just at initial evaluation?
Performance/behavioral drift	Is model output quality/behavior monitored over time to catch drift or degradation that could indicate data pipeline issues or tampering?
Alerting integration	Do these signals reach the same SIEM/monitoring stack as other production systems, rather than living in a separate ML-only dashboard nobody in security reviews?

8. Third-Party / Foundation Model API-Specific Considerations

Check	What You're Looking For
Data retention policy	Does the vendor retain submitted prompts/data, and for how long — is a zero/limited-retention option available and enabled?
Training-use policy	Is customer data explicitly excluded from being used for the vendor's own model training/improvement, contractually, not just by default setting?
Vendor security posture	Does the vendor publish independent audit results (SOC 2, ISO 27001) and their own red-team/safety evaluation results?
Regional/residency requirements	Does the vendor's processing location meet your data residency and regulatory requirements?
Incident notification	Does the vendor contract commit to timely notification if a security incident affects the model or the data processed through it?

Risk Tiering

Tier	Criteria	Review Bar
Low	API-only consumption of an established vendor's model, no fine-tuning, non-sensitive use case	Standard checklist review; vendor due-diligence documentation on file
Medium	Fine-tuning on internal (non-regulated) data, or API use with confidential data under a reviewed data-processing agreement	Full checklist; data-handling and retention terms explicitly verified
High	Self-hosted open-weight model, regulated or highly sensitive training/fine-tuning data, or public-facing inference endpoint	Full checklist + architecture review; adversarial/red-team testing before launch; weight access control audited
Critical	Model trained/fine-tuned on regulated data at scale, high-consequence use case (safety, financial, medical decisioning), or unverified/community-sourced checkpoint in the lineage	Default deny pending remediation; independent red team; documented accountable owner; compensating monitoring controls — handled as a security exception per Application Control

Pre-Deployment Checklist

#	Item	Status
1	Model provenance, publisher, and model card reviewed	☐
2	Weight/checkpoint checksum or signature verified; version pinned	☐
3	Training/fine-tuning data sources and PII exposure reviewed	☐
4	Weight files encrypted at rest with access restricted from inference-only roles	☐
5	Adversarial robustness and jailbreak-resistance testing completed	☐
6	Extraction-attack mitigations in place (rate limiting, anomaly detection)	☐
7	Independent input/output guardrail layer implemented	☐
8	Inference API authenticated, rate-limited, and network-scoped appropriately	☐
9	Inference logging wired into central monitoring/SIEM	☐
10	Vendor data retention/training-use policy reviewed (third-party models)	☐
11	Risk tier assigned and matching controls applied	☐
12	Accountable owner and re-review trigger (retrain, fine-tune, version change) documented	☐

Ongoing Assessment — Recertification Triggers

Any retraining or fine-tuning event — behavior can shift even with the same architecture and a nominally similar dataset.
Any base model version change for a vendor API, including "silent" backend model swaps behind a stable API name.
New data sources added to a training or fine-tuning pipeline.
Newly published attack techniques relevant to the model's architecture or modality (monitor MITRE ATLAS and vendor security advisories).
Anomalous monitoring signals — unusual query patterns, output drift, or a spike in guardrail-layer rejections.
Scheduled recertification on your standard cadence even absent a specific trigger.

MITRE ATLAS — Tactic Coverage

MITRE ATLAS catalogs adversary tactics and techniques specific to AI systems, structured analogously to MITRE ATT&CK. Use it as the basis for red-team scoping — map planned test coverage against each tactic rather than testing ad hoc:

ATLAS Tactic	Assessment Activity
Reconnaissance	Review what's discoverable about the model publicly (architecture, training data hints, API documentation) that would aid an attacker
Resource Development	Assess whether an attacker could cheaply stand up a similar model or acquire training data to develop attacks offline
Initial Access	Review how the model/API is exposed and what access controls gate reaching it at all
ML Model Access	Distinguish and separately assess API-only access vs. full white-box weight access — the attack surface differs substantially
Execution / Persistence	For self-hosted deployments, review whether a compromised model artifact could establish persistence in the serving environment
Defense Evasion	Jailbreak and content-filter bypass testing (Domain 4/5 above)
Discovery / Collection	Test resistance to systematic probing intended to map model behavior or extract training data (membership inference)
ML Attack Staging	Assess backdoor/trojan risk in third-party checkpoints before they reach production (Domain 1)
Exfiltration	Extraction-attack resistance testing, weight-file access control (Domains 3–4)
Impact	Evaluate downstream consequences if the model is successfully manipulated, extracted, or poisoned, informing the risk tier

NIST AI RMF (AI RMF 1.0) Function Mapping

Function	Applied to Model Assessment
Govern	Accountable ownership and approval policy for models before deployment, independent of who requested it
Map	Document the model's intended use, data lineage, and deployment context specifically — the same model can carry different risk in different use cases
Measure	Execute the domain checklist above — provenance, robustness, extraction resistance, guardrail effectiveness
Manage	Apply tiered controls, monitor continuously, and recertify on the triggers above

NIST AI 600-1 (Generative AI Profile) — Relevant Risk Categories

GAI Risk Category	Relevance to Model Assessment
Information Security	Weight protection, extraction resistance, adversarial robustness (Domains 3–4)
Data Privacy	Membership inference/model inversion risk, training data PII exposure (Domains 2, 4)
Confabulation	Relevant to output reliability evaluation feeding into guardrail design (Domain 5)
Value Chain and Component Integration	Pretrained/fine-tuned checkpoint provenance and supply chain (Domain 1)
Intellectual Property	License review, training data rights (Domain 1)
CBRN Information or Capabilities	For general-purpose foundation models, specific evaluation of uplift risk in high-consequence domains as part of red-team scope

NIST SP 800-218A — Secure Development for Generative AI / Dual-Use Foundation Models

NIST SP 800-218A extends the Secure Software Development Framework specifically to AI model development and deployment. Map your model lifecycle against its practice areas rather than treating model development as exempt from your existing SSDF program:

Practice Area	Applied Here
Prepare the organization	Accountable ownership, risk-tiering policy (Govern function above)
Secure the training/tuning data and pipeline	Domain 2
Secure the model itself	Domains 3–4
Secure the model's deployment/serving environment	Domain 6
Monitor and respond	Domain 7, recertification triggers

NIST SP 800-53 Rev. 5 Control Mapping

Control	Title	Applied Here
AC-3 / AC-6	Access Enforcement / Least Privilege	Separating weight access from inference-only access (Domain 3)
SC-28	Protection of Information at Rest	Weight file encryption at rest
SC-8	Transmission Confidentiality/Integrity	TLS for weight transfer and inference traffic
SI-4	System Monitoring	Anomalous query detection, output monitoring
SI-10	Information Input Validation	Input constraints ahead of the model (Domain 5)
SA-11	Developer Testing and Evaluation	Adversarial robustness and jailbreak testing before deployment
SR-3 / SR-4	Supply Chain Controls / Provenance	Pretrained checkpoint verification (Domain 1)
PT-2 / PT-3	Authority to Process PII / PII Processing Purposes	Training data privacy review (Domain 2)
RA-5	Vulnerability Monitoring and Scanning	Backdoor/trojan scanning of third-party checkpoints
CA-7	Continuous Monitoring	Recertification triggers and ongoing drift/behavior monitoring