Adversarial machine learning attacks, taxonomies, and mitigations across the ML lifecycle.
Adversarial ML gives useful language for thinking about evasion, poisoning, privacy attacks, and model misuse. It becomes especially valuable when teams need to distinguish classic model risk from the broader application-layer risks found in modern LLM systems.
These items are included because they add useful evidence, framing, implementation detail, or upcoming context for teams working in this area.
NIST finalizes AI 100-2e2025, providing a terminology and taxonomy for adversarial machine learning across predictive and generative AI systems.
PMLR Volume 299 collects fourteen peer-reviewed CAMLIS papers spanning typographic prompt injection, system-level AI red teaming, white-box LLM backdoors, scam agents, LLM attack defenses, poisoned-model restoration, security knowledge graphs, cloud identity analysis, and production cyber-defense agents. Individual entries provide stable abstracts, citations, and open PDFs, with code or supplemental material where available.
NVIDIA’s 2023 AI red-team introduction organizes assessments across the ML lifecycle, infrastructure and organizational risk. It combines conventional security testing, model attacks and harm scenarios, then illustrates lifecycle boundaries, privilege separation and tabletop exercises. The framework helps teams identify affected components and assign responsibility across data collection, training, deployment and monitoring.
Play video
Edward Raff and collaborators introduce Maximum Violated Multi-Objective attacks for manipulating financial statements while simultaneously reducing model-generated fraud scores. Their evaluation finds roughly 20 times more successful dual-objective attacks than standard methods; in about half of tested cases, earnings could be inflated 100–200% while fraud scores fell 15%.
Play video
Ji'an Zhou and Lei Lu show how a malicious model artifact can move beyond familiar pickle or Lambda-layer deserialization bugs into native memory corruption. Their Black Hat briefing builds an end-to-end three-stage chain from a crafted model file through controlled heap layout and control-flow hijacking to reliable code execution, then evaluates the attack against real inference systems.
Unit 42 presents a two-forward-pass method for identifying feed-forward neurons causally tied to a target behavior. In Qwen3-4B, disabling 50 of 350,208 neurons changed the refusal format on 80% of 520 harmful prompts; across 13 tested models, an FFN/Skip ratio explained 81% of measured vulnerability to small targeted changes.
garak 0.17.0 adds EU AI Act reference mappings and improves AgentBreaker judging, package-hallucination detection, exfiltration probes and report analysis. The open-source tool also changes supported Python versions and fixes hangs against unreachable compatible endpoints.
Play video
Sanket Badhe presents ScamAgent, an autonomous multi-turn framework that combines planning, conversational memory, deceptive framing, and text-to-speech to produce realistic scam calls. Evaluation against current model safeguards shows that distributing malicious intent across apparently benign turns can bypass prompt-level refusal and content filtering.
Play video
The AgentFuzz researchers present directed greybox fuzzing for finding paths from attacker-controlled natural-language input to security-sensitive agent operations. Their evaluation across 20 open-source agents combines generated seed prompts, semantic and distance feedback, and argument-aware mutation, reporting 34 high-risk zero-days and 23 assigned CVEs.
NVIDIA shows how a deliberately placed Pickle-backed model can beacon through a DNS canary when loaded, turning unauthorized use of a model artifact into a detection signal. The technique complements access controls and safer formats such as safetensors; it does not make untrusted Pickle files safe to load.
Play video
Ashley Song and collaborators evaluate test-time compute strategies on two operational cybersecurity agents: a container vulnerability analysis workflow and a server-alert triage system. The study examines whether allocating more inference-time reasoning can improve both answer accuracy and consistency across repeated runs.
Play video
BlackIce packages fourteen open-source responsible-AI, LLM-security, and adversarial-ML tools into a reproducible, version-pinned container with a unified command-line interface. The CAMLIS presentation explains tool selection, coverage, dependency isolation, image architecture, and a working assessment demonstration rather than presenting the bundle as a substitute for test design.
garak 0.14.0 adds JSON configuration, redesigned HTML reports, detector calibration and benchmarking guidance, new generator integrations, and broader report records. It also changes the eval and digest JSONL formats and aggregate scoring, making the release operationally important for teams that parse reports or compare historical baselines.
garak v0.16.0 begins context-aware scanning with technique and intent annotations and IntentProbe, adds native Anthropic and adaptive-attack plugins, and introduces a unified selection grammar plus revised report output.
NVIDIA argues that most proposed model CVEs actually describe vulnerable serving applications, unsafe serialization and supply-chain formats, access-control failures, or statistical behaviors shared by a model class. A narrow exception may exist for deliberately poisoned training that creates a reproducible backdoor in a specific weight artifact.
NVIDIA analyzed nearly 140 GB of Meta's Kaggle for Code corpus and found more than 140 active plaintext third-party credentials, widespread pickle deserialization, common import typos, and no imports of several adversarial-testing libraries. The study cautions that isolated competition notebooks still shape code and habits that migrate into production.
Play video
Text2VLM is a reproducible pipeline that extracts harmful concepts from text-only safety datasets and renders them as typographic images for multimodal evaluation. Human validation supports the transformation pipeline, and tests of open-source visual language models find greater prompt-injection susceptibility when the same concepts arrive through images instead of plain text.
GuardFall tests 11 open-source coding and computer-use agents against shell-command transformations that evade string and regex deny lists, including quote removal, $IFS expansion, command substitution, and encoded payloads. The study finds configuration- and model-dependent failures and shows that local or auto-approve modes can turn untrusted repository content into host command execution; it is vendor-authored research, not an independent benchmark.
TrustFall shows how project-defined MCP configuration can turn a generic “trust this folder” decision into unsandboxed command execution in several coding agents, with zero-click variants in unattended CI. The vendor-authored research traces the issue to conflating permission to read or edit a workspace with permission to start repository-supplied executables.
Trail of Bits bypassed multiple agent-skill scanners with compiled Python hidden beside benign source and with prompt-like prose that persuaded an LLM classifier to accept a malicious configuration. The experiments show recurring blind spots around unreferenced files, binaries, assets, and ambiguous installer behavior, and also explain why legitimate skills can contain patterns that look malicious.
Play video
The NDSS-backed research identifies six inference-time cache attacks across vLLM, SGLang, GPTCache, and related stacks. Weak prefix and image cache keys plus semantic near-match errors can make distinct inputs share cached state, enabling poisoned responses, information leakage, and moderation bypass; the authors provide experimental artifacts and vendor disclosures.
Google introduced Gemini Omni Flash, a multimodal model that combines text, image, audio, and video references to generate and iteratively edit video through natural-language conversation. Generated videos include a SynthID watermark.
NVIDIA demonstrates a model supply-chain attack in which a privileged adversary edits a tokenizer JSON file so visible words map to different token IDs. The change can make the model interpret "deny" as "allow" or corrupt decoded output while leaving the model weights untouched.
CISA, NSA and FBI allege industrial-scale extraction of US model capabilities through distributed accounts, proxies and aggregators. Their advisory recommends correlating prompts, usage and account behavior across providers; it distinguishes these alleged campaigns from legitimate model distillation.