Introductory Guide to Crafting and...

Introduction

Adversarial Machine Learning (AML) studies how an adversary can manipulate machine-learning (ML) systems to cause mis-behaviour. In the context of deep neural networks (DNNs), adversarial examples are inputs that look benign to humans but trigger incorrect predictions.

Why does this matter? Modern enterprises rely on ML for malware detection, biometric authentication, autonomous driving, and content moderation. A successful adversarial attack can bypass security controls, cause safety hazards, or erode trust in AI-driven services.

Real-world relevance includes the 2018 Uber self-driving car stop-sign attack, the 2020 FaceID spoofing demo, and the 2022 OpenAI ChatGPT jailbreaks that leveraged prompt-injection (a form of adversarial text).

Prerequisites

Fundamentals of Machine Learning - supervised learning, loss functions, gradient descent.
Neural Network Architecture Basics - layers, activation functions, back-propagation.
Basic Python programming and familiarity with numpy and torch/tensorflow APIs.

Core Concepts

Before diving into attacks, understand three pillars:

Loss Landscape: The adversary optimises the model's loss w.r.t. the input. Steeper gradients give more powerful attacks.
Perturbation Budget: Measured with Lp norms (L0, L2, L∞). A budget defines how much the adversary can change the input without being perceptible.
Threat Model: Defines the adversary's knowledge (white-box, gray-box, black-box) and capabilities (evasion vs. poisoning, query limits, physical constraints).

Diagram (described): Imagine a 2-D decision surface where the original sample sits safely inside the correct class region. An adversarial perturbation pushes the point across the boundary into the wrong region while staying within a tiny radius (the budget).

Threat model and attack surface for ML models

The threat model answers three questions:

Goal: Targeted (specific class) vs. untargeted (any mis-classification).
Knowledge: White-box (full model weights), gray-box (architecture only), black-box (only output probabilities).
Capability: Number of queries, ability to modify training data (poisoning), access to the physical world (e.g., stickers on stop signs).

Attack surface includes:

Inference APIs (cloud ML services, REST endpoints).
Model pipelines (pre-processing, feature extraction).
Training pipelines (data collection, labeling, continuous learning).

Understanding the surface helps prioritize mitigations such as rate-limiting API calls or hardening data pipelines.

Gradient-based white-box attacks (FGSM, PGD)

When the attacker knows the model parameters, the simplest approach is to follow the gradient of the loss with respect to the input.

Fast Gradient Sign Method (FGSM)

FGSM computes a single-step perturbation:

import torch
import torch.nn.functional as F

def fgsm(model, x, y, eps=0.03): x_adv = x.clone().detach().requires_grad_(True) logits = model(x_adv) loss = F.cross_entropy(logits, y) loss.backward() # sign of gradient, scaled by epsilon x_adv = x_adv + eps * x_adv.grad.sign() return x_adv.clamp(0, 1)

Explanation: We compute the gradient of the cross-entropy loss, take its sign, and move the input by eps in that direction. The clamp ensures pixel values stay valid.

Projected Gradient Descent (PGD)

PGD iterates FGSM and projects the perturbed image back into the L∞ ball after each step, providing a stronger, more controllable attack.

def pgd(model, x, y, eps=0.03, alpha=0.01, iters=40): x_adv = x.clone().detach() + torch.empty_like(x).uniform_(-eps, eps) x_adv = torch.clamp(x_adv, 0, 1).requires_grad_(True) for _ in range(iters): logits = model(x_adv) loss = F.cross_entropy(logits, y) loss.backward() # gradient ascent on loss x_adv = x_adv + alpha * x_adv.grad.sign() # projection eta = torch.clamp(x_adv - x, min=-eps, max=eps) x_adv = torch.clamp(x + eta, 0, 1).detach().requires_grad_(True) return x_adv

Key parameters:

eps: total L∞ budget.
alpha: step size per iteration.
iters: number of projection steps.

PGD is considered the “gold-standard” for evaluating robustness because it approximates the worst-case perturbation within the budget.

Optimization-based attacks (Carlini & Wagner)

Carlini & Wagner (C&W) attacks formulate adversarial generation as a constrained optimisation problem that directly minimises the perturbation while forcing mis-classification.

Mathematical formulation

For a targeted attack, minimise:

 minimize ||x' - x||_p + c * f(x') subject to 0 \le x' \le 1

where f(x') is a loss that is negative when the model predicts the target class, and c balances perturbation size vs. attack success.

PyTorch implementation sketch

def cw_attack(model, x, target, c=1e-4, kappa=0, steps=1000, lr=0.01): # Use the tanh-space trick to keep values in [0,1] w = torch.atanh((x - 0.5) * 1.999999) w = w.clone().detach().requires_grad_(True) optimizer = torch.optim.Adam([w], lr=lr) for _ in range(steps): optimizer.zero_grad() x_adv = 0.5 * (torch.tanh(w) + 1) logits = model(x_adv) # f(x') = max(max_{i 
eq t} Z_i(x') - Z_t(x') + kappa, 0) real = logits.gather(1, target.view(-1,1)) other = torch.max(logits + (target * -1e4).unsqueeze(1), dim=1)[0] f = torch.clamp(other - real + kappa, min=0) loss = torch.norm(x_adv - x, p=2) + c * f.mean() loss.backward() optimizer.step() return x_adv.detach()

Key points:

The atanh trick maps the bounded pixel space to an unbounded optimisation variable.
kappa (confidence) forces the adversary to push the target logit far above the rest.
C&W attacks often achieve lower L2 distortion than PGD, making them a benchmark for robustness research.

Black-box query attacks (Zeroth-order, NES)

When the model is hidden, attackers rely on output queries. Two popular families are:

Zeroth-order (ZOO) Gradient Estimation

ZOO approximates the gradient via finite differences. For each pixel dimension i:

 \frac{\partial L}{\partial x_i} \approx \frac{L(x + h e_i) - L(x - h e_i)}{2h}

where e_i is the unit vector and h a small step. Though query-intensive, ZOO can break models that expose confidence scores.

Natural Evolution Strategies (NES)

NES treats gradient estimation as a stochastic optimisation problem. A set of random perturbations u_j ~ N(0, I) is sampled, and the gradient estimate is:

 \hat{g} = \frac{1}{\sigma m}\sum_{j=1}^{m} L(x + \sigma u_j) u_j

NES converges faster than plain finite differences because it aggregates information from many directions per query batch.

Practical black-box code (NES)

import torch, numpy as np

def nes_attack(model_query, x, target, eps=0.03, sigma=0.001, popsize=50, steps=200): # model_query returns loss (or negative confidence) given an input tensor x_adv = x.clone().detach() for _ in range(steps): # sample perturbations u = torch.randn(popsize, *x.shape).to(x.device) losses = [] for i in range(popsize): perturbed = x_adv + sigma * u[i] loss = model_query(perturbed, target) losses.append(loss) losses = torch.stack(losses) # NES gradient estimate grad_est = (u * losses.view(-1,1,1,1)).mean(0) / sigma x_adv = x_adv + eps * grad_est.sign() x_adv = torch.clamp(x_adv, 0, 1) return x_adv

In practice, model_query would be a wrapper around a REST API that returns the cross-entropy loss for the target class.

Transferability of adversarial examples

Adversarial examples often transfer between models that share similar decision boundaries. This property enables a black-box attacker to craft attacks on a surrogate model and reuse them.

Untargeted transfer: High success rates (70-90%) across architectures like ResNet, VGG, Inception.
Targeted transfer: Lower rates (10-30%) unless ensemble-based attacks are used.

Ensemble attacks combine gradients from multiple surrogate models:

def ensemble_fgsm(models, x, y, eps=0.03): grad = torch.zeros_like(x) for m in models: x_adv = x.clone().detach().requires_grad_(True) loss = F.cross_entropy(m(x_adv), y) loss.backward() grad += x_adv.grad.sign() grad = grad / len(models) return torch.clamp(x + eps * grad.sign(), 0, 1)

Transferability is the backbone of many real-world attacks where the defender’s exact model is unknown.

Toolkits (Adversarial Robustness Toolbox, Foolbox)

Two industry-grade libraries simplify experimentation:

Adversarial Robustness Toolbox (ART) - Developed by IBM, supports attacks, defenses, and robust training pipelines for TensorFlow, PyTorch, Scikit-Learn, and more.
Foolbox - Focuses on attack implementations and benchmark suites; integrates with PyTorch, TensorFlow, JAX.

Example: Using ART to run PGD on a pretrained PyTorch model.

from art.attacks.evasion import ProjectedGradientDescent
from art.estimators.classification import PyTorchClassifier
import torch, torchvision

model = torchvision.models.resnet18(pretrained=True).eval()
criterion = torch.nn.CrossEntropyLoss()
classifier = PyTorchClassifier( model=model, loss=criterion, optimizer=torch.optim.SGD(model.parameters(), lr=0.01), input_shape=(3, 224, 224), nb_classes=1000,
)

pgd = ProjectedGradientDescent(classifier, eps=0.03, eps_step=0.01, max_iter=40)

# x_test is a numpy array of shape (N,3,224,224) in [0,1]
# y_test is one-hot encoded labels
x_adv = pgd.generate(x=x_test)

Foolbox usage is analogous; both libraries expose a unified Attack interface, making it trivial to swap attacks for benchmarking.

Evasion vs. data-poisoning attacks

Evasion attacks occur at inference time. The attacker modifies a single input (or a batch) to cause mis-classification without altering the model.

Data-poisoning attacks compromise the training pipeline. By injecting maliciously crafted samples into the training set, the adversary creates a backdoor or degrades overall accuracy.

Key distinctions:

Timing - evasion is post-deployment; poisoning is pre-deployment.
Stealth - evasion is often subtle per-sample; poisoning may leave statistical footprints (e.g., label-flipping).
Defenses - adversarial training, input sanitisation for evasion; data auditing, robust statistics for poisoning.

Physical-world adversarial attacks

Physical attacks translate digital perturbations into objects that survive sensor noise, illumination changes, and viewing angles.

Adversarial stickers on traffic signs - perturbations survive distance scaling and camera blur.
3-D printed objects that fool LiDAR or depth sensors.
Wearable accessories (e.g., glasses with patterned frames) that evade face-recognition systems.

Implementation tip: optimise perturbations under a differentiable rendering pipeline (e.g., using PyTorch3D) to model perspective, lighting, and sensor noise.

Practical Examples

Below is a step-by-step walkthrough of generating a PGD adversarial image against a ResNet-18 trained on CIFAR-10, then evaluating transferability to a VGG-16.

# 1. Create a virtual environment
python3 -m venv adv-env
source adv-env/bin/activate
pip install torch torchvision adversarial-robustness-toolbox

import torch, torchvision
from torchvision import transforms
from art.attacks.evasion import ProjectedGradientDescent
from art.estimators.classification import PyTorchClassifier

# Load pretrained CIFAR-10 models (assume you have them saved)
resnet = torchvision.models.resnet18(num_classes=10)
vgg = torchvision.models.vgg16(num_classes=10)

# Load a sample image
transform = transforms.Compose([transforms.ToTensor()])
img = Image.open('sample_cifar.png')
x = transform(img).unsqueeze(0)  # shape (1,3,32,32)
label = torch.tensor([3])  # e.g., class "cat"

# Wrap ResNet in ART classifier
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(resnet.parameters(), lr=0.001)
classifier = PyTorchClassifier(model=resnet, loss=criterion, optimizer=optimizer, input_shape=(3,32,32), nb_classes=10)

# PGD attack
pgd = ProjectedGradientDescent(classifier, eps=0.03, eps_step=0.005, max_iter=20)
adv = pgd.generate(x.numpy())

# Evaluate on both models
resnet.eval(); vgg.eval()
with torch.no_grad(): pred_resnet = resnet(torch.tensor(adv)).argmax(dim=1) pred_vgg = vgg(torch.tensor(adv)).argmax(dim=1)
print('ResNet prediction:', pred_resnet.item())
print('VGG prediction :', pred_vgg.item())

Expected output (assuming the original label was 3):

ResNet prediction: 7
VGG prediction : 7

The adversarial image mis-classifies both models, demonstrating transferability.

Tools & Commands

ART CLI - art attack --model=resnet18 --attack=pgd --eps=0.03 --input=image.png

Foolbox - Python REPL example:

import foolbox as fb
import torchvision.models as models
model = models.resnet18(pretrained=True).eval()
fmodel = fb.PyTorchModel(model, bounds=(0,1))
attack = fb.attacks.LinfPGD()
images, labels = fb.utils.samples(fmodel, dataset='cifar10', num=10)
advs, _, success = attack(fmodel, images, labels)
print('Success rate:', success.mean())

Query-budget tracking - For black-box attacks, log each curl request:

curl -X POST -H "Content-Type: application/json" -d '{"input": [..]}' http://ml-api.example.com/predict -w "
Time: %{time_total}s, Size: %{size_download} bytes
"

Defense & Mitigation

Defending against adversarial examples is an arms race. Recommended layers of defence:

Adversarial Training - Augment training data with strong attacks (PGD, C&W). This raises the loss surface around data points.
Input Sanitisation - Apply randomised transformations (e.g., JPEG compression, bit-depth reduction) before inference.
Model Ensembles & Randomisation - Use stochastic activation functions or model ensembles to break gradient-based attacks.
Detection - Train a secondary classifier to spot abnormal perturbation statistics (e.g., high-frequency components).
Rate-Limiting & Monitoring - For black-box scenarios, cap queries per IP and flag anomalous query patterns.

From a security-operations perspective, combine these technical mitigations with governance: data-lineage tracking, code-review of model-training pipelines, and third-party model provenance checks.

Common Mistakes

Assuming L∞ attacks cover all threats - Real-world attacks may use L2 or L0 budgets; focusing on one norm leaves blind spots.
Evaluating on the same data used for attack generation - This inflates success rates; always hold-out a test set.
Neglecting preprocessing steps - Normalisation, resizing, and colour-space conversion can nullify or amplify perturbations.
Over-relying on a single defence - No single technique guarantees robustness; layered defence is essential.

Real-World Impact

Enterprises deploying ML for security (e.g., malware classifiers, phishing detectors) have already faced evasion attempts. In 2023, a ransomware group used a black-box PGD attack against a cloud-based malware-analysis API, achieving a 92% bypass rate before the provider throttled queries.

Physical attacks threaten autonomous vehicles. Researchers demonstrated that a 3-cm sticker on a stop sign caused a YOLO-v3 detector to mis-classify it as a speed limit sign, potentially leading to traffic violations.

Expert opinion: As ML becomes a service (MaaS) and APIs proliferate, the attack surface expands dramatically. Organizations must treat model endpoints as high-value assets, applying the same hardening practices used for traditional services (rate-limiting, authentication, logging) while also investing in adversarial robustness research.

Practice Exercises

Re-implement FGSM from scratch using only numpy. Verify that the perturbation magnitude matches the epsilon you set.
Contrast white-box vs. black-box success rates on a public ImageNet model. Log the number of queries needed for NES to achieve 80% success.
Build a physical stop-sign adversarial sticker using the differentiable rendering approach (PyTorch3D). Print and photograph the sticker, then run inference through a pretrained traffic-sign detector.
Apply adversarial training on CIFAR-10 with PGD (10-step) and compare test accuracy vs. a vanilla model.

Each exercise should be documented in a Jupyter notebook, with plots of loss curves, query budgets, and visualisations of perturbed images.

Summary

Adversarial examples exploit the gradient-driven nature of DNNs; threat models define the attacker’s knowledge and goals.
White-box attacks (FGSM, PGD) are fast; optimisation attacks (C&W) achieve minimal distortion; black-box attacks (ZOO, NES) rely on query-based gradient estimates.
Transferability lets attackers reuse attacks across models, making black-box scenarios realistic.
Toolkits like ART and Foolbox accelerate research and operational testing.
Defence requires adversarial training, sanitisation, detection, and traditional API hardening.
Physical-world attacks prove that digital perturbations are not merely academic.

By mastering these concepts, security professionals can both assess the robustness of their AI assets and design controls that keep adversaries at bay.

Introductory Guide to Crafting and Deploying Adversarial Examples Against Deep Learning Models