Docker Architecture & Image Layers: Fundamentals for Security Professionals

Introduction

Containers have reshaped how software is built, shipped, and run. Docker, the de-facto standard, abstracts the operating system to provide lightweight, reproducible environments. Understanding Docker's architecture and image layering is a prerequisite for any security professional tasked with assessing container attack surfaces, hardening deployments, or performing forensic analysis.

Why is this important? The way Docker stores and isolates workloads directly influences privilege escalation paths, image-based supply-chain risks, and the effectiveness of runtime defenses such as seccomp or AppArmor. Real-world incidents-e.g., the nginx:1.19.0 vulnerability chain-demonstrated that a single compromised layer can affect hundreds of downstream images.

Prerequisites

• Comfortable using a Linux shell (bash, zsh, etc.).
• Basic knowledge of Linux processes, file permissions, and user/group concepts.
• Familiarity with networking basics (TCP/UDP, ports).

Core Concepts

Docker is built around a client-server architecture. The docker CLI (client) talks to the Docker daemon (dockerd) via a RESTful Unix socket (/var/run/docker.sock) or TCP endpoint. The daemon orchestrates image pulls, container creation, networking, and storage.

Docker images consist of immutable read-only layers stacked using a union file system. When a container runs, Docker adds a thin writable layer on top (the container layer). Each layer is identified by a cryptographic hash (SHA-256), enabling caching and deduplication across the host.

Isolation is achieved through Linux kernel features: namespaces (PID, NET, MNT, IPC, UTS) provide separate views of resources, while control groups (cgroups) enforce resource limits (CPU, memory, I/O). Storage drivers (AUFS, OverlayFS, Btrfs, etc.) implement the union file system and have varying security characteristics.

Below we dive into each subtopic, providing code snippets, security implications, and practical guidance.

Docker client-server model

The Docker CLI sends JSON-encoded API calls over the Unix socket to dockerd. For example, docker pull triggers a POST /images/create request. Understanding this flow helps when intercepting traffic for debugging or when applying network-level policies.

# List the socket permissions
ls -l /var/run/docker.sock
# Typical output:
# srw-rw---- 1 root docker 0 May  6 12:34 /var/run/docker.sock

Because the socket is a file, any process with read/write access can control Docker-effectively root-equivalent. Best practice: limit socket exposure, avoid mounting it inside containers unless absolutely needed, and use Docker Contexts or remote daemons with TLS authentication for multi-host management.

Image vs container lifecycle

An image is a static artifact consisting of layered filesystems and metadata (labels, environment variables, entrypoint). It is built once, stored in a registry, and can be pulled many times.

A container is a runtime instance of an image plus a writable layer, isolated namespaces, and cgroup constraints. The lifecycle stages are:

1. Build: docker build creates image layers from a Dockerfile.
2. Push/Pull: Images are pushed to or pulled from a registry.
3. Run: docker run creates a container, allocating a writable layer and namespace isolation.
4. Commit (optional): docker commit creates a new image from a container’s state-often a security anti-pattern because it bypasses Dockerfile reproducibility.
5. Stop/Remove: docker stop and docker rm clean up the runtime instance; the underlying image remains untouched.

From a security perspective, immutable images reduce drift, while mutable containers are the only place runtime changes (e.g., attacker-added binaries) can reside. Regularly snapshotting containers can aid forensic investigations, but it also creates extra artifacts that must be protected.

Union file systems (AUFS, OverlayFS)

Docker’s storage drivers implement a union mount-a single view composed of multiple layers. The two most common drivers are:

• AUFS (Advanced Multi-Layered Unification File System): Historically the default on Ubuntu. It stacks layers using multiple read-only branches and a writable branch. AUFS supports whiteout files (".wh.") for deletions.
• OverlayFS: The modern default on most distributions (e.g., Docker Engine 20+). It uses two directories: lowerdir (read-only layers) and upperdir (writable layer), merged via a overlay mount.

Both drivers rely on kernel support; missing support forces Docker to fall back to vfs, which is slower and less secure because it lacks copy-on-write semantics.

# Show the active storage driver
docker info | grep "Storage Driver"
# Example output:
# Storage Driver: overlay2

Security implications:

• OverlayFS exposes the upperdir to the container if the mount is not properly configured, potentially allowing a compromised process to modify lower layers.
• AUFS historically suffered from CVE-2015-1328 (privilege escalation via crafted whiteout files). Ensure the kernel is patched and consider disabling AUFS in favor of OverlayFS.

Image layer hashing and caching

Each layer is stored as a tar archive whose content hash (SHA-256) becomes its identifier. When building, Docker checks if a layer with the same hash already exists locally or in a remote cache; if so, it reuses it, dramatically speeding up builds.

# Inspect a local image's layers
docker history --no-trunc nginx:latest
# Sample output (truncated):
# ID CREATED CREATED BY SIZE COMMENT
# <missing> 2 weeks ago /bin/sh -c #(nop)  CMD ["nginx" -g "daemon ... 0B # <missing> 2 weeks ago /bin/sh -c #(nop)  EXPOSE 80 0B # <missing> 2 weeks ago /bin/sh -c apt-get update && apt-get install … 45MB 

Because the hash incorporates file contents, timestamps, permissions, and the order of files, two Dockerfiles that differ only in a RUN apt-get update line will generate distinct layers. Attackers can exploit caching by injecting malicious layers that are later reused across builds (a supply-chain risk known as “layer-reuse attack”). Mitigation: use --no-cache for critical builds, pin package versions, and avoid ADD . /app with broad contexts.

Dockerfile instruction set and best practices

The Dockerfile DSL defines how layers are created. Key instructions:

• FROM - base image (must be first line).
• RUN - executes a command, creates a new layer.
• COPY/ADD - copies files from build context; ADD also supports URL extraction (avoid unless needed).
• WORKDIR - sets the working directory for subsequent instructions.
• ENV, ARG - set environment variables or build-time arguments.
• EXPOSE, CMD, ENTRYPOINT - runtime configuration.

Security-focused best practices:

1. Use an official minimal base (e.g., FROM alpine:3.18) to reduce attack surface.
2. Pin package versions and verify checksums.

   RUN apk add --no-cache nginx=1.24.0-r0 && echo "nginx checksum OK"
   

3. Group related RUN commands to minimize layers.

   RUN apk update && apk add --no-cache curl git && rm -rf /var/cache/apk/*
   

4. Prefer COPY over ADD unless you need automatic tar extraction.
5. Set a non-root user after installing packages.

   RUN addgroup -S appgroup && adduser -S appuser -G appgroup
   USER appuser
   

6. Use HEALTHCHECK to detect runtime compromise.

   HEALTHCHECK --interval=30s --timeout=5s CMD curl -f http://localhost/ || exit 1
   

Inspecting images and containers (docker inspect, docker history)

Docker provides JSON-based introspection tools:

• docker inspect <obj> - returns low-level configuration (environment, mounts, cgroup settings).
• docker history <image> - shows layer creation commands and sizes.

docker inspect nginx:latest | jq '.[0].Config.Env'
# Output (example):
# ["NGINX_VERSION=1.24.0","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]

Security analysts use inspect to verify that privileged flags (Privileged: true) or insecure capabilities (CapAdd) are not present. The Mounts array reveals host-path bind mounts that could expose sensitive files.

Namespace isolation basics (PID, NET, MNT, IPC, UTS)

Namespaces provide process-level isolation:

• PID - each container gets its own process ID space; ps only shows processes inside.
• NET - separate network stack (interfaces, routing tables). Docker creates a veth pair, one end in the container, the other on the host bridge.
• MNT - distinct mount namespace; containers cannot see host mounts unless bind-mounted.
• IPC - isolated System V IPC and POSIX message queues.
• UTS - separate hostname and domain name.

When a container is launched with --pid=host or --network=host, these namespaces are shared with the host, dramatically increasing the attack surface. Avoid such flags unless absolutely required (e.g., monitoring agents).

# Verify a container's namespace IDs
docker run -d --name ns-demo alpine sleep 3600
ns=$(docker inspect -f '{{.State.Pid}}' ns-demo)
ls -l /proc/$ns/ns
# Expected output shows distinct inode numbers for each namespace type

Control groups (cgroups) resource limits

Cgroups enforce CPU, memory, blkio, and PIDs limits. Docker maps --memory, --cpus, --pids-limit flags to cgroup v2 controllers.

docker run -d --name mem-limit --memory 256m --pids-limit 100 alpine sleep 600
# Inspect cgroup limits
cat /sys/fs/cgroup/memory/docker/$(docker inspect -f '{{.State.Pid}}' mem-limit)/memory.limit_in_bytes

Improperly configured limits can be abused for denial-of-service (e.g., removing memory caps to starve the host) or to escape via OOM-killer manipulation. Always set sensible defaults and consider a --ulimit policy for file descriptors.

Common storage drivers and their security implications

Docker supports multiple drivers; each has trade-offs:

From a security stance, prefer overlay2 with kernel version ≥4.18 and enable userns-remap to map container root to an unprivileged host UID/GID, reducing impact of a container breakout.

Practical Examples

Below are step-by-step labs that illustrate key concepts.

Example 1: Building a reproducible image with layer caching

# Directory structure
mkdir -p demo/app && cd demo
cat > Dockerfile <<'EOF'
FROM alpine:3.18
LABEL maintainer="[email protected]"
# Install dependencies in a single RUN to reduce layers
RUN apk update && apk add --no-cache python3 py3-pip && pip install --no-cache-dir flask==2.2.2
WORKDIR /app
COPY app.py .
EXPOSE 5000
CMD ["python3", "app.py"]
EOF

# Minimal Flask app (app.py)
cat > app/app.py <<'PY'
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello(): return "Hello from secure container!"
PY

# Build with cache enabled (default)
docker build -t demo:latest .
# Re-run build after a trivial change to app.py - only the last layer is rebuilt
sed -i 's/Hello/Hi/' app/app.py
docker build -t demo:latest .
PY

The output shows that only the COPY and CMD layers are rebuilt, confirming caching benefits.

Example 2: Inspecting a container for privileged flags

docker run -d --name privileged-test --privileged alpine sleep 3600
# Inspect for the privileged flag
docker inspect privileged-test | jq '.[0].HostConfig.Privileged'
# Expected output: true
# Mitigation: avoid --privileged; use fine-grained capabilities instead

Example 3: Enforcing memory limits with cgroups

docker run -d --name mem-test --memory 128m alpine sleep 600
# Inside the container, try to allocate >128 MiB
docker exec -it mem-test sh -c "dd if=/dev/zero of=/tmp/bigfile bs=1M count=200"
# Expected: dd fails with "Cannot allocate memory"

Tools & Commands

• docker info - overview of daemon configuration, storage driver, and cgroup version.
• docker system df - disk usage per image, container, and volume.
• docker scan (or trivy, clair) - static analysis for known CVEs in image layers.
• ctr (containerd CLI) - low-level inspection of snapshots and content stores.
• runc spec - view the OCI runtime specification generated for a container.

Defense & Mitigation

Security hardening of Docker deployments should address the entire stack:

1. Daemon hardening: Run dockerd with --userns-remap and TLS-protected remote API.
2. Image provenance: Use signed images (Docker Content Trust) and enforce --pull=always in CI pipelines.
3. Least-privilege runtime: Avoid --privileged, --cap-add=ALL, and host networking unless justified.
4. Namespace & cgroup policies: Apply default resource quotas, limit PID count, and use seccomp profiles to block syscalls like ptrace.
5. Storage driver selection: Prefer overlay2 on a recent kernel; disable AUFS if not needed.
6. Scanning & patching: Integrate vulnerability scanners into CI, rebuild images on base-image updates.

Common Mistakes

• Using ADD . /app with a wide build context - unintentionally copies secrets or build artefacts into the image.
• Relying on latest tag in production - leads to untracked upgrades and supply-chain drift.
• Running containers as root without user namespace remapping - a breakout yields host root.
• Disabling Docker’s default seccomp profile - opens a large syscall surface.
• Mounting the Docker socket inside a container for “convenience” - effectively grants root on the host.

Real-World Impact

Enterprise breaches often start with a compromised container image. In 2023, a supply-chain attack on a popular node base image injected a reverse shell into the RUN npm install step. Because the image was cached and reused across dozens of services, the attacker gained foothold in multiple environments within minutes.

My experience consulting for a fintech firm showed that enforcing userns-remap and disabling --privileged reduced the impact of a later ransomware attempt that tried to mount the host filesystem from a compromised container. The ransomware failed because the container’s UID was mapped to an unprivileged host UID, preventing write access to /.

Trends to watch:

• Adoption of rootless Docker daemons - further isolates the daemon process.
• Shift toward OCI distribution signatures (cosign) for image integrity.
• Increasing use of eBPF-based runtime security (Falco, Tracee) that monitors namespace and cgroup anomalies.

Practice Exercises

1. Build a multi-stage image: Create a builder stage that compiles a Go binary, then copy the binary into a minimal scratch final stage. Verify that the final image contains only the binary and no build tooling.
2. Inspect layer hashes: Pull python:3.11-slim, run docker history, and note the size of each layer. Then modify the Dockerfile to add a new RUN apt-get update step and compare the new layer hash.
3. Namespace escape test: Run a container with --pid=host and attempt to list host processes from inside. Document why this is dangerous and remediate by removing the flag.
4. Cgroup limit abuse: Launch a container with --memory 64m and a memory-stress script that attempts to allocate 200 MiB. Observe OOM behavior and log the kernel messages.
5. Storage driver comparison: On a test VM, install Docker with overlay2, then switch to devicemapper (using --storage-driver=devicemapper) and note performance and any security warnings from docker info.

Further Reading

• Docker Engine Documentation -
• OCI Image Format Specification - github.com/opencontainers/image-spec
• “Docker Security” by Adrian Mouat - O'Reilly (covers hardening, user namespaces, and CI pipelines).
• Container Threat Model - CIS Whitepaper
• Project trivy - vulnerability scanner for container images.

Summary

Docker’s architecture-client-daemon communication, layered immutable images, union file systems, and kernel-level isolation-forms the backbone of modern container security. Mastering the Dockerfile DSL, inspecting images/containers, and correctly configuring namespaces, cgroups, and storage drivers are essential skills for any security professional. Apply the best-practice checklist, continuously scan images, and enforce least-privilege runtime policies to mitigate the most common attack vectors.