Container Registries: Content-Addressable Storage, Push/Pull Protocols, Tags vs Digests

During an emergency production hotfix, a DevOps engineer rebuilds an image, tags it with an existing version name—payment-service:v2.1.0—and pushes it to the enterprise container registry.

The Kubernetes cluster triggers a rolling update across 20 pod replicas.

Unexpectedly, half the application pods execute the new hotfix code, while the remaining pods continue executing the old code.

Debugging reveals the cause: 10 worker nodes had imagePullPolicy: IfNotPresent and possessed a local cached copy of payment-service:v2.1.0, so they skipped pulling the updated layer blobs from the registry.

The team fell victim to Image Tag Mutability Drift.

How do container registries store, deduplicate, and serve container images over HTTP networks?

To prevent deployment inconsistencies, we must explore the OCI Distribution Specification and the difference between Mutable Tags and Immutable Digests.

1. Container Registry Architecture

A Container Registry (such as Docker Hub, Amazon ECR, or GitHub Container Registry) is a web service implementing the OCI Distribution Specification (HTTP API V2).

A registry does not store images as monolithic single files. Instead, it stores two distinct types of data in Content-Addressable Blob Storage:

+-------------------------------------------------------------------+
|                        CONTAINER REGISTRY                         |
+-------------------------------------------------------------------+
| 1. Tag Pointers (Key-Value Store)                                 |
|    "latest"    -> sha256:7a8b9c1d2e3f...                          |
|    "v2.1.0"    -> sha256:7a8b9c1d2e3f...                          |
+-------------------------------------------------------------------+
| 2. Content-Addressable Blob Store                                 |
|    /blobs/sha256/7a8b9c... (Image Manifest JSON)                 |
|    /blobs/sha256/e3b0c4... (Image Config JSON)                   |
|    /blobs/sha256/1a2b3c... (Layer 1 Tarball)                     |
|    /blobs/sha256/9f8e7d... (Layer 2 Tarball)                     |
+-------------------------------------------------------------------+

2. Anatomy of a `docker pull` Operation

When you execute docker pull registry.example.com/web-app:v1.0, the local container engine executes a 4-step HTTP exchange with the registry:

Docker Engine (Client)                               Registry HTTP API v2
  |                                                          |
  |--- 1. GET /v2/web-app/manifests/v1.0 ------------------>|
  |<-- 2. Returns Manifest JSON (Lists Layer Digests) -------|
  |                                                          |
  | [ Inspects Local Content Store ]                         |
  | (Layer 1 sha256:1a2b... EXISTS locally -> SKIP!)        |
  | (Layer 2 sha256:9f8e... MISSING locally!)                |
  |                                                          |
  |--- 3. GET /v2/web-app/blobs/sha256:9f8e... ------------->|
  |<-- 4. Streams compressed layer tarball payload ----------|

Protocol Steps:

Fetch Manifest: The client requests the image manifest associated with tag v1.0.
Inspect Local Store: The client reads the list of SHA-256 layer digests inside the returned manifest and compares them against its local containerd content store.
Download Missing Blobs: If a layer digest already exists on the local host (from another image build), the client skips downloading it. The client issues HTTP GET requests only for missing layer digests, downloading them concurrently in parallel.
Unpack Layers: The client verifies that the SHA-256 digest of the downloaded tarball matches the manifest digest exactly, unpacking it into the local storage driver.

3. Mutable Tags vs Immutable Digests

Understanding the distinction between an Image Tag and an Image Digest is critical for production security and reliability.

       MUTABLE TAG POINTER                         IMMUTABLE DIGEST
   payment-service:v2.1.0               payment-service@sha256:7a8b9c1d...
             |                                             |
             v                                             v
Points to Manifest A (Today)                  Points PERMANENTLY to Manifest A
Points to Manifest B (Tomorrow!)              (Guaranteed byte-for-byte identical!)

A. Image Tags (Pointers)

An Image Tag (nginx:latest, myapp:v1.0) is a mutable string pointer.

The registry owner can re-assign v1.0 to point to a completely different manifest digest at any time.
Relying on tags in deployment manifests (deploy.yaml) introduces non-deterministic builds and security vulnerability risks.

B. Image Digests (Cryptographic Hashes)

An Image Digest (myapp@sha256:7a8b9c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b) is a cryptographic SHA-256 hash computed over the image manifest.

Digests are strictly immutable. A digest refers to one, and only one, exact byte payload forever.
If an attacker alters a single byte in the image manifest, the digest changes, breaking the reference.

Production Deployment Best Practice

In Kubernetes production manifests, reference images using Immutable SHA-256 Digests:

# AVOID (Subject to tag mutability drift):
image: registry.example.com/payment-service:v2.1.0

# RECOMMENDED (Guarantees byte-for-byte immutability across all nodes):
image: registry.example.com/payment-service@sha256:7a8b9c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b

4. Layer Deduplication Across Registries

Because layer blobs are content-addressed by SHA-256 hashes, registries perform Global Storage Deduplication.

If 100 teams push different application images built on ubuntu:22.04 to an enterprise registry:

The registry stores the 75MB Ubuntu base layer blob exactly once on physical storage.
Pushing a new image version that shares base layers takes less than 1 second because the registry responds to the initial layer upload check with 200 OK (Blob Exists).

Tag vs Digest Comparison Matrix

Property	Image Tag (`:v1.0`)	Image Digest (`@sha256:...`)
Mutability	Mutable (Can be reassigned to new builds)	Immutable (Calculated via SHA-256 hash)
Human Readability	High (`v1.0.4`, `latest`)	Low (`sha256:7a8b9c1d...`)
Deployment Consistency	Vulnerable to caching drift across host nodes	Guaranteed 100% byte-for-byte identical
Security Risk	High (Tag squatting / overwrite attacks)	Low (Cryptographically tamper-proof)
Recommended Environment	Local Development / Feature Branches	Production Kubernetes Clusters & CI/CD

Summary & Next Steps

Container registries use content-addressable storage to optimize image delivery and storage:

The OCI Distribution API v2 streams manifest JSONs and layer tarball blobs over HTTP.
Content-Addressable Storage (CAS) deduplicates identical layers across images using SHA-256 digests.
Image Tags (:v1.0) are mutable pointers susceptible to deployment drift.
Image Digests (@sha256:...) provide immutable, tamper-proof references essential for production deployments.

In the next article, we open Module 3 with From Image to Container: Read-Only Layers, Writable Layers, and Execution State.

References & Further Reading

Linux Man-Pages Project. capabilities(7) — Overview of Linux Capabilities. Linux Man Pages.
Linux Kernel Organization. Seccomp BPF (SECure COMPuting with filters). Linux Kernel Docs.
Canonical. AppArmor Profile Syntax and Security Architecture. AppArmor Docs.