Update container images on air-gapped environments

[almet]

On air-gapped environments, we will need to have another way to install new container images, rather than using the Github Container Registry.

For Linux, we should be able to use specific packages (see #1067), but we should still be able to install specific container images alongside their signatures by another mean, especially for Windows and macOS environments.

Having a CLI option for doing this would be useful.

[almet]

In order to handle air-gapped environments, we need a format which provides the following:

1. Layer "blobs" (e.g. where the podman/docker layers are stored)
2. The cosign signatures
3. A way to ensure that the cosign signatures have been applied to this image format, once loaded.

podman save

When saving a container image from podman with podman save, the image digest is actually lost:

$ podman image list ghcr.io/almet/dangerzone/dangerzone --format "{{.Digest}}"
sha256:fa948726aac29a6ac49f01ec8fbbac18522b35b2491fdf716236a0b3502a2ca7

$ podman save ghcr.io/almet/dangerzone/dangerzone -o archive.tar

$ podman rmi ghcr.io/almet/dangerzone/dangerzone:latest
Untagged: ghcr.io/almet/dangerzone/dangerzone:latest
Deleted: 97808d20e505674268604beb67515c6444d0d46257a74d9d80c7450788e477a9

$ podman load -i archive.tar

$ podman image list ghcr.io/almet/dangerzone/dangerzone --format "{{.Digest}}"
sha256:118e38fabe3ccd0ec89b2e21bb7704b62fae92e106166ed02b4ee301151319e

Interestingly, after doing a podman pull from the repository, the repo digest is added, so we have:

[
  {
    "Id": "97808d20e505674268604beb67515c6444d0d46257a74d9d80c7450788e477a9",
    "ParentId": "",
    "RepoTags": null,
    "RepoDigests": [
      "ghcr.io/almet/dangerzone/dangerzone@sha256:118e38fabe3ccd0ec89b2e21bb7704b62fae92e106166ed02b4ee301151319e8",
      "ghcr.io/almet/dangerzone/dangerzone@sha256:fa948726aac29a6ac49f01ec8fbbac18522b35b2491fdf716236a0b3502a2ca7"
    ],
    "Size": 1321336831,
    "SharedSize": 0,
    "VirtualSize": 1321336831,
    "Labels": {
      "io.buildah.version": "1.33.7"
    },
    "Containers": 0,
    "Digest": "sha256:118e38fabe3ccd0ec89b2e21bb7704b62fae92e106166ed02b4ee301151319e8",
    "History": [
      "ghcr.io/almet/dangerzone/dangerzone:latest"
    ],
    "Names": [
      "ghcr.io/almet/dangerzone/dangerzone:latest"
    ],
    "Created": 1737389331,
    "CreatedAt": "2025-01-20T16:08:51Z"
  }
]

cosign save

cosign save provides an OCI-image archive that cannot be loaded as is, but can if tweaked a bit (removing the signatures from the top-level index.json)

# Save the container + signatures in a folder, then tar it.
cosign save ghcr.io/almet/dangerzone/dangerzone@sha256:fa948726aac29a6ac49f01ec8fbbac18522b35b2491fdf716236a0b3502a2ca7 --dir container
cd container && tar -cvf ../container.tar

If loaded directly without tweaks, it results in the following error:

Error: payload does not match any of the supported image formats:
 * oci: open here.tar/index.json: not a directory
 * oci-archive: loading index: more than one image in oci, choose an image
 * docker-archive: loading tar component "manifest.json": file does not exist
 * dir: open here.tar/manifest.json: not a directory

This is because the index.json contains a link to the signatures as well as the image itself, like this:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1566,
      "digest": "sha256:fa948726aac29a6ac49f01ec8fbbac18522b35b2491fdf716236a0b3502a2ca7",
      "annotations": {
        "kind": "dev.cosignproject.cosign/image"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 4369,
      "digest": "sha256:e5cb967032b8ae1c1d28f98d3b509513a287a1a982e58dde518ebd3fe1d8ab2b",
      "annotations": {
        "kind": "dev.cosignproject.cosign/sigs"
      }
    }
  ]
}

Removing the signatures before taring the folder comes up with a loadable container image.

cat container/index.json | jq '.manifests |= [.[0]]' > container/index.json
cd container && tar -cvf ../container.tar
podman load -i container.tar

Unfortunately, it doesn't retain the image name, but the image Digest is the right one!

podman image list --format "{{.Digest}}"
sha256:fa948726aac29a6ac49f01ec8fbbac18522b35b2491fdf716236a0b3502a2ca7

To retag it:

IMAGE_ID=$(podman image list -f digest="sha256:fa948726aac29a6ac49f01ec8fbbac18522b35b2491fdf716236a0b3502a2ca7" --format "{{.Id}}")
podman tag $IMAGE_ID ghcr.io/almet/dangerzone/dangerzone

[almet]

The approach mentioned in the previous posts (cosign save && podman load -i with a modified index.json) works for podman, but the same approach isn't working with docker, unfortunately, as it seems that docker load -i cannot accept OCI image tarballs.

There is another import command that is available for both podman and docker, and can import an OCI image tarball, but it seems to behave a bit differently on each "runtime":

Create the tarball (out of cosign save, see steps above):

./dev_scripts/dangerzone-image prepare-archive ghcr.io/almet/dangerzone/dangerzone@sha256:7b21dbdebffed855621dfcdeaa52230dc6566997f852ef5d62b0338b46796e01
INFO:dangerzone.updater:Downloading image ghcr.io/almet/dangerzone/dangerzone@sha256:7b21dbdebffed855621dfcdeaa52230dc6566997f852ef5d62b0338b46796e0
1.
It might take a while.
✅ Archive dangerzone-airgapped.tar created

Then, for docker:

$ docker import dangerzone-airgapped.tar
sha256:342d0756391936813551780e9c132ebc3811cfb3382686e9ec8dbb829919eb5e
$ docker images --format json | jq
{
  "Containers": "N/A",
  "CreatedAt": "2025-03-03 17:20:30 +0100 CET",
  "CreatedSince": "4 minutes ago",
  "Digest": "<none>",
  "ID": "342d07563919",
  "Repository": "<none>",
  "SharedSize": "N/A",
  "Size": "944MB",
  "Tag": "<none>",
  "UniqueSize": "N/A",
  "VirtualSize": "943.7MB"
}

And for podman:

$ podman import dangerzone-airgapped.tar
Getting image source signatures
Copying blob d489d97339e6 done   |
Copying config aed18f757c done   |
Writing manifest to image destination
sha256:aed18f757c1ea9fff80606e6c29398e425e2190f6daa4f906e13fe9d03bfee3e

$ podman images --format json | jq
[
  {
    "Id": "aed18f757c1ea9fff80606e6c29398e425e2190f6daa4f906e13fe9d03bfee3e",
    "ParentId": "",
    "RepoTags": null,
    "RepoDigests": [],
    "Size": 943739989,
    "SharedSize": 0,
    "VirtualSize": 943739989,
    "Labels": null,
    "Containers": 0,
    "Dangling": true,
    "Digest": "sha256:2e31a2aea19d0690e2a029703630aa28b58d55a02357dc777ddbdf8d165a67bb",
    "Created": 1741018777,
    "CreatedAt": "2025-03-03T16:19:37Z"
  }
]

I do not understand where these digests are coming from, as they are not any of the contained blobs, nor the sha256 digest of the index.json...

[almet]

regctl seem to have a way to create tarballs from specific architectures. That can be used like this:

(here is an OCI directory, as provided by cosign save, and the digest used here is the one of a specific architecture):

regctl image export ocidir://here@sha256:752bb0410f7f30d67355fe97ebc0b8413daf7686eb944549c3e118a87fa2955c > newimage.tar

The generated tarball can be loaded in docker with docker load -i newimage.tar.

It works, but... the output of the docker images --format json command doesn't really help either, as it doesn't contain the digest we're looking for 🤷 :

{
  "Containers": "N/A",
  "CreatedAt": "2025-02-06 17:55:29 +0100 CET",
  "CreatedSince": "3 weeks ago",
  "Digest": "<none>",
  "ID": "2824272007d4",
  "Repository": "localhost/here",
  "SharedSize": "N/A",
  "Size": "1.04GB",
  "Tag": "latest",
  "UniqueSize": "N/A",
  "VirtualSize": "1.042GB"
}

[apyrgio]

Hm, I see. I wonder if we can replace cosign save with something like docker save && cosign download signature && tar cvf dz-image-sig.tar

[almet]

replace cosign save with something like docker save && cosign download signature && tar cvf dz-image-sig.tar

Yeah, it might be practical.

Using cosign save provides us a way to verify the resulting tarball with cosign verify --local-image, which would not be the case with docker save, but if that gets us cross-runtime compatibility (keeping the metadata, please), i believe the bargain would be worth it.

[almet]

As asked by @apyrgio out of band, here is the results of an repro-build analyze (from this PR) on a tarball created by cosign save:

./dev_scripts/repro-build analyze dangerzone-airgapped.tar
The OCI tarball contains an index and 7 manifest(s):
                                                                                                                                                      
Image digest: sha256:7b21dbdebffed855621dfcdeaa52230dc6566997f852ef5d62b0338b46796e01
                                                                                                                                                      
Index (./index.json):
  Digest: sha256:19de16ae6c1fa3d469b0c0ed334571374758d644638ac08f8174b89f491e4596
  Media type: application/vnd.oci.image.index.v1+json
  Platform: -
  Contents: {   "schemaVersion": 2,   "mediaType": "application/vnd.oci.image.index.v1+json",   "manifests": [      {         "mediaType": "application/vnd.oci.image.index.v1+json",         "size": 1609,         "digest": "sha256:7b21dbdebffed855621dfcdeaa52230dc6566997f852ef5d62b0338b46796e01",         "annotations": {            "kind": "dev.cosignproject.cosign/imageIndex"         }      },      {         "mediaType": "application/vnd.oci.image.manifest.v1+json",         "size": 1621,         "digest": "sha256:4a3f44189387dffe8ac8e1bc1cd3b1b5e3c7a324fcd568f5e7c199bba4e07e84",         "annotations": {    [... 355 characters omitted. Pass --show-contents to print them in their entirety]
                                                                                                                                                      
Manifest 1 (./blobs/sha256/7b21dbdebffed855621dfcdeaa52230dc6566997f852ef5d62b0338b46796e01):
  Digest: sha256:7b21dbdebffed855621dfcdeaa52230dc6566997f852ef5d62b0338b46796e01
  Media type: application/vnd.oci.image.index.v1+json
  Platform: -
  Contents: {  "schemaVersion": 2,  "mediaType": "application/vnd.oci.image.index.v1+json",  "manifests": [    {      "mediaType": "application/vnd.oci.image.manifest.v1+json",      "digest": "sha256:099bb54a180c8aecdb4686cc474adeda7e8b97c85e73652c6fbbe334f8dd6330",      "size": 2377,      "platform": {        "architecture": "arm64",        "os": "linux"      }    },    {      "mediaType": "application/vnd.oci.image.manifest.v1+json",      "digest": "sha256:cc9c1c546c07166056a393922df88d0de8d052dad1ec89d288fc382b74088243",      "size": 567,      "annotations": {        "vnd.docker.reference.digest": "sha  [... 960 characters omitted. Pass --show-contents to print them in their entirety]
                                                                                                                                                      
Manifest 2 (./blobs/sha256/099bb54a180c8aecdb4686cc474adeda7e8b97c85e73652c6fbbe334f8dd6330):
  Digest: sha256:099bb54a180c8aecdb4686cc474adeda7e8b97c85e73652c6fbbe334f8dd6330
  Media type: application/vnd.oci.image.manifest.v1+json
  Platform: linux/arm64
  Contents: {  "schemaVersion": 2,  "mediaType": "application/vnd.oci.image.manifest.v1+json",  "config": {    "mediaType": "application/vnd.oci.image.config.v1+json",    "digest": "sha256:49914d7f3c8ab91408d3a0c8af2a6880532c0055d65374ab7c4801f2bfcafc35",    "size": 3708  },  "layers": [    {      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",      "digest": "sha256:9c6f603389ae3b65d779c458d2da68676670d0bc3375c8a6701c5156378a9714",      "size": 236    },    {      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",      "digest": "sha256:52a46e94d0916512d53af06c02ab9b595d76a5a930c0046  [... 1712 characters omitted. Pass --show-contents to print them in their entirety]
                                                                                                                                                      
Manifest 3 (./blobs/sha256/cc9c1c546c07166056a393922df88d0de8d052dad1ec89d288fc382b74088243):
  Digest: sha256:cc9c1c546c07166056a393922df88d0de8d052dad1ec89d288fc382b74088243
  Media type: application/vnd.oci.image.manifest.v1+json
  Platform: unknown/unknown
  Contents: {  "schemaVersion": 2,  "mediaType": "application/vnd.oci.image.manifest.v1+json",  "config": {    "mediaType": "application/vnd.oci.image.config.v1+json",    "digest": "sha256:99f8b396bb5d24f02d6cc76c21d370c7920616555f7c673ea29dc6f702de3e40",    "size": 167  },  "layers": [    {      "mediaType": "application/vnd.in-toto+json",      "digest": "sha256:ee2e22235cc86a6b949876ffa258d847c081f4d9c3ffb7b4382985fde5141183",      "size": 47774,      "annotations": {        "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"      }    }  ]}

Manifest 4 (./blobs/sha256/752bb0410f7f30d67355fe97ebc0b8413daf7686eb944549c3e118a87fa2955c):
  Digest: sha256:752bb0410f7f30d67355fe97ebc0b8413daf7686eb944549c3e118a87fa2955c
  Media type: application/vnd.oci.image.manifest.v1+json
  Platform: linux/amd64
  Contents: {  "schemaVersion": 2,  "mediaType": "application/vnd.oci.image.manifest.v1+json",  "config": {    "mediaType": "application/vnd.oci.image.config.v1+json",    "digest": "sha256:2824272007d49a260a5612fd6e11391a10f80d3a40578a203fea2bdf384e4ae8",    "size": 3712  },  "layers": [    {      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",      "digest": "sha256:9aa67d6d73707ab71725c1c93325c8b385461b6bc0e6e8d0badec9f5de087042",    "size": 238    },    {      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",      "digest": "sha256:846b33c634e989a0a941c7cf25da92124f5fa9ca181b110  [... 1712 characters omitted. Pass --show-contents to print them in their entirety]

Manifest 5 (./blobs/sha256/8888acdfd202d8df14241d1a230fd5c1ed70356d1eaccf0486348a1c1bb7a629):
  Digest: sha256:8888acdfd202d8df14241d1a230fd5c1ed70356d1eaccf0486348a1c1bb7a629
  Media type: application/vnd.oci.image.manifest.v1+json
  Platform: unknown/unknown
  Contents: {  "schemaVersion": 2,  "mediaType": "application/vnd.oci.image.manifest.v1+json",  "config": {    "mediaType": "application/vnd.oci.image.config.v1+json",    "digest": "sha256:cbe0e81a449386e8b539dc994128005e3d1d86a6a61710f68920cf3dbc9d50f1",    "size": 167  },  "layers": [    {      "mediaType": "application/vnd.in-toto+json",      "digest": "sha256:bc1d0dfe8b8e94923c3b0e7a1385e10ffec3f255f14d1a5a5c3fb16cfd0709a3",      "size": 47773,      "annotations": {        "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"      }    }  ]}

Manifest 6 (./blobs/sha256/4a3f44189387dffe8ac8e1bc1cd3b1b5e3c7a324fcd568f5e7c199bba4e07e84):
  Digest: sha256:4a3f44189387dffe8ac8e1bc1cd3b1b5e3c7a324fcd568f5e7c199bba4e07e84
  Media type: application/vnd.oci.image.manifest.v1+json
  Platform: -
  Contents: {"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":233,"digest":"sha256:cbabac12486b113f06b66320b573c37d28e735b4a893bac8e2fa24a88ee05fbc"},"layers":[{"mediaType":"application/vnd.dev.cosign.simplesigning.v1+json","size":251,"digest":"sha256:3ed05bea76eaf30fac570e78e80efd143bed171c3ec0691b6057b7a0034a238e","annotations":{"dev.cosignproject.cosign/signature":"MEUCIQC2WlJH+B8VuX1c6i4sDwEGEZc53hXUD6/ds9TMJ3HrfwIgCxSnrNYRD2c8XENqfqc+Ik1gx0DK9kPNsn/Lt8V/dCo=","dev.sigstore.cosign/bundle":"{\"SignedEntryT  [... 1021 characters omitted. Pass --show-contents to print them in their entirety]

Manifest 7 (./blobs/sha256/56bbf5d45dade2a24e973658a8df66ca754cc26bf11fae77966dae71e642e5ab):
  Digest: sha256:56bbf5d45dade2a24e973658a8df66ca754cc26bf11fae77966dae71e642e5ab
  Media type: application/vnd.oci.image.manifest.v1+json
  Platform: -
  Contents: {"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":233,"digest":"sha256:e834c38b8462da230536f36a7fc334cfcf20349f6cb23ea7d06e19e7d8793e37"},"layers":[{"mediaType":"application/vnd.dsse.envelope.v1+json","size":11936,"digest":"sha256:6b9e861e18d74107a413828814825e41e8d7e869979a538fc981b1e8c634b691","annotations":{"dev.cosignproject.cosign/signature":"","dev.sigstore.cosign/bundle":"{\"SignedEntryTimestamp\":\"MEUCIQDMubkZRqw0l5e81Z7Fk6ckM3S9HXsQYn9unprapKpKgQIgLDJ0Otu/ehzPoxxoRub43zy9wmd8o5npiU4AnrC+  [... 9715 characters omitted. Pass --show-contents to print them in their entirety]

[almet]

replace cosign save with something like docker save && cosign download signature && tar cvf dz-image-sig.tar

I tried to do this today without luck. I won't report the exact details here as I don't think they would be very valuable, but when doing docker save and then importing the resulting tarball, the digest of the manifest we sign is not present in the imported image (queried by {podman,docker} inspect.

I've been trying to use alternative tools for this (I'm thinking crane, regctl and the like). Here is what I found:

• You can build a local image with crane pull, but when loading the resulting image it doesn't contain the digest we'd like to see.
• rgctl provides something akin with the regctl image export command. I've been able to import the produced on podman and it keeps the digest, but the tarball doesn't seem to please Docker, failing with the following error:

/var/lib/docker/tmp/docker-import-3376484789/blobs/json: no such file or directory