From f6363cee2573d58acd7be1bbd66a1c546825b0d1 Mon Sep 17 00:00:00 2001 From: HarshwardhanPatil07 Date: Sat, 11 Jul 2026 15:30:51 +0530 Subject: [PATCH 1/2] docs: add guide for building custom node images Document how to build a disk image with a custom or patched package and test it with bink. Covers two approaches: - Runtime update via bootc switch (no disk rebuild) - Full disk image rebuild for boot-critical changes Includes concrete examples for COPR repos, local RPMs, binary replacement, and building bootc from source using the bootc repo's Dockerfile with the bink node image as base. Closes: #91 Signed-off-by: HarshwardhanPatil07 --- README.md | 4 + docs/custom-node-images.md | 337 +++++++++++++++++++++++++++++++++++++ 2 files changed, 341 insertions(+) create mode 100644 docs/custom-node-images.md diff --git a/README.md b/README.md index 6186ba8..7a82976 100644 --- a/README.md +++ b/README.md @@ -122,3 +122,7 @@ bink cluster stop --remove-data ```bash source <(bink completion bash) ``` + +## Documentation + +- [Building Custom Node Images](docs/custom-node-images.md) — How to build and test with custom or patched packages diff --git a/docs/custom-node-images.md b/docs/custom-node-images.md new file mode 100644 index 0000000..f0d47c8 --- /dev/null +++ b/docs/custom-node-images.md @@ -0,0 +1,337 @@ +# Building Custom Node Images + +## Overview + +When debugging a bug in a node component (bootc, CRI-O, kubelet, kernel, etc.), you need to replace the package or binary with a custom build and verify the fix. There are two approaches: + +1. **Runtime update** — Build a custom bootc OCI image and use `bootc switch` to deploy it on a running node. Faster, no disk rebuild needed. +2. **Disk image rebuild** — Rebuild the entire qcow2 disk image. Required when the fix must be present from first boot (kernel, bootc itself, initrd). + +**When to use which:** + +| Situation | Approach | +|-----------|----------| +| Patching a userspace component (CRI-O, kubelet, a CLI tool) | Runtime update | +| Patching bootc itself, the kernel, or initrd | Disk image rebuild | +| Need the fix before the first boot completes | Disk image rebuild | + +## Prerequisites + +- Podman with socket running (`systemctl --user start podman.socket`) +- `/dev/kvm` and `/dev/fuse` accessible +- bink binary built (see [README](../README.md)) +- A running bink cluster, or you will create one + +## Inspecting the Current Node State + +Before making changes, inspect what is currently deployed. SSH into a node: + +```bash +bink node ssh node1 --cluster-name +``` + +Then run: + +```bash +# Check the ostree deployment +ostree admin status + +# Show ostree commit details +ostree show --repo=/sysroot/ostree/repo + +# Check the origin file (which container image the node tracks) +sudo cat /sysroot/ostree/deploy/default/deploy/*.origin + +# Full bootc status +sudo bootc status --json | jq + +# Check specific package versions +rpm -qa | grep -E 'bootc|cri-o|kubeadm' +``` + +## Approach 1: Runtime Update via `bootc switch` + +The simpler path when the patch does not need to be in the boot image. Build a custom bootc OCI image, push it to bink's local registry, and switch the node to it. + +### Example: Building bootc from Source + +The [bootc repository](https://github.com/bootc-dev/bootc) has a Dockerfile with `ARG base=quay.io/centos-bootc/centos-bootc:stream10`. Override this to use the bink node image as the base, producing a new bootable container image with bootc compiled from any git commit: + +```bash +git clone https://github.com/bootc-dev/bootc.git +cd bootc +git checkout + +podman build \ + --build-arg base=ghcr.io/bootc-dev/bink/node:v1.35-fedora-44 \ + -t localhost/custom-bootc-node:latest \ + . +``` + +This compiles bootc from source and installs it into an image derived from the bink node image, preserving all the existing packages (kubernetes, CRI-O, etc.). + +### Example: Layering a Custom Package + +For simpler cases (installing a different package version), write a minimal Containerfile: + +```dockerfile +FROM ghcr.io/bootc-dev/bink/node:v1.35-fedora-44 +RUN dnf -y install +``` + +Build it: + +```bash +podman build -t localhost/custom-node:latest -f Containerfile.custom . +``` + +### Push to the Local Registry and Switch + +Bink runs a local OCI registry on port 5000. From the host, push to `localhost:5000`. Inside the VMs, the registry is reachable at `registry.cluster.local:5000`. + +```bash +podman push --tls-verify=false \ + localhost/custom-bootc-node:latest \ + localhost:5000/custom-bootc-node:latest +``` + +SSH into the node and switch: + +```bash +bink node ssh node1 --cluster-name + +sudo bootc switch \ + registry.cluster.local:5000/custom-bootc-node:latest +``` + +To switch by digest: + +```bash +sudo bootc switch \ + registry.cluster.local:5000/custom-bootc-node@sha256: +``` + +Reboot to apply: + +```bash +sudo reboot +``` + +### Verify After Reboot + +```bash +bink node ssh node1 --cluster-name + +ostree admin status +sudo bootc status --json | jq '.status.booted.image' +rpm -q bootc +``` + +### Using `--target-imgref` for New Clusters + +Instead of switching after boot, set the tracked image reference at cluster creation: + +```bash +bink cluster start \ + --cluster-name test \ + --target-imgref registry.cluster.local:5000/custom-bootc-node:latest +``` + +This rewrites the ostree origin file during cloud-init so the node tracks your custom image from the start. Running `bootc upgrade` will pull updates from your custom image reference. + +## Approach 2: Rebuild the Disk Image + +Required when the fix must be present from first boot (kernel, bootc, initrd changes). + +### Build Pipeline Overview + +The node disk image is built in two stages from `node-images/fedora/`: + +``` +Containerfile -> bootc OCI image (bootc-base-imagectl build-rootfs) +Containerfile.disk -> qcow2 disk image (bcvk to-disk) +``` + +- **Stage 1** (`Containerfile`): Builds a bootc OCI image from `quay.io/fedora/fedora-bootc:44`. Uses `bootc-base-imagectl build-rootfs` with `--install` flags for packages (kubernetes, CRI-O, etc.). Validated with `bootc container lint`. +- **Stage 2** (`Containerfile.disk`): Converts the bootc OCI image to a qcow2 disk using `bcvk to-disk`. The final container image contains only `/disk.qcow2` and `/images.txt`. + +> **Important:** Do not run `podman build` directly on `node-images/fedora/Containerfile`. The `build-rootfs` command requires elevated privileges. Always use the Makefile targets (`make build-bootc-image`, `make build-disk-image`) which pass the correct flags (`--cap-add=all --security-opt=label=disable --device /dev/fuse`). + +### Scenario A: Custom Package from a COPR Repo + +Modify `node-images/fedora/Containerfile` to add a COPR repo before the `build-rootfs` command: + +```dockerfile +RUN dnf -y install 'dnf5-command(copr)' && \ + dnf -y copr enable / fedora-44-x86_64 +``` + +The COPR architecture string must match the base image (e.g., `fedora-44-x86_64` for Fedora 44). + +Build and test: + +```bash +cd node-images/fedora + +make build-bootc-image BOOTC_IMAGE=localhost/custom-node:latest +make build-disk-image \ + BOOTC_IMAGE=localhost/custom-node:latest \ + NODE_IMAGE=localhost/custom-node:disk + +cd ../.. + +bink cluster start \ + --node-image localhost/custom-node:disk \ + --cluster-name custom-test +``` + +> **Note:** COPR repos should be used as an opt-in customization for debugging and testing, not as a permanent change to the default image. Upstream COPR updates can introduce unexpected changes that break CI. + +### Scenario B: Local RPM File + +Copy the RPM into the build context and install it into the target rootfs after `build-rootfs` completes: + +```dockerfile +COPY my-package.rpm /tmp/my-package.rpm + +# After the build-rootfs RUN instruction: +RUN dnf --installroot=/target-rootfs install -y /tmp/my-package.rpm +``` + +The `--installroot=/target-rootfs` flag is required because `build-rootfs` assembles the filesystem at `/target-rootfs`, not in the builder's own root. + +Build and test with the same `make` commands as Scenario A. + +### Scenario C: Custom Binary Replacement + +Use a multi-stage build to compile the binary and copy it into the node image: + +```dockerfile +FROM fedora:44 AS custom-build +RUN dnf -y install git golang make +COPY my-source/ /src +WORKDIR /src +RUN make build + +FROM quay.io/fedora/fedora-bootc:44 AS builder +ARG KUBE_MINOR=1.35 +ARG KERNEL_VERSION=6.19.14-300.fc44 +RUN /usr/libexec/bootc-base-imagectl build-rootfs \ + --manifest=minimal \ + --no-docs \ + --lock kernel-${KERNEL_VERSION} \ + --lock kernel-core-${KERNEL_VERSION} \ + --lock kernel-modules-${KERNEL_VERSION} \ + --lock kernel-modules-core-${KERNEL_VERSION} \ + --install NetworkManager \ + --install openssh-server \ + --install openssh-clients \ + --install kubernetes \ + --install kubernetes${KUBE_MINOR}-kubeadm \ + --install kubernetes${KUBE_MINOR}-client \ + --install cri-o${KUBE_MINOR} \ + --install qemu-guest-agent \ + --install bind-utils \ + --install iputils \ + --install cloud-init \ + --install dnsmasq \ + --install bubblewrap \ + --install sudo \ + --install vim-minimal \ + --install jq \ + --install less \ + /target-rootfs + +FROM scratch AS root +COPY --from=builder /target-rootfs/ / +COPY --from=custom-build /src/my-binary /usr/bin/my-binary + +RUN passwd -d root +RUN sed -i 's|"/opt/cni/bin"|"/var/lib/cni/bin"|g' /etc/crio/crio.conf && \ + sed -i 's|"/opt/cni/net.d"|"/etc/cni/net.d"|g' /etc/crio/crio.conf +RUN bootc container lint + +LABEL containers.bootc 1 +LABEL ostree.bootable 1 +STOPSIGNAL SIGRTMIN+3 +CMD ["/sbin/init"] +``` + +Build and test with the same `make` commands as Scenario A. + +### Composefs Variant + +To build the disk image with the composefs backend: + +```bash +make build-disk-image-composefs \ + BOOTC_IMAGE=localhost/custom-node:latest \ + NODE_IMAGE=localhost/custom-node:disk-composefs +``` + +## Testing with Multiple Nodes + +Use two nodes to compare stock vs. custom behavior: + +```bash +# Start a cluster with the stock image +bink cluster start --cluster-name test + +# Add a worker with the custom disk image +bink node add node2 --cluster-name test \ + --node-image localhost/custom-node:disk + +# SSH into each to compare +bink node ssh node1 --cluster-name test +bink node ssh node2 --cluster-name test +``` + +Alternatively, switch a single worker at runtime: + +```bash +bink node add node2 --cluster-name test + +# SSH into node2 and switch to the custom image +bink node ssh node2 --cluster-name test +sudo bootc switch \ + registry.cluster.local:5000/custom-node:latest +sudo reboot +``` + +## Makefile Variable Reference + +Variables in `node-images/fedora/Makefile`: + +| Variable | Default | Description | +|----------|---------|-------------| +| `KUBE_MINOR` | `1.35` | Kubernetes minor version | +| `FEDORA_VERSION` | `44` | Fedora base version | +| `DISK_SIZE` | `10G` | VM disk size | +| `BUILD_MEMORY` | `4G` | Memory for bcvk build | +| `BOOTC_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44` | Bootc OCI image name | +| `NODE_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk` | Disk image name | +| `BCVK_EXTRA_ARGS` | (none) | Extra flags for `bcvk to-disk` | + +## Troubleshooting + +### KVM Permission Errors + +``` +qemu-system-x86_64: Could not access KVM kernel module: Permission denied +``` + +Add your user to the `kvm` group or set permissions: `sudo chmod 666 /dev/kvm`. + +### `bootc container lint` Failures + +The Containerfile runs `bootc container lint` at the end. Custom modifications must preserve: +- `/sbin/init` as the entrypoint (`CMD ["/sbin/init"]`) +- Labels: `containers.bootc 1` and `ostree.bootable 1` + +### Disk Space + +The qcow2 build via bcvk needs approximately 15 GB of temporary space. Free disk space or reduce the disk size with `DISK_SIZE=8G`. + +### COPR Repo Not Found + +Verify the Fedora version and architecture string matches the base image. For `quay.io/fedora/fedora-bootc:44`, use `fedora-44-x86_64`. From 143e4e5b64bd43cf33d87e0a9b2e1caefb69366b Mon Sep 17 00:00:00 2001 From: HarshwardhanPatil07 Date: Mon, 27 Jul 2026 17:48:22 +0530 Subject: [PATCH 2/2] docs: address PR review feedback - Add composefs origin path variant alongside ostree path - Add composefs verification commands after reboot - Add bootc status store field check to identify deployment backend - Expand composefs variant section with build, verify, and integration test commands using bootc status JSON fields (.store, .composefs) - Add composefs row to decision table and NODE_IMAGE_COMPOSEFS to Makefile reference - Rework privilege warning to N.B. with full podman build example - Remove COPR opt-in warning note (contradicts current direction) - Drop "Testing with Multiple Nodes" section (superfluous) - Drop entire Troubleshooting section (lint is self-explanatory, rest unnecessary) - Simplify Scenario C to use bootc node image as base instead of duplicating build-rootfs Signed-off-by: HarshwardhanPatil07 --- docs/custom-node-images.md | 145 ++++++++++++++----------------------- 1 file changed, 56 insertions(+), 89 deletions(-) diff --git a/docs/custom-node-images.md b/docs/custom-node-images.md index f0d47c8..2a3b311 100644 --- a/docs/custom-node-images.md +++ b/docs/custom-node-images.md @@ -14,6 +14,7 @@ When debugging a bug in a node component (bootc, CRI-O, kubelet, kernel, etc.), | Patching a userspace component (CRI-O, kubelet, a CLI tool) | Runtime update | | Patching bootc itself, the kernel, or initrd | Disk image rebuild | | Need the fix before the first boot completes | Disk image rebuild | +| Testing composefs-specific behavior or fs-verity | Disk image rebuild (composefs variant) | ## Prerequisites @@ -39,12 +40,18 @@ ostree admin status # Show ostree commit details ostree show --repo=/sysroot/ostree/repo -# Check the origin file (which container image the node tracks) +# Check the origin file (ostree backend) sudo cat /sysroot/ostree/deploy/default/deploy/*.origin +# Check the origin file (composefs backend) +sudo cat /sysroot/state/deploy/*.origin + # Full bootc status sudo bootc status --json | jq +# Determine the deployment backend (ostree or composefs) +sudo bootc status --json | jq '.status.booted.store' + # Check specific package versions rpm -qa | grep -E 'bootc|cri-o|kubeadm' ``` @@ -125,6 +132,15 @@ bink node ssh node1 --cluster-name ostree admin status sudo bootc status --json | jq '.status.booted.image' rpm -q bootc + +# Determine the deployment backend +sudo bootc status --json | jq '.status.booted.store' + +# Check the origin file (ostree backend) +sudo cat /sysroot/ostree/deploy/default/deploy/*.origin + +# Check the origin file (composefs backend) +sudo cat /sysroot/state/deploy/*.origin ``` ### Using `--target-imgref` for New Clusters @@ -155,7 +171,18 @@ Containerfile.disk -> qcow2 disk image (bcvk to-disk) - **Stage 1** (`Containerfile`): Builds a bootc OCI image from `quay.io/fedora/fedora-bootc:44`. Uses `bootc-base-imagectl build-rootfs` with `--install` flags for packages (kubernetes, CRI-O, etc.). Validated with `bootc container lint`. - **Stage 2** (`Containerfile.disk`): Converts the bootc OCI image to a qcow2 disk using `bcvk to-disk`. The final container image contains only `/disk.qcow2` and `/images.txt`. -> **Important:** Do not run `podman build` directly on `node-images/fedora/Containerfile`. The `build-rootfs` command requires elevated privileges. Always use the Makefile targets (`make build-bootc-image`, `make build-disk-image`) which pass the correct flags (`--cap-add=all --security-opt=label=disable --device /dev/fuse`). +> **N.B.** If you need to run `podman build` directly instead of using the Makefile targets, you must pass the required privilege flags: +> +> ```bash +> podman build \ +> --cap-add=all \ +> --security-opt=label=disable \ +> --device /dev/fuse \ +> --build-arg KUBE_MINOR=1.35 \ +> -t localhost/custom-node:latest \ +> -f Containerfile \ +> . +> ``` ### Scenario A: Custom Package from a COPR Repo @@ -185,8 +212,6 @@ bink cluster start \ --cluster-name custom-test ``` -> **Note:** COPR repos should be used as an opt-in customization for debugging and testing, not as a permanent change to the default image. Upstream COPR updates can introduce unexpected changes that break CI. - ### Scenario B: Local RPM File Copy the RPM into the build context and install it into the target rootfs after `build-rootfs` completes: @@ -213,89 +238,54 @@ COPY my-source/ /src WORKDIR /src RUN make build -FROM quay.io/fedora/fedora-bootc:44 AS builder -ARG KUBE_MINOR=1.35 -ARG KERNEL_VERSION=6.19.14-300.fc44 -RUN /usr/libexec/bootc-base-imagectl build-rootfs \ - --manifest=minimal \ - --no-docs \ - --lock kernel-${KERNEL_VERSION} \ - --lock kernel-core-${KERNEL_VERSION} \ - --lock kernel-modules-${KERNEL_VERSION} \ - --lock kernel-modules-core-${KERNEL_VERSION} \ - --install NetworkManager \ - --install openssh-server \ - --install openssh-clients \ - --install kubernetes \ - --install kubernetes${KUBE_MINOR}-kubeadm \ - --install kubernetes${KUBE_MINOR}-client \ - --install cri-o${KUBE_MINOR} \ - --install qemu-guest-agent \ - --install bind-utils \ - --install iputils \ - --install cloud-init \ - --install dnsmasq \ - --install bubblewrap \ - --install sudo \ - --install vim-minimal \ - --install jq \ - --install less \ - /target-rootfs - -FROM scratch AS root -COPY --from=builder /target-rootfs/ / +FROM ghcr.io/bootc-dev/bink/node:v1.35-fedora-44 COPY --from=custom-build /src/my-binary /usr/bin/my-binary - -RUN passwd -d root -RUN sed -i 's|"/opt/cni/bin"|"/var/lib/cni/bin"|g' /etc/crio/crio.conf && \ - sed -i 's|"/opt/cni/net.d"|"/etc/cni/net.d"|g' /etc/crio/crio.conf -RUN bootc container lint - -LABEL containers.bootc 1 -LABEL ostree.bootable 1 -STOPSIGNAL SIGRTMIN+3 -CMD ["/sbin/init"] ``` Build and test with the same `make` commands as Scenario A. ### Composefs Variant -To build the disk image with the composefs backend: +[Composefs](https://github.com/composefs/composefs) is an alternative to the default ostree deployment backend. It uses EROFS for filesystem metadata and overlayfs for content-addressed file mounts, with optional fs-verity integrity checking. + +Build the disk image with the composefs backend: ```bash +cd node-images/fedora + +make build-bootc-image BOOTC_IMAGE=localhost/custom-node:latest make build-disk-image-composefs \ BOOTC_IMAGE=localhost/custom-node:latest \ - NODE_IMAGE=localhost/custom-node:disk-composefs + NODE_IMAGE_COMPOSEFS=localhost/custom-node:disk-composefs + +cd ../.. + +bink cluster start \ + --node-image localhost/custom-node:disk-composefs \ + --cluster-name composefs-test ``` -## Testing with Multiple Nodes +`make build-disk-image-composefs` wraps `make build-disk-image` with `BCVK_EXTRA_ARGS="--composefs-backend"`. Override the image name via `NODE_IMAGE_COMPOSEFS` (not `NODE_IMAGE`). -Use two nodes to compare stock vs. custom behavior: +Verify the node is using the composefs backend: ```bash -# Start a cluster with the stock image -bink cluster start --cluster-name test +bink node ssh node1 --cluster-name composefs-test + +# Should return "composefs" +sudo bootc status --json | jq '.status.booted.store' -# Add a worker with the custom disk image -bink node add node2 --cluster-name test \ - --node-image localhost/custom-node:disk +# Composefs-specific metadata (verity status, boot digest) +sudo bootc status --json | jq '.status.booted.composefs' -# SSH into each to compare -bink node ssh node1 --cluster-name test -bink node ssh node2 --cluster-name test +# Origin file (composefs path) +sudo cat /sysroot/state/deploy/*.origin ``` -Alternatively, switch a single worker at runtime: +Run the composefs-specific integration tests: ```bash -bink node add node2 --cluster-name test - -# SSH into node2 and switch to the custom image -bink node ssh node2 --cluster-name test -sudo bootc switch \ - registry.cluster.local:5000/custom-node:latest -sudo reboot +make test-integration-composefs ``` ## Makefile Variable Reference @@ -310,28 +300,5 @@ Variables in `node-images/fedora/Makefile`: | `BUILD_MEMORY` | `4G` | Memory for bcvk build | | `BOOTC_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44` | Bootc OCI image name | | `NODE_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk` | Disk image name | +| `NODE_IMAGE_COMPOSEFS` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk-composefs` | Composefs disk image name | | `BCVK_EXTRA_ARGS` | (none) | Extra flags for `bcvk to-disk` | - -## Troubleshooting - -### KVM Permission Errors - -``` -qemu-system-x86_64: Could not access KVM kernel module: Permission denied -``` - -Add your user to the `kvm` group or set permissions: `sudo chmod 666 /dev/kvm`. - -### `bootc container lint` Failures - -The Containerfile runs `bootc container lint` at the end. Custom modifications must preserve: -- `/sbin/init` as the entrypoint (`CMD ["/sbin/init"]`) -- Labels: `containers.bootc 1` and `ostree.bootable 1` - -### Disk Space - -The qcow2 build via bcvk needs approximately 15 GB of temporary space. Free disk space or reduce the disk size with `DISK_SIZE=8G`. - -### COPR Repo Not Found - -Verify the Fedora version and architecture string matches the base image. For `quay.io/fedora/fedora-bootc:44`, use `fedora-44-x86_64`.