Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.18'
go-version: "1.25"
- name: Login ghcr
uses: docker/login-action@v3
with:
Expand All @@ -26,7 +26,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build
run: |
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.48.0
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/HEAD/install.sh | sh -s -- -b $(go env GOPATH)/bin v2.5.0
export PATH=$PATH:$(go env GOPATH)/bin
make check
make build
Expand Down
41 changes: 24 additions & 17 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -1,22 +1,29 @@
# https://golangci-lint.run/usage/configuration#config-file

version: "2"
linters:
enable:
- structcheck
- varcheck
- staticcheck
- unconvert
- gofmt
- goimports
- revive
- ineffassign
- vet
- unused
- misspell
- revive
- unconvert
disable:
- errcheck

run:
deadline: 4m
skip-dirs:
- misc
exclusions:
generated: lax
presets:
- comments
- common-false-positives
- legacy
- std-error-handling
paths:
- third_party$
- builtin$
- examples$
formatters:
enable:
- gofmt
- goimports
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
84 changes: 84 additions & 0 deletions WARP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# WARP.md

This file provides guidance to WARP (warp.dev) when working with code in this repository.

## Common commands

- Build (Go 1.25 pinned via `go.mod` and CI):
- Default (linux/amd64): `make build`
- Cross-compile examples: `GOOS=linux GOARCH=arm64 make build`
- Output: `bin/nydus-store`
- Lint (golangci-lint uses .golangci.yml):
- `make check`
- If missing locally: `curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/HEAD/install.sh | sh -s -- -b "$(go env GOPATH)"/bin v2.5.0`
- Unit tests:
- All: `go test ./...`
- Package: `go test ./pkg/fs -v`
- Single test: `go test ./pkg/manager -run '^TestName$' -v`
- Integration smoke (Linux with Podman and root privileges):
- Install nydus (version used in CI):
- `NYDUS_VERSION=v2.1.6`
- `wget https://github.com/dragonflyoss/image-service/releases/download/$NYDUS_VERSION/nydus-static-$NYDUS_VERSION-linux-amd64.tgz`
- `sudo tar xzvf nydus-static-$NYDUS_VERSION-linux-amd64.tgz --wildcards --strip-components=1 -C /usr/bin/ nydus-static/*`
- Configure storage and nydusd:
- `sudo mkdir -p /var/lib/nydus-store`
- `sudo cp misc/nydus-config.json /etc/nydusd-config.json`
- `sudo cp misc/storage.conf /etc/containers/storage.conf`
- Run store and verify:
- `nohup sudo bin/nydus-store --log-to-stdout --log-level info --config-path /etc/nydusd-config.json --root /var/lib/nydus-store &`
- `sudo podman run -it ghcr.io/dragonflyoss/image-service/nginx:nydus-latest echo hello word`

## Running the plugin locally

- Requires Linux kernel with FUSE, Podman/CRI-O using containers/storage, and `nydusd`/`nydus-image` installed in PATH.
- Typical launch:
- `sudo bin/nydus-store --log-to-stdout --log-level info --config-path /etc/nydusd-config.json --root /var/lib/nydus-store`
- Optional file mode overrides (octal without leading 0o):
- `--fs-file-mode 0400 --fs-dir-mode 0500 --fs-link-mode 0400`
- Credentials resolution obeys (in order):
- Docker config.json
- Podman-compatible `auth.json`: `REGISTRY_AUTH_FILE`, `$XDG_RUNTIME_DIR/containers/auth.json`, `$HOME/.config/containers/auth.json`

## High-level architecture

- Entry point (`cmd/store/main.go`)
- Uses flags from `containerd/nydus-snapshotter` to parse config (`--root`, `--config-path`, logging), sets up `slog`.
- Builds a registry resolver with credentials from both Docker config and Podman `auth.json`.
- Initializes the `LayerManager`, mounts a FUSE filesystem under `<root>/store`, and blocks until SIGINT. On exit, releases mounts.
- Resolver & Keychains (`pkg/services/...`)
- `resolver.RegistryHostsFromConfig` constructs `docker.RegistryHost` with retryable HTTP client and request timeouts.
- Credential sources:
- Docker: `pkg/services/keychain/dockerconfig` (supports identity token, user/pass; Docker Hub host normalized).
- Podman: `pkg/services/keychain/podmanauth` (searches `REGISTRY_AUTH_FILE`, XDG runtime, then `$HOME/.config/containers/auth.json`; supports identity token and base64 `auth`).
- Layer management (`pkg/manager`)
- Verifies signatures (configurable public key, optional validation).
- Spawns and tracks `nydusd` processes via snapshotter’s process manager; stores state in an embedded DB under `<root>`.
- Resolves image manifests/configs, identifies Nydus meta layers, downloads bootstrap, mounts via Nydus FS, waits for readiness.
- Exposes mounted content by bind-mounting Nydus mountpoints to `<root>/store/<snapshotID>/<digest>/diff` (read-only), reference-counted per layer.
- Crash recovery: on startup, attempts to unmount any orphaned bind mounts found under `<root>/store/*/*/diff`.
- OS-specific shims: Linux-specific mount helpers with `mount_shim_linux.go`, safe fallbacks in `mount_shim_other.go` for non-Linux builds.
- FUSE filesystem (`pkg/fs`)
- go-fuse v2 based; presents a structured view with directories and symlinks: `pool`, `diff`, `blob`, `info`, and `use` markers.
- Default permission modes are intentionally restrictive; can be overridden at mount time via `WithModes` (wired to CLI flags).
- Detects `fusermount`/`fusermount3`; if absent, attempts direct mount. Waits for server mount completion before returning.
- Integration with containers/storage
- `misc/storage.conf` declares `additionallayerstores = [ "/var/lib/nydus-store/store:ref" ]` to register the plugin’s store.
- Podman/CRI-O can then lazy-mount Nydus layers referenced by images.

## Project rules for Warp agents

- Use Go 1.25 toolchain.
- Maintain compatibility with Podman `auth.json` discovery and precedence; do not regress Docker config support.
- Prefer FUSE3 (`fusermount3`) when available; fallback paths must remain functional.
- Keep FS access modes configurable via CLI flags and plumbed through `pkg/fs`.
- Preserve and improve crash/unmount recovery semantics in `LayerManager` (e.g., `RecoverOrphanMounts`, `ReleaseAll`).

## Notable files

- `Makefile` — build and lint targets (`build`, `check`).
- `.golangci.yml` — enabled linters/formatters.
- `misc/nydus-config.json`, `misc/storage.conf` — sample runtime configs.
- `cmd/store/main.go` — CLI entry.
- `pkg/manager/*` — layer lifecycle, mounting, recovery.
- `pkg/fs/*` — FUSE filesystem and wiring.
- `pkg/services/{resolver,keychain}/` — registry access and auth.
145 changes: 123 additions & 22 deletions cmd/store/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@

import (
"fmt"
"log/slog"
"os"
"os/signal"
"path/filepath"
"runtime"
"syscall"

"github.com/containerd/containerd/log"
"github.com/containerd/nydus-snapshotter/cmd/containerd-nydus-grpc/pkg/command"
"github.com/containerd/nydus-snapshotter/cmd/containerd-nydus-grpc/pkg/logging"
"github.com/containerd/nydus-snapshotter/config"
"github.com/containerd/nydus-snapshotter/pkg/errdefs"
"github.com/pkg/errors"
Expand All @@ -17,6 +18,7 @@
"github.com/containers/nydus-storage-plugin/pkg/fs"
"github.com/containers/nydus-storage-plugin/pkg/manager"
"github.com/containers/nydus-storage-plugin/pkg/services/keychain/dockerconfig"
podmanauth "github.com/containers/nydus-storage-plugin/pkg/services/keychain/podmanauth"
"github.com/containers/nydus-storage-plugin/pkg/services/resolver"
)

Expand All @@ -26,61 +28,160 @@
<-c
}

func parseOctalMode(s string) (uint32, error) {
var v uint32
for i := 0; i < len(s); i++ {
c := s[i]
if c < '0' || c > '7' {
return 0, fmt.Errorf("invalid octal: %s", s)
}
v = (v << 3) | uint32(c-'0')
}
return v, nil
}

func setupSlog(level string, toStdout bool, logDir string) error {
var lvl slog.Level
switch level {
case "debug":
lvl = slog.LevelDebug
case "warn", "warning":
lvl = slog.LevelWarn
case "error":
lvl = slog.LevelError
default:
lvl = slog.LevelInfo
}
var w *os.File
if toStdout || logDir == "" {
w = os.Stdout
} else {
path := filepath.Join(logDir, "nydus-store.log")
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
if err != nil {
return err
}
w = f
}
h := slog.NewTextHandler(w, &slog.HandlerOptions{Level: lvl})
slog.SetDefault(slog.New(h))
return nil
}

func main() {
flags := command.NewFlags()
app := &cli.App{
Name: "crio nydus store",
Usage: "crio nydus store plugin",
Version: "0.0.0",
Flags: flags.F,
Action: func(c *cli.Context) error {
if err := logging.SetUp(flags.Args.LogLevel, flags.Args.LogToStdout, flags.Args.LogDir, flags.Args.RootDir); err != nil {
Flags: append(flags.F,
&cli.StringFlag{Name: "fs-file-mode", Usage: "octal file mode for files (e.g. 0400)"},
&cli.StringFlag{Name: "fs-dir-mode", Usage: "octal dir mode (e.g. 0500)"},
&cli.StringFlag{Name: "fs-link-mode", Usage: "octal symlink mode (e.g. 0400)"},
&cli.BoolFlag{Name: "fs-allow-other", Value: true, Usage: "enable allow_other on FUSE mount"},
&cli.BoolFlag{Name: "fs-direct-mount", Value: false, Usage: "force direct mount (bypass fusermount)"},
&cli.BoolFlag{Name: "fs-mount-suid", Value: false, Usage: "add suid to fusermount mount options"},
),
Action: func(c *cli.Context) error {

Check failure on line 85 in cmd/store/main.go

View workflow job for this annotation

GitHub Actions / Build

File is not properly formatted (gofmt)
if err := setupSlog(flags.Args.LogLevel, flags.Args.LogToStdout, flags.Args.LogDir); err != nil {
return errors.Wrap(err, "failed to prepare logger")
}

// Fail fast on unsupported platforms to avoid false-positive "mounted" states.
if runtime.GOOS != "linux" {
slog.ErrorContext(c.Context, "nydus-store requires Linux (FUSE) to mount; current OS unsupported", "GOOS", runtime.GOOS)
return errors.New("platform not supported: requires linux with FUSE")
}

var cfg config.Config
if err := command.Validate(flags.Args, &cfg); err != nil {
return errors.Wrap(err, "invalid argument")
}

mountPoint := fmt.Sprintf("%s/store", flags.Args.RootDir)
if _, err := os.Stat(mountPoint); err != nil {
if os.IsNotExist(err) {
if err := os.MkdirAll(mountPoint, 0755); err != nil {
return errors.Wrapf(err, "create root directory %s", mountPoint)
}
} else {
return errors.Wrapf(err, "stat root directory %s", mountPoint)
}
if err := os.MkdirAll(mountPoint, 0755); err != nil {
return errors.Wrapf(err, "create root directory %s", mountPoint)
}

// replace it with nydus-snapshotter resolver.
hosts := resolver.RegistryHostsFromConfig([]resolver.Credential{dockerconfig.NewDockerconfigKeychain(c.Context)}...)
hosts := resolver.RegistryHostsFromConfig(
[]resolver.Credential{
dockerconfig.NewDockerconfigKeychain(c.Context),
// Podman-compatible auth.json
podmanauth.NewPodmanAuthKeychain(c.Context),
}...,
)
layManager, err := manager.NewLayerManager(c.Context, flags.Args.RootDir, hosts, &cfg)
if err != nil {
panic(err)
}

if err := fs.Mount(c.Context, mountPoint, flags.Args.RootDir, true, layManager); err != nil {
log.G(c.Context).WithError(err).Fatalf("failed to mount fs at %q", mountPoint)
// Parse optional FS modes
fileMode := fs.DefaultFileMode()
dirMode := fs.DefaultDirMode()
linkMode := fs.DefaultLinkMode()
if v := c.String("fs-file-mode"); v != "" {
if m, err := parseOctalMode(v); err == nil {
fileMode = m
}
}
if v := c.String("fs-dir-mode"); v != "" {
if m, err := parseOctalMode(v); err == nil {
dirMode = m
}
}
if v := c.String("fs-link-mode"); v != "" {
if m, err := parseOctalMode(v); err == nil {
linkMode = m
}
}

// Recover orphan bind mounts from previous crashes
_ = layManager.RecoverOrphanMounts(c.Context)

slog.InfoContext(c.Context, "Starting FUSE mount",
"mountPoint", mountPoint,
"rootDir", flags.Args.RootDir,
"fileMode", fmt.Sprintf("0%o", fileMode),
"dirMode", fmt.Sprintf("0%o", dirMode),
"linkMode", fmt.Sprintf("0%o", linkMode),
"allowOther", c.Bool("fs-allow-other"))

if err := fs.Mount(
c.Context,
mountPoint,
flags.Args.RootDir,
true,
layManager,
fs.WithModes(fileMode, dirMode, linkMode),
fs.WithAllowOther(c.Bool("fs-allow-other")),
fs.WithDirectMount(c.Bool("fs-direct-mount")),
fs.WithMountSuid(c.Bool("fs-mount-suid")),
); err != nil {
slog.ErrorContext(c.Context, "failed to mount fs", "mountPoint", mountPoint, "err", err)
return err
}
defer func() {
// Best-effort: release bind mounts first
layManager.ReleaseAll(c.Context)
err := syscall.Unmount(mountPoint, 0)
if err != nil {
log.G(c.Context).Error(err)
// Try a normal unmount of the FUSE mountpoint
if err := syscall.Unmount(mountPoint, 0); err != nil {
slog.WarnContext(c.Context, "unmount busy; retry lazy unmount", "err", err)
if derr := lazyUnmount(mountPoint); derr != nil {
slog.ErrorContext(c.Context, "lazy unmount failed", "err", derr)
}
}
log.G(c.Context).Info("Exiting")
slog.InfoContext(c.Context, "Exiting")
}()
waitForSIGINT()
return nil
},
}
if err := app.Run(os.Args); err != nil {
if errdefs.IsConnectionClosed(err) {
log.L.Info("snapshotter exited")
slog.Info("snapshotter exited")
return
}
log.L.WithError(err).Fatal("failed to start crio nydus store")
slog.Error("failed to start crio nydus store", "err", err)
}
}
10 changes: 10 additions & 0 deletions cmd/store/umount_linux.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
//go:build linux

package main

import "golang.org/x/sys/unix"

// lazyUnmount performs a lazy unmount (MNT_DETACH) on Linux.
func lazyUnmount(target string) error {
return unix.Unmount(target, unix.MNT_DETACH)
}

Check failure on line 10 in cmd/store/umount_linux.go

View workflow job for this annotation

GitHub Actions / Build

File is not properly formatted (gofmt)
6 changes: 6 additions & 0 deletions cmd/store/umount_other.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
//go:build !linux

package main

// lazyUnmount is a no-op fallback on non-Linux platforms.
func lazyUnmount(_ string) error { return nil }
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/containers/nydus-storage-plugin

go 1.18
go 1.25

require (
github.com/containerd/containerd v1.6.19
Expand Down
2 changes: 1 addition & 1 deletion pkg/cache/lrucache.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ type LRUCache struct {
// NewLRUCache creates new lru cache.
func NewLRUCache(maxEntries int) *LRUCache {
inner := lru.New(maxEntries)
inner.OnEvicted = func(key lru.Key, value interface{}) {
inner.OnEvicted = func(_ lru.Key, value interface{}) {
// Decrease the ref count incremented in Add().
// When nobody refers to this value, this value will be finalized via refCounter.
value.(*refCounter).finalize()
Expand Down
Loading
Loading