Add Go CLI to improve developer experience

[jcardozo-eth]

Add Go CLI to replace justfile and improve developer experience

Summary

As the justfile grew, its shell recipes became difficult to read, debug, and maintain — shell is the wrong language for non-trivial logic like conditional deployments, health checks, or environment validation. This PR replaces the justfile with dap, a purpose-built CLI written in Go where each command is real, structured code that developers can read, step through, and reason about. Because the CLI is proper software — unit-tested, linted, statically typed — the developer tooling can now be maintained to the same standard as production code.

What changed

The CLI itself

A new cli/ directory contains the full Go source for the dap binary, built on Cobra with terminal styling via Lipgloss. Commands are organized into five groups:

Group	Commands	Purpose
Development	dev, test, check, lint, typecheck	Day-to-day Python development
Environment	versions, clean, reset, welcome	Environment introspection and maintenance
Dagster	materialize, run	Pipeline operations
Kubernetes	k8s up/down/restart/status/logs/shell	Local K8s deployment lifecycle
CLI Development	cli build/test/lint	Meta commands for working on the CLI itself (lint supports --fix for auto-fixing)

Two internal packages support all commands:

• internal/ui — terminal output styled with the ETH corporate design color palette, with adaptive light/dark terminal support via lipgloss.AdaptiveColor. Output degrades gracefully: in a TTY you get full colors and an ASCII art DAP-O banner; in CI or piped output, colors are disabled automatically and the banner falls back to a plain text one-liner. Supports the NO_COLOR standard and a global --no-color flag.
• internal/exec — thin wrapper around os/exec providing Run, RunInteractive, RunPassthrough, and Which helpers

Nix integration

• Split devShells: The monolithic nix develop shell is replaced by four purpose-built shells — default (everything), minimal (Python + CLI), k8s (adds kubectl/helm), and cli-dev (adds Go toolchain). CI pipelines can use smaller shells for faster startup.
• CLI packaging: A dedicated cli/flake.nix uses gomod2nix to build the CLI hermetically. The root flake composes it as a flake input and exposes dap in all devShells.
• Simplified .envrc: The 37-line .envrc with inline bash for venv creation, dependency sync, and a hand-rolled welcome banner is replaced by 19 lines that delegate to dap welcome and a small uv-sync.sh script.

CI/CD

• CLI test & lint workflow (.github/workflows/cli.yml): Runs dap cli test and dap cli lint on pushes/PRs touching cli/.
• Release workflow (.github/workflows/release.yml): Triggers on cli/v* tags, runs tests, then builds cross-platform binaries (linux/darwin, amd64/arm64) via GoReleaser.
• Path filtering: The existing pipeline CI now ignores cli/** changes to avoid unnecessary Python test runs.
• Pre-commit hooks: Added .pre-commit-config.yaml with trailing-whitespace, ruff lint/format, and automatic gomod2nix.toml regeneration when go.mod/go.sum change.

Removed

• justfile (213 lines) — fully replaced by the CLI.
• just dependency from flake.nix — no longer needed.
• jq, curl, openssl dependencies — version-check functions that shelled out to these tools are replaced by native Go equivalents. Password generation for K8s secrets now uses crypto/rand instead of openssl rand.

Docs

• cli/CONTRIBUTING.md — comprehensive guide covering project structure, adding commands, UI guidelines, error handling, testing, linting, building, and the Nix build chain.
• Updated README.md — all command references updated from just to dap, devShell table added, project structure updated to show cli/, tooling section updated.

Why Go?

• Testable and lintable: The CLI has unit tests for every command group and both internal packages, runnable with go test. Linting uses go vet and gofmt.
• Statically typed: Catches errors at compile time that a shell script surfaces only at runtime.
• Single binary: Compiles to one binary with no runtime dependencies — easy to distribute and fast to start.
• CLI ecosystem: Cobra provides subcommands, help generation, flag parsing, and shell completion out of the box. Lipgloss and termenv handle cross-platform terminal styling. The CLI also supports dap --version with version, commit, and build date injected at compile time via go build -ldflags.
• Performance: Compiled to a native binary with instant startup, unlike a Python-based CLI where interpreter overhead adds latency to every command
• Distribution: Can be installed via Nix (nix develop or nix build .#dap) or downloaded as a pre-built binary from GitHub releases.

Why this matters for onboarding and daily work

Onboarding friction typically comes from two places: setting up the environment and discovering what commands exist. This branch eliminates both. nix develop provides a complete, reproducible environment with Python, uv, Go, all linters and formatters included — nothing to install manually. The dap CLI organizes every development task into discoverable command groups with built-in documentation. Contributing new tooling means writing Go with tests, not appending shell scripts. The Nix-based setup also prepares the project for GitHub Codespaces, where the same dev environment can be replicated in the cloud with zero additional configuration.

Appendix

dap welcome (TTY)

  ██████╗  █████╗ ██████╗          ██████╗
  ██╔══██╗██╔══██╗██╔══██╗        ██╔═══██╗
  ██║  ██║███████║██████╔╝ █████╗ ██║   ██║
  ██║  ██║██╔══██║██╔═══╝  ╚════╝ ██║   ██║
  ██████╔╝██║  ██║██║             ╚██████╔╝
  ╚═════╝ ╚═╝  ╚═╝╚═╝              ╚═════╝
  Data Archive Pipeline (DAP) - Orchestrator
  ETH Library Zurich

Versions
  python         3.12.12
  uv             0.9.28
  dagster        1.12.2

Environment
  nix flake      /Users/jc/PycharmProjects/dap ✓
  python venv    /Users/jc/PycharmProjects/dap/.venv ✓
  DAGSTER_HOME   /Users/jc/PycharmProjects/dap/.dagster ✓

Quick Start
  dap dev        Start the Dagster development server
  dap test       Run tests (pytest)
  dap check      Run all quality checks (ruff, mypy, pytest)

  Run 'dap --help' for all commands

dap --help

dap is the CLI for the Data Archive Pipeline (DAP) Orchestrator.

A Dagster-based orchestrator for processing digital assets following
the OAIS reference model. This tool provides commands for local development,
testing, code quality, and Kubernetes deployment.

Usage:
  dap [command]

Development:
  check       Run all quality checks
  dev         Start Dagster development server
  lint        Check code style and formatting
  test        Run pytest tests
  typecheck   Run type checking

Environment:
  clean       Remove .venv and caches
  reset       Clean and reinstall dependencies
  versions    Display tool versions
  welcome     Show welcome banner and environment info

Dagster:
  materialize Materialize Dagster assets
  run         Run Dagster job

Kubernetes:
  k8s         Kubernetes operations

CLI Development:
  cli         CLI development commands

Additional Commands:
  help        Help about any command

Flags:
  -h, --help       help for dap
      --no-color   Disable colored output
  -v, --version    version for dap

Use "dap [command] --help" for more information about a command.