tooling: deploy Nemo block explorer on tooling server

README.md

@@ -100,14 +100,14 @@ NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --aggregato
 
 After validator nodes are spun up, leanpoint is deployed so it can monitor them. Behavior depends on deployment mode:
 
-- **Local deployment** (`NETWORK_DIR=local-devnet`, `deployment_mode: local`): Leanpoint runs **locally**. `sync-leanpoint-upstreams.sh` generates `upstreams.json` (with `--docker` so the container can reach host validators at `host.docker.internal`), writes it to `<NETWORK_DIR>/data/upstreams.json`, pulls the latest image, and starts a local Docker container. UI at http://localhost:5555. The container is removed on Ctrl+C cleanup or when you run with `--stop`.
+- **Local deployment** (`NETWORK_DIR=local-devnet`, `deployment_mode: local`): Leanpoint runs **locally**. `sync-leanpoint-upstreams.sh` generates `upstreams.json` (with `--docker` so the container can reach host validators at `host.docker.internal`), writes it to `<NETWORK_DIR>/data/upstreams.json`, pulls the latest image, and starts a local Docker container. UI at **http://localhost:5555** (host port **`LEANPOINT_HOST_PORT`**, default **5555** — kept separate from Nemo’s default host port **5455**). The container is removed on Ctrl+C cleanup or when you run with `--stop`.
 - **Ansible/remote deployment**: Leanpoint is updated on the **tooling server**. The script rsyncs `upstreams.json` to the server, pulls the latest image there, and recreates the remote container.
 
 **What runs:**
 1. `convert-validator-config.py` reads `validator-config.yaml` and generates `upstreams.json` (validator URLs for health checks).
 2. `sync-leanpoint-upstreams.sh` either deploys leanpoint locally (local devnet) or syncs to the tooling server and recreates the remote container (Ansible).
 
-**Remote defaults:** Tooling server `46.225.10.32`, user `root`, remote path `/etc/leanpoint/upstreams.json`, container name `leanpoint`. Override with env vars (see script header in `sync-leanpoint-upstreams.sh`).
+**Remote defaults:** Tooling server `46.225.10.32`, user `root`, remote path `/etc/leanpoint/upstreams.json`, container name `leanpoint`, published port **5555→5555** (`LEANPOINT_HOST_PORT`, default **5555**). Override with env vars (see script header in `sync-leanpoint-upstreams.sh`).
 
 **SSH key for remote sync:** When using Ansible deployment, the tooling server may require a specific SSH key. Pass `--sshKey ~/.ssh/id_ed25519_github` (or `--private-key`) so the sync can succeed.
 
@@ -124,6 +124,22 @@ python3 convert-validator-config.py local-devnet/genesis/validator-config.yaml u
 
 Requires Python 3 and PyYAML (`pip install pyyaml`).
 
+### Nemo (block explorer) on the tooling server
+
+After validators are up, **Nemo** (Lean consensus block/slot explorer; image `0xpartha/nemo:latest`) can run on the same **tooling server** as leanpoint (`46.225.10.32` by default). **Ports:** leanpoint uses host **5555** by default; Nemo publishes host **5455** → container **5053** by default — they do not overlap. If you change either port, set **`NEMO_HOST_PORT`** and **`LEANPOINT_HOST_PORT`** so they stay different; `sync-nemo-tooling.sh` exits with an error if they match. **HTTPS / nginx:** see [`docs/tooling-server-nemo-nginx.md`](docs/tooling-server-nemo-nginx.md) and [`tooling/nginx-nemo.conf.example`](tooling/nginx-nemo.conf.example).
+
+It uses **`LEAN_API_URL`**: a comma-separated list of `http://<validator-ip>:<apiPort>` for **every** entry in `validator-config.yaml` (same IPs/ports as the devnet HTTP APIs). Rows with **empty `enrFields.ip`** are skipped (e.g. placeholder nodes until an IP is set).
+
+- **Ansible deploy:** `sync-nemo-tooling.sh` writes `/etc/nemo/nemo.env`, runs **`docker pull`** on **`NEMO_IMAGE`** (default `0xpartha/nemo:latest`), then **`docker run --pull=always`** and recreates the `nemo` container. The SQLite data dir is **cleared only when `spin-node.sh` is run with `--generateGenesis`** (or when you set **`NEMO_RESET_DB=1`** manually). Otherwise the existing DB under **`/opt/nemo/data`** is reused. When **`NEMO_IMAGE`** is a **multi-arch** manifest that lists the host CPU (`linux/arm64` or `linux/amd64`), the script passes **`--platform`** for that arch so Docker pulls the matching variant (no platform-mismatch warning). For **single-arch** tags, it omits **`--platform`** so pull/run still succeed. Optional **`NEMO_DOCKER_PLATFORM`** overrides the detected platform when the manifest includes it.
+- **Local devnet:** Same pull + `--pull=always` behavior; Nemo runs in Docker with `host.docker.internal` and data under `<NETWORK_DIR>/data/nemo-data` (wiped only with **`--generateGenesis`**, same as remote).
+
+UI: `http://<tooling-host>:5455` (override with `NEMO_HOST_PORT`). Skip with **`--skip-nemo`** or **`NEMO_SYNC_DISABLED=1`**. Env vars: see `sync-nemo-tooling.sh`.
+
+```sh
+# Print LEAN_API_URL for the current ansible devnet config
+python3 convert-validator-config.py --print-lean-api-url ansible-devnet/genesis/validator-config.yaml
+```
+
 ### Remote Observability Stack
 
 Every Ansible deployment automatically deploys an observability stack alongside each lean node on remote hosts. No additional flags are needed.
@@ -706,6 +722,7 @@ This quickstart includes automated configuration parsing:
 
 - **Official Genesis Generation**: Uses PK's `eth-beacon-genesis` docker tool from [PR #36](https://github.com/ethpandaops/eth-beacon-genesis/pull/36)
 - **Leanpoint upstreams sync**: After nodes are spun up, `convert-validator-config.py` and `sync-leanpoint-upstreams.sh` generate `upstreams.json` from `validator-config.yaml`, rsync it to the tooling server, and restart the leanpoint container (see [Leanpoint upstreams sync](#leanpoint-upstreams-sync-tooling-server))
+- **Nemo tooling sync**: `sync-nemo-tooling.sh` builds `LEAN_API_URL` for all validators, deploys Nemo on the tooling server (or locally), and clears its SQLite data on each restart (see [Nemo (block explorer) on the tooling server](#nemo-block-explorer-on-the-tooling-server))
 - **Complete File Set**: Generates `validators.yaml`, `nodes.yaml`, `genesis.json`, `genesis.ssz`, and `.key` files
 - **QUIC Port Detection**: Automatically extracts QUIC ports from `validator-config.yaml` using `yq`
 - **Node Detection**: Dynamically discovers available nodes from the validator configuration

convert-validator-config.py

@@ -1,17 +1,23 @@
 #!/usr/bin/env python3
 """
-Convert validator-config.yaml to upstreams.json for leanpoint.
+Convert validator-config.yaml to upstreams.json for leanpoint,
+or emit Nemo LEAN_API_URL / env file.
 
 This script reads a validator-config.yaml file (used by lean-quickstart)
 and generates an upstreams.json file that leanpoint can use to monitor
-multiple lean nodes.
+multiple lean nodes. It can also build the comma-separated LEAN_API_URL
+that Nemo expects (one base URL per validator, apiPort/httpPort).
 
 Usage:
     python3 convert-validator-config.py [validator-config.yaml] [output.json] [--docker]
 
+    python3 convert-validator-config.py --print-lean-api-url [validator-config.yaml] [--docker]
+
+    python3 convert-validator-config.py --write-nemo-env <output.env> <validator-config.yaml> [--docker]
+
 Options:
-    --docker  Use host.docker.internal so leanpoint running in Docker can
-              reach a devnet on the host (e.g. upstreams-local-docker.json).
+    --docker  Use host.docker.internal so a container on the host can reach
+              validators on the host (local devnet + Docker).
 
 Examples:
     python3 convert-validator-config.py \\
@@ -62,8 +68,9 @@ def convert_validator_config(
 
         # Try to get IP from enrFields, default to localhost
         ip = "127.0.0.1"
-        if 'enrFields' in validator and 'ip' in validator['enrFields']:
-            ip = validator['enrFields']['ip']
+        enr = validator.get('enrFields')
+        if isinstance(enr, dict) and enr.get('ip') is not None:
+            ip = enr['ip']
         if docker_host:
             ip = "host.docker.internal"
 
@@ -77,24 +84,121 @@ def convert_validator_config(
         }
 
         upstreams.append(upstream)
-    
+
     output = {"upstreams": upstreams}
-    
+
     with open(output_path, 'w') as f:
         json.dump(output, f, indent=2)
-    
+
     print(f"✅ Converted {len(upstreams)} validators to {output_path}")
     print(f"\nGenerated upstreams:")
     for u in upstreams:
         print(f"  - {u['name']}: {u['url']}{u['path']}")
-    
+
     print(f"\n💡 To use: leanpoint --upstreams-config {output_path}")
 
 
+def nemo_lean_api_url_string(
+    yaml_path: str,
+    base_port: int = 8081,
+    docker_host: bool = False,
+) -> str:
+    """
+    Comma-separated Lean HTTP API base URLs for Nemo (no path), one per validator.
+
+    Skips validators with empty enrFields.ip when not using docker_host (e.g. placeholder ansible rows).
+    """
+    with open(yaml_path, 'r') as f:
+        config = yaml.safe_load(f)
+
+    if 'validators' not in config:
+        raise ValueError("No 'validators' key found in config")
+
+    bases: list[str] = []
+    for idx, validator in enumerate(config['validators']):
+        name = validator.get('name', f'validator_{idx}')
+        ip = "127.0.0.1"
+        enr = validator.get('enrFields')
+        if isinstance(enr, dict) and enr.get('ip') is not None:
+            ip = enr['ip']
+        if docker_host:
+            ip = "host.docker.internal"
+        elif isinstance(ip, str) and not ip.strip():
+            print(
+                f"Warning: skipping validator '{name}' for Nemo LEAN_API_URL: empty enrFields.ip",
+                file=sys.stderr,
+            )
+            continue
+
+        http_port = validator.get('apiPort') or validator.get('httpPort') or (base_port + idx)
+        bases.append(f"http://{ip}:{http_port}")
+
+    if not bases:
+        raise ValueError(
+            "No validators with a usable IP for Nemo; assign enrFields.ip or use --docker for local."
+        )
+    return ",".join(bases)
+
+
+def write_nemo_env_file(
+    yaml_path: str,
+    env_out_path: str,
+    base_port: int = 8081,
+    docker_host: bool = False,
+) -> None:
+    """Write a docker --env-file compatible file for Nemo (LEAN_API_URL + defaults)."""
+    url_string = nemo_lean_api_url_string(yaml_path, base_port=base_port, docker_host=docker_host)
+    if any(c in url_string for c in (' ', '"', '\n', '\\')):
+        escaped = url_string.replace("\\", "\\\\").replace('"', '\\"')
+        line = f'LEAN_API_URL="{escaped}"\n'
+    else:
+        line = f"LEAN_API_URL={url_string}\n"
+    with open(env_out_path, "w") as f:
+        f.write(line)
+        f.write("NEMO_PORT=5053\n")
+        f.write("NEMO_DB_PATH=/data/nemo.db\n")
+        f.write("SYNC_INTERVAL_SEC=4\n")
+
+
 def main():
-    args = [a for a in sys.argv[1:] if a != "--docker"]
+    argv = [a for a in sys.argv[1:] if a != "--docker"]
     docker_host = "--docker" in sys.argv
 
+    if "--print-lean-api-url" in argv:
+        argv = [a for a in argv if a != "--print-lean-api-url"]
+        yaml_path = argv[0] if argv else "local-devnet/genesis/validator-config.yaml"
+        try:
+            print(nemo_lean_api_url_string(yaml_path, docker_host=docker_host))
+        except (OSError, ValueError, yaml.YAMLError) as e:
+            print(f"Error: {e}", file=sys.stderr)
+            sys.exit(1)
+        return
+
+    if "--write-nemo-env" in argv:
+        i = argv.index("--write-nemo-env")
+        try:
+            env_out = argv[i + 1]
+            yaml_path = argv[i + 2]
+        except IndexError:
+            print(
+                "Usage: convert-validator-config.py --write-nemo-env <output.env> "
+                "<validator-config.yaml> [--docker]",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+        # Remove consumed args so stray args don't confuse
+        rest = argv[:i] + argv[i + 3:]
+        if rest:
+            print(f"Warning: ignoring extra arguments: {rest}", file=sys.stderr)
+        try:
+            write_nemo_env_file(yaml_path, env_out, docker_host=docker_host)
+            print(f"✅ Wrote Nemo env file to {env_out}")
+        except (OSError, ValueError, yaml.YAMLError) as e:
+            print(f"Error: {e}", file=sys.stderr)
+            sys.exit(1)
+        return
+
+    args = argv
     if len(args) < 2:
         if len(args) == 0:
             print(__doc__)

docs/tooling-server-nemo-nginx.md

@@ -0,0 +1,27 @@
+# Nemo on the tooling server (port + nginx)
+
+## Port
+
+`sync-nemo-tooling.sh` publishes Nemo on the **host** using **`NEMO_HOST_PORT`** (default **5455**). The container still listens on **5053** inside the image (`-p ${NEMO_HOST_PORT}:5053`).
+
+After changing the default or env, redeploy Nemo (e.g. run `spin-node.sh` without `--skip-nemo`, or invoke `sync-nemo-tooling.sh` manually) so Docker recreates the `nemo` container with the new publish port.
+
+## nginx
+
+1. Copy the example site and edit `server_name` (and SSL stanzas if you use HTTPS):
+
+   ```sh
+   sudo cp /path/to/lean-quickstart/tooling/nginx-nemo.conf.example /etc/nginx/sites-available/nemo
+   sudoedit /etc/nginx/sites-available/nemo
+   ```
+
+2. Ensure the **`upstream`** `server` line matches **`NEMO_HOST_PORT`** (default `127.0.0.1:5455`).
+
+3. Enable and reload:
+
+   ```sh
+   sudo ln -sf /etc/nginx/sites-available/nemo /etc/nginx/sites-enabled/
+   sudo nginx -t && sudo systemctl reload nginx
+   ```
+
+If you previously proxied to port **5053**, update that line to **5455** (or whatever you set for `NEMO_HOST_PORT`).

parse-env.sh

@@ -104,6 +104,10 @@ while [[ $# -gt 0 ]]; do
       skipLeanpoint=true
       shift
       ;;
+    --skip-nemo)
+      skipNemo=true
+      shift
+      ;;
     --prepare)
       prepareMode=true
       shift
@@ -157,4 +161,5 @@ echo "coreDumps = ${coreDumps:-disabled}"
 echo "checkpointSyncUrl = ${checkpointSyncUrl:-<not set>}"
 echo "restartClient = ${restartClient:-<not set>}"
 echo "skipLeanpoint = ${skipLeanpoint:-false}"
+echo "skipNemo = ${skipNemo:-false}"
 echo "dryRun = ${dryRun:-false}"

spin-node.sh

@@ -122,6 +122,7 @@ if [ -n "$prepareMode" ] && [ "$prepareMode" == "true" ]; then
   [ -n "$popupTerminal" ]       && ignored_flags+=("--popupTerminal")
   [ -n "$dockerWithSudo" ]      && ignored_flags+=("--dockerWithSudo")
   [ -n "$skipLeanpoint" ]       && ignored_flags+=("--skip-leanpoint")
+  [ -n "$skipNemo" ]            && ignored_flags+=("--skip-nemo")
   [ -n "$validatorConfig" ] && [ "$validatorConfig" != "genesis_bootnode" ] \
                                 && ignored_flags+=("--validatorConfig")
 
@@ -475,6 +476,14 @@ if [ "$deployment_mode" == "ansible" ]; then
     fi
   fi
 
+  if [ -z "$skipNemo" ]; then
+    _nemo_reset_db=0
+    [ -n "$generateGenesis" ] && _nemo_reset_db=1
+    if ! NEMO_RESET_DB="$_nemo_reset_db" "$scriptDir/sync-nemo-tooling.sh" "$validator_config_file" "$scriptDir" "$sshKeyFile" "$useRoot"; then
+      echo "Warning: Nemo tooling sync failed. Pass --sshKey <path-to-key> if the tooling server requires it, or use --skip-nemo to skip."
+    fi
+  fi
+
   # Ansible deployment succeeded, exit normally
   exit 0
 fi
@@ -527,8 +536,10 @@ if [ -n "$stopNodes" ] && [ "$stopNodes" == "true" ]; then
   # Stop local leanpoint container if running
   if [ -n "$dockerWithSudo" ]; then
     sudo docker rm -f leanpoint 2>/dev/null || echo "  Container leanpoint not found or already stopped"
+    sudo docker rm -f nemo 2>/dev/null || echo "  Container nemo not found or already stopped"
   else
     docker rm -f leanpoint 2>/dev/null || echo "  Container leanpoint not found or already stopped"
+    docker rm -f nemo 2>/dev/null || echo "  Container nemo not found or already stopped"
   fi
 
   echo "✅ Local nodes stopped successfully!"
@@ -705,6 +716,18 @@ if [ -z "$skipLeanpoint" ]; then
   fi
 fi
 
+# Nemo explorer: same tooling server (Ansible) or local Docker; DB reset only with --generateGenesis
+local_nemo_deployed=0
+if [ -z "$skipNemo" ]; then
+  _nemo_reset_db=0
+  [ -n "$generateGenesis" ] && _nemo_reset_db=1
+  if NEMO_RESET_DB="$_nemo_reset_db" "$scriptDir/sync-nemo-tooling.sh" "$validator_config_file" "$scriptDir" "$sshKeyFile" "$useRoot" "$dataDir"; then
+    local_nemo_deployed=1
+  else
+    echo "Warning: Nemo deploy failed. Pass --sshKey if needed, or --skip-nemo to skip."
+  fi
+fi
+
 container_names="${spin_nodes[*]}"
 process_ids="${spinned_pids[*]}"
 
@@ -728,6 +751,12 @@ cleanup() {
     eval "$execCmd" 2>/dev/null || true
   fi
 
+  if [ "${local_nemo_deployed:-0}" = "1" ]; then
+    execCmd="docker rm -f nemo"
+    [ -n "$dockerWithSudo" ] && execCmd="sudo $execCmd"
+    eval "$execCmd" 2>/dev/null || true
+  fi
+
   # try for process ids
   execCmd="kill -9 $process_ids"
   echo "$execCmd"

sync-leanpoint-upstreams.sh

@@ -21,6 +21,8 @@
 #   REMOTE_UPSTREAMS_PATH   Remote path for upstreams.json (default: /etc/leanpoint/upstreams.json)
 #   LEANPOINT_CONTAINER     Docker container name (default: leanpoint)
 #   LEANPOINT_IMAGE         Docker image to pull and run (default: 0xpartha/leanpoint:latest)
+#   LEANPOINT_HOST_PORT     Host port published for leanpoint HTTP (default: 5555).
+#   NEMO_HOST_PORT          Used only for clash check (default 5053); must differ from LEANPOINT_HOST_PORT.
 #   LEANPOINT_SYNC_DISABLED Set to 1 to skip (e.g. when tooling server is not used)
 
 set -e
@@ -37,6 +39,13 @@ LEANPOINT_DIR="${LEANPOINT_DIR:-$scriptDir}"
 REMOTE_UPSTREAMS_PATH="${REMOTE_UPSTREAMS_PATH:-/etc/leanpoint/upstreams.json}"
 LEANPOINT_CONTAINER="${LEANPOINT_CONTAINER:-leanpoint}"
 LEANPOINT_IMAGE="${LEANPOINT_IMAGE:-0xpartha/leanpoint:latest}"
+LEANPOINT_HOST_PORT="${LEANPOINT_HOST_PORT:-5555}"
+NEMO_HOST_PORT="${NEMO_HOST_PORT:-5053}"
+
+if [ "${LEANPOINT_HOST_PORT}" = "${NEMO_HOST_PORT}" ]; then
+  echo "Error: LEANPOINT_HOST_PORT (${LEANPOINT_HOST_PORT}) must not equal NEMO_HOST_PORT (Nemo also binds that host port)." >&2
+  exit 1
+fi
 
 if [ "${LEANPOINT_SYNC_DISABLED:-0}" = "1" ]; then
   echo "Leanpoint sync disabled (LEANPOINT_SYNC_DISABLED=1), skipping."
@@ -65,9 +74,10 @@ if [ -n "$local_data_dir" ]; then
   docker pull "$LEANPOINT_IMAGE"
   docker stop "$LEANPOINT_CONTAINER" 2>/dev/null || true
   docker rm "$LEANPOINT_CONTAINER" 2>/dev/null || true
-  docker run -d --name "$LEANPOINT_CONTAINER" --restart unless-stopped -p 5555:5555 \
+  docker run -d --name "$LEANPOINT_CONTAINER" --restart unless-stopped \
+    -p "${LEANPOINT_HOST_PORT}:5555" \
     -v "$local_upstreams:/etc/leanpoint/upstreams.json:ro" "$LEANPOINT_IMAGE"
-  echo "Leanpoint deployed locally at http://localhost:5555 (upstreams: $local_upstreams)."
+  echo "Leanpoint deployed locally at http://localhost:${LEANPOINT_HOST_PORT} (upstreams: $local_upstreams)."
   exit 0
 fi
 
@@ -93,6 +103,6 @@ remote_dir=$(dirname "$REMOTE_UPSTREAMS_PATH")
 $ssh_cmd "$remote_target" "mkdir -p $remote_dir"
 rsync -e "$ssh_cmd" "$out_file" "${remote_target}:${REMOTE_UPSTREAMS_PATH}"
 
-$ssh_cmd "$remote_target" "docker pull $LEANPOINT_IMAGE && docker stop $LEANPOINT_CONTAINER 2>/dev/null || true; docker rm $LEANPOINT_CONTAINER 2>/dev/null || true; docker run -d --name $LEANPOINT_CONTAINER --restart unless-stopped -p 5555:5555 -v $REMOTE_UPSTREAMS_PATH:/etc/leanpoint/upstreams.json:ro $LEANPOINT_IMAGE"
+$ssh_cmd "$remote_target" "docker pull $LEANPOINT_IMAGE && docker stop $LEANPOINT_CONTAINER 2>/dev/null || true; docker rm $LEANPOINT_CONTAINER 2>/dev/null || true; docker run -d --name $LEANPOINT_CONTAINER --restart unless-stopped -p ${LEANPOINT_HOST_PORT}:5555 -v $REMOTE_UPSTREAMS_PATH:/etc/leanpoint/upstreams.json:ro $LEANPOINT_IMAGE"
 
 echo "Leanpoint upstreams synced to $TOOLING_SERVER, image $LEANPOINT_IMAGE pulled, container '$LEANPOINT_CONTAINER' recreated."