Add deploy/ scaffold for reproducible server bootstrap

.gitignore

@@ -12,6 +12,6 @@ CLAUDE.md
 .docs/
 icicle
 
-# Deployment (machine-specific)
-deploy/
+# Deployment: track templates / configs, NOT per-server secrets.
+# The deploy/.env (matched by *.env above) holds CLICKHOUSE_PASSWORD etc. and stays ignored.
 Makefile

deploy/.env.example

@@ -0,0 +1,35 @@
+# Copy this file to .env and fill in real values.
+# .env is gitignored. Never commit it.
+
+# ---- Required ----
+
+# Domain this server will serve the API on (e.g. api.example.com)
+API_DOMAIN=
+
+# OS user the icicle services run as. Created by setup.sh if missing.
+ICICLE_USER=icicle
+
+# Strong random password for the ClickHouse `default` user.
+# Generate with: openssl rand -base64 30 | tr -dc 'A-Za-z0-9' | head -c 32
+CLICKHOUSE_PASSWORD=
+
+# sha256 hex of CLICKHOUSE_PASSWORD. Used inside the CH config XML.
+# Compute with: printf '%s' "$CLICKHOUSE_PASSWORD" | sha256sum | awk '{print $1}'
+CLICKHOUSE_PASSWORD_SHA256=
+
+# Bearer token gating /metrics (Prometheus auth). Generate with: openssl rand -hex 32
+ICICLE_METRICS_TOKEN=
+
+# ---- Optional (sensible defaults) ----
+
+# ClickHouse data dir on the host (bind-mounted into the container)
+CH_DATA_DIR=/home/${ICICLE_USER}/clickhouse-data
+
+# ClickHouse config dir on the host (bind-mounted into the container)
+CH_CONFIG_DIR=/home/${ICICLE_USER}/clickhouse-config
+
+# Memory caps for the systemd services
+ICICLE_API_MEMORY_HIGH=4G
+ICICLE_API_MEMORY_MAX=6G
+ICICLE_INDEXER_MEMORY_HIGH=8G
+ICICLE_INDEXER_MEMORY_MAX=12G

deploy/README.md

@@ -0,0 +1,115 @@
+# Icicle Deploy
+
+Reproducible setup for an Icicle indexer + API server. Each new server is bootstrapped from the files in this directory; nothing lives only in someone's head.
+
+## What's in here
+
+| File | Role |
+| --- | --- |
+| `setup.sh` | Idempotent installer. Run once per fresh server. |
+| `.env.example` | Template for the per-server secrets file. |
+| `docker-compose.yml` | ClickHouse container with the right bind-mounts and loopback ports. |
+| `clickhouse/users.d/default-user.xml.template` | Renders the `default` user with a sha256 password. |
+| `clickhouse/access-setup.sql` | Creates the read-only `anonymous` / `anonymous_heavy` users for dashboards. |
+| `nginx/api.conf` | Reverse-proxy vhost. Forwards `X-Forwarded-For` correctly. |
+| `systemd/icicle-api.service`, `icicle-indexer.service` | Service units (with memory caps). |
+| `iptables/rules.v4` | Default-deny firewall. Allows 22/80/443/9651 only. |
+
+## Bootstrap a new server (~20 minutes)
+
+Assumes Ubuntu 24.04 with root SSH access and DNS already pointing your domain at the box.
+
+```bash
+# 1. Get the repo onto the server
+ssh root@<server>
+cd /root && git clone <repo-url> icicle && cd icicle/deploy
+
+# 2. Create .env with real values
+cp .env.example .env
+nano .env
+#    - Generate CLICKHOUSE_PASSWORD: openssl rand -base64 30 | tr -dc 'A-Za-z0-9' | head -c 32
+#    - Compute CLICKHOUSE_PASSWORD_SHA256:
+#        printf '%s' "$CLICKHOUSE_PASSWORD" | sha256sum | awk '{print $1}'
+#    - Generate ICICLE_METRICS_TOKEN: openssl rand -hex 32
+#    - Set API_DOMAIN to the domain you're using
+
+# 3. Run the installer
+./setup.sh
+```
+
+The installer:
+1. Installs packages (Docker, nginx, certbot, iptables-persistent)
+2. Loads firewall rules
+3. Creates the `icicle` OS user
+4. Renders the CH password into the bind-mounted users.d
+5. Starts ClickHouse via docker-compose
+6. Writes systemd env files (mode 600) and unit files
+7. Installs the nginx vhost (HTTP only — TLS in step 4 below)
+
+After `setup.sh` finishes, do the four manual steps it prints:
+1. Clone the Icicle source and `go build -o icicle .`
+2. `clickhouse-client < clickhouse/access-setup.sql` to create the read-only users
+3. `certbot --nginx -d $API_DOMAIN` to get a TLS cert (rewrites the nginx vhost)
+4. `systemctl enable --now icicle-api icicle-indexer`
+
+## Verifying a deploy
+
+After everything is up:
+
+```bash
+systemctl is-active icicle-api icicle-indexer        # both 'active'
+curl -fsS https://${API_DOMAIN}/health                # 200
+sudo journalctl -u icicle-api -n 20 --no-pager        # 'ClickHouse connected successfully'
+
+# DB should NOT be reachable from outside (run from your laptop):
+nc -zv <server-ip> 8123     # times out
+nc -zv <server-ip> 9000     # times out
+
+# Empty CH password should NOT work:
+docker exec icicle-clickhouse clickhouse-client \
+  --user default --password "" --query "SELECT 1"     # AUTHENTICATION_FAILED
+```
+
+## Updating the deployed app
+
+```bash
+cd ~/icicle && git pull && go build -o icicle . && sudo systemctl restart icicle-api icicle-indexer
+```
+
+## Rotating the ClickHouse password
+
+```bash
+# Pick a new password
+NEW_PW=$(openssl rand -base64 30 | tr -dc 'A-Za-z0-9' | head -c 32)
+HASH=$(printf '%s' "$NEW_PW" | sha256sum | awk '{print $1}')
+
+# Update the host config
+sudo sed -i "s|<password_sha256_hex>.*</password_sha256_hex>|<password_sha256_hex>$HASH</password_sha256_hex>|" \
+  /home/icicle/clickhouse-config/users.d/default-user.xml
+
+# CH auto-reloads users.d. Verify old password is rejected:
+docker exec icicle-clickhouse clickhouse-client --user default --password "OLD_PW" --query "SELECT 1"
+
+# Update the env files
+sudo sed -i "s|^CLICKHOUSE_PASSWORD=.*|CLICKHOUSE_PASSWORD=$NEW_PW|" /etc/icicle/api.env /etc/icicle/indexer.env
+
+# Restart services
+sudo systemctl restart icicle-api icicle-indexer
+```
+
+## Things this deploy does NOT include (yet)
+
+- **Avalanchego** — assumed already running, either on this host or reachable via RPC. Add its installation separately if needed.
+- **Observability stack** (Prometheus / Grafana / Loki) — operator-side, not customer-facing. See the main repo's `.docs/` for the setup that was used in the L1Beat dev environment.
+- **ClickHouse backup** — only the indexer RPC cache is currently backed up (`backup_cache.sh`). Real CH backups via `clickhouse-backup` or volume snapshots are a TODO.
+- **TLS cert auto-renewal** — certbot installs a systemd timer that handles renewals automatically; nothing to do here. Just confirm with `systemctl list-timers | grep certbot`.
+
+## Security checklist after deploy
+
+- [ ] `ss -tlnp | grep -E ':8123|:9000'` shows `127.0.0.1` only (never `0.0.0.0`)
+- [ ] `docker exec icicle-clickhouse clickhouse-client --user default --password ""` fails with `AUTHENTICATION_FAILED`
+- [ ] `nc -zv <server-ip> 8123` (from outside) times out
+- [ ] `sudo iptables -L INPUT -n` shows default policy DROP
+- [ ] `systemctl is-enabled netfilter-persistent` reports `enabled` (firewall survives reboots)
+- [ ] `curl -sI https://<domain>/metrics` returns 401 (Prometheus auth gating works)
+- [ ] In `journalctl -u icicle-api`, `HTTP request` lines contain `remote=` and `xff=` fields (IP logging works end-to-end)

deploy/clickhouse/access-setup.sql

@@ -0,0 +1,73 @@
+-- Read-only users for the front-end and any dashboard tool.
+-- Apply after ClickHouse is up and after the indexer has created the schema.
+-- Run with:
+--   docker exec -i icicle-clickhouse clickhouse-client --user default --password "$CLICKHOUSE_PASSWORD" < access-setup.sql
+--
+-- This file is a copy of the canonical anonymous_user.sql at the repo root. Keep them in sync.
+
+CREATE ROLE IF NOT EXISTS frontend_reader;
+GRANT SELECT ON default.* TO frontend_reader;
+GRANT SHOW ON *.* TO frontend_reader;
+GRANT SELECT ON system.parts TO frontend_reader;
+GRANT SELECT ON system.tables TO frontend_reader;
+GRANT SELECT ON system.columns TO frontend_reader;
+GRANT SELECT ON system.databases TO frontend_reader;
+
+DROP SETTINGS PROFILE IF EXISTS anonymous_profile;
+CREATE SETTINGS PROFILE anonymous_profile SETTINGS
+    readonly = 1,
+    allow_ddl = 0,
+    allow_introspection_functions = 0,
+    max_concurrent_queries_for_user = 10,
+    max_threads = 1,
+    max_result_rows  = 1000,
+    max_result_bytes = 64000000,
+    result_overflow_mode = 'break',
+    max_rows_to_read  = 1000000,
+    max_bytes_to_read = 1000000000,
+    max_execution_time = 3,
+    max_memory_usage   = 1000000000;
+
+DROP SETTINGS PROFILE IF EXISTS anonymous_heavy_profile;
+CREATE SETTINGS PROFILE anonymous_heavy_profile SETTINGS
+    readonly = 1,
+    allow_ddl = 0,
+    allow_introspection_functions = 0,
+    max_concurrent_queries_for_user = 2,
+    max_rows_to_read  = 0,
+    max_bytes_to_read = 0,
+    max_partitions_to_read = 0,
+    max_result_rows  = 1000,
+    max_result_bytes = 64000000,
+    result_overflow_mode = 'break',
+    max_execution_time = 60,
+    max_memory_usage   = 0;
+
+DROP QUOTA IF EXISTS anonymous_quota;
+CREATE QUOTA anonymous_quota KEYED BY ip_address
+    FOR INTERVAL 1 minute
+        MAX QUERIES 4000,
+        MAX ERRORS 200;
+
+DROP QUOTA IF EXISTS anonymous_heavy_quota;
+CREATE QUOTA anonymous_heavy_quota KEYED BY ip_address
+    FOR INTERVAL 1 minute
+        MAX QUERIES 10,
+        MAX ERRORS 10;
+
+CREATE USER IF NOT EXISTS anonymous
+    IDENTIFIED WITH no_password
+    HOST ANY
+    DEFAULT ROLE frontend_reader
+    SETTINGS PROFILE 'anonymous_profile'
+    DEFAULT DATABASE default;
+
+CREATE USER IF NOT EXISTS anonymous_heavy
+    IDENTIFIED WITH no_password
+    HOST ANY
+    DEFAULT ROLE frontend_reader
+    SETTINGS PROFILE 'anonymous_heavy_profile'
+    DEFAULT DATABASE default;
+
+ALTER QUOTA anonymous_quota TO anonymous;
+ALTER QUOTA anonymous_heavy_quota TO anonymous_heavy;

deploy/clickhouse/users.d/default-user.xml.template

@@ -0,0 +1,17 @@
+<clickhouse>
+  <!-- Overrides the image's default `default` user. Removes the no-password
+       definition that ships with the image, then re-defines with a real password.
+       The setup script renders this template by substituting __CLICKHOUSE_PASSWORD_SHA256__. -->
+  <users>
+    <default remove="remove"></default>
+    <default>
+      <profile>default</profile>
+      <networks>
+        <ip>::/0</ip>
+      </networks>
+      <password_sha256_hex>__CLICKHOUSE_PASSWORD_SHA256__</password_sha256_hex>
+      <quota>default</quota>
+      <access_management>1</access_management>
+    </default>
+  </users>
+</clickhouse>

deploy/docker-compose.yml

@@ -0,0 +1,25 @@
+# ClickHouse for the Icicle indexer.
+#
+# Run from this directory after `cp .env.example .env` and filling in values:
+#   docker compose up -d
+#
+# Key choices:
+# - Ports are bound to 127.0.0.1 only. The DB must NEVER be on 0.0.0.0.
+# - users.d is bind-mounted from host so the `default` password survives container recreation.
+# - Data volume on host, not a docker-managed volume, so it's easy to back up and reason about.
+
+services:
+  clickhouse:
+    image: clickhouse/clickhouse-server:25.10
+    container_name: icicle-clickhouse
+    restart: unless-stopped
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+    volumes:
+      - ${CH_DATA_DIR}:/var/lib/clickhouse
+      - ${CH_CONFIG_DIR}/users.d:/etc/clickhouse-server/users.d:ro
+    ports:
+      - "127.0.0.1:8123:8123"
+      - "127.0.0.1:9000:9000"

deploy/iptables/rules.v4

@@ -0,0 +1,47 @@
+# iptables rules for an Icicle indexer server.
+# Loaded at boot by netfilter-persistent (apt install iptables-persistent).
+# Apply with: iptables-restore < /etc/iptables/rules.v4
+#
+# Docker adds its own rules to DOCKER / DOCKER-USER chains at runtime — those don't
+# belong here. This file covers host-facing INPUT/FORWARD/OUTPUT and a couple of
+# defense-in-depth NAT-level drops.
+
+*filter
+:INPUT DROP [0:0]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [0:0]
+
+# Allow established/related (return traffic for outbound connections)
+-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+
+# Loopback (required for inter-service localhost talk)
+-A INPUT -i lo -j ACCEPT
+
+# Drop invalid packets
+-A INPUT -m conntrack --ctstate INVALID -j DROP
+
+# SSH — consider restricting to known source IPs in production
+-A INPUT -p tcp --dport 22 -j ACCEPT
+
+# Public web (HTTP for ACME challenges + redirect, HTTPS for the API)
+-A INPUT -p tcp --dport 80 -j ACCEPT
+-A INPUT -p tcp --dport 443 -j ACCEPT
+
+# avalanchego P2P (only needed if avalanchego runs on this host)
+-A INPUT -p tcp --dport 9651 -j ACCEPT
+-A INPUT -p udp --dport 9651 -j ACCEPT
+
+COMMIT
+
+# Defense in depth: block any non-loopback source from reaching 127.0.0.1
+# on the ClickHouse ports, even if docker-proxy is ever misconfigured to 0.0.0.0.
+*nat
+:PREROUTING ACCEPT [0:0]
+:INPUT ACCEPT [0:0]
+:OUTPUT ACCEPT [0:0]
+:POSTROUTING ACCEPT [0:0]
+
+-A PREROUTING -d 127.0.0.1/32 ! -i lo -p tcp --dport 8123 -j DROP
+-A PREROUTING -d 127.0.0.1/32 ! -i lo -p tcp --dport 9000 -j DROP
+
+COMMIT

deploy/nginx/api.conf

@@ -0,0 +1,32 @@
+# nginx vhost for the Icicle API.
+# Place at /etc/nginx/sites-available/api and symlink into sites-enabled.
+# Replace __API_DOMAIN__ with the real domain (setup.sh does this from $API_DOMAIN).
+# Run `certbot --nginx -d <domain>` after enabling — certbot rewrites this file to add the TLS listener.
+
+server {
+    server_name __API_DOMAIN__;
+
+    # Block dotfile scanners (.env, .git, etc.)
+    location ~ /\. {
+        return 444;
+    }
+
+    location / {
+        proxy_pass http://127.0.0.1:8080;
+        proxy_http_version 1.1;
+
+        # WebSocket support
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+
+        # Standard proxy headers — critical for the API's rate limiter and
+        # access logs to see real client IPs.
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+
+    listen 80;
+    # certbot --nginx will add the listen 443 ssl block and TLS cert paths here.
+}