Skip to content

Run production services from a main worktree #156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jakobtfaber wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Run production services from a main worktree #156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions docs/operations/dashboard.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,25 @@ Health checks: `systemctl status dsa110-dashboard`, `systemctl list-timers 'dsa1

## Worktree topology

The dashboard service serves the `dashboard-production` branch from a dedicated worktree at
`/data/dsa110-continuum-dashboard` (create once with
`git -C /data/dsa110-continuum worktree add /data/dsa110-continuum-dashboard dashboard-production`).
The unit's `DSA110_REPO_ROOT=/data/dsa110-continuum` keeps pipeline launches executing from the
live checkout, which may sit on a different branch. Upgrade procedure:
Both production services run `main` from the dedicated worktree at
`/data/dsa110-continuum-production`. Agent work stays in separate worktrees and never changes the
code used by a running service. Create the production worktree once with:

```bash
git -C /data/dsa110-continuum-dashboard pull --ff-only
git -C /data/dsa110-continuum worktree add /data/dsa110-continuum-production main
```

Deploy a new `main` revision explicitly from that worktree:

```bash
git -C /data/dsa110-continuum-production pull --ff-only
sudo cp /data/dsa110-continuum-production/ops/systemd/dsa110-dashboard.service \
/data/dsa110-continuum-production/ops/systemd/dsa110-autopipeline.service \
/data/dsa110-continuum-production/ops/systemd/dsa110-autopipeline.timer \
/etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl restart dsa110-dashboard
sudo systemctl restart dsa110-autopipeline.timer
```

Per-artifact QA pages: `/artifacts/caltable/` (BP/G/K tables with acquisition provenance),
Expand Down
4 changes: 2 additions & 2 deletions ops/systemd/dsa110-autopipeline.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ Description=DSA-110 continuum daily batch pipeline launcher
[Service]
Type=oneshot
User=ubuntu
WorkingDirectory=/data/dsa110-continuum
Environment=PYTHONPATH=/data/dsa110-continuum
WorkingDirectory=/data/dsa110-continuum-production
Environment=PYTHONPATH=/data/dsa110-continuum-production
ExecStart=/opt/miniforge/envs/casa6/bin/python scripts/auto_pipeline.py
6 changes: 3 additions & 3 deletions ops/systemd/dsa110-dashboard.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ After=network.target

[Service]
User=ubuntu
WorkingDirectory=/data/dsa110-continuum-dashboard
Environment=PYTHONPATH=/data/dsa110-continuum-dashboard
Environment=DSA110_REPO_ROOT=/data/dsa110-continuum
WorkingDirectory=/data/dsa110-continuum-production
Environment=PYTHONPATH=/data/dsa110-continuum-production
Environment=DSA110_REPO_ROOT=/data/dsa110-continuum-production
EnvironmentFile=-/home/ubuntu/.config/dsa110/dashboard.env
ExecStart=/opt/miniforge/envs/casa6/bin/python -m uvicorn scripts.qa_server:app --host 0.0.0.0 --port 8767 --log-level warning
ExecStartPost=/bin/sh -c 'for attempt in $$(seq 1 60); do /usr/bin/curl --fail --silent --show-error --max-time 5 http://127.0.0.1:8767/health > /dev/null && exit 0; sleep 1; done; exit 1'
Expand Down