Skip to content

docs(selfhost): document a git-pull-safe path to enabling Alertmanager notifications - #3875

Merged
JSONbored merged 1 commit into
mainfrom
feat-alertmanager-discord-notifications
Jul 7, 2026
Merged

docs(selfhost): document a git-pull-safe path to enabling Alertmanager notifications#3875
JSONbored merged 1 commit into
mainfrom
feat-alertmanager-discord-notifications

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • alertmanager/alertmanager.yml ships with every alert routed to a silent "null" receiver by design (so a fresh docker compose --profile observability up -d always comes up green), but the existing docs told operators to enable a real receiver by editing that committed file in place. Self-host operators git pull this repo to deploy, so a local edit to a tracked file either blocks the next pull or gets silently discarded by it — this was a real, live gap: on our own self-hosted instance, GittensoryQueueBacklogHigh and other alert rules fired continuously for ~9 hours during a queue-stall incident with zero notification, because the receiver was never actually wired up.
  • Documents copying the template to a gitignored alertmanager/alertmanager.local (already covered by the existing *.local ignore rule, so no .gitignore change needed) and pointing Alertmanager at it via docker-compose.override.yml's command: override — the same pattern this repo already uses for other per-operator customization (co-located CPU tuning, runner registration).
  • Mounts the whole alertmanager/ directory (not just alertmanager.yml) into the container, so an operator's local config and any secret file it references (e.g. a Discord webhook_url_file) are visible inside the container without a docker-compose.yml edit.
  • No linked issue: found and fixed while investigating why a live incident's alerts never reached a human; small, self-contained docs + mount fix.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage — no src/** logic changed (docs + a compose volume mount only), so no new coverage surface.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate (0 vulnerabilities)
  • Full npm run test:ci gate run locally end-to-end (green)

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. — N/A.
  • API/OpenAPI/MCP behavior is updated and tested where needed. — no API/OpenAPI/MCP surface touched.
  • UI changes use live API data or real empty/error/loading states. — N/A, docs-only text/code-sample change, no data-driven UI.
  • Visible UI changes include a UI Evidence section. — N/A, plain docs prose/code-block change, not a visual/layout change.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

Notes

  • Verified live on edge-us-01: applied the documented alertmanager.local + docker-compose.override.yml pattern, confirmed Alertmanager successfully delivered a Discord notification for an active alert after the change (reusing the existing gittensory Discord channel webhook already configured for this repo's per-PR notifications).

…r notifications

alertmanager.yml ships with every alert routed to a silent "null" receiver
by design, and the existing guidance told operators to edit that
committed file in place to enable a real one. Since self-host operators
git pull this repo to deploy, that either blocks the next pull or gets
silently discarded by it.

Document copying to a gitignored alertmanager/alertmanager.local (already
covered by the *.local ignore rule) and pointing Alertmanager at it via
docker-compose.override.yml instead, so the real receiver config survives
every future deploy.

Also mount the whole alertmanager/ directory (not just the yml) so an
operator's local config and any secret file it references show up inside
the container with no docker-compose.yml edit required.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 8f38b92 Commit Preview URL

Branch Preview URL
Jul 06 2026, 11:59 PM

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 7, 2026
@loopover-orb

loopover-orb Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-07 00:17:28 UTC

2 files · 1 AI reviewer · 2 blockers · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

Review summary
This PR fixes a real operational gap: the old docs told operators to edit the git-tracked alertmanager.yml directly, which either blocks git pull or gets silently clobbered by it, explaining why alerts fired silently for hours during a live incident. The fix is sound — it documents copying to a gitignored alertmanager.local and repointing Alertmanager's command via docker-compose.override.yml (an existing pattern in this repo), and switches the docker-compose.yml mount from a single-file bind to mounting the whole ./alertmanager directory read-only so the local config and any secret file it references (e.g. webhook_url_file) are visible in the container without further docker-compose.yml edits. The storage.path volume for Alertmanager's own state remains a separate named volume, so the wider directory mount doesn't collide with anything Alertmanager needs to write.

Nits — 6 non-blocking
  • docker-compose.yml's new comment block above the volumes entry is six lines long for what could be a two-line explanation — consider trimming to just the local-config + secret-file pointer.
  • The doc's suggested docker-compose.override.yml command block hardcodes `--storage.path=/alertmanager` alongside `--config.file`; worth a one-line note that operators must keep both flags or storage.path will silently reset to the Alertmanager default rather than the named volume path.
  • Mounting the entire alertmanager/ directory read-only now exposes any stray file an operator drops in there to the container at the same relative path — harmless for a self-hosted single-operator setup but worth a one-line callout that only alertmanager-relevant files should live in that directory.
  • In docker-compose.yml, shorten the comment above `- ./alertmanager:/etc/alertmanager:ro` to reference the doc section instead of re-explaining the whole mechanism inline.
  • In the docs, consider linking directly to the `*.local` gitignore rule (or naming the exact glob) so operators can confirm `alertmanager.local` matches it without having to open .gitignore themselves.
  • Code changes lack test evidence — Add focused regression tests or explain why existing coverage is sufficient.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
Signal Result Evidence
Code review ❌ 2 blockers 1 reviewer
Linked issue ✅ No-issue rationale PR body explains why no issue is linked.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 51 registered-repo PR(s), 43 merged, 352 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 51 PR(s), 352 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
  • Official Gittensor activity: 51 PR(s), 352 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 7, 2026
@JSONbored
JSONbored merged commit 2daea2f into main Jul 7, 2026
11 checks passed
@JSONbored
JSONbored deleted the feat-alertmanager-discord-notifications branch July 7, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant