feat(publisher): generate sitemap.xml and robots.txt

[artfusion]

Summary

Published sites currently have no crawler-facing route index. This adds two reserved public routes, resolved ahead of the public-slug resolver:

• GET /sitemap.xml — a urlset of every published, directly-routable URL with a <lastmod>: standalone pages plus content-row routes.
• GET /robots.txt — allow-all except /admin, plus a Sitemap: line.

Both derive from the published database per request. Sitemaps are crawled infrequently and the response is cheap to rebuild, so this deliberately avoids adding another artefact to the publish pipeline.

Two details worth reviewing:

The route set mirrors what publicRouter actually answers 200 for. Template pages (entry templates, everywhere layouts, the notFound page) are never directly routable, so they are excluded by checking isTemplatePage against the published SiteDocument. A published page absent from the latest snapshot (an unusual incremental-publish edge) is kept, since it still resolves by slug and omitting it would hide a live URL. Content-row routes come from listPublishedRowRoutes — the same list a full publish bakes.

Absolute URLs are anchored to the configured public origin, via a new canonicalPublicOrigin(url) in server/auth/security.ts. Behind a TLS-terminating edge the request URL scheme is plain http, so raw url.origin would leak http:// locs into crawler-facing output. A configured origin whose host matches the request host wins (multi-domain installs emit the domain the crawler actually fetched), else the canonical first entry, else the request origin when nothing is configured.

The vite.config.ts change opts both paths past the dev-proxy file-extension rejection, for the same reason as the existing /_instatic/ routes.

Developed against a live self-hosted install and currently serving that sites sitemap in production.

Verification

• bun run build
• bun test
• bun run lint
• Docker/deployment check, if relevant

bun test is 6581 pass / 1 fail. The single failure is the ContentPage-*.js bundle-size budget (89.1 kB vs 87.9 kB), which reproduces on a clean main with none of this branch applied — pre-existing and unrelated to this change. The sitemap suite itself is 10/10.

Checklist

• Tests cover behavior changes.
• Docs were updated when behavior, config, deployment, or public surfaces changed.
• No compatibility shim was added for old pre-release behavior.
• No secrets, local databases, uploads, or generated artifacts are included.