Require optional admin auth for /metrics endpoint - #1537
Conversation
The /metrics endpoint was unauthenticated, leaking internal label values such as resource paths and TEE types. Require a valid admin JWT (via the configured admin authentication/authorization backend) before serving Prometheus metrics, and map failures to AdminAuthAccess like the other admin-protected endpoints. Relax the regex_acl anchor validation from '^/kbs' to '^/' so that /metrics can be granted to admin roles via allowed_endpoints; existing '^/kbs...$' rules remain valid. Add an integration test covering no token, valid token, DenyAll admin backend, and a restricted ACL that excludes /metrics. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Add http_server.require_admin_auth_metrics (default false) so operators can opt in to protecting /metrics with admin JWT auth instead of it being mandatory. The default preserves unauthenticated metric scraping for existing deployments; enabling it applies the check_admin_access guard added previously. Update the metrics integration test to opt in to the protection, add a test asserting /metrics stays accessible without a token by default, and document the new flag in config.md and metrics.md. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
d950629 to
f814d3b
Compare
|
This was discussed when the metrics feature was first added. Note that neither of thing things in your example are considered secret. They are both conveyed revealed by the KBS protocol itself (unless TLS Is enabled). On the other hand, maybe it's reasonable to add this option. wdyt @Xynnn007 @pmores |
|
Oh. This is a following question for admin module integration. imo, we do not need this in KBS logic, as it would make the core KBS logic complexer. Usually, the API only exposes in intranet (with proper deployment with k8s) thus would not be called by outer clients. If the API needs to be exposed to outer callers in some cases, it's rercommended to use a network gateway in front of KBS. That gate way can control the inbound/outbound network flow and the allowlists. This is about the deployment model, than core logic. Furthurmore, in future some more APIs about HTTP service status/liveness/... should not be covered by admin auth imo. |
|
I don't really have a strong opinion of this as it's really much more about auth than Prometheus. That said, the option doesn't seem likely to break anything, and it seems to bring the metric endpoint to the same level of auth as the admin endpoints, for whatever that's worth. The change isn't overly complex either. On the other hand, extending the set of auth'd endpoints this way does seem excessive. |
|
I am open to it. In theory metrics don't reveal any information, but perhaps better safe than sorry. If something is potentially sensitive, I think we we should provide a way to lock it down without needing any extra tools. |
4 participants
Protect /metrics with admin auth (opt-in)
The
/metricsendpoint was open to anyone who could reach KBS, which leaks internal stuff like resource paths and TEE types in the metric labels. Not great.This adds the option to require a valid admin token for
/metricsso you can lock it down if you want to.What changed
New config flag
http_server.require_admin_auth_metrics(defaultfalse). Set it totrueto require an admin JWT to scrape/metrics:When enabled,
/metricsgoes through the same admin auth/authorization as the other admin endpoints. Scrapers need to send a bearer admin JWT, and their role'sallowed_endpointshas to cover/metrics(e.g.^/(kbs/v0/.*|metrics)$). If the admin backend isDenyAll,/metricsis fully locked.Relaxed the ACL regex check so roles can be granted
/metrics(was only^/kbs...rules before). Existing rules still work.Docs updated (
config.md,admin.md,metrics.md).