I'd like to propose adding a /metrics endpoint to the appview, served on a separate listener, plus a small HTTP middleware that records RED (rate, error, duration) metrics for every request. Today, knotmirror is the only Tangled service that exports metrics. The appview, knot, and spindle are all uninstrumented to my knowledge. I started exporting these metrics on my local server to debug reachability and memory issues.
Starting with the appview, we could add the appview/metrics package and start with standard HTTP measures for each request (duration, # in-flight) broken down by {route, method, status}.
They will be computed by a separate, simpler chi middleware that runs independent of the very stateful appview/middleware/middleware.go handler.
Later on, the knot server can be instrumented.
Label cardinality is bounded since the route label uses the chi route pattern (e.g. /{user}/{repo}/issues/{issue}), not the raw path; unmatched requests can be labeled unmatched.
The metrics can be exposes via a /metrics endpoint on a separate server only accessible to the host by default. I'm not sure what the monitoring situation for the deployment is, but defaulting to local is best and can still be scraped in k8s etc.
And fun fact! The prometheus go client, bsky indigo, and jetstream libraries already export really useful metrics already like: go_gc_duration_seconds, atproto_identity_redisdir_resolve_handle_duration, jetstream_client_events_read, etc.