Instrument HTTP requests & responses

[Mirko-von-Leipzig]

Currently remote context is handled via gRPC middleware which does not have access to the http request, only the gRPC metadata. This means that we cannot trace client/server IP or other nice to have http data.

See here for context, but it will likely include switching to a tower layer instead. Consider simplifying the RPC span creation as well, as mentioned here.

Also see the open-telemetry standard on what to consider including as additional metadata.

[sergerad]

Hey @Mirko-von-Leipzig just checking my understanding that the main unlock here would be to leverage an impl like this on the server so as to get HTTP metadata via http::Request type and potentially the body via the generic ReqBody: Send + 'static, (e.g. + Display or + Debug)? TY

[Mirko-von-Leipzig]

@sergerad I haven't dug in too deeply, my current knowledge is mostly based on tonic's Interceptor docs:

If you need more powerful middleware, tower is the recommended approach. You can find examples of how to use tower with tonic here.

Additionally, interceptors is not the recommended way to add logging to your service. For that a tower middleware is more appropriate since it can also act on the response. For example tower-http’s Trace middleware supports gRPC out of the box.

I know tower has built-in grpc-tracing support but that is just some default implementation based on gRPC headers.

We probably need to customise things, in which case I believe the trace is the next level of abstraction. Hopefully this is enough, though maybe we have to drop down to the example you've linked.