HTTP Instrumentation on Quarkus/Netty doesn't work for TLS + HTTP/2

[ahus1]

Describe the bug

When I start Keycloak 21.1.2 or 22.0.0 (which is based on Quarkus and Netty) with TLS and HTTP/2, I don't get http_server_duration_milliseconds* metrics.

When I start it with HTTP only (which will fallback to HTTP/1.1), I get them:

http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.netty-4.1",otel_scope_version="1.27.0-alpha",http_method="GET",http_route="/",http_scheme="http",http_status_code="200",net_host_name="localhost",net_host_port="8080",net_protocol_name="http",net_protocol_version="1.1"} 1.0 1689095194824

Steps to reproduce

• Download Keycloak 22.0.0 and extract - https://www.keycloak.org/archive/downloads-22.0.0.html
• Change to the folder of Keycloak
• Set some common options via environment variables

  export JAVA_OPTS_APPEND="-javaagent:opentelemetry-javaagent_1_27_0.jar -Dotel.javaagent.debug=true"
  export OTEL_METRICS_EXPORTER=prometheus
  export KC_HEALTH_ENABLED=true
  export KC_METRICS_ENABLED=true
  

• Start with HTTP only

  bin/kc.sh build
  eval exec bin/kc.sh start --hostname-strict=false --hostname-strict-https=false --http-enabled=true
  

• Open http://localhost:8080/ to generate some metrics (the will be no metrics unless a page is being loaded)
• Open http://localhost:9464/ to look at the metrics -> http_server_duration_milliseconds_count is present
• Stop Keycloak by Ctrl+C on the CLI
• create TLS key pair
• Start Keycloak with TLS

  export KC_HTTPS_CERTIFICATE_FILE=tls.crt
  export KC_HTTPS_CERTIFICATE_KEY_FILE=tls.key
  bin/kc.sh build
  eval exec bin/kc.sh start --hostname-strict=false --hostname-strict-https=false
  

• Open start page with TLS https://localhost:8443/
• Open http://localhost:9464/ to look at the metrics -> http_server_duration_milliseconds_count is missing

What did you expect to see?

The metrics http_server_* should be visible both for TLS and non-TLS

What did you see instead?

The metrics http_server_* are missing for TLS and are only present in non-TLS

What version are you using?

1.27.0

Environment

$ java --version
openjdk 11.0.16.1 2022-08-12
OpenJDK Runtime Environment Temurin-11.0.16.1+1 (build 11.0.16.1+1)
OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1 (build 11.0.16.1+1, mixed mode)
$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"

Additional context

I'm testing this to do performance tests on Keycloak for the upcoming version of Keycloak. I'm happy to support by testing this / running pre-releases / debugging etc.

[laurit]

Are you making request with HTTP/2 or HTTP/1.1?

[ahus1]

Thanks for pointing towards the protocol. I tested again, and the metrics are not recorded for HTTP/2.
Once I force HTTP/1.1, the metrics are recorded.

# using HTTP/2 - no metrics recorded
curl https://localhost:8443/ -kv

# forcing HTTP/1.1 for TLS - metrics recorded
curl https://localhost:8443/ -kv --http1.1

[ahus1]

Updated issue description and title to reflect HTTP/2 connection

[trask]

Related: #3227