Add causality test with multiple concurrent requests

[iNikem]

Several things happen in this PR, sorry for such a soup. I can split it into several PR if needed...

First, a new "causality test" was introduced to the HttpClientTest. It stresses our instrumentations of http clients in highly concurrent settings. Please read the test comments.

Second, this new test has to be disabled for a lot of http clients. This is very concerning, see #2155

Third, Reactor Netty 0.9 test was converted to the subclass of HttpClientTest.

Fourth, integration with Reactor Netty 1.0 was introduced.

[trask]

First, a new "causality test" was introduced to the HttpClientTest. It stresses our instrumentations of http clients in highly concurrent settings. Please read the test comments.

❤️

Second, this new test has to be disabled for a lot of http clients. This is very concerning, see #2155

😢

Third, Reactor Netty 0.9 test was converted to the subclass of HttpClientTest.

👍

Fourth, integration with Reactor Netty 1.0 was introduced.

🎉

[iNikem]

@mateuszrzeszutek @trask Can you help me? I don't understand why muzzle fails on Reactor Netty with

MUZZLE PASSED NettyInstrumentationModule BUT FAILURE WAS EXPECTED

> Task :instrumentation:reactor-netty-0.9:javaagent:muzzle-AssertFail-io.projectreactor.netty-reactor-netty-1.0.3 FAILED

Why does it fail due to NettyInstrumentationModule?

[mateuszrzeszutek]

Hmm, it looks like for fail directives the instrumentation classloader is missing the reactor-netty instrumentation module:

(example for 1.0)

> Task :instrumentation:reactor-netty-1.0:javaagent:muzzle-AssertFail-io.projectreactor.netty-reactor-netty-0.9.12.RELEASE FAILED
CHECKING NettyInstrumentationModule
MUZZLE PASSED NettyInstrumentationModule BUT FAILURE WAS EXPECTED

> Task :instrumentation:reactor-netty-1.0:javaagent:muzzle-AssertPass-io.projectreactor.netty-reactor-netty-http-1.0.3
CHECKING NettyInstrumentationModule
CHECKING ReactorNettyInstrumentationModule

There must be something wrong with how MuzzlePlugin prepares classloaders

[iNikem]

@mateuszrzeszutek I look at io.opentelemetry.javaagent.tooling.muzzle.matcher.MuzzleGradlePluginUtil#assertInstrumentationMuzzled and I see that

        if (passed && !assertPass) {
          System.err.println(
              "MUZZLE PASSED "
                  + instrumentationModule.getClass().getSimpleName()
                  + " BUT FAILURE WAS EXPECTED");
          throw new RuntimeException("Instrumentation unexpectedly passed Muzzle validation");

Meaning that even if there are 2 modules present, the plugin throws exception and fails the build as soon as any of them pass muzzle. Which does not make sense in my case. I depend on classes from netty instrumentation, thus NettyInstrumentationModule is present on my classpath, but it certainly passes all muzzle checks, regardless of Reactor Netty version.

[mateuszrzeszutek]

Meaning that even if there are 2 modules present, the plugin throws exception and fails the build as soon as any of them pass muzzle.

Oh, that's true! I was so concentrated on the CL & gradle plugin thing that I missed the loop 😅
Yes, I suppose that's where the problem is - this passed && !assertPass if should be outside of the loop.

[iNikem]

I went another way... Let's see...

[johnbley]

LGTM, thanks so much for pushing on this much-needed test type.

[trask]

Really great test addition 👍

Two last suggestions, no worries if you disagree please ignore.