Skip to content

Create configure_tests convention plugin #9859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Create configure_tests convention plugin #9859

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2194059
Create configure-tests convention plugin
sarahchen6 Oct 24, 2025
96f586b
Use lazy providers for property evaluation
sarahchen6 Oct 24, 2025
7ff614b
Use more lazy API configurations
sarahchen6 Oct 24, 2025
be41722
Remove old file references and add missing configuration
sarahchen6 Oct 24, 2025
d85b522
Alphabetize instrumentations
sarahchen6 Oct 27, 2025
afa8ed9
Comment tasks
sarahchen6 Oct 27, 2025
f654867
Add comment for forked test memory limit
sarahchen6 Oct 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions buildSrc/src/main/kotlin/datadog.configure-tests.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
import org.gradle.api.tasks.testing.Test
import org.gradle.api.tasks.testing.junitplatform.JUnitPlatformOptions
import org.gradle.api.services.BuildService
import org.gradle.api.services.BuildServiceParameters
import org.gradle.testing.base.TestingExtension
import org.gradle.api.plugins.jvm.JvmTestSuite
import java.time.Duration
import java.time.temporal.ChronoUnit

fun isTestingInstrumentation(project: Project): Boolean {
return listOf(
"junit-4.10",
"cucumber",
"cucumber-junit-4",
"junit-4.13",
"munit-junit-4",
"junit-5.3",
"junit-5.8",
"cucumber-junit-5",
"spock-junit-5",
"testng-6",
"testng-7",
"karate",
"scalatest",
"selenium",
"weaver"
).contains(project.name)
}

// Need concrete implementation of BuildService in Kotlin
abstract class ForkedTestLimit : BuildService<BuildServiceParameters.None>

val forkedTestLimit = gradle.sharedServices.registerIfAbsent("forkedTestLimit", ForkedTestLimit::class.java) {
maxParallelUsages.set(3)
}

extensions.findByType(TestingExtension::class.java)?.apply {
suites.withType(JvmTestSuite::class.java).configureEach {
// Use JUnit 5 to run tests
useJUnitJupiter()
}
}

// Use lazy providers to avoid evaluating the property until it is needed
val skipTestsProvider = rootProject.providers.gradleProperty("skipTests")
val skipForkedTestsProvider = rootProject.providers.gradleProperty("skipForkedTests")
val skipFlakyTestsProvider = rootProject.providers.gradleProperty("skipFlakyTests")
val runFlakyTestsProvider = rootProject.providers.gradleProperty("runFlakyTests")
val activePartitionProvider = providers.provider {
project.extra.properties["activePartition"] as? Boolean ?: true
}
Comment on lines +47 to +53
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: Maybe, these could end-up in the extension I mentioned earlier. I'm kind if undecided there.


// Go through the Test tasks and configure them
tasks.withType(Test::class.java).configureEach {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick:

Suggested change
tasks.withType(Test::class.java).configureEach {
tasks.withType<Test>().configureEach {

enabled = activePartitionProvider.get()
Comment on lines +51 to +57
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: I wonder if this "trait" or "feature" should be moved over the ci convention plugin ?


// Disable all tests if skipTests property was specified
onlyIf { !skipTestsProvider.isPresent }

// Enable force rerun of tests with -Prerun.tests.${project.name}
outputs.upToDateWhen {
!rootProject.providers.gradleProperty("rerun.tests.${project.name}").isPresent
}

// Avoid executing classes used to test testing frameworks instrumentation
if (isTestingInstrumentation(project)) {
exclude("**/TestSucceed*")
exclude("**/TestFailed*")
exclude("**/TestFailedWithSuccessPercentage*")
exclude("**/TestError*")
exclude("**/TestSkipped*")
exclude("**/TestSkippedClass*")
exclude("**/TestInheritance*", "**/BaseTestInheritance*")
exclude("**/TestFactory*")
exclude("**/TestParameterized*")
exclude("**/TestRepeated*")
exclude("**/TestTemplate*")
exclude("**/TestDisableTestTrace*")
exclude("**/TestAssumption*", "**/TestSuiteSetUpAssumption*")
exclude("**/TestUnskippable*")
exclude("**/TestWithSetup*")
}

// Split up tests that want to run forked in their own separate JVM for generated tasks
if (name.startsWith("forkedTest") || name.endsWith("ForkedTest")) {
setExcludes(emptyList())
setIncludes(listOf("**/*ForkedTest*"))
forkEvery = 1
// Limit the number of concurrent forked tests
usesService(forkedTestLimit)
onlyIf { !skipForkedTestsProvider.isPresent }
} else {
exclude("**/*ForkedTest*")
}

// Set test timeout for 20 minutes. Default job timeout is 1h (configured on CI level).
timeout.set(Duration.of(20, ChronoUnit.MINUTES))
}

tasks.register("allTests") {
dependsOn(providers.provider {
tasks.withType<Test>().filter { testTask ->
!testTask.name.contains("latest", ignoreCase = true) && testTask.name != "traceAgentTest"
}
})
Comment on lines +105 to +109
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: I think the provider is not needed there, but filter here is a faux-ami in this case as it works against the collection (which means the tasks have to be "realized"). Instead to filter lazily a TaskCollection (and any gradle *DomainObject*), the matching method is more adapted to the situation.

Suggested change
dependsOn(providers.provider {
tasks.withType<Test>().filter { testTask ->
!testTask.name.contains("latest", ignoreCase = true) && testTask.name != "traceAgentTest"
}
})
dependsOn(tasks.withType<Test>().matching { testTask ->
!testTask.name.contains("latest", ignoreCase = true) && testTask.name != "traceAgentTest"
})

}

tasks.register("allLatestDepTests") {
dependsOn(providers.provider {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: same as above.

tasks.withType<Test>().filter { testTask ->
testTask.name.contains("latest", ignoreCase = true)
}
})
}

tasks.named("check") {
dependsOn(tasks.withType<Test>())
}
Comment on lines 104 to 126
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be cool to comment each register block for future refernce

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this: afa8ed9?


tasks.withType(Test::class.java).configureEach {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick:

Suggested change
tasks.withType(Test::class.java).configureEach {
tasks.withType<Test>().configureEach {

// Flaky tests management for JUnit 5
if (testFramework is JUnitPlatformOptions) {
val junitPlatform = testFramework as JUnitPlatformOptions
if (skipFlakyTestsProvider.isPresent) {
junitPlatform.excludeTags("flaky")
} else if (runFlakyTestsProvider.isPresent) {
junitPlatform.includeTags("flaky")
}
}

// Flaky tests management for Spock
if (skipFlakyTestsProvider.isPresent) {
jvmArgs("-Drun.flaky.tests=false")
} else if (runFlakyTestsProvider.isPresent) {
jvmArgs("-Drun.flaky.tests=true")
}
}

// tasks.withType(Test).configureEach {
// // https://docs.gradle.com/develocity/flaky-test-detection/
// // https://docs.gradle.com/develocity/gradle-plugin/current/#test_retry
// develocity.testRetry {
// if (providers.environmentVariable("CI").isPresent()) {
// maxRetries = 3
// }
// }
// }
Copy link
Contributor Author

@sarahchen6 sarahchen6 Oct 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how to convert this part to Kotlin... Any help or recommendations would be appreciated!

Since CI is passing, I wonder if we can remove this section, but perhaps these CI runs did not need to leverage test retries whereas other runs will need to.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be kept, the issue is likely to be that you need to acces the develocity types. Usually this involves referring to the plugin without applying it. But since this is develocity, I'm not sure.

2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/armeria/armeria-grpc-0.84/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,6 @@ protobuf {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These files should already be applying this plugin via gradle/java_no_deps.gradle.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the cleanup !

finalizedBy 'latestDepForkedTest'
}
Expand Down
1 change: 0 additions & 1 deletion dd-java-agent/instrumentation/armeria/armeria-jetty-1.24/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ muzzle {
}

apply from: "$rootDir/gradle/java.gradle"
apply from: "$rootDir/gradle/configure_tests.gradle"

addTestSuiteForDir('latestDepTest', 'test')

Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/aws-java/aws-java-sqs-1.0/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ apply from: "$rootDir/gradle/java.gradle"
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

dependencies {
compileOnly group: 'com.amazonaws', name: 'aws-java-sdk-sqs', version: '1.11.0'
compileOnly group: 'com.amazonaws', name: 'amazon-sqs-java-messaging-lib', version: '1.0.0'
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/aws-java/aws-java-sqs-2.0/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ apply from: "$rootDir/gradle/java.gradle"
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

dependencies {
compileOnly group: 'software.amazon.awssdk', name: 'sqs', version: '2.2.0'
compileOnly group: 'com.amazonaws', name: 'amazon-sqs-java-messaging-lib', version: '2.0.0'
Expand Down
1 change: 0 additions & 1 deletion dd-java-agent/instrumentation/jakarta-jms/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ muzzle {
}

apply from: "$rootDir/gradle/java.gradle"
apply from: "$rootDir/gradle/configure_tests.gradle"

repositories {
maven {
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/jms/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,6 @@ repositories {
addTestSuite('latestDepTest')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
3 changes: 0 additions & 3 deletions dd-java-agent/instrumentation/kafka/kafka-clients-0.11/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@ apply from: "$rootDir/gradle/java.gradle"
addTestSuite('latestDepTest')
addTestSuite('iastLatestDepTest3')

apply from: "$rootDir/gradle/configure_tests.gradle"

dependencies {
compileOnly group: 'org.apache.kafka', name: 'kafka-clients', version: '0.11.0.0'
implementation project(':dd-java-agent:instrumentation:kafka:kafka-common')
Expand All @@ -39,7 +37,6 @@ dependencies {
testRuntimeOnly project(':dd-java-agent:instrumentation:reactive-streams')
testImplementation project(':dd-java-agent:agent-iast:iast-test-fixtures')


// IAST testing dependencies
testRuntimeOnly project(':dd-java-agent:instrumentation:iast-instrumenter')
testRuntimeOnly project(':dd-java-agent:instrumentation:java:java-lang:java-lang-1.8')
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/scala-promise/scala-promise-2.10/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,6 @@ apply plugin: 'scala' // Don't use test-with-scala since we want to pick our own
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'forkedTest')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/scala-promise/scala-promise-2.13/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,6 @@ apply plugin: 'scala' // Don't use test-with-scala since we want to pick our own
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'forkedTest')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/spring/spring-jms-3.1/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ apply from: "$rootDir/gradle/java.gradle"
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
1 change: 0 additions & 1 deletion dd-java-agent/instrumentation/spymemcached-2.10/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ muzzle {
}

apply from: "$rootDir/gradle/java.gradle"
apply from: "$rootDir/gradle/configure_tests.gradle"

addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-mysql-client/vertx-mysql-client-3.9/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ muzzle {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-mysql-client/vertx-mysql-client-4.0/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ muzzle {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-mysql-client/vertx-mysql-client-4.4.2/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ muzzle {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-pg-client/vertx-pg-client-4.0/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ muzzle {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-pg-client/vertx-pg-client-4.4.2/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ muzzle {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
1 change: 0 additions & 1 deletion dd-java-agent/instrumentation/vertx/vertx-redis-client-3.9/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ muzzle {
}

apply from: "$rootDir/gradle/java.gradle"
apply from: "$rootDir/gradle/configure_tests.gradle"

addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-rx-3.5/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@ apply from: "$rootDir/gradle/java.gradle"
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-web/vertx-web-3.4/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@ muzzle {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
2 changes: 0 additions & 2 deletions dd-java-agent/instrumentation/vertx/vertx-web/vertx-web-3.5/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@ muzzle {
addTestSuiteForDir('latestDepTest', 'test')
addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

apply from: "$rootDir/gradle/configure_tests.gradle"

tasks.named("latestDepTest", Test) {
finalizedBy 'latestDepForkedTest'
}
Expand Down
Loading