Custom Gradle source set added to main source set is absent from the image

[fthouraud]

Hi 👋

Environment:

• Jib version: 3.3.0
• Build tool: Gradle 7.5.1
• OS: macOS Monterey 12.6

Description of the issue:
The runtimeClasspath of the main source set is incomplete.

Given I have two source sets in the same module (I excluded tests source sets):

• src/main/kotlin
• src/shared/kotlin

And I add the shared source set to the runtime classpath of the main one:

sourceSets {
  val shared by creating
  getByName("main") {
    it.compileClasspath += shared.output
    it.runtimeClasspath += shared.output
  }
}

When I look into the image, I cannot see the classes from the shared source set. But when I run this custom task:

tasks.create("checkMainClasspath") {
  doFirst {
    println(sourceSets.getByName("main").runtimeClasspath.joinToString("\n") { it.toRelativeString(buildDir) })
  }
}

I can see the expected output (truncated for conciseness):

classes/java/main
classes/kotlin/main
resources/main
../../../../../../../.gradle/caches/modules-2/files-2.1/io.insert-koin/koin-ktor/3.2.2/e24cae73a4ad209ed5b57febed16b7a02b7ad423/koin-ktor-3.2.2.jar
[...]
../../../../../../../.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar
classes/java/shared
classes/kotlin/shared
resources/shared

Also, the jib-classpath-file file does not reference this shared source set.

Expected behavior:

I would have expected the image to contain everything from the configuration used.

Nonetheless, it is possible to get around this by using the extraDirectories closure:

extraDirectories {
    paths {
      path {
        setFrom("build/classes/kotlin/shared")
        into = "/app/classes"
        excludes.add("META-INF/*")
      }
    }
  }

Steps to reproduce:

1. Add a custom source set to a Gradle module,
2. Add this source set's output to the main runtime classpath,
3. Add at least a class to this new source set,
4. Run gradle jibBuildTar,
5. Run find build/jib-cache/layers -type f -print -exec tar -tvzf {} \;

The file from the shared source set should be missing.

jib-gradle-plugin Configuration:

jib {
  from {
    image = "eclipse-temurin:17-jre"
    platforms {
      platform {
        architecture = if (System.getProperty("os.arch") == "aarch64") "arm64" else "amd64"
        os = "linux"
      }
    }
  }
  container {
    jvmFlags = listOf("-XX:+UseZGC")
    labels.put("APPLICATION_VERSION", project.version.toString())
  }
}

Additional Information: I didn't try to reproduce it on a minimal Gradle project.

Regards.

[chanseokoh]

Also, the jib-classpath-file file does not reference this shared source set.

Just a minor point: jib-classpath-file is for the classpath used to launch a JVM in the container. It won't talk anything about the structure of your Gradle source project.

[emmileaf]

Hmm, I think the current behavior in Jib includes files from main SourceSet output and the gradle Configuration (defaults to runtimeClasspath, which do not include the shared source set files added to the modified main.runtimeClasspath), which might explain this observation.

As support for custom gradle configurations was added in #3034, perhaps you could try overriding the default runtimeClasspath configuration with something like:

jib {
   ...
   configurationName = "mainRuntimeClasspath"
}
configurations {
   mainRuntimeClasspath.extendsFrom runtimeClasspath
}

In general, Jib does not currently support configuration of custom gradle SourceSets, and the extraDirectories workaround you provided is another recommended approach to include these files. More context on this can be found in an earlier discussion in #1778.

[fthouraud]

@chanseokoh thank you for the precision. I was trying to prove that the shared source set was removed at some point which wasn't very clear.

@emmileaf thank you for the details! Considering Jib figures out the runtime dependencies using the configuration I'm not even sure I could expend unless I provide a JAR.

Do you plan on supporting this kind of structure in the future? It maybe isn't common practice but it feels weird to me that the additional source set is purely ignored.

[emmileaf]

Unfortunately support for this is not on the team’s current roadmap, but we will keep this issue open as a feature request. Community contributions with a design proposal would also be welcomed.

@fthouraud Thank you for outlining the details of this use case and workaround!

[nathanfallet]

Is this related to #4316 (and so to #4396)?