CASSSIDECAR-423 Fix circle CI pipelines OOM#334
Conversation
jyothsnakonisa
left a comment
jyothsnakonisa
left a comment
There was a problem hiding this comment.
Looks good overall. just a few comments
| INTEGRATION_MAX_PARALLEL_FORKS: 3 | ||
| INTEGRATION_MAX_HEAP_SIZE: "2500M" | ||
| INTEGRATION_MAX_PARALLEL_FORKS: 2 | ||
| INTEGRATION_MAX_HEAP_SIZE: "2000M" |
There was a problem hiding this comment.
Curious how is reducing INTEGRATION_MAX_HEAP_SIZE from "2500M" to "2000M" fixes the CI
There was a problem hiding this comment.
Light weight tests don't need 2500M heap, they can run just fine with lower heap as well. The main fix is --max-workers and reducing the parallelism. Reduced max heap value to ensure we don't go beyond calculations and hit OOM again.
There was a problem hiding this comment.
Nice, setting INTEGRATION_MAX_PARALLEL_FORKS: 2 makes sense since we are setting --max-workers=2
| /** | ||
| * A test for the SidecarLoadBalancingPolicy | ||
| */ | ||
| @Tag("heavy") |
There was a problem hiding this comment.
How did you find out that only these tests have to be marked as heavy to resolve CI issues
There was a problem hiding this comment.
- High node count (5-10) — each in-JVM Cassandra instance holds its own heap, memtables,
caches, etc. - network=true — enables full network simulation in the dtest framework, substantially
increasing memory per node (used by Joining, Moving, Replacement tests) - Dynamic node operations — bootstrap/move/replace require ByteBuddy instrumentation and
maintaining complex state transitions across multiple node lifecycles
When INTEGRATION_MAX_PARALLEL_FORKS=2, multiple 5-10 node tests could run
simultaneously, easily exceeding the CI container's memory. Moving them to the separate
heavyweight job ensures they run in isolation
sarankk
left a comment
sarankk
left a comment
There was a problem hiding this comment.
+1 Thanks Shailaja for fixing the pipelines. Seeing green CI.
| at: dtest-jars | ||
| - run: ./scripts/install-shaded-dtest-jar-local.sh | ||
| - run: ./gradlew --no-daemon -PdtestVersion=4.0.16 -Dcassandra.sidecar.versions_to_test="4.0" checkstyleIntegrationTest integrationTestLightWeight --stacktrace | ||
| - run: ./gradlew --no-daemon --max-workers=2 -PdtestVersion=4.0.16 -Dcassandra.sidecar.versions_to_test="4.0" checkstyleIntegrationTest integrationTestLightWeight --stacktrace |
There was a problem hiding this comment.
Thanks @skoppu22 for the fix, could you also add a comment, why this setting is important, will be good for referring back.
| INTEGRATION_MAX_PARALLEL_FORKS: 3 | ||
| INTEGRATION_MAX_HEAP_SIZE: "2500M" | ||
| INTEGRATION_MAX_PARALLEL_FORKS: 2 | ||
| INTEGRATION_MAX_HEAP_SIZE: "2000M" |
There was a problem hiding this comment.
Nice, setting INTEGRATION_MAX_PARALLEL_FORKS: 2 makes sense since we are setting --max-workers=2
3 participants
Tune parallelism and heap in Circle CI config. Also mark some tests as heavy