Skip to content

CASSSIDECAR-423 Fix circle CI pipelines OOM #334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jyothsnakonisa merged 1 commit into from
Apr 9, 2026
Merged

CASSSIDECAR-423 Fix circle CI pipelines OOM #334

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,16 +119,16 @@ jobs:
docker:
- image: cimg/openjdk:11.0
environment:
INTEGRATION_MAX_PARALLEL_FORKS: 3
INTEGRATION_MAX_HEAP_SIZE: "2500M"
INTEGRATION_MAX_PARALLEL_FORKS: 2
INTEGRATION_MAX_HEAP_SIZE: "2000M"
Copy link
Copy Markdown
Contributor

@jyothsnakonisa jyothsnakonisa Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious how is reducing INTEGRATION_MAX_HEAP_SIZE from "2500M" to "2000M" fixes the CI

Copy link
Copy Markdown
Contributor Author

@skoppu22 skoppu22 Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Light weight tests don't need 2500M heap, they can run just fine with lower heap as well. The main fix is --max-workers and reducing the parallelism. Reduced max heap value to ensure we don't go beyond calculations and hit OOM again.

Copy link
Copy Markdown
Contributor

@sarankk sarankk Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, setting INTEGRATION_MAX_PARALLEL_FORKS: 2 makes sense since we are setting --max-workers=2

resource_class: large
steps:
- install_common
- checkout
- attach_workspace:
at: dtest-jars
- run: ./scripts/install-shaded-dtest-jar-local.sh
- run: ./gradlew --no-daemon -PdtestVersion=4.0.16 -Dcassandra.sidecar.versions_to_test="4.0" checkstyleIntegrationTest integrationTestLightWeight --stacktrace
- run: ./gradlew --no-daemon --max-workers=2 -PdtestVersion=4.0.16 -Dcassandra.sidecar.versions_to_test="4.0" checkstyleIntegrationTest integrationTestLightWeight --stacktrace
Copy link
Copy Markdown
Contributor

@sarankk sarankk Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @skoppu22 for the fix, could you also add a comment, why this setting is important, will be good for referring back.


- store_artifacts:
path: build/reports
Expand Down Expand Up @@ -174,16 +174,16 @@ jobs:
docker:
- image: cimg/openjdk:11.0
environment:
INTEGRATION_MAX_PARALLEL_FORKS: 3
INTEGRATION_MAX_HEAP_SIZE: "2500M"
INTEGRATION_MAX_PARALLEL_FORKS: 2
INTEGRATION_MAX_HEAP_SIZE: "2000M"
resource_class: large
steps:
- install_common
- checkout
- attach_workspace:
at: dtest-jars
- run: ./scripts/install-shaded-dtest-jar-local.sh
- run: ./gradlew --no-daemon -PdtestVersion=5.0.3 -Dcassandra.sidecar.versions_to_test="5.0" checkstyleIntegrationTest integrationTestLightWeight --stacktrace
- run: ./gradlew --no-daemon --max-workers=2 -PdtestVersion=5.0.3 -Dcassandra.sidecar.versions_to_test="5.0" checkstyleIntegrationTest integrationTestLightWeight --stacktrace

- store_artifacts:
path: build/reports
Expand Down Expand Up @@ -229,16 +229,16 @@ jobs:
docker:
- image: cimg/openjdk:11.0
environment:
INTEGRATION_MAX_PARALLEL_FORKS: 3
INTEGRATION_MAX_HEAP_SIZE: "2500M"
INTEGRATION_MAX_PARALLEL_FORKS: 2
INTEGRATION_MAX_HEAP_SIZE: "2000M"
resource_class: large
steps:
- install_common
- checkout
- attach_workspace:
at: dtest-jars
- run: ./scripts/install-shaded-dtest-jar-local.sh
- run: ./gradlew --no-daemon -PdtestVersion=5.1 -Dcassandra.sidecar.versions_to_test="5.1" checkstyleIntegrationTest integrationTestLightWeight --stacktrace
- run: ./gradlew --no-daemon --max-workers=2 -PdtestVersion=5.1 -Dcassandra.sidecar.versions_to_test="5.1" checkstyleIntegrationTest integrationTestLightWeight --stacktrace

- store_artifacts:
path: build/reports
Expand Down
2 changes: 2 additions & 0 deletions ...tegration/org/apache/cassandra/sidecar/cluster/driver/SidecarLoadBalancingPolicyTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@

import com.google.common.util.concurrent.Uninterruptibles;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Tag;

import com.datastax.driver.core.DriverUtils;
import com.datastax.driver.core.Host;
Expand All @@ -42,6 +43,7 @@
/**
* A test for the SidecarLoadBalancingPolicy
*/
@Tag("heavy")
Copy link
Copy Markdown
Contributor

@jyothsnakonisa jyothsnakonisa Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you find out that only these tests have to be marked as heavy to resolve CI issues

Copy link
Copy Markdown
Contributor Author

@skoppu22 skoppu22 Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. High node count (5-10) — each in-JVM Cassandra instance holds its own heap, memtables,
    caches, etc.
  2. network=true — enables full network simulation in the dtest framework, substantially
    increasing memory per node (used by Joining, Moving, Replacement tests)
  3. Dynamic node operations — bootstrap/move/replace require ByteBuddy instrumentation and
    maintaining complex state transitions across multiple node lifecycles

When INTEGRATION_MAX_PARALLEL_FORKS=2, multiple 5-10 node tests could run
simultaneously, easily exceeding the CI container's memory. Moving them to the separate
heavyweight job ensures they run in isolation

public class SidecarLoadBalancingPolicyTest extends IntegrationTestBase
{

Expand Down
2 changes: 2 additions & 0 deletions ...tion/org/apache/cassandra/sidecar/routes/tokenrange/BasicMultiDCSingleReplicatedTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import java.util.Collections;

import com.google.common.collect.ImmutableMap;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.extension.ExtendWith;

import io.vertx.junit5.VertxExtension;
Expand All @@ -34,6 +35,7 @@
* Note: Some related test classes are broken down to have a single test case to parallelize test execution and
* therefore limit the instance size required to run the tests from CircleCI as the in-jvm-dtests tests are memory bound
*/
@Tag("heavy")
@ExtendWith(VertxExtension.class)
class BasicMultiDCSingleReplicatedTest extends BaseTokenRangeIntegrationTest
{
Expand Down
2 changes: 2 additions & 0 deletions server/src/test/integration/org/apache/cassandra/sidecar/routes/tokenrange/BasicRf3Test.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import java.util.Collections;

import com.google.common.collect.ImmutableMap;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.extension.ExtendWith;

import io.vertx.junit5.VertxExtension;
Expand All @@ -34,6 +35,7 @@
* Note: Some related test classes are broken down to have a single test case to parallelize test execution and
* therefore limit the instance size required to run the tests from CircleCI as the in-jvm-dtests tests are memory bound
*/
@Tag("heavy")
@ExtendWith(VertxExtension.class)
class BasicRf3Test extends BaseTokenRangeIntegrationTest
{
Expand Down
2 changes: 2 additions & 0 deletions ...est/integration/org/apache/cassandra/sidecar/routes/tokenrange/JoiningTestSingleNode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import java.util.Map;

import com.google.common.collect.Range;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.extension.ExtendWith;

import io.vertx.junit5.VertxExtension;
Expand All @@ -40,6 +41,7 @@
* Note: Some related test classes are broken down to have a single test case to parallelize test execution and
* therefore limit the instance size required to run the tests from CircleCI as the in-jvm-dtests tests are memory bound
*/
@Tag("heavy")
@ExtendWith(VertxExtension.class)
public class JoiningTestSingleNode extends JoiningBaseTest
{
Expand Down
2 changes: 2 additions & 0 deletions server/src/test/integration/org/apache/cassandra/sidecar/routes/tokenrange/MovingTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import java.util.Map;

import com.google.common.collect.Range;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.extension.ExtendWith;

import io.vertx.junit5.VertxExtension;
Expand All @@ -40,6 +41,7 @@
/**
* Node movement scenarios integration tests for token range replica mapping endpoint with the in-jvm dtest framework.
*/
@Tag("heavy")
@ExtendWith(VertxExtension.class)
public class MovingTest extends MovingBaseTest
{
Expand Down
2 changes: 2 additions & 0 deletions .../src/test/integration/org/apache/cassandra/sidecar/routes/tokenrange/ReplacementTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import java.util.function.BiConsumer;

import com.google.common.collect.Range;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.extension.ExtendWith;

import io.vertx.junit5.VertxExtension;
Expand All @@ -46,6 +47,7 @@
/**
* Host replacement scenario integration tests for token range replica mapping endpoint with the in-jvm dtest framework.
*/
@Tag("heavy")
@ExtendWith(VertxExtension.class)
class ReplacementTest extends ReplacementBaseTest
{
Expand Down