[GOBBLIN-2174] GoT YarnService Integration with DynamicScaling #4077

Blazer-007 · 2024-11-23T18:05:41Z

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-2174

Description

☑️ Here are some details about my PR, including screenshots (if applicable):

In [GOBBLIN-2170] Define Gobblin-on-Temporal dynamic ScalingDirectives with parser and FsScalingDirectiveSource #4068 , Functionality for enabling GoT Dynamic Scaling was introduced.
This PR aims to extend that functionality to GoT YarnService to have dynamically launch new containers at runtime based on ScalingDirectives

Tests

☑️ My PR adds the following unit tests OR does not need testing for this extremely good reason:

Tested E2E using newly created DummyDynamicScalingYarnServiceManager which uses DummyScalingDirectiveSource which returns set of fixed profiles and their set points
YarnServiceTest
- testBaselineWorkerProfileCreatedWithPassedConfigs and testBuildContainerCommand
DynamicScalingYarnServiceManagerTest

Logs from container

2024-11-29 05:22:33 PST [,,] INFO  [DynamicScalingYarnService STARTING] org.apache.gobblin.temporal.yarn.YarnService  - Requesting initial containers using baselineWorkerProfile
2024-11-29 05:22:33 PST [,,] INFO  [DynamicScalingYarnService STARTING] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 2 containers with resource=<memory:8192, vCores:2> and allocation request id = Optional.of(0)
2024-11-29 05:22:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 2 new containers for profile secondProfile having currently 0 containers
2024-11-29 05:22:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 2 containers with resource=<memory:2048, vCores:2> and allocation request id = Optional.of(1)
2024-11-29 05:22:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 3 new containers for profile firstProfile having currently 0 containers
2024-11-29 05:22:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 3 containers with resource=<memory:2048, vCores:2> and allocation request id = Optional.of(2)
...
...
2024-11-29 05:23:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 1 new containers for profile secondProfile having currently 2 containers
2024-11-29 05:23:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 1 containers with resource=<memory:2048, vCores:2> and allocation request id = Optional.of(3)
2024-11-29 05:23:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.DynamicScalingYarnService  - Requesting 2 new containers for profile firstProfile having currently 3 containers
2024-11-29 05:23:59 PST [,,] INFO  [DynamicScalingExecutor] org.apache.gobblin.temporal.yarn.YarnService  - Requesting 2 containers with resource=<memory:2048, vCores:2> and allocation request id = Optional.of(4)

Commits

✔️ My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

…ve`s with parser

Blazer-007 · 2024-11-23T18:07:16Z

Previous PR review done here - phet#1

Also please ignore initial commit history as those got added while rebasing

phet

looks really good. are there any mocks that would help test YarnService?

...l/src/main/java/org/apache/gobblin/temporal/loadgen/dynamic/DummyScalingDirectiveSource.java

gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java

...emporal/src/main/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnServiceManager.java

gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnService.java

Blazer-007 · 2024-11-29T18:02:22Z

are there any mocks that would help test YarnService?

Have added unit test for buildContainerCommand

phet

this YARN integration is coming along nicely and seems very close!

...src/main/java/org/apache/gobblin/temporal/yarn/AbstractDynamicScalingYarnServiceManager.java

...src/main/java/org/apache/gobblin/temporal/yarn/FsSourceDynamicScalingYarnServiceManager.java

...l/src/main/java/org/apache/gobblin/temporal/loadgen/dynamic/DummyScalingDirectiveSource.java

...ral/src/test/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnServiceManagerTest.java

phet · 2024-12-05T00:16:01Z

...ral/src/test/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnServiceManagerTest.java

+    testDynamicScalingYarnServiceManager.startUp();
+    Thread.sleep(5000); // 5 seconds sleep so that GetScalingDirectivesRunnable.run() is called for 5 times
+    testDynamicScalingYarnServiceManager.shutDown();
+    Mockito.verify(mockDynamicScalingYarnService, Mockito.times(3)).reviseWorkforcePlanAndRequestNewContainers(Mockito.anyList());


I'm tempted to verify further what it actually did. e.g. that it called requestContainers. what do you think?

i dont think it will be useful or good to do as we are working with mock of dynamicscalingyarn service and requestContainers is called from internal of reviseWorkforcePlanAndRequestNewContainers.

Keeping in mind added DynamicScalingYarnServiceTest to test reviseWorkforcePlanAndRequestNewContainers

phet · 2024-12-05T00:17:54Z

.../java/org/apache/gobblin/temporal/loadgen/dynamic/DummyDynamicScalingYarnServiceManager.java

+/**
+ * {@link DummyScalingDirectiveSource} based implementation of {@link AbstractDynamicScalingYarnServiceManager}.
+ *  This class is meant to be used for testing purposes only.
+ */


how is this to be used? are you hard coding when making a private build or passing by config? if not the latter yet, would it be worth getting it to work that way?

yes for doing e2e testing for which logs i have added in pr description is passed through config -
gobblin.yarn.app.master.serviceClasses=org.apache.gobblin.temporal.loadgen.dynamic.DummyDynamicScalingYarnServiceManager in prod we can simply replace this with FsScaling one

gobblin-temporal/src/main/java/org/apache/gobblin/temporal/dynamic/WorkerProfile.java

...src/main/java/org/apache/gobblin/temporal/yarn/AbstractDynamicScalingYarnServiceManager.java

gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnService.java

phet · 2024-12-05T19:21:58Z

...src/main/java/org/apache/gobblin/temporal/yarn/FsSourceDynamicScalingYarnServiceManager.java

+  public final static String DYNAMIC_SCALING_DIRECTIVES_DIR = GobblinTemporalConfigurationKeys.DYNAMIC_SCALING_PREFIX + "directives.dir";
+  public final static String DYNAMIC_SCALING_ERRORS_DIR = GobblinTemporalConfigurationKeys.DYNAMIC_SCALING_PREFIX + "errors.dir";


these are fine for now, but I suspect we'll move to basing these directories from the JobStateUtils::getWorkDirRoot

Yes, will move it.
added a todo for reminder

gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java

phet

NOTE: checkstyle Errors:

[ERROR] /home/runner/work/gobblin/gobblin/gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnService.java:23:8: Unused import - org.apache.hadoop.yarn.api.records.Resource. [UnusedImports]

> Task :gobblin-temporal:checkstyleMain FAILED
[ERROR] /home/runner/work/gobblin/gobblin/gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnService.java:26:8: Unused import - com.google.common.base.Optional. [UnusedImports]

UPDATE: sorry, NVM, not an issue... my browser hadn't updated at first to indicate a successful exec happened since.

gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java

...n-temporal/src/test/java/org/apache/gobblin/temporal/yarn/DynamicScalingYarnServiceTest.java

gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java

phet

excellent work here!

codecov-commenter · 2024-12-09T11:29:06Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 38.77%. Comparing base (e5d897e) to head (2e78702).
Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (e5d897e) and HEAD (2e78702). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (e5d897e) HEAD (2e78702)

2 1

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #4077       +/-   ##
=============================================
- Coverage     51.52%   38.77%   -12.76%     
+ Complexity     7563     1598     -5965     
=============================================
  Files          1387      388      -999     
  Lines         52132    16006    -36126     
  Branches       5724     1586     -4138     
=============================================
- Hits          26863     6206    -20657     
+ Misses        22979     9301    -13678     
+ Partials       2290      499     -1791

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

phet and others added 7 commits November 23, 2024 23:13

Define Gobblin-on-TemporalWorkforcePlan and dynamic `ScalingDirecti…

8b8bb5a

…ve`s with parser

change package name from dynscale to dynamic and add a few comments

8eb3d4e

initial changes for yarn integration for dynamic scaling

a788a8f

added worker profile config to be used while starting container

f1c859c

removed getter

7e7113c

addressed few comments

0528f71

adding dummy scaling directive source for testing

8a51f6c

Blazer-007 changed the title ~~[GOBBLIN-] GoT YarnService Integration with DynamicScaling~~ [GOBBLIN-2174] GoT YarnService Integration with DynamicScaling Nov 23, 2024

adding changes that was removed during rebase

64eb6e2

phet reviewed Nov 26, 2024

View reviewed changes

Blazer-007 added 3 commits November 28, 2024 21:32

addressed few review comments

7094023

changed initial container request to use baseline worker profile

9ccf8ae

added abstractdynamicscalingmanger

606349e

Blazer-007 marked this pull request as ready for review November 29, 2024 13:56

Blazer-007 added 2 commits November 29, 2024 19:47

fix typo and added one extra log line while launching initial containers

0c87a75

added unit test for buildcontainercommand

c7a6d1a

Blazer-007 added 4 commits November 29, 2024 23:51

added unit test for baselineworkerprofilecreation

3883837

added unit tests for runnable

c478eb5

refactored tests

15396fc

minor comment update

e511968

phet reviewed Dec 5, 2024

View reviewed changes

addressed few comments

88cd732

phet reviewed Dec 5, 2024

View reviewed changes

Blazer-007 added 4 commits December 6, 2024 10:37

fixed yarn service test

b593ca5

added DynamicScalingYarnServiceTest

fbd4890

removed unused line

ae1fb4c

use renderName while printing profile name

4503325

phet reviewed Dec 9, 2024

View reviewed changes

corrected tests

2e78702

phet approved these changes Dec 9, 2024

View reviewed changes

phet merged commit 7309060 into apache:master Dec 9, 2024
6 checks passed

Blazer-007 deleted the dynamic-scaling-test-branch branch December 9, 2024 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GOBBLIN-2174] GoT YarnService Integration with DynamicScaling #4077

[GOBBLIN-2174] GoT YarnService Integration with DynamicScaling #4077

Blazer-007 commented Nov 23, 2024 •

edited

Loading

Blazer-007 commented Nov 23, 2024

phet left a comment

Blazer-007 commented Nov 29, 2024 •

edited

Loading

phet left a comment

phet Dec 5, 2024

Blazer-007 Dec 6, 2024

phet Dec 5, 2024

Blazer-007 Dec 5, 2024

phet Dec 5, 2024 •

edited

Loading

Blazer-007 Dec 6, 2024

phet left a comment •

edited

Loading

phet left a comment

codecov-commenter commented Dec 9, 2024

		public final static String DYNAMIC_SCALING_DIRECTIVES_DIR = GobblinTemporalConfigurationKeys.DYNAMIC_SCALING_PREFIX + "directives.dir";
		public final static String DYNAMIC_SCALING_ERRORS_DIR = GobblinTemporalConfigurationKeys.DYNAMIC_SCALING_PREFIX + "errors.dir";

[GOBBLIN-2174] GoT YarnService Integration with DynamicScaling #4077

[GOBBLIN-2174] GoT YarnService Integration with DynamicScaling #4077

Conversation

Blazer-007 commented Nov 23, 2024 • edited Loading

JIRA

Description

Tests

Commits

Blazer-007 commented Nov 23, 2024

phet left a comment

Choose a reason for hiding this comment

Blazer-007 commented Nov 29, 2024 • edited Loading

phet left a comment

Choose a reason for hiding this comment

phet Dec 5, 2024

Choose a reason for hiding this comment

Blazer-007 Dec 6, 2024

Choose a reason for hiding this comment

phet Dec 5, 2024

Choose a reason for hiding this comment

Blazer-007 Dec 5, 2024

Choose a reason for hiding this comment

phet Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Blazer-007 Dec 6, 2024

Choose a reason for hiding this comment

phet left a comment • edited Loading

Choose a reason for hiding this comment

phet left a comment

Choose a reason for hiding this comment

codecov-commenter commented Dec 9, 2024

Codecov Report

Blazer-007 commented Nov 23, 2024 •

edited

Loading

Blazer-007 commented Nov 29, 2024 •

edited

Loading

phet Dec 5, 2024 •

edited

Loading

phet left a comment •

edited

Loading