Flaky E2E TestModelBoosterSelfHealing

[FAUST-BENCHOU]

https://github.com/volcano-sh/kthena/actions/runs/28858012483/job/85589288712?pr=1314
https://github.com/volcano-sh/kthena/actions/runs/28913547764/job/85775704598?pr=1311

=== RUN   TestModelBoosterSelfHealing
    model_booster_test.go:244: Waiting for webhook server to accept requests
    model_booster_test.go:256: Created Model CR: kthena-e2e-controller-eoyam/self-healing-test-model
    model_booster_test.go:259: 
        	Error Trace:	/home/runner/work/kthena/kthena/test/e2e/controller-manager/model_booster_test.go:259
        	Error:      	Condition never satisfied
        	Test:       	TestModelBoosterSelfHealing
        	Messages:   	Model did not become Active
--- FAIL: TestModelBoosterSelfHealing (300.02s)

[avinxshKD]

@FAUST-BENCHOU can I look at it??

[avinxshKD]

/assign

[katara-Jayprakash]

It's a bit strange because I didn't run into this issue when I raised the PR: #1149.

Also, if the concern is that the E2E tests are flaky, could you explain why the changes are being made in pkg/model-booster-controller/controller/model_booster_controller.go?

@avinxshKD

[avinxshKD]

#1149 passing makes this less obvious i feel. I looked here because the timeout is on the ModelBooster Active condition, which is set in this controller after checking ModelServing availability. should verify the failed run conditions/events before assuming this is the root cause. @katara-Jayprakash

[avinxshKD]

checked the artifact. Looks like the self-healing pod was waiting on previous test-model cleanup/300s termination, so this probably isn’t a controller reconcile issue.

[katara-Jayprakash]

[update]
TestModelBoosterSelfHealing
https://github.com/volcano-sh/kthena/actions/runs/28990791465/job/86029868231?pr=1326#step:7:9799
TestModelServingPartitionScaleDown
https://github.com/volcano-sh/kthena/actions/runs/28990791465/job/86029868231?pr=1326#step:7:9972
these two e2e test are failing
if no one is working then i would love to take them

[hzxuzhonghu]

Please check the logs from the failure tests

2s
Node 20 is being deprecated. This workflow is running with Node 24 by default. If you need to temporarily use Node 20, you can set the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true environment variable. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Run actions/upload-artifact@v4
(node:99561) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
With the provided path, there will be 50 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
(node:99561) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
Uploaded bytes 1305909
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact zip is ecd937b199df56735e0070d20754c598925b2485864e4004dc20d7b2093b7a61
Finalizing artifact upload
Artifact e2e-logs-controller-manager-28990791465.zip successfully finalized. Artifact ID 8188133032
Artifact e2e-logs-controller-manager-28990791465 has been successfully uploaded! Final size is 1305909 bytes. Artifact ID is 8188133032
Artifact download URL: https://github.com/volcano-sh/kthena/actions/runs/28990791465/artifacts/8188133032

[avinxshKD]

Please check the logs from the failure tests

2s
Node 20 is being deprecated. This workflow is running with Node 24 by default. If you need to temporarily use Node 20, you can set the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true environment variable. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Run actions/upload-artifact@v4
(node:99561) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
With the provided path, there will be 50 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
(node:99561) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
Uploaded bytes 1305909
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact zip is ecd937b199df56735e0070d20754c598925b2485864e4004dc20d7b2093b7a61
Finalizing artifact upload
Artifact e2e-logs-controller-manager-28990791465.zip successfully finalized. Artifact ID 8188133032
Artifact e2e-logs-controller-manager-28990791465 has been successfully uploaded! Final size is 1305909 bytes. Artifact ID is 8188133032
Artifact download URL: https://github.com/volcano-sh/kthena/actions/runs/28990791465/artifacts/8188133032

checked this artifact too. Self-healing still looks blocked by the previous pod cleanup before the new pod starts. i don’t see that pod startup stall for partition scale down from node logs

[verma-garv]

[update] TestModelBoosterSelfHealing https://github.com/volcano-sh/kthena/actions/runs/28990791465/job/86029868231?pr=1326#step:7:9799 TestModelServingPartitionScaleDown https://github.com/volcano-sh/kthena/actions/runs/28990791465/job/86029868231?pr=1326#step:7:9972 these two e2e test are failing if no one is working then i would love to take them

hey @katara-Jayprakash, i would like to work on this.

[katara-Jayprakash]

sure!