[CI] ShardFollowTaskReplicationTests testChangeFollowerHistoryUUID failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #8021 / ubuntu-2004_platform-support-unix
• elasticsearch-pull-request #70242 / part-2
• elasticsearch-pull-request #70194 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:ccr:test" --tests "org.elasticsearch.xpack.ccr.action.ShardFollowTaskReplicationTests.testChangeFollowerHistoryUUID" -Dtests.seed=7FF94D6CE6CB7B86 -Dtests.locale=mn-MN -Dtests.timezone=Pacific/Pohnpei -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: shard [test][0], node[s0], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=43gDJAAAQAClXah7AAAAAA], unassigned_info[[reason=REINITIALIZED], at[2025-05-04T05:37:05.095Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0] is not a primary shard in primary mode

Issue Reasons:

• [main] 3 failures in test testChangeFollowerHistoryUUID (0.9% fail rate in 329 executions)
• [main] 2 failures in step part-2 (1.9% fail rate in 104 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.9% fail rate in 103 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testChangeFollowerHistoryUUID (0.9% fail rate in 329 executions)
• [main] 2 failures in step part-2 (1.9% fail rate in 104 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (1.9% fail rate in 103 executions)

Build Scans:

• elasticsearch-periodic-platform-support #8021 / ubuntu-2004_platform-support-unix
• elasticsearch-pull-request #70242 / part-2
• elasticsearch-pull-request #70194 / part-2

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[kingherc]

Assigning low risk, because according to the last year's failure dashboard, this problem has sporadically re-appeared, with the earliest exactly same failure being this build scan on Mar 28th. I believe it's some test infrastructure issue to solve.