Fix ServerStatusService.statusWatchers concurrent modification.

[weixiang1862]

Fix ServerStatusService.statusWatchers concurrent modification.

• Add a unit test to verify that the fix works.
• Explain briefly why the bug exists and how to fix it.

• If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
• Update the CHANGES log.

ServerStatusService#registerWatcher called in main thread and ServerStatusService#rebalancedCluster called in
cluster coordinator thread, this may cause concurrent modification of statusWatchers ArrayList when oap server bootstrap.

[wu-sheng]

I think we should wait here rather than return? If we skip this step, then rebalancedTime and the telemetry metric of cluster_rebalanced_time are not being updated. Worse, ServerStatusWatcher implementations would not get any notification.

[weixiang1862]

ServerStatusWatcher will be notified by the method ServerStatusService#bootNow right after bootstrap complete.

I think notice once is enough in bootstrap stage. If you concern about cluster_rebalanced_time telemetry metric, how about move this return judge after cluster_rebalanced_time creating?

[wu-sheng]

OK, thanks for the explanation. Then you need to move bootingStatus.setBooted(true); to the last line of the bootedNow method. Otherwise, the race condition could still happen, right?

I think notice once is enough in bootstrap stage. If you concern about cluster_rebalanced_time telemetry metric, how about move this return judge after cluster_rebalanced_time creating?

That is not correct. Because actually, we don't do rebalancing notifications.