-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AGENTRUN-142] compute instanceID in the server for configcheck command and configcheck flare #34321
[AGENTRUN-142] compute instanceID in the server for configcheck command and configcheck flare #34321
Conversation
Uncompressed package size comparisonComparison with ancestor Diff per package
Decision |
6df3c62
to
2ddd23b
Compare
@@ -75,7 +75,7 @@ func GetClusterChecks(w io.Writer, checkName string) error { | |||
if len(cr.Dangling) > 0 { | |||
fmt.Fprintf(w, "=== %s configurations ===\n", color.RedString("Unassigned")) | |||
for _, c := range cr.Dangling { | |||
flare.PrintConfig(w, c, checkName) | |||
flare.PrintClusterCheckConfig(w, c, checkName) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@clamoriniere For this PR I decided to leave the cluster check v1
untouched. We currently do not scrub the integration config information, which is something that we need to do, but it would require more changes. This PR is already big enough, that I think is best to limit the size.
Once we merge this one and validate everything works as expected we can open a new PR that focus on scrubbing the integration config and precompute the check ID for the cluster check. Once we do that, we can unify the printing function into a single one as we had before
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good, But if the difference is only scrubbing can we use the same function and add an option to decide if the function scrub or not?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The difference is not just the scrubbing, but the actual response from the server. The cluster agent uses a different struct for the response. So the two functions have different signature. I rather not increase the scope of the PR and work on the cluster agent on a separate PR 😄
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: ef7a913 Optimization Goals: ✅ No significant changes detected
|
perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
---|---|---|---|---|---|---|
➖ | file_to_blackhole_1000ms_latency | egress throughput | +0.47 | [-0.35, +1.28] | 1 | Logs |
➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | +0.37 | [-0.10, +0.84] | 1 | Logs |
➖ | quality_gate_logs | % cpu utilization | +0.28 | [-2.73, +3.29] | 1 | Logs |
➖ | file_tree | memory utilization | +0.15 | [+0.09, +0.21] | 1 | Logs |
➖ | file_to_blackhole_500ms_latency | egress throughput | +0.05 | [-0.76, +0.86] | 1 | Logs |
➖ | file_to_blackhole_100ms_latency | egress throughput | +0.01 | [-0.68, +0.71] | 1 | Logs |
➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.29, +0.29] | 1 | Logs |
➖ | file_to_blackhole_0ms_latency_http2 | egress throughput | -0.00 | [-0.81, +0.81] | 1 | Logs |
➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.02, +0.02] | 1 | Logs |
➖ | file_to_blackhole_0ms_latency_http1 | egress throughput | -0.01 | [-0.83, +0.82] | 1 | Logs |
➖ | file_to_blackhole_300ms_latency | egress throughput | -0.01 | [-0.64, +0.62] | 1 | Logs |
➖ | file_to_blackhole_0ms_latency | egress throughput | -0.01 | [-0.80, +0.77] | 1 | Logs |
➖ | quality_gate_idle | memory utilization | -0.21 | [-0.25, -0.17] | 1 | Logs bounds checks dashboard |
➖ | quality_gate_idle_all_features | memory utilization | -0.40 | [-0.46, -0.35] | 1 | Logs bounds checks dashboard |
➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.58 | [-1.44, +0.28] | 1 | Logs |
➖ | tcp_syslog_to_blackhole | ingress throughput | -1.19 | [-1.25, -1.13] | 1 | Logs |
Bounds Checks: ✅ Passed
perf | experiment | bounds_check_name | replicates_passed | links |
---|---|---|---|---|
✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http1 | lost_bytes | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http1 | memory_usage | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http2 | lost_bytes | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http2 | memory_usage | 10/10 | |
✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_1000ms_latency_linear_load | memory_usage | 10/10 | |
✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_300ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_500ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
✅ | quality_gate_idle | intake_connections | 10/10 | bounds checks dashboard |
✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
✅ | quality_gate_idle_all_features | intake_connections | 10/10 | bounds checks dashboard |
✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
✅ | quality_gate_logs | intake_connections | 10/10 | |
✅ | quality_gate_logs | lost_bytes | 10/10 | |
✅ | quality_gate_logs | memory_usage | 10/10 |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
Test changes on VMUse this command from test-infra-definitions to manually test this PR changes on a VM: inv aws.create-vm --pipeline-id=56735063 --os-family=ubuntu Note: This applies to commit 538a197 |
Static quality checks ✅Please find below the results from static quality gates Successful checksInfo
|
/merge |
View all feedbacks in Devflow UI.
This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
The median merge time in
|
…nd and configcheck flare (#34321)
What does this PR do?
Modify the
config-check
response to include the checkID of all the integration config instances.Motivation
Ensure the instance ID from the
status
andconfigcheck
command matches. Prior to this change the instance ID in the configcheck command was wrong. The problem is that we compute the check ID in the client, using scrubbed integration config information. That causes the information to be different from the status commandDescribe how you validated your changes
status
andconfigcheck
command display the same check IDconfig-check.log
from the flare matches the check ID as well.Screenshot provides showing the checks ID for the disk and ntp check between the status and configchec command
Also, uploaded the
configcheck.log
file from a flare I requested locallyconfig-check.log
Possible Drawbacks / Trade-offs
Additional Notes