-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate CI to deva #33587
base: main
Are you sure you want to change the base?
Migrate CI to deva #33587
Conversation
7d9d6bd
to
e6cbe38
Compare
Uncompressed package size comparisonComparison with ancestor Diff per package
Decision✅ Passed |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 18800a4 Optimization Goals: ✅ No significant changes detected
|
perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
---|---|---|---|---|---|---|
➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +1.11 | [+0.23, +1.98] | 1 | Logs |
➖ | quality_gate_logs | % cpu utilization | +1.04 | [-1.98, +4.05] | 1 | Logs |
➖ | file_to_blackhole_1000ms_latency | egress throughput | +0.31 | [-0.46, +1.08] | 1 | Logs |
➖ | quality_gate_idle | memory utilization | +0.16 | [+0.10, +0.22] | 1 | Logs bounds checks dashboard |
➖ | quality_gate_idle_all_features | memory utilization | +0.04 | [-0.01, +0.09] | 1 | Logs bounds checks dashboard |
➖ | file_to_blackhole_0ms_latency | egress throughput | +0.02 | [-0.80, +0.84] | 1 | Logs |
➖ | file_to_blackhole_500ms_latency | egress throughput | +0.02 | [-0.77, +0.80] | 1 | Logs |
➖ | file_to_blackhole_0ms_latency_http1 | egress throughput | +0.01 | [-0.82, +0.84] | 1 | Logs |
➖ | file_to_blackhole_0ms_latency_http2 | egress throughput | +0.01 | [-0.79, +0.80] | 1 | Logs |
➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.01 | [-0.02, +0.03] | 1 | Logs |
➖ | file_to_blackhole_100ms_latency | egress throughput | +0.00 | [-0.65, +0.66] | 1 | Logs |
➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | -0.00 | [-0.47, +0.47] | 1 | Logs |
➖ | uds_dogstatsd_to_api | ingress throughput | -0.01 | [-0.30, +0.28] | 1 | Logs |
➖ | file_to_blackhole_300ms_latency | egress throughput | -0.01 | [-0.64, +0.61] | 1 | Logs |
➖ | file_tree | memory utilization | -0.18 | [-0.24, -0.13] | 1 | Logs |
➖ | tcp_syslog_to_blackhole | ingress throughput | -0.79 | [-0.85, -0.73] | 1 | Logs |
Bounds Checks: ✅ Passed
perf | experiment | bounds_check_name | replicates_passed | links |
---|---|---|---|---|
✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http1 | lost_bytes | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http1 | memory_usage | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http2 | lost_bytes | 10/10 | |
✅ | file_to_blackhole_0ms_latency_http2 | memory_usage | 10/10 | |
✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_1000ms_latency_linear_load | memory_usage | 10/10 | |
✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_300ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 | |
✅ | file_to_blackhole_500ms_latency | lost_bytes | 10/10 | |
✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
✅ | quality_gate_idle | intake_connections | 10/10 | bounds checks dashboard |
✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
✅ | quality_gate_idle_all_features | intake_connections | 10/10 | bounds checks dashboard |
✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
✅ | quality_gate_logs | intake_connections | 10/10 | |
✅ | quality_gate_logs | lost_bytes | 10/10 | |
✅ | quality_gate_logs | memory_usage | 10/10 |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
e6cbe38
to
ed554f2
Compare
d28f562
to
a16a8ec
Compare
@ofek Can you convert the PR into a draft while it has |
0521b65
to
0c16f73
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few remarks:
- I asked the question with a specific comment: if the syntax is now
deva inv
instead ofinv
I think there are some occurrences where it was justdeva
instead ofinv
, and we should fix them to be fully uniform - I tried to look for places where we used
invoke
instead ofinv
and gave you links. Please check if I was right or wrong, thanks in advance.
/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable DEPLOY_AGENT=true --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on |
View all feedbacks in Devflow UI.
Started pipeline #56082383 |
0c16f73
to
c9d2338
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you have a chance to test the workflows to make sure they still work ?
The same change in buildimages broke a similar workflow...
4d24738
to
1ab9efd
Compare
1ab9efd
to
3c681e3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might also be worth making the change for the https://github.com/DataDog/datadog-agent/blob/main/skaffold.yaml file?
I had a quick look and the first build using deva inv
leads to this failure:
DEBU[0009] Running command: [sh -c docker exec datadog_agent_devcontainer bash -c " deva --verbose inv agent.hacky-dev-image-build --target-image=$IMAGE"] subtask=agent task=Build
Building [agent]...
Target platforms: [linux/arm64]
Using CPython 3.12.8 interpreter at: /usr/bin/python3
Creating virtual environment with seed packages at: /root/.local/share/deva/venvs/legacy
+ pip==25.0.1
Prepared 3 packages in 127ms
error: Failed to install: awscli-1.29.45-py3-none-any.whl (awscli==1.29.45)
Caused by: failed to copy file from /root/.local/share/deva/venvs/legacy/lib/python3.12/site-packages/awscli-1.29.45.data/scripts/aws_zsh_completer.sh to /root/.local/share/deva/venvs/legacy/bin/aws_zsh_completer.sh: No such file or directory (os error 2)
Command failed with exit code 2: ['/usr/local/bin/uv', 'sync', '--frozen', '--no-install-project', '--inexact', '--only-group', 'legacy-tasks']
Build [agent] failed: building custom artifact: exit status 1
Do you know what could be causing this behavior?
Subsequent builds are good. Next run looks like this:
DEBU[0011] Running command: [sh -c docker exec datadog_agent_devcontainer bash -c " deva --verbose inv agent.hacky-dev-image-build --target-image=$IMAGE"] subtask=agent task=Build
Audited 94 packages in 0.06ms
Unable to find image 'gcr.io/datadoghq/agent:7.63.0' locally
7.63.0: Pulling from datadoghq/agent
b32c3a0de16e: Pulling fs layer
b32c3a0de16e: Download complete
b32c3a0de16e: Pull complete
I'm trying to reproduce but the skaffold tasks aren't working on Windows. I'll try to debug that later today but are all subsequent builds successful? |
I did not try multiple times afterwards but I can try, it did appears to work after the first failure though. |
This was the issue #34253 |
What does this PR do?
Motivation
Continuation of DataDog/datadog-agent-buildimages#741
Describe how you validated your changes
rg -. "requirements.+txt"
rg -. "inv "