[FA] Add python packages info #34320
Conversation
github-actions
bot
added
medium review
coignetp
added
changelog/no-changelog
qa/done
Test changes on VMUse this command from test-infra-definitions to manually test this PR changes on a VM: inv aws.create-vm --pipeline-id=56848984 --os-family=ubuntuNote: This applies to commit 6a8b0a0 |
Uncompressed package size comparisonComparison with ancestor Diff per package
Decision |
Static quality checks ✅Please find below the results from static quality gates Successful checksInfo
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: cd2d6fa Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | +0.18 | [-0.29, +0.65] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.02 | [-0.84, +0.88] | 1 | Logs |
| ➖ | file_to_blackhole_300ms_latency | egress throughput | +0.02 | [-0.61, +0.65] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.02 | [-0.25, +0.29] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http2 | egress throughput | +0.01 | [-0.78, +0.80] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.01 | [-0.63, +0.64] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.03, +0.02] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http1 | egress throughput | -0.01 | [-0.78, +0.75] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | -0.09 | [-0.87, +0.70] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.14 | [-1.04, +0.76] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.20 | [-0.26, -0.13] | 1 | Logs |
| ➖ | file_tree | memory utilization | -0.24 | [-0.31, -0.18] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.33 | [-1.10, +0.44] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.33 | [-0.38, -0.28] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | -0.54 | [-0.58, -0.50] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | -1.61 | [-4.49, +1.27] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | links |
|---|---|---|---|---|
| ✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency_linear_load | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | |
| ✅ | quality_gate_logs | lost_bytes | 10/10 | |
| ✅ | quality_gate_logs | memory_usage | 10/10 |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
| - `python` - **object**: | ||
| - `python_version` - **string**: The version of the embedded python. | ||
| - `python_packages` - **dict of string to string**: All the python packages installed in the embedded python with their versions. |
There was a problem hiding this comment.
How do you plan to store this in REDAPL ? We can't have a row per python packages since we want a single row per Agent. Are we storing a JSON blob and using SQL JSON operator ?
If not, maybe a dedicated table would be more valuable, WDYT ?
I guess the question is: how do you plan to query this ?
There was a problem hiding this comment.
That would most likely be a dedicated REDAPL table yes, used by fleet services to know what is installed/what is expected to be installed
comp/metadata/inventoryagent/inventoryagentimpl/inventoryagent.go
Outdated
Show resolved
Hide resolved
comp/metadata/inventoryagent/inventoryagentimpl/inventoryagent.go
Outdated
Show resolved
Hide resolved
comp/metadata/inventoryagent/inventoryagentimpl/inventoryagent.go
Outdated
Show resolved
Hide resolved
pkg/collector/python/init.go
Outdated
| args := []string{ | ||
| "-m", | ||
| "pip", | ||
| "freeze", | ||
| } | ||
| pipCmd := exec.Command(pythonBinPath, args...) |
There was a problem hiding this comment.
The result of pip freeze can be massive. We have limitation regarding payload size imposed by the intake. We're also need to be mindful of the size in REDAPL. We currently have ~320 result from pip freeze, 320 rows for each Agent out there is way to much I think.
Could we filter the result to only include python package not ship by default with the Agent ?
There was a problem hiding this comment.
Related to #34320 (comment)
This helps us having a reliable source of what's currently running within the agent, without merging multiple sources and try to correct error of what kind go wrong (what happens for marketplace integration? If a user upgraded a datadog dep manually?)
pkg/collector/python/init.go
Outdated
| // Get the python packages version | ||
| // New packages can be installed, but they're not taken into account until the agent is restarted, | ||
| // so it's safe to cache the versions here. | ||
| cache.Cache.Set(pythonPackagesCacheKey, getPythonPackagesVersion(pythonBinPath), cache.NoExpiration) |
There was a problem hiding this comment.
What if someone start the agent, install a new package and upgrade the Agent ? Unless I'm missing something we would miss the newly installed package, right ?
Is that an issue ?
There was a problem hiding this comment.
If they upgrade the agent there will be a "restart", so we should re-run this part
github-actions
bot
added
long review
pkg/collector/python/init.go
Outdated
| // Get the python packages version | ||
| // New packages can be installed, but they're not taken into account until the agent is restarted, | ||
| // so it's safe to cache the versions here. | ||
| cache.Cache.Set(pythonPackagesCacheKey, getPythonPackagesVersion(pythonBinPath), cache.NoExpiration) |
There was a problem hiding this comment.
As this can take nontrivial amount of time, and doesn't depend on the actual state of the embedded Python interpreter, I think this probably shouldn't be done during init. And with #32611 this won't even happen at startup any more, so you may be missing data in a hard to predict way.
Given that this doesn't depend on the embedded interpreter at all, I would suggest moving this code to pkg/util instead and driving the collection directly from the inventories code, rather than plugging it here and depending on the cache.
pkg/collector/python/init.go
Outdated
| // Or `package @ url` | ||
| // Or `-e package` | ||
| pkgVersion := strings.SplitN(string(line), "==", 2) | ||
| pkgURL := strings.SplitN(string(line), "@", 2) |
There was a problem hiding this comment.
Each string(line) creates a new copy, which is unnecessary.
Since the package and url format includes spaces, would it make sense to use that as a separator? Otherwise we probably should remove whitespace from the fields when we use them.
| pkgURL := strings.SplitN(string(line), "@", 2) | |
| pkgURL := strings.SplitN(line, " @ ", 2) |
pkg/collector/python/init.go
Outdated
| // This is a local package, we don't care about the version | ||
| packageVersions[strings.TrimPrefix(string(line), "-e ")] = "local" | ||
| } else { | ||
| log.Infof("Unable to parse python package version, it won't appear in the metadata payload: %s", line) |
There was a problem hiding this comment.
Is this something that the customer can fix? If not, it should probably be debug level.
| log.Infof("Unable to parse python package version, it won't appear in the metadata payload: %s", line) | |
| log.Infof("Unable to parse python package version, it won't appear in the metadata payload: %q", line) |
What does this PR do?
Add python packages information in the metadata payload.
Motivation
The goal is for the backend to know in which state the agent is running. This will help understanding which custom packages are installed by the customers, as well as providing better support for extras & marketplace installation.
This is a pre-requisite if we want to re-install marketplace integrations through the agent upgrade workflow in fleet-automation.
Example:
{ ... "python": { "python_packages": { "cryptography": " https://agent-int-packages.datadoghq.com/built/cryptography/cryptography-43.0.1-20250218143943-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=a7d7383a7eb5f2f60cb32046dba2b6997d923a57e7e8f518cb19897b88c0966d", "datadog-activemq": "5.0.0", "datadog-activemq-xml": "5.1.0" }, "python_version": "3.12.8" } ... }Describe how you validated your changes
Possible Drawbacks / Trade-offs
Additional Notes