feat(data-pipeline): Cross-Language Logging Integration #1018
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1018 +/- ##
==========================================
+ Coverage 71.52% 71.55% +0.02%
==========================================
Files 339 341 +2
Lines 50628 50790 +162
==========================================
+ Hits 36214 36345 +131
- Misses 14414 14445 +31
🚀 New features to boost your workflow:
|
BenchmarksComparisonBenchmark execution time: 2025-04-16 13:59:07 Comparing candidate commit e0e5c8a in PR branch Found 0 performance improvements and 10 performance regressions! Performance is the same for 42 metrics, 2 unstable metrics. scenario:credit_card/is_card_number/ 3782-8224-6310-005
scenario:credit_card/is_card_number/ 378282246310005
scenario:credit_card/is_card_number/378282246310005
scenario:credit_card/is_card_number/x371413321323331
scenario:credit_card/is_card_number_no_luhn/x371413321323331
CandidateCandidate benchmark detailsGroup 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Group 9
Group 10
Group 11
Group 12
Group 13
BaselineOmitted due to size. |
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
ganeshnj
changed the title
|
The symbol prefixes I know are already in use are:
Can we name this |
actually this threw me off a few times, currently we have all sort of prefixes depending on the context (crate name vs dir name vs rust type name vs ffi method names) do you mean anything specific like FFI methods or in general? I agree that atleast ffi methods should be |
|
There are conventions for the full name too, but they are less standardized. I was specifically talking about the FFI symbol prefixes with this comment. Within a package, you should be consistent with the FFI casing, types, etc, but across packages, I want to be consistent with And also I would prefer |
crate name: datadog-log and datadog-log-ffi dir name: log and log-ffi prefix: ddog_log_
A lot of cognitive load... but anyway I would stick to what we have already. crate name: datadog-log and datadog-log-ffi |
ekump
reviewed
Apr 15, 2025
ekump
reviewed
Apr 15, 2025
| @@ -0,0 +1,46 @@ | |||
| // Copyright 2025-Present Datadog, Inc. https://www.datadoghq.com/ | |||
There was a problem hiding this comment.
Do you think there is any value in adding something for log-ffi to examples?
There was a problem hiding this comment.
good call, a dumb example would be good for people understand the usage.
There was a problem hiding this comment.
I didn't add its own example but updated the trace exporter which actually emits some logs
➜ ./examples/ffi/build/trace_exporter
ERROR :: Error deserializing trace from request body :: err: "Invalid format: Unable to read array len for trace count"
Operation failed with error: 18, reason: Invalid format: Unable to read array len for trace count
TRACE :: checkout waiting for idle connection: ("http", localhost:8126) ::
TRACE :: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8126)) ::
TRACE :: checkout dropped for ("http", localhost:8126) ::
ERROR :: Error while fetching /info :: err: client error (Connect)
ekump
approved these changes
Apr 16, 2025
ekump
reviewed
Apr 16, 2025
| @@ -0,0 +1,210 @@ | |||
| # RFC: Cross-Language Logging Integration for libdatadog | |||
There was a problem hiding this comment.
Given that we may want to clarify the RFC a bit, do you maybe want to add this in a separate PR? (Or add it now and change it later, up to you)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This pull request includes several changes to add new logging functionality and update dependencies across multiple files. The most important changes include adding new
ddlogandddlog-fficrates, updating dependencies to usetracinginstead oflog.It introduces callback based mechanism to allow lanagues to setup the callback which is called on every logging action, depending on the configured log level.
There are two important APIs introduced
Motivation
Fixes https://datadoghq.atlassian.net/browse/APMSP-1835
This is part of bigger effort to allow language integration access to the logs happening in the libdatadog.
Check the RFC for more details. Also, available internally in Google docs.
Note
This is blocking issue for data-pipeline .NET integration.
Why separate crate?
Holistically speaking, this functionality not only works for data-pipeline but for any integration. Hence, I decided to move to its own crate just like telemetry. This also gives a place to start adding any other additional crate agnostic logging logic.
Additional Notes
Note
I'm still exploring ways to make it a linter rule (or script) so we don't ever introduce a logging statement with formatted string.
Note
One thing I'm not sure, what will be the impact on non data-pipeline products such as profiler which I'm still investigating.
How to test the change?
For now, I used below script to check there is no formatted logging is happening in the dependencies of data-pipeline
The trace exporter example is updated with log usage