DSM optimizations - major refactoring to get rid of LinkedHashMap #9151
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 11 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.048 s) : 0, 1048085
Total [baseline] (10.69 s) : 0, 10690004
Agent [candidate] (1.041 s) : 0, 1041398
Total [candidate] (10.651 s) : 0, 10651337
section appsec
Agent [baseline] (1.219 s) : 0, 1218592
Total [baseline] (10.787 s) : 0, 10787396
Agent [candidate] (1.218 s) : 0, 1218469
Total [candidate] (10.738 s) : 0, 10738320
section iast
Agent [baseline] (1.181 s) : 0, 1181369
Total [baseline] (10.867 s) : 0, 10866954
Agent [candidate] (1.192 s) : 0, 1192040
Total [candidate] (10.939 s) : 0, 10938541
section profiling
Agent [baseline] (1.183 s) : 0, 1183476
Total [baseline] (10.802 s) : 0, 10801814
Agent [candidate] (1.19 s) : 0, 1190148
Total [candidate] (10.893 s) : 0, 10892675
gantt
title petclinic - break down per module: candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.434 ms) : 0, 1434
crashtracking [candidate] (1.426 ms) : 0, 1426
BytebuddyAgent [baseline] (736.57 ms) : 0, 736570
BytebuddyAgent [candidate] (731.492 ms) : 0, 731492
GlobalTracer [baseline] (243.22 ms) : 0, 243220
GlobalTracer [candidate] (242.14 ms) : 0, 242140
AppSec [baseline] (30.701 ms) : 0, 30701
AppSec [candidate] (30.392 ms) : 0, 30392
Debugger [baseline] (6.095 ms) : 0, 6095
Debugger [candidate] (6.011 ms) : 0, 6011
Remote Config [baseline] (656.036 µs) : 0, 656
Remote Config [candidate] (650.548 µs) : 0, 651
Telemetry [baseline] (8.279 ms) : 0, 8279
Telemetry [candidate] (8.199 ms) : 0, 8199
section appsec
crashtracking [baseline] (1.43 ms) : 0, 1430
crashtracking [candidate] (1.428 ms) : 0, 1428
BytebuddyAgent [baseline] (752.445 ms) : 0, 752445
BytebuddyAgent [candidate] (751.732 ms) : 0, 751732
GlobalTracer [baseline] (234.06 ms) : 0, 234060
GlobalTracer [candidate] (235.128 ms) : 0, 235128
AppSec [baseline] (166.946 ms) : 0, 166946
AppSec [candidate] (168.773 ms) : 0, 168773
Debugger [baseline] (7.94 ms) : 0, 7940
Debugger [candidate] (7.875 ms) : 0, 7875
Remote Config [baseline] (616.15 µs) : 0, 616
Remote Config [candidate] (594.11 µs) : 0, 594
Telemetry [baseline] (10.245 ms) : 0, 10245
Telemetry [candidate] (8.084 ms) : 0, 8084
IAST [baseline] (23.627 ms) : 0, 23627
IAST [candidate] (23.563 ms) : 0, 23563
section iast
crashtracking [baseline] (1.441 ms) : 0, 1441
crashtracking [candidate] (1.456 ms) : 0, 1456
BytebuddyAgent [baseline] (853.906 ms) : 0, 853906
BytebuddyAgent [candidate] (861.612 ms) : 0, 861612
GlobalTracer [baseline] (232.044 ms) : 0, 232044
GlobalTracer [candidate] (234.477 ms) : 0, 234477
AppSec [baseline] (26.852 ms) : 0, 26852
AppSec [candidate] (30.005 ms) : 0, 30005
Debugger [baseline] (8.31 ms) : 0, 8310
Debugger [candidate] (6.731 ms) : 0, 6731
Remote Config [baseline] (589.781 µs) : 0, 590
Remote Config [candidate] (611.876 µs) : 0, 612
Telemetry [baseline] (7.972 ms) : 0, 7972
Telemetry [candidate] (8.111 ms) : 0, 8111
IAST [baseline] (29.217 ms) : 0, 29217
IAST [candidate] (27.821 ms) : 0, 27821
section profiling
crashtracking [baseline] (1.405 ms) : 0, 1405
crashtracking [candidate] (1.404 ms) : 0, 1404
BytebuddyAgent [baseline] (758.78 ms) : 0, 758780
BytebuddyAgent [candidate] (760.923 ms) : 0, 760923
GlobalTracer [baseline] (220.432 ms) : 0, 220432
GlobalTracer [candidate] (222.248 ms) : 0, 222248
AppSec [baseline] (30.338 ms) : 0, 30338
AppSec [candidate] (30.598 ms) : 0, 30598
Debugger [baseline] (6.967 ms) : 0, 6967
Debugger [candidate] (6.383 ms) : 0, 6383
Remote Config [baseline] (661.052 µs) : 0, 661
Remote Config [candidate] (681.282 µs) : 0, 681
Telemetry [baseline] (14.06 ms) : 0, 14060
Telemetry [candidate] (13.599 ms) : 0, 13599
ProfilingAgent [baseline] (102.455 ms) : 0, 102455
ProfilingAgent [candidate] (105.617 ms) : 0, 105617
Profiling [baseline] (102.478 ms) : 0, 102478
Profiling [candidate] (105.64 ms) : 0, 105640
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044589
Total [baseline] (8.643 s) : 0, 8643322
Agent [candidate] (1.04 s) : 0, 1040393
Total [candidate] (8.611 s) : 0, 8611486
section iast
Agent [baseline] (1.175 s) : 0, 1174517
Total [baseline] (9.287 s) : 0, 9286888
Agent [candidate] (1.186 s) : 0, 1185909
Total [candidate] (9.318 s) : 0, 9318494
gantt
title insecure-bank - break down per module: candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.434 ms) : 0, 1434
crashtracking [candidate] (1.422 ms) : 0, 1422
BytebuddyAgent [baseline] (732.937 ms) : 0, 732937
BytebuddyAgent [candidate] (730.619 ms) : 0, 730619
GlobalTracer [baseline] (242.741 ms) : 0, 242741
GlobalTracer [candidate] (241.821 ms) : 0, 241821
AppSec [baseline] (30.745 ms) : 0, 30745
AppSec [candidate] (30.57 ms) : 0, 30570
Debugger [baseline] (6.014 ms) : 0, 6014
Debugger [candidate] (6.006 ms) : 0, 6006
Remote Config [baseline] (669.055 µs) : 0, 669
Remote Config [candidate] (651.193 µs) : 0, 651
Telemetry [baseline] (9.024 ms) : 0, 9024
Telemetry [candidate] (8.236 ms) : 0, 8236
section iast
crashtracking [baseline] (1.424 ms) : 0, 1424
crashtracking [candidate] (1.458 ms) : 0, 1458
BytebuddyAgent [baseline] (848.224 ms) : 0, 848224
BytebuddyAgent [candidate] (857.84 ms) : 0, 857840
GlobalTracer [baseline] (231.523 ms) : 0, 231523
GlobalTracer [candidate] (232.677 ms) : 0, 232677
AppSec [baseline] (26.597 ms) : 0, 26597
AppSec [candidate] (28.42 ms) : 0, 28420
Debugger [baseline] (8.434 ms) : 0, 8434
Debugger [candidate] (5.799 ms) : 0, 5799
Remote Config [baseline] (581.359 µs) : 0, 581
Remote Config [candidate] (589.638 µs) : 0, 590
Telemetry [baseline] (7.926 ms) : 0, 7926
Telemetry [candidate] (7.923 ms) : 0, 7923
IAST [baseline] (28.936 ms) : 0, 28936
IAST [candidate] (30.059 ms) : 0, 30059
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 2 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section baseline
no_agent (4.363 ms) : 4313, 4413
. : milestone, 4363,
iast (9.494 ms) : 9329, 9658
. : milestone, 9494,
iast_FULL (13.923 ms) : 13654, 14191
. : milestone, 13923,
iast_GLOBAL (10.181 ms) : 9989, 10372
. : milestone, 10181,
profiling (8.779 ms) : 8632, 8926
. : milestone, 8779,
tracing (7.493 ms) : 7380, 7607
. : milestone, 7493,
section candidate
no_agent (4.389 ms) : 4340, 4439
. : milestone, 4389,
iast (9.323 ms) : 9170, 9475
. : milestone, 9323,
iast_FULL (14.366 ms) : 14080, 14652
. : milestone, 14366,
iast_GLOBAL (10.254 ms) : 10070, 10439
. : milestone, 10254,
profiling (8.744 ms) : 8594, 8895
. : milestone, 8744,
tracing (7.502 ms) : 7389, 7616
. : milestone, 7502,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section baseline
no_agent (35.898 ms) : 35613, 36184
. : milestone, 35898,
appsec (47.49 ms) : 47081, 47898
. : milestone, 47490,
code_origins (45.431 ms) : 45041, 45821
. : milestone, 45431,
iast (44.961 ms) : 44571, 45352
. : milestone, 44961,
profiling (47.165 ms) : 46728, 47602
. : milestone, 47165,
tracing (45.106 ms) : 44722, 45491
. : milestone, 45106,
section candidate
no_agent (36.963 ms) : 36660, 37266
. : milestone, 36963,
appsec (48.684 ms) : 48230, 49138
. : milestone, 48684,
code_origins (46.166 ms) : 45777, 46555
. : milestone, 46166,
iast (44.049 ms) : 43661, 44437
. : milestone, 44049,
profiling (48.716 ms) : 48278, 49154
. : milestone, 48716,
tracing (43.916 ms) : 43544, 44287
. : milestone, 43916,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section baseline
no_agent (1.482 ms) : 1470, 1493
. : milestone, 1482,
appsec (3.674 ms) : 3457, 3892
. : milestone, 3674,
iast (2.216 ms) : 2152, 2279
. : milestone, 2216,
iast_GLOBAL (2.262 ms) : 2198, 2326
. : milestone, 2262,
profiling (2.07 ms) : 2018, 2122
. : milestone, 2070,
tracing (2.026 ms) : 1978, 2075
. : milestone, 2026,
section candidate
no_agent (1.477 ms) : 1466, 1489
. : milestone, 1477,
appsec (3.676 ms) : 3459, 3893
. : milestone, 3676,
iast (2.207 ms) : 2144, 2270
. : milestone, 2207,
iast_GLOBAL (2.256 ms) : 2192, 2319
. : milestone, 2256,
profiling (2.054 ms) : 2003, 2106
. : milestone, 2054,
tracing (2.035 ms) : 1985, 2084
. : milestone, 2035,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~929749978d, baseline=1.52.0-SNAPSHOT~13efd646bc
dateFormat X
axisFormat %s
section baseline
no_agent (14.973 s) : 14973000, 14973000
. : milestone, 14973000,
appsec (15.093 s) : 15093000, 15093000
. : milestone, 15093000,
iast (18.371 s) : 18371000, 18371000
. : milestone, 18371000,
iast_GLOBAL (17.797 s) : 17797000, 17797000
. : milestone, 17797000,
profiling (15.243 s) : 15243000, 15243000
. : milestone, 15243000,
tracing (14.78 s) : 14780000, 14780000
. : milestone, 14780000,
section candidate
no_agent (15.458 s) : 15458000, 15458000
. : milestone, 15458000,
appsec (15.018 s) : 15018000, 15018000
. : milestone, 15018000,
iast (18.483 s) : 18483000, 18483000
. : milestone, 18483000,
iast_GLOBAL (17.893 s) : 17893000, 17893000
. : milestone, 17893000,
profiling (16.044 s) : 16044000, 16044000
. : milestone, 16044000,
tracing (14.949 s) : 14949000, 14949000
. : milestone, 14949000,
|
Kafka / producer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
Kafka / consumer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
dougqh
reviewed
Jul 14, 2025
| result.put(DIRECTION_TAG, DIRECTION_OUT); | ||
| result.put(TYPE_TAG, "grpc"); | ||
| return DataStreamsContext.fromTags(result); | ||
| return DataStreamsContext.fromTags( |
There was a problem hiding this comment.
I would avoid builder idioms in the critical path. That's still extra allocation that we really don't need.
dougqh
reviewed
Jul 14, 2025
|
|
||
| public DataStreamsTags(DataStreamsTagsBuilder builder) { | ||
| this.builder = builder; | ||
| this.size = |
There was a problem hiding this comment.
This is introducing a capturing lambda. I think a regular loop would probably be better here.
AlexeyKuznetsov-DD
requested changes
Jul 14, 2025
Comment on lines
93
to
102
| processor.process(BUS_TAG, this.builder.bus); | ||
| count += 1; | ||
| } | ||
|
|
||
| if (this.builder.direction == Direction.Inbound) { | ||
| count += 1; | ||
| processor.process(DIRECTION_TAG, DIRECTION_IN); | ||
| } else if (this.builder.direction == Direction.Outbound) { | ||
| count += 1; | ||
| processor.process(DIRECTION_TAG, DIRECTION_OUT); |
There was a problem hiding this comment.
Suggestion: Just a matter of taste, but probably count++; Would be a better option (here and other similar places)?
Comment on lines
286
to
299
| + "," | ||
| + ", kafkaClusterId='" | ||
| + this.builder.kafkaClusterId | ||
| + "," | ||
| + ", partition='" | ||
| + this.builder.partition | ||
| + "," | ||
| + ", hash=" | ||
| + hash | ||
| + "," | ||
| + ", aggregationHash=" | ||
| + aggregationHash | ||
| + "," | ||
| + ", size=" |
There was a problem hiding this comment.
I think this code will produce String with TWO commas. Is that expected?
There was a problem hiding this comment.
Yes, you are right... AI autosuggest :D
There was a problem hiding this comment.
Nice :), just FYI IDEA has toString() method generation too,
dougqh
reviewed
Jul 14, 2025
| result.put(TYPE_TAG, "grpc"); | ||
| return result; | ||
| private static final DataStreamsTags createServerPathwaySortedTags() { | ||
| return new DataStreamsTagsBuilder() |
There was a problem hiding this comment.
I think we can probably still keep these pretty clean without needing a builder. I've put suggestions throughout...
- createGrpcInbound() <-- bonus, that's cacheable
dougqh
reviewed
Jul 14, 2025
| DataStreamsContext dsmContext = DataStreamsContext.fromTags(getTags(eventBusName)); | ||
| DataStreamsTags tags = | ||
| new DataStreamsTagsBuilder() | ||
| .withDirection(DataStreamsTags.Direction.Outbound) |
There was a problem hiding this comment.
createBusOutbound(eventBusName)
dougqh
reviewed
Jul 14, 2025
| sortedTags.put(TagsProcessor.TOPIC_TAG, bucket); | ||
| sortedTags.put(TagsProcessor.TYPE_TAG, "s3"); | ||
| DataStreamsTags tags = | ||
| new DataStreamsTagsBuilder() |
There was a problem hiding this comment.
createS3Outbound(bucket, key)
dougqh
reviewed
Jul 14, 2025
| sortedTags.put(TYPE_TAG, "kinesis"); | ||
| DataStreamsTags tags = | ||
| new DataStreamsTagsBuilder() | ||
| .withType("kinesis") |
There was a problem hiding this comment.
createKinesisInbound(streamArn)
dougqh
reviewed
Jul 14, 2025
|
|
||
| return sortedTags; | ||
| private DataStreamsTags getTags(String snsTopicName) { | ||
| return new DataStreamsTagsBuilder() |
There was a problem hiding this comment.
createSnsOutbound(snsTopicName)
There was a problem hiding this comment.
If you're in refactoring mode, can you have a look (made in another PR if it's easier) to complete the context refactoring?
In short, PathwayContext should not be stored in span data model (DDSpanContext).
Instead, it must be stored with Context as DataStreamsContext and AgentSpan.
You may create a second Context entry if the DataStreamsContext holds different kind of data (or have a different lifecycle) than what PathwayContext should hold.
I already done some refactoring (adding DataStreamsContext, creating a dedicated DSM propagator) but it miss the last step of moving away PathwayContext from span.
Would that be something you could have a look at? Would you like us to pair on this task? The end goal would be to decouple Tracing from DSM. So you can save even more resources.
dougqh
reviewed
Jul 15, 2025
| @@ -89,7 +85,9 @@ private String getTraceContextToInject( | |||
| // Inject context | |||
| datadog.context.Context context = span; | |||
| if (traceConfig().isDataStreamsEnabled()) { | |||
| DataStreamsContext dsmContext = DataStreamsContext.fromTags(getTags(eventBusName)); | |||
| DataStreamsTags tags = | |||
| DataStreamsTags.createWithBus("bus", DataStreamsTags.Direction.Outbound, eventBusName); | |||
There was a problem hiding this comment.
Why pass "bus" as argument to withBus helper? Is there a case where the value isn't "bus"?
There was a problem hiding this comment.
Yes, the type is always bus, no need to pass it here.
dougqh
reviewed
Jul 15, 2025
| private long completeHash; | ||
| private int nonNullSize; | ||
|
|
||
| // hash tags |
There was a problem hiding this comment.
These should probably be final
Right now, strictly speaking this probably isn't thread safe
dougqh
reviewed
Jul 15, 2025
| kafkaClusterId != null ? KAFKA_CLUSTER_ID_TAG + ":" + kafkaClusterId : null; | ||
| this.partition = partition != null ? PARTITION_TAG + ":" + partition : null; | ||
|
|
||
| // hashable tags are 0-4 |
There was a problem hiding this comment.
Nice, that should unroll and inline nicely
I'm definitely for refactoring the context propagation logic, but I don't want to bundle the changes into a single PR. |
kr-igor
added
type: enhancement
dougqh
approved these changes
Jul 18, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change is aimed to reduce DSM overhead by using a lightweight object instead of linked maps.
Introduced
DataStreamsTagswhich are now used instead ofLinkedHashMap.Updated all integrations and tests.
My tests show ~20% throughput increase with a significant latency decrease without increasing CPU load. (left side is v1.51.1, right side is this branch)
