LogsDB: Large stack traces not available in synthetic _source

[felixbarny]

Elasticsearch Version

9.3.0-SNAPSHOT

Installed Plugins

No response

Java Version

bundled

OS Version

n/a

Problem Description

Ingesting a large stack trace into an OTel data stream (LogsDB) succeeds but the _source doesn't contain the attributes.exception.stacktrace.

Steps to Reproduce

The following patch adds a yaml rest test case that reproduces the issue:

Click to expand patch ...

Index: x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml b/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml
--- a/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml	(revision 82df0825778dc24773236299d5ecff6ae48d6786)
+++ b/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml	(date 1761814905928)
@@ -72,6 +72,193 @@
   - match: { hits.hits.0.fields.error\.exception\.message: ["foo"] }
   - match: { hits.hits.0.fields.error\.stack_trace: ["Exception in thread \"main\" java.lang.RuntimeException: Test exception\n at com.example.GenerateTrace.methodB(GenerateTrace.java:13)\n at com.example.GenerateTrace.methodA(GenerateTrace.java:9)\n at com.example.GenerateTrace.main(GenerateTrace.java:5)"] }
 ---
+"Long stacktrace":
+  - do:
+      bulk:
+        index: logs-generic.otel-default
+        refresh: true
+        body:
+          - create: { }
+          - "@timestamp": 2024-07-18T14:48:33.467654000Z
+            resource:
+              attributes:
+                service.name: my-service
+            attributes:
+              exception.stacktrace: |-
+                Exception in thread "main" java.lang.RuntimeException: Simulated deep stacktrace for testing
+                  at com.example.project.moduleA.ClassA.methodA(ClassA.java:10)
+                  at com.example.project.moduleA.ClassA.methodB(ClassA.java:15)
+                  at com.example.project.moduleB.ClassB.methodC(ClassB.java:22)
+                  at com.example.project.moduleB.ClassB.methodD(ClassB.java:27)
+                  at com.example.project.moduleC.ClassC.methodE(ClassC.java:34)
+                  at com.example.project.moduleC.ClassC.methodF(ClassC.java:39)
+                  at com.example.project.moduleD.ClassD.methodG(ClassD.java:46)
+                  at com.example.project.moduleD.ClassD.methodH(ClassD.java:51)
+                  at com.example.project.moduleE.ClassE.methodI(ClassE.java:58)
+                  at com.example.project.moduleE.ClassE.methodJ(ClassE.java:63)
+                  at com.example.project.moduleF.ClassF.methodK(ClassF.java:70)
+                  at com.example.project.moduleF.ClassF.methodL(ClassF.java:75)
+                  at com.example.project.moduleG.ClassG.methodM(ClassG.java:82)
+                  at com.example.project.moduleG.ClassG.methodN(ClassG.java:87)
+                  at com.example.project.moduleH.ClassH.methodO(ClassH.java:94)
+                  at com.example.project.moduleH.ClassH.methodP(ClassH.java:99)
+                  at com.example.project.moduleI.ClassI.methodQ(ClassI.java:106)
+                  at com.example.project.moduleI.ClassI.methodR(ClassI.java:111)
+                  at com.example.project.moduleJ.ClassJ.methodS(ClassJ.java:118)
+                  at com.example.project.moduleJ.ClassJ.methodT(ClassJ.java:123)
+                  at com.example.project.moduleK.ClassK.methodU(ClassK.java:130)
+                  at com.example.project.moduleK.ClassK.methodV(ClassK.java:135)
+                  at com.example.project.moduleL.ClassL.methodW(ClassL.java:142)
+                  at com.example.project.moduleL.ClassL.methodX(ClassL.java:147)
+                  at com.example.project.moduleM.ClassM.methodY(ClassM.java:154)
+                  at com.example.project.moduleM.ClassM.methodZ(ClassM.java:159)
+                  at com.example.project.moduleN.ClassN.methodAA(ClassN.java:166)
+                  at com.example.project.moduleN.ClassN.methodAB(ClassN.java:171)
+                  at com.example.project.moduleO.ClassO.methodAC(ClassO.java:178)
+                  at com.example.project.moduleO.ClassO.methodAD(ClassO.java:183)
+                  at com.example.project.moduleP.ClassP.methodAE(ClassP.java:190)
+                  at com.example.project.moduleP.ClassP.methodAF(ClassP.java:195)
+                  at com.example.project.moduleQ.ClassQ.methodAG(ClassQ.java:202)
+                  at com.example.project.moduleQ.ClassQ.methodAH(ClassQ.java:207)
+                  at com.example.project.moduleR.ClassR.methodAI(ClassR.java:214)
+                  at com.example.project.moduleR.ClassR.methodAJ(ClassR.java:219)
+                  at com.example.project.moduleS.ClassS.methodAK(ClassS.java:226)
+                  at com.example.project.moduleS.ClassS.methodAL(ClassS.java:231)
+                  at com.example.project.moduleT.ClassT.methodAM(ClassT.java:238)
+                  at com.example.project.moduleT.ClassT.methodAN(ClassT.java:243)
+                  at com.example.project.moduleU.ClassU.methodAO(ClassU.java:250)
+                  at com.example.project.moduleU.ClassU.methodAP(ClassU.java:255)
+                  at com.example.project.moduleV.ClassV.methodAQ(ClassV.java:262)
+                  at com.example.project.moduleV.ClassV.methodAR(ClassV.java:267)
+                  at com.example.project.moduleW.ClassW.methodAS(ClassW.java:274)
+                  at com.example.project.moduleW.ClassW.methodAT(ClassW.java:279)
+                  at com.example.project.moduleX.ClassX.methodAU(ClassX.java:286)
+                  at com.example.project.moduleX.ClassX.methodAV(ClassX.java:291)
+                  at com.example.project.moduleY.ClassY.methodAW(ClassY.java:298)
+                  at com.example.project.moduleY.ClassY.methodAX(ClassY.java:303)
+                  at com.example.project.moduleZ.ClassZ.methodAY(ClassZ.java:310)
+                  at com.example.project.moduleZ.ClassZ.methodAZ(ClassZ.java:315)
+                  at com.example.project.moduleAA.ClassAA.methodBA(ClassAA.java:322)
+                  at com.example.project.moduleAA.ClassAA.methodBB(ClassAA.java:327)
+                  at com.example.project.moduleAB.ClassAB.methodBC(ClassAB.java:334)
+                  at com.example.project.moduleAB.ClassAB.methodBD(ClassAB.java:339)
+                  at com.example.project.moduleAC.ClassAC.methodBE(ClassAC.java:346)
+                  at com.example.project.moduleAC.ClassAC.methodBF(ClassAC.java:351)
+                  at com.example.project.moduleAD.ClassAD.methodBG(ClassAD.java:358)
+                  at com.example.project.moduleAD.ClassAD.methodBH(ClassAD.java:363)
+                  at com.example.project.moduleAE.ClassAE.methodBI(ClassAE.java:370)
+                  at com.example.project.moduleAE.ClassAE.methodBJ(ClassAE.java:375)
+                  at com.example.project.moduleAF.ClassAF.methodBK(ClassAF.java:382)
+                  at com.example.project.moduleAF.ClassAF.methodBL(ClassAF.java:387)
+                  at com.example.project.moduleAG.ClassAG.methodBM(ClassAG.java:394)
+                  at com.example.project.moduleAG.ClassAG.methodBN(ClassAG.java:399)
+                  at com.example.project.moduleAH.ClassAH.methodBO(ClassAH.java:406)
+                  at com.example.project.moduleAH.ClassAH.methodBP(ClassAH.java:411)
+                  at com.example.project.moduleAI.ClassAI.methodBQ(ClassAI.java:418)
+                  at com.example.project.moduleAI.ClassAI.methodBR(ClassAI.java:423)
+                  at com.example.project.moduleAJ.ClassAJ.methodBS(ClassAJ.java:430)
+                  at com.example.project.moduleAJ.ClassAJ.methodBT(ClassAJ.java:435)
+                  at com.example.project.moduleAK.ClassAK.methodBU(ClassAK.java:442)
+                  at com.example.project.moduleAK.ClassAK.methodBV(ClassAK.java:447)
+                  at com.example.project.moduleAL.ClassAL.methodBW(ClassAL.java:454)
+                  at com.example.project.moduleAL.ClassAL.methodBX(ClassAL.java:459)
+                  at com.example.project.moduleAM.ClassAM.methodBY(ClassAM.java:466)
+                  at com.example.project.moduleAM.ClassAM.methodBZ(ClassAM.java:471)
+                  at com.example.project.Main.main(Main.java:478)
+                Caused by: java.lang.IllegalArgumentException: Invalid argument encountered
+                  at com.example.project.util.Validator.check(Validator.java:55)
+                  at com.example.project.moduleA.ClassA.methodA(ClassA.java:8)
+                  at com.example.project.moduleA.ClassA.methodB(ClassA.java:15)
+                  at com.example.project.moduleB.ClassB.methodC(ClassB.java:22)
+                  at com.example.project.moduleB.ClassB.methodD(ClassB.java:27)
+                  at com.example.project.moduleC.ClassC.methodE(ClassC.java:34)
+                  at com.example.project.moduleC.ClassC.methodF(ClassC.java:39)
+                  at com.example.project.moduleD.ClassD.methodG(ClassD.java:46)
+                  at com.example.project.moduleD.ClassD.methodH(ClassD.java:51)
+                  at com.example.project.moduleE.ClassE.methodI(ClassE.java:58)
+                  at com.example.project.moduleE.ClassE.methodJ(ClassE.java:63)
+                  at com.example.project.moduleF.ClassF.methodK(ClassF.java:70)
+                  at com.example.project.moduleF.ClassF.methodL(ClassF.java:75)
+                  at com.example.project.moduleG.ClassG.methodM(ClassG.java:82)
+                  at com.example.project.moduleG.ClassG.methodN(ClassG.java:87)
+                  at com.example.project.moduleH.ClassH.methodO(ClassH.java:94)
+                  at com.example.project.moduleH.ClassH.methodP(ClassH.java:99)
+                  at com.example.project.moduleI.ClassI.methodQ(ClassI.java:106)
+                  at com.example.project.moduleI.ClassI.methodR(ClassI.java:111)
+                  at com.example.project.moduleJ.ClassJ.methodS(ClassJ.java:118)
+                  at com.example.project.moduleJ.ClassJ.methodT(ClassJ.java:123)
+                  at com.example.project.moduleK.ClassK.methodU(ClassK.java:130)
+                  at com.example.project.moduleK.ClassK.methodV(ClassK.java:135)
+                  at com.example.project.moduleL.ClassL.methodW(ClassL.java:142)
+                  at com.example.project.moduleL.ClassL.methodX(ClassL.java:147)
+                  at com.example.project.moduleM.ClassM.methodY(ClassM.java:154)
+                  at com.example.project.moduleM.ClassM.methodZ(ClassM.java:159)
+                  at com.example.project.moduleN.ClassN.methodAA(ClassN.java:166)
+                  at com.example.project.moduleN.ClassN.methodAB(ClassN.java:171)
+                  at com.example.project.moduleO.ClassO.methodAC(ClassO.java:178)
+                  at com.example.project.moduleO.ClassO.methodAD(ClassO.java:183)
+                  at com.example.project.moduleP.ClassP.methodAE(ClassP.java:190)
+                  at com.example.project.moduleP.ClassP.methodAF(ClassP.java:195)
+                  at com.example.project.moduleQ.ClassQ.methodAG(ClassQ.java:202)
+                  at com.example.project.moduleQ.ClassQ.methodAH(ClassQ.java:207)
+                  at com.example.project.moduleR.ClassR.methodAI(ClassR.java:214)
+                  at com.example.project.moduleR.ClassR.methodAJ(ClassR.java:219)
+                  at com.example.project.moduleS.ClassS.methodAK(ClassS.java:226)
+                  at com.example.project.moduleS.ClassS.methodAL(ClassS.java:231)
+                  at com.example.project.moduleT.ClassT.methodAM(ClassT.java:238)
+                  at com.example.project.moduleT.ClassT.methodAN(ClassT.java:243)
+                  at com.example.project.moduleU.ClassU.methodAO(ClassU.java:250)
+                  at com.example.project.moduleU.ClassU.methodAP(ClassU.java:255)
+                  at com.example.project.moduleV.ClassV.methodAQ(ClassV.java:262)
+                  at com.example.project.moduleV.ClassV.methodAR(ClassV.java:267)
+                  at com.example.project.moduleW.ClassW.methodAS(ClassW.java:274)
+                  at com.example.project.moduleW.ClassW.methodAT(ClassW.java:279)
+                  at com.example.project.moduleX.ClassX.methodAU(ClassX.java:286)
+                  at com.example.project.moduleX.ClassX.methodAV(ClassX.java:291)
+                  at com.example.project.moduleY.ClassY.methodAW(ClassY.java:298)
+                  at com.example.project.moduleY.ClassY.methodAX(ClassY.java:303)
+                  at com.example.project.moduleZ.ClassZ.methodAY(ClassZ.java:310)
+                  at com.example.project.moduleZ.ClassZ.methodAZ(ClassZ.java:315)
+                  at com.example.project.moduleAA.ClassAA.methodBA(ClassAA.java:322)
+                  at com.example.project.moduleAA.ClassAA.methodBB(ClassAA.java:327)
+                  at com.example.project.moduleAB.ClassAB.methodBC(ClassAB.java:334)
+                  at com.example.project.moduleAB.ClassAB.methodBD(ClassAB.java:339)
+                  at com.example.project.moduleAC.ClassAC.methodBE(ClassAC.java:346)
+                  at com.example.project.moduleAC.ClassAC.methodBF(ClassAC.java:351)
+                  at com.example.project.moduleAD.ClassAD.methodBG(ClassAD.java:358)
+                  at com.example.project.moduleAD.ClassAD.methodBH(ClassAD.java:363)
+                  at com.example.project.moduleAE.ClassAE.methodBI(ClassAE.java:370)
+                  at com.example.project.moduleAE.ClassAE.methodBJ(ClassAE.java:375)
+                  at com.example.project.moduleAF.ClassAF.methodBK(ClassAF.java:382)
+                  at com.example.project.moduleAF.ClassAF.methodBL(ClassAF.java:387)
+                  at com.example.project.moduleAG.ClassAG.methodBM(ClassAG.java:394)
+                  at com.example.project.moduleAG.ClassAG.methodBN(ClassAG.java:399)
+                  at com.example.project.moduleAH.ClassAH.methodBO(ClassAH.java:406)
+                  at com.example.project.moduleAH.ClassAH.methodBP(ClassAH.java:411)
+                  at com.example.project.moduleAI.ClassAI.methodBQ(ClassAI.java:418)
+                  at com.example.project.moduleAI.ClassAI.methodBR(ClassAI.java:423)
+                  at com.example.project.moduleAJ.ClassAJ.methodBS(ClassAJ.java:430)
+                  at com.example.project.moduleAJ.ClassAJ.methodBT(ClassAJ.java:435)
+                  at com.example.project.moduleAK.ClassAK.methodBU(ClassAK.java:442)
+                  at com.example.project.moduleAK.ClassAK.methodBV(ClassAK.java:447)
+                  at com.example.project.moduleAL.ClassAL.methodBW(ClassAL.java:454)
+                  at com.example.project.moduleAL.ClassAL.methodBX(ClassAL.java:459)
+                  at com.example.project.moduleAM.ClassAM.methodBY(ClassAM.java:466)
+                  at com.example.project.moduleAM.ClassAM.methodBZ(ClassAM.java:471)
+                  ... 58 more
+                Caused by: java.lang.NullPointerException: Null value found
+                  at com.example.project.util.Validator.requireNonNull(Validator.java:60)
+                  at com.example.project.util.Validator.check(Validator.java:53)
+                  ... 77 more
+
+  - is_false: errors
+  - do:
+      search:
+        index: logs-generic.otel-default
+  - length: { hits.hits: 1 }
+  - exists: hits.hits.0._source.attributes.exception\.stacktrace
+  - exists: hits.hits.0._ignored
+---
 "resource.attributes.host.name @timestamp should be used as sort fields":
   - do:
       bulk:

The issue revolves around how the stack trace is mapped and how the LogsDB default ignore_above is applied (or not applied)

elasticsearch/x-pack/plugin/otel-data/src/main/resources/component-templates/logs-otel@mappings.yaml

Lines 26 to 30 in b6d9132

	exception.stacktrace:
	type: wildcard
	fields:
	text:
	type: match_only_text

elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java

Line 141 in b6d9132

public static final int IGNORE_ABOVE_DEFAULT_VALUE_FOR_LOGSDB_INDICES = 8191;

The following changes fix the issue

• Adding an explicit ignore_above: 1024 to the mappings
• Adding an explicit ignore_above: 8190 to the mappings
• Changing the mapping from a wildcard multi-field to keyword and explicitly setting ignore_above: 8192 (unclear why default ignore_above doesn't get applied)

The following changes do not fix the issue

• Changing the mapping from a wildcard multi-field to keyword
• Adding an explicit ignore_above: 8191 to the mappings. Something is happening in the step from 8190 to 8191. Keyword fields work with this higher mapping. I suppose it has to do with wildcard fields useing up more space than keyword fields in binary doc_values.
• Reducing IGNORE_ABOVE_DEFAULT_VALUE_FOR_LOGSDB_INDICES from 8191 to 1024

Proposed next steps:

• Set an explicit ignore_above: 1024 for the wildcard field in the OTel mappings. OTel logs data streams: avoid loss of large stacktraces #137370
• Look at why keyword fields don't get a default ignore_above in this case.
• Ensure that whenever a field is _ignored, we can still reconstruct the _source.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[Kubik42]

In our code, we have two default values for ignore_above: 8191 for logsdb indices and Integer.MAX_VALUE for everything else. If ignore_above is explicitly configured to either of these, we treat it as not being set. This is a bit counterintuitive, but we've always done it like this.

So this line ends up evaluating to false.