Skip to content

Conversation

@marcsanmi
Copy link
Contributor

@marcsanmi marcsanmi commented Oct 9, 2025

Fixes #4383

Problem

After commit ac947a1 removed stub creation, unsymbolized OTLP profiles were skipped when symbolizer is disabled. If the symbolizer is enabled, symbolizer itself will populate those stubs.

Fix

Create stubs at query time. When symbolizer is disabled and blocks have
unsymbolized profiles, generate stubs functions like "libjvm.so 0xcafebabe".

Why read-path? Preserves raw profiles in storage, allowing future symbolization when users toggle the symbolizer on.

⚠️ The fix is only added to v2. Adding it to v1 would require either:

  • Checking every profile for unsymbolized locations on every query (performance overhead), or
  • Adding equivalent to block metadata checking to v1's querier to avoid checking every profile at read time.

Considerations

We could also try adding those stubs at ingest time. However, if symbolizer is enabled, the system would skip stub functions preventing real symbolization unless we re-process data or implement stub detection logic

@marcsanmi marcsanmi requested a review from alsoba13 as a code owner October 9, 2025 09:23
@marcsanmi marcsanmi marked this pull request as draft October 9, 2025 09:23
@marcsanmi marcsanmi marked this pull request as ready for review October 9, 2025 09:54
@marcsanmi marcsanmi marked this pull request as draft October 9, 2025 10:14
@oleg-kozlyuk-grafana
Copy link
Contributor

Check #4470 - this case may be already covered

@marcsanmi marcsanmi force-pushed the marcsanmi/add-otel-stub-creation-back branch 2 times, most recently from 689424b to 971291a Compare October 13, 2025 17:54
@marcsanmi marcsanmi changed the title fix: add otlp stub creation back for unsymbolized profiles fix: add read-path stub creation for unsymbolized profiles Oct 13, 2025
@marcsanmi marcsanmi force-pushed the marcsanmi/add-otel-stub-creation-back branch from 971291a to dad7f49 Compare October 13, 2025 18:00
@marcsanmi
Copy link
Contributor Author

Check #4470 - this case may be already covered

#4470 fixed the HasFunctions flag so blocks are properly tagged as unsymbolized. This PR adds the stub creation logic that runs when symbolizer is disabled. Both changes were needed, but I'd say address slightly different things.

@marcsanmi marcsanmi force-pushed the marcsanmi/add-otel-stub-creation-back branch 2 times, most recently from a95c059 to 7820238 Compare October 14, 2025 09:06
@marcsanmi marcsanmi marked this pull request as ready for review October 14, 2025 09:22
@marcsanmi marcsanmi requested review from a team and bryanhuhta as code owners October 14, 2025 09:22
@marcsanmi marcsanmi force-pushed the marcsanmi/add-otel-stub-creation-back branch from 7820238 to fbfe8f3 Compare October 14, 2025 10:24
Copy link
Contributor

@simonswine simonswine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marcsanmi marcsanmi requested a review from simonswine October 17, 2025 09:05
Copy link
Contributor

@simonswine simonswine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@korniltsev-grafanista
Copy link
Contributor

I've run the branch, but I see no stubs
image

@korniltsev-grafanista
Copy link
Contributor

does this only fix pprof query? and not tree query?

@marcsanmi
Copy link
Contributor Author

does this only fix pprof query? and not tree query?

Yes, you're right. It needs to be applied to tree as well to update the flamegraph. It'll just work for pprof (export)... Going to update it

@marcsanmi
Copy link
Contributor Author

Closing this PR since this needs to be done at the query backend level since we skip the locations there; so we don't have them available in the query frontend. Since this would imply a bigger change, I'm closing this and creating an issue to address it in the future.

Closing this PR since this needs to be done at the query backend level, where we skip the locations; so we don't have them available in the query frontend. Since this would imply a bigger change, I'm closing this and creating an issue to properly address it.

@marcsanmi marcsanmi closed this Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

After upgrade to the latest version (bump from otlp v1.4.0 to v1.7.0), unsymbolized frames disappear from query

4 participants