rocprofv3: surface inter-stream HIP event-sync dependencies on wait/sync records

[ajassani]

Motivation

rocprofv3 already shows the per-record stream ID on hipStreamWaitEvent and hipEventSynchronize tool records, but it does not yet surface which stream and which hipEventRecord call the waiter is actually depending on. As a result, tools that want to reconstruct a cross-stream / critical-path dependency graph from a captured trace have to re-run the workload under a separate analysis pass or guess at the topology from timestamps.

This PR closes that gap by adding two new fields on the HIP API tool record:

• wait_on_stream — the stream that recorded the awaited hipEvent_t
• wait_on_correlation_id — the rocprofv3 correlation id of that hipEventRecord call

Both default to 0 (the "no resolved dependency" sentinel) when the awaited event was never observed being recorded earlier in the same trace, so the change is non-breaking from a record-shape standpoint.

Technical Details

Event-lifecycle tracking utility

A new utility in source/lib/rocprofiler-sdk-tool/event_producer_map.{hpp,cpp} maintains two process-global, shared/unique-locked maps using the existing common::Synchronized + common::static_object building blocks:

• producer map — hipEvent_t -> { stream_id, hipEventRecord corr_id }. Written on PHASE_EXIT of hipEventRecord (and _spt / WithFlags variants); erased on hipEventDestroy.
• resolved-wait map — waiter corr_id -> { wait_on_stream, wait_on_corr_id }. Written on PHASE_EXIT of hipStreamWaitEvent (and _spt) and hipEventSynchronize after a producer lookup; drained at buffer-flush time when the corresponding tool record is materialized.

Callback wiring

A dedicated callback context is registered on ROCPROFILER_CALLBACK_TRACING_HIP_RUNTIME_API filtered to just the seven operation IDs involved in event lifecycle and synchronization. The context is gated on hip_runtime_api_trace so there is zero overhead when HIP tracing is off. The callback runs on PHASE_EXIT only (after the underlying HIP call has completed) so the producer state is consistent with what other threads will observe via lookup, and the body is wrapped in a top-level try/catch (the underlying std::unordered_map ops can throw std::bad_alloc under memory pressure and this code runs synchronously inside the SDK callback dispatch path).

Buffer-flush integration

At the existing buffered_tracing_callback flush site, the materialization of tool_buffer_tracing_hip_api_ext_record_t consumes any stashed resolved-wait entry for the record's correlation id and routes through an extended constructor that fills the new fields.

The lookup is gated on the record's operation kind — only hipStreamWaitEvent, hipStreamWaitEvent_spt, and hipEventSynchronize can ever carry a resolved dependency in the side-table, so the (locking) map call is skipped for the other ~99.95% of HIP API records on a representative DDP workload.

Reattach hygiene

Both maps live in common::static_object and persist across tool_init → tool_detach → tool_init cycles. hipEvent_t pointers can be reused across sessions, so we clear both maps at tool_init right before the dependency context is created. A single ROCP_INFO is also emitted at the same site so users see the service being enabled in the tool init log.

Output coverage

• CSV (out_hip_api_trace.csv) widens from 7 to 10 columns: the existing 7 + Stream_Id + Wait_On_Stream_Id + Wait_On_Correlation_Id. A dedicated hip_api_csv_encoder is used only for HIP records so other CSV outputs are unchanged.
• JSON (out_results.json) picks up wait_on_stream (as {"handle": N}) and wait_on_correlation_id on hip_api buffer records via cereal serialization. No schema breakage; the fields default to 0.
• Perfetto pftrace emits two new args (wait_on_stream_ID, wait_on_corr_id) on the relevant slices and, when wait_on_correlation_id != 0, adds a second Flow::ProcessScoped terminating at the resolved producer so the cross-stream dependency shows up as an arrow in the Perfetto UI.

Deferred to follow-up PRs

• OTF2 emitter changes need schema-level attribute IDs that deserve discussion with downstream consumers (Vampir, Score-P, etc.) on its own PR.
• Rocpd column / schema migration for the new fields is deferred to its own PR for the SQL-migration review.

Both formatters continue to emit valid records today; they just don't yet carry the new fields.

⚠️ Breaking changes

The HIP API CSV (<out>_hip_api_trace.csv) now has 10 columns instead of 7. Consumers that parse this CSV positionally or hard-code the column count will need to be updated.

The change is contained: only HIP records use the new hip_api_csv_encoder<10>; other domains (kernels, markers, memory, etc.) keep their existing 7-column shape via the original api_csv_encoder<7>.

The added columns are:

Column	Meaning
Stream_Id	the call's own stream (previously dropped from the CSV even though it was always available on the record)
Wait_On_Stream_Id	resolved producer stream — 0 if no resolved dependency
Wait_On_Correlation_Id	resolved producer hipEventRecord correlation id — 0 if no resolved dependency

If a downstream gating flag for the new columns is preferred, happy to add an opt-in option (--hip-trace-deps or similar) so the default shape stays at 7 columns until consumers have migrated.

The JSON and Perfetto outputs are additive: new fields appear on existing records but no existing fields are renamed, reshaped, or removed.

JIRA ID

N/A — external contributor (no SWDEV access). Happy to have an internal champion attach a ticket if that helps with the merge gate.

Test Plan

Unit tests

source/lib/tests/tool/event_producer_map.cpp — 13 cases against the new tool-tests binary (a focused gtest unit-test executable for the tool-side state machine):

Case	What it covers
lookup_unknown_event_returns_nullopt	base read-miss
lookup_null_event_returns_nullopt	null sentinel
record_then_lookup_round_trips	basic write/read
record_overwrites_prior_entry	re-recording an event picks up the latest producer
record_null_event_is_silent_noop	null event write is rejected silently
forget_event_removes_entry	hipEventDestroy cleanup
forget_unknown_event_is_silent_noop	destroying an unrecorded event is safe
distinct_events_are_independent	no cross-talk between event keys
stash_then_consume_resolved_wait_drains	side-table is single-shot per waiter
consume_unknown_corr_id_returns_nullopt	no-resolve case
stash_zero_corr_id_is_silent_noop	0 is reserved as the no-dep sentinel
clear_all_resets_both_maps	tool-init / test-isolation hook
concurrent_record_and_forget_does_not_corrupt	4-writer × 1000-op stress

Integration tests

tests/rocprofv3/hip-interstream-sync/ — 1 execute + 5 validate cases. A small HIP workload exercises a cross-stream hipStreamWaitEvent plus a host hipEventSynchronize, then the validator asserts:

• the new CSV columns are present;
• every hipStreamWaitEvent and hipEventSynchronize row carries a non-zero Wait_On_Correlation_Id whose value matches some prior hipEventRecord row's Correlation_Id (no dangling references);
• waiter rows report a Wait_On_Stream_Id distinct from their own Stream_Id (cross-stream is the entire point);
• non-wait/sync rows still carry the 0 sentinel (no leakage onto records that shouldn't carry a dependency);
• the same checks pass against the JSON output;
• resolved-dependency counts agree between CSV and JSON for the same trace.

Local validation

• tool-tests gtest binary: 13/13 pass on an MI300X / MI250 debug build.
• ctest -R hip-interstream-sync: 6/6 pass (1 execute + 5 validate).
• Manual rocprofv3 --hip-trace --kernel-trace --output-format json csv pftrace on a 2-rank PyTorch DDP workload: JSON, CSV, and pftrace all show non-zero wait_on_correlation_id for the cross-stream waits; reconstructed dependency DAG matches the expected backward → reduce topology (78 of 149,872 records carry resolved metadata, ≈0.05%).
• Output-size impact measured on the same DDP trace: CSV +5.6%, JSON +3.2%, pftrace +0.006% (pftrace only emits the extra args + Flow when there's a resolved dependency, so its overhead scales with the number of sync events rather than the total HIP call count).

Test Result

$ /path/to/build/bin/tool-tests
[==========] Running 13 tests from 1 test suite.
[----------] 13 tests from EventProducerMap
... 13/13 OK ...
[==========] 13 tests from 1 test suite ran. (1 ms total)
[  PASSED  ] 13 tests.

$ ctest --test-dir build -R hip-interstream-sync --output-on-failure
        Start 718: tests.integration.execute.rocprofv3-test-hip-interstream-sync
1/6 Test #718: ............................................................   Passed    0.31 sec
        Start 713: ...validate.rocprofv3-test-hip-interstream-sync.test_hip_csv_has_inter_stream_columns
2/6 Test #713: ............................................................   Passed    0.19 sec
... 6/6 OK ...
100% tests passed, 0 tests failed out of 6

Submission Checklist

• Looked over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
• Unit tests added.
• Integration test added (gated on integration-tests label, runs under existing GPU-required test infrastructure).
• Reviewed for breaking changes (CSV column-count widening — flagged above).
• OTF2 emitter follow-up PR (deferred).
• Rocpd schema follow-up PR (deferred).

Draft status

This PR is open on the fork (ajassani/rocm-systems) for review and CI iteration before being promoted to an upstream PR against ROCm/rocm-systems. The OTF2 and Rocpd follow-ups will gate the upstream submission.