Partial WEKA Trace Format Support (without subagents) by SkiHatDuckie · Pull Request #972 · vllm-project/guidellm

SkiHatDuckie · 2026-07-28T19:23:03Z

Summary

Adds WEKA trace format support to GuideLLM. This PR includes basic handling and parsing of the format, updated tests and documentation, as well as some tweaking of preexisting formats. Subagent conversations, tool call events and nonlinear histories are still in active development, and will be added in a separate PR (ideally after #935 is merged).

This include new utilities for unwrapping JSON in utils/json_unwrap.py, used to parse and flatten conversations for trace replay.

Details

trace_weka.py (new)

Based off of this spec
Handles hash IDs and conversation-scoping
Currently missing:
- Subagent conversations
- Tool call events
- Nonlinear histories
- Handling of fields such as models, block_size, output_types, etc... (variable importance)
Accommodates for datasets which may not match the spec exactly by accepting any non-negative hash ID as valid and dropping partially filled hash IDs if they exist

test_trace_weka.py (new)

json_unwrap.py (new)

Added VirtualColumnLocation
Added functions construct_virtual_column_locations, unzip_virtual_column_locations, try_json_load, is_json_serializable and get_json_column_names

test_json_unwrap.py (new)

trace_common.py

Added conversation_id_column to TraceDataArgs
Rearranged helper function to fit new required checks
Added/updated helper functions to handle wrapped columns
Modified TraceExamplesIterable to handle localize relative timestamps to the conversation (if one exists)
Added optional method reset to TraceFormatBase
Moved dataset loading and validation up to the deserializer instead of the examples iterable.

trace_mooncake.py

Switch to storing token blocks as set[tuple[int, ...]] instead of list[list[int]] for minor performance boost
Switch to storing hash IDs in a dictionary instead of a lookup table

test_trace_common.py, test_trace_minimal.py & test_trace_mooncake.py

Minor tweaks to helper functions
New test for input token length matching or bordering the hash ID block size in test_trace_mooncake.py
New test for virtual columns in test_trace_common.py

trace_replay.md

Added documentation for weka
Included a warning about WIP features

datasets.md

Updated --data kind options list

Test Plan

tox -e lint-check && tox -e type-check
tox -e test-unit

Example datasets:
semianalysisai/cc-traces-weka-no-subagents-051226

Related Issues

Partial completion of #967

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes code generated or substantially modified by an AI agent
Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

git log

commit 8868aaa
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 13:57:55 2026 -0400

Add files

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 75371ff
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 14:05:52 2026 -0400

Update field description

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit e049ce8
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 14:18:54 2026 -0400

Add stubs

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit dc1920e
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 14:19:39 2026 -0400

Add missing trailing whitespace

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit a965b40
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 16:32:54 2026 -0400

Implement trace format & args

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 60cfcbd
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 11:39:42 2026 -0400

Add tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 2e1d969
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 12:12:17 2026 -0400

Test prompts matching or bordering block size

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit f28a347
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 13:20:29 2026 -0400

Update tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 1c37aaf
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 13:31:56 2026 -0400

Update trace format docs

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 7fef74a
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 14:10:31 2026 -0400

Update trace_replay.md

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 3450fc3
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Mon Jul 13 13:46:03 2026 -0400

Reapir tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 4edc5da
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 15 13:52:15 2026 -0400

Indicate private functions

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 58d8dab
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 15 13:56:37 2026 -0400

Update trace_replay.md

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 581f89b
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 15 13:58:55 2026 -0400

Make mdformat happy

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 4d7f58d
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 21 15:59:50 2026 -0400

Add json utils + impl in trace_common

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit ee391f0
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 21 16:20:35 2026 -0400

Refactor

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 16758aa
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 21 16:37:37 2026 -0400

Handle unserializable python dicts

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit fdb864f
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 22 12:00:52 2026 -0400

Improve json_unwrap tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 4aec222
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 22 12:45:52 2026 -0400

Improve trace_common test coverage

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 8310217
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Mon Jul 27 14:35:22 2026 -0400

Handle multi-conversation datasets

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit c1e2312
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Mon Jul 27 17:19:45 2026 -0400

Store token blocks as tuples

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 8622c0d
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 07:50:51 2026 -0400

Repair tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 9d04e2b
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 10:28:24 2026 -0400

Tolerate partially filled hash ID

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit a8872bb
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 11:40:27 2026 -0400

Improve test coverage

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit de082bb
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 12:02:54 2026 -0400

Fix linting

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit a833784
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 14:25:33 2026 -0400

Update docs

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 5fe0d30
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 14:54:45 2026 -0400

Fix typo

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Signed-off-by: SkiHatDuckie SkiHatDuckie@gmail.com

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

jaredoconnell

Looks good. I have a few comments.

Does this automatically support column names t / in / out? Sources like this state that it's what the spec is.

jaredoconnell · 2026-07-29T15:19:42Z

+            processor,
+            faker,
+        )
+        return prompt + " " + remainder


Either the prompt or the remainder could be empty, right? If so, the logic here needs to return "" if both are empty, the non empty one if only one is non-empty, and use the current logic if both are not empty.

jaredoconnell · 2026-07-29T15:45:56Z

+class ColumnSearchResult:
+    status: Status
+    checked_json_columns: bool
+    apropos_column_names: Sequence[str | VirtualColumnLocation]


What is apropos supposed to mean?

Apropos in this case means "relevant". I had some trouble coming up with a good name since the field could either be a list of missing required columns or a list of the locations of all virtual column locations found. Maybe I should just change it to relevant_column_names?

jaredoconnell · 2026-07-29T15:57:37Z

+    """Required columns must all be stored within the same column."""
+    completely_missing = set(target_columns)
+    for col in json_column_names:
+        parsed = sample_row[col]


Is it possible that a call to try_json_load should go here? The AI analysis determined that this could be a JSON string that is then rejected by _is_supported_json_data_type

If so, a matching call in _make_columns_from_virtual may be needed, too.

jaredoconnell · 2026-07-29T15:59:57Z

+    )
+
+
+def _create_prompt_from_hash_ids(


Can this be extracted to trace_common and shared with mooncake?

dbutenhof

Just a few initial comments. I haven't gotten far: and after spending most of the morning wrestling with AIPCC and GitLab workflows, I'm now heading out to hike (in the rain) before continuing this afternoon. Figured I might as well drop what I already got ...

dbutenhof · 2026-07-29T15:51:20Z

+
+**NOTE:** :construction: While the format is accepted, some features such as subagent conversations, tool call events and non-linear histories are still in active development. The results from datasets including these features will be unreliable.
+
+The WEKA format expects a column with conversation UUIDs that is not wrapped within another column. The timestamp, input token length, output token length and hash IDs columns will very likely be wrapped inside another column (ex. "requests"), representing an entire conversation. Ensure these four virtual columns are wrapped in the same column. GuideLLM will handle the unwrapping and parsing of these trace sessions internally.


Instead of the predictive "will very likely be", how about a more direct allowance ("may be").

Also, this suggests they "may not be", in which case the following sentence ("ensure [they] are wrapped in the same column") becomes misleading. These conditions should be combined: so maybe, "The [...] columns may be wrapped inside another column [...]: if so, they must all be wrapped inside a single column." And perhaps even followed by saying that if they're not all wrapped inside a single column, they must each appear as a separate column? (That is, it sounds like one cannot combine "some wrapped" with "some not wrapped"?)

dbutenhof · 2026-07-29T15:54:30Z

+
+The WEKA format expects a column with conversation UUIDs that is not wrapped within another column. The timestamp, input token length, output token length and hash IDs columns will very likely be wrapped inside another column (ex. "requests"), representing an entire conversation. Ensure these four virtual columns are wrapped in the same column. GuideLLM will handle the unwrapping and parsing of these trace sessions internally.
+
+Similar to Mooncake, WEKA uses prefix-based cache hash IDs. The original [specification](https://github.com/callanjfox/agentic-coding-analysis/blob/master/docs/TRACE_FORMAT.md) for the trace requires hash IDs to be 1 or greater, and for trailing hash IDs to be dropped if there are not enough input tokens to fill the hash ID block size. To accommodate for datasets which may not follow the specification exactly (ex. [semianalysisai/cc-traces-weka-no-subagents-051226](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-no-subagents-051226)), GuideLLM will accept any non-negative integer as a valid hash ID, and will drop partially filled hash IDs if they still exist.


Is the "still" in "if they still exist" operational here? (It's not clear from previous text exactly what might have already removed them.) Should this just be "if they exist".

dbutenhof · 2026-07-29T16:00:30Z

                self.config, row, self.processor, self.faker
            )
-            relative_timestamp = timestamps[row_idx] - timestamps[0]
+            relative_timestamp = timestamps[row_idx] - (conv_start_ts or timestamps[0])


Since you have timestamps outside the loop, can't you just set conv_start_ts = timestamps[0] in the first place and avoid the or???

SkiHatDuckie · 2026-07-29T16:15:28Z

Does this automatically support column names t / in / out? Sources like this state that it's what the spec is.

Currently no. I'll change the defaults for WEKA so that it fits the more common names.

SkiHatDuckie added 27 commits July 28, 2026 14:27

Add files

8868aaa

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update field description

75371ff

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add stubs

e049ce8

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add missing trailing whitespace

dc1920e

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Implement trace format & args

a965b40

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add tests

60cfcbd

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Test prompts matching or bordering block size

2e1d969

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update tests

f28a347

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update trace format docs

1c37aaf

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update trace_replay.md

7fef74a

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Reapir tests

3450fc3

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Indicate private functions

4edc5da

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update trace_replay.md

58d8dab

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Make mdformat happy

581f89b

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add json utils + impl in trace_common

4d7f58d

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Refactor

ee391f0

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Handle unserializable python dicts

16758aa

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Improve json_unwrap tests

fdb864f

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Improve trace_common test coverage

4aec222

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Handle multi-conversation datasets

8310217

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Store token blocks as tuples

c1e2312

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Repair tests

8622c0d

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Tolerate partially filled hash ID

9d04e2b

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Improve test coverage

a8872bb

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Fix linting

de082bb

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update docs

a833784

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Fix typo

5fe0d30

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

dbutenhof assigned SkiHatDuckie Jul 29, 2026

dbutenhof added internal filed by core contributor or associate priority-critical labels Jul 29, 2026

dbutenhof added the feature Represents a new user-visible feature label Jul 29, 2026

dbutenhof added this to the v0.8.0 milestone Jul 29, 2026

dbutenhof requested review from dbutenhof, jaredoconnell and sjmonson July 29, 2026 14:51

sjmonson mentioned this pull request Jul 29, 2026

Subagent Conversation Support #935

Open

3 tasks

jaredoconnell requested changes Jul 29, 2026

View reviewed changes

dbutenhof reviewed Jul 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Partial WEKA Trace Format Support (without subagents) - #972

Partial WEKA Trace Format Support (without subagents)#972
SkiHatDuckie wants to merge 27 commits into
vllm-project:mainfrom
SkiHatDuckie:weka-trace

SkiHatDuckie commented Jul 28, 2026 •

edited by github-actions Bot

Loading

Uh oh!

jaredoconnell left a comment

Uh oh!

jaredoconnell Jul 29, 2026

Uh oh!

jaredoconnell Jul 29, 2026

Uh oh!

SkiHatDuckie Jul 29, 2026

Uh oh!

jaredoconnell Jul 29, 2026

Uh oh!

jaredoconnell Jul 29, 2026

Uh oh!

dbutenhof left a comment

Uh oh!

dbutenhof Jul 29, 2026

Uh oh!

dbutenhof Jul 29, 2026

Uh oh!

dbutenhof Jul 29, 2026

Uh oh!

SkiHatDuckie commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		NOTE: :construction: While the format is accepted, some features such as subagent conversations, tool call events and non-linear histories are still in active development. The results from datasets including these features will be unreliable.

		The WEKA format expects a column with conversation UUIDs that is not wrapped within another column. The timestamp, input token length, output token length and hash IDs columns will very likely be wrapped inside another column (ex. "requests"), representing an entire conversation. Ensure these four virtual columns are wrapped in the same column. GuideLLM will handle the unwrapping and parsing of these trace sessions internally.


		The WEKA format expects a column with conversation UUIDs that is not wrapped within another column. The timestamp, input token length, output token length and hash IDs columns will very likely be wrapped inside another column (ex. "requests"), representing an entire conversation. Ensure these four virtual columns are wrapped in the same column. GuideLLM will handle the unwrapping and parsing of these trace sessions internally.

		Similar to Mooncake, WEKA uses prefix-based cache hash IDs. The original [specification](https://github.com/callanjfox/agentic-coding-analysis/blob/master/docs/TRACE_FORMAT.md) for the trace requires hash IDs to be 1 or greater, and for trailing hash IDs to be dropped if there are not enough input tokens to fill the hash ID block size. To accommodate for datasets which may not follow the specification exactly (ex. [semianalysisai/cc-traces-weka-no-subagents-051226](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-no-subagents-051226)), GuideLLM will accept any non-negative integer as a valid hash ID, and will drop partially filled hash IDs if they still exist.

		)


		def _create_prompt_from_hash_ids(

Uh oh!

Conversation

SkiHatDuckie commented Jul 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Related Issues

Use of AI

git log

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SkiHatDuckie commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SkiHatDuckie commented Jul 28, 2026 •

edited by github-actions Bot

Loading