Partial WEKA Trace Format Support (without subagents)

docs/guides/datasets.md

@@ -21,6 +21,7 @@ The following arguments configure datasets and their processing:
   - `huggingface` (alias `hf`) — loads from HuggingFace Hub or a local directory/file. Required field: `source` (dataset ID or path). Pass dataset loading arguments (for example `split`, `name`) via `load_kwargs`.
   - `json_file`, `csv_file`, `text_file`, `parquet_file`, `arrow_file`, `hdf5_file`, `db_file`, `tar_file` — loads from a local file. Required field: `path`.
   - `trace_synthetic`, `mooncake` — loads a JSONL, JSON, CSV, or Parquet trace file for replay benchmarking. Required field: `path`. Optional: `timestamp_column` (default: `timestamp`), `prompt_tokens_column` (default: `input_length`), `output_tokens_column` (default: `output_length`).
+  - `weka` - Similar to `trace_synthetic`, but with the following changes to the optional field defaults: `timestamp_column` (default: `t`), `prompt_tokens_column` (default: `in`), `output_tokens_column` (default: `out`).
 
 In addition, you can specify additional arguments to the dataset loading with the data argument `load_kwargs`:
 

docs/guides/trace_replay.md

@@ -9,7 +9,8 @@ Detailed use of the replay profile and file-based datasets as a whole is explain
 These are passed to the `--data` argument as `kind=format`:
 
 - `trace_synthetic`: A trace format that does the bare minimum needed to complete a fully functioning trace replay benchmark with synthetic prompt generation
-- `mooncake`: The trace format used by the serving platform Mooncake, as defined in [https://doi.org/10.48550/arXiv.2407.00079](https://doi.org/10.48550/arXiv.2407.00079)
+- `mooncake`: The trace format used by the serving platform *Mooncake*, as defined in [https://doi.org/10.48550/arXiv.2407.00079](https://doi.org/10.48550/arXiv.2407.00079)
+- `weka`: The trace format used by WEKA's *Augmented Memory Grid*, as specified [in the original research repository](https://github.com/callanjfox/agentic-coding-analysis/blob/master/docs/TRACE_FORMAT.md)
 
 ## Format-Agnostic Data Arguments
 
@@ -24,8 +25,8 @@ All trace formats can accept the following optional data arguments:
 These are passed through the `--data` argument like below:
 
 ```bash
-guidellm benchmark \
-    --target http://localhost:8000 \
+guidellm run \
+    --backend kind=openai_http,target=http://localhost:8000 \
     --profile kind=replay \
     --data "kind=trace_synthetic,path=replay.jsonl,timestamp_column=ts,prompt_tokens_column=input_tokens,output_tokens_column=generated_tokens"
 ```
@@ -36,9 +37,33 @@ guidellm benchmark \
 
 ### `mooncake`
 
-The Mooncake format expects an additional column for hash IDs. During prompt generation, hash IDs sharing the same previous ID are required to represent distinct blocks of token ids.
+The Mooncake format expects an additional column for prefix-based cache hash IDs. During prompt generation, hash IDs sharing the same previous ID are required to represent distinct blocks of token ids.
 
 | Argument             | Default    | Description                                         |
 | -------------------- | ---------- | --------------------------------------------------- |
 | `hash_ids_column`    | "hash_ids" | Column name for lists of hash IDs in the trace file |
 | `hash_id_block_size` | 512        | Amount of tokens represented by one hash ID         |
+
+### `weka`
+
+**NOTE:** :construction: While the format is accepted, some features such as subagent conversations, tool call events and non-linear histories are still in active development. The results from datasets including these features will be unreliable.
+
+The WEKA format expects a column with conversation UUIDs that is not wrapped within another column. The timestamp, input token length, output token length and hash IDs columns must either be top level columns, or must all be wrapped inside the same JSON column (ex. "requests").
+
+Similar to Mooncake, WEKA uses prefix-based cache hash IDs. The original [specification](https://github.com/callanjfox/agentic-coding-analysis/blob/master/docs/TRACE_FORMAT.md) for the trace requires hash IDs to be 1 or greater, and for trailing hash IDs to be dropped if there are not enough input tokens to fill the hash ID block size. To accommodate for datasets which may not follow the specification exactly (ex. [semianalysisai/cc-traces-weka-no-subagents-051226](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-no-subagents-051226)), GuideLLM will accept any non-negative integer as a valid hash ID, and will drop partially filled hash IDs if they exist.
+
+GuideLLM will generate prompts starting from the first conversation. When the conversation ends, the next conversation will be used. Hash IDs and relative timestamps are local to the conversation. After a conversation ends, the hash ID tree is reset and the relative timestamp returns to 0.0.
+
+| Argument                 | Default    | Description                                          |
+| ------------------------ | ---------- | ---------------------------------------------------- |
+| `conversation_id_column` | "id"       | Column name for conversation UUIDs in the trace file |
+| `hash_ids_column`        | "hash_ids" | Column name for lists of hash IDs in the trace file  |
+| `hash_id_block_size`     | 64         | Amount of tokens represented by one hash ID          |
+
+Modified defaults:
+
+| Argument               | New Default |
+| ---------------------- | ----------- |
+| `timestamp_column`     | "t"         |
+| `prompt_tokens_column` | "in"        |
+| `output_tokens_column` | "out"       |

src/guidellm/data/deserializers/__init__.py

@@ -43,11 +43,14 @@
     TraceDatasetDeserializer,
     TraceFormatBase,
     TraceFormatRegistry,
+    create_distinct_token_block,
+    create_prompt_from_hash_ids,
     decode_prompt,
     generate_token_ids,
 )
 from .trace_minimal import MinimalTraceFormatArgs
 from .trace_mooncake import MooncakeTraceFormatArgs
+from .trace_weka import WEKATraceFormatArgs
 
 __all__ = [
     "ArrowFileDatasetDeserializer",
@@ -85,6 +88,9 @@
     "TraceDatasetDeserializer",
     "TraceFormatBase",
     "TraceFormatRegistry",
+    "WEKATraceFormatArgs",
+    "create_distinct_token_block",
+    "create_prompt_from_hash_ids",
     "decode_prompt",
     "generate_token_ids",
 ]