Subagent Conversation Support by jaredoconnell · Pull Request #935 · vllm-project/guidellm

jaredoconnell · 2026-07-17T04:40:08Z

Summary

This PR includes the initial implementation of the DAG, and basic synthetic data support to utilize that.

Details

Implements a DAG that uses nodes and edges. The edges specify dependency requirements for history.
Improves the conversation script to color code it and show the node's parents, and a full history path.
It's designed in a way to hopefully easily support trace dataset formats in the future.
The input format from the data deserializers is not the flattened design used in existing formats. This is to match both how the DAG needs it, and how typical trace dataset formats actually nest subagents.

Test Plan

I would recommend starting by running a benchmark using the examples given in the markdown to create simple subagents.

Related Issues

TBD

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes code generated or substantially modified by an AI agent
Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

git log

commit 26bad64
Author: Jared O'Connell joconnel@redhat.com
Date: Wed Jul 8 11:42:13 2026 -0400

Initial DAG implementation

Generated-by: Cursor AI Claude Opus 4.6
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit 2599bc5
Author: Jared O'Connell joconnel@redhat.com
Date: Mon Jul 13 19:49:05 2026 -0400

Integrate DAG into full synthetic data pipeline

Generated-by: Cursor AI Claude Opus 4.6
Signed-off-by: Jared O'Connell <joconnel@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

commit 7980c20
Author: Jared O'Connell joconnel@redhat.com
Date: Wed Jul 15 17:50:15 2026 -0400

Address review comments

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit 6a81f9e
Author: Jared O'Connell joconnel@redhat.com
Date: Thu Jul 16 18:18:05 2026 -0400

Implemented requeue delay to DAG, and fixed formatting

Assisted-by: Cursor AI Claude Opus 4.6
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit d090274
Author: Jared O'Connell joconnel@redhat.com
Date: Fri Jul 17 00:30:57 2026 -0400

Redesign input format from dataset deserializers for subagent conversations

Generated-by: Cursor AI Claude Opus 4.6
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit 345b2aa
Author: Jared O'Connell joconnel@redhat.com
Date: Fri Jul 17 01:03:47 2026 -0400

Make code compliant with all CI checks

Generated-by: Cursor AI Claude Opus 4.6
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit e45dd1f
Author: Jared O'Connell joconnel@redhat.com
Date: Thu Jul 23 19:23:55 2026 -0400

Added way to make conversation extraction script less verbose

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit bdcab73
Author: Jared O'Connell joconnel@redhat.com
Date: Thu Jul 23 20:11:02 2026 -0400

Allow setting the merge turn for synthetic subagent branches

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit ec3d0eb
Author: Jared O'Connell joconnel@redhat.com
Date: Thu Jul 23 20:23:53 2026 -0400

Clarify synthetic data sampling behavior

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit d59ee9b
Author: Jared O'Connell joconnel@redhat.com
Date: Thu Jul 23 22:35:47 2026 -0400

Address review comment

Assisted-by: Cursor AI
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit e98decc
Author: Jared O'Connell joconnel@redhat.com
Date: Thu Jul 23 22:41:46 2026 -0400

Address review comment. Switch root_node_ids to computed field

Generated-by: Cursor AI
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit 49699dc
Author: Jared O'Connell joconnel@redhat.com
Date: Fri Jul 24 16:04:04 2026 -0400

Remove sorting from DAG

This makes the DAG function order based on the DAG node insertion order.

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit f64f943
Author: Jared O'Connell joconnel@redhat.com
Date: Fri Jul 24 17:26:07 2026 -0400

Address review feedback

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit c3009b1
Author: Jared O'Connell joconnel@redhat.com
Date: Fri Jul 24 17:59:43 2026 -0400

Refactor history building

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit df949c9
Author: Jared O'Connell joconnel@redhat.com
Date: Tue Jul 28 14:15:20 2026 -0400

Fix linter errors

Generated-by: Cursor AI
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit a32c57d
Author: Jared O'Connell joconnel@redhat.com
Date: Tue Jul 28 18:52:53 2026 -0400

Address review comments

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit d0dd8ef
Author: Jared O'Connell joconnel@redhat.com
Date: Wed Jul 29 15:29:58 2026 -0400

Replace turn index with history length

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit 425a45e
Author: Jared O'Connell joconnel@redhat.com
Date: Wed Jul 29 17:42:43 2026 -0400

Fix multiturn synthetic sampling bug, and add first prompt settings

This will allow us to better mock real world scenarios.

Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit 7943bb9
Author: Jared O'Connell joconnel@redhat.com
Date: Wed Jul 29 18:00:56 2026 -0400

Address review comments

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Assisted-by: Cursor AI
Assisted-by: Cursor AI Claude Opus 4.6
Co-authored-by: Cursor cursoragent@cursor.com
Generated-by: Cursor AI
Generated-by: Cursor AI Claude Opus 4.6
Generated-by: Cursor AI Grok 4.5
Signed-off-by: Jared O'Connell joconnel@redhat.com

SkiHatDuckie

Some comments on dag.py. Many of these are questions that will also help me better understand what the DAG is doing and why.

There will be multiple batches of comment/requests as I go through this, but one of them will be dedicated to seeing how well this interface would work with trace replays like WEKA.

SkiHatDuckie · 2026-07-20T14:57:13Z

+
+        :return: Sorted list of incomplete node IDs.
+        """
+        return sorted(nid for nid in self._graph.nodes if nid not in self._completed)


What is the purpose of sorting the list in alphanumeric order?

I removed the sorting. The ordering is now based on DAG node insertion order.

dbutenhof

I still haven't actually been able to finish this, but before I head out to hike I might as well submit my initial comments from Friday and this morning. 😁

SkiHatDuckie

Treat these as one suggestion. I can't find a good way of suggesting changes across different lines without making a large diff.

dbutenhof

Some additional comments / questions on the first (DAG) commit. (Whew.)

dbutenhof

OK, I started going through by commit, thinking it'd be easier, and then found out you removed the ENUMs and reorganized everything, so that exercise proved pointless. AH, well: I'll resume/restart tomorrow.

Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Assisted-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

…ations Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Assisted-by: Cursor AI Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Generated-by: Cursor AI Signed-off-by: Jared O'Connell <joconnel@redhat.com>

This makes the DAG function order based on the DAG node insertion order. Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Generated-by: Cursor AI Signed-off-by: Jared O'Connell <joconnel@redhat.com>

sjmonson

Some initial notes on the worker code. Overall looks good, it was pretty easy to follow the code. Nits are a few inefficiencies and one bug.

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

sjmonson

Few more comments

sjmonson · 2026-07-28T20:44:54Z

+                        node_id=node_id,
+                        agent_id=node.agent_id,
+                        parent_node_ids=incoming_map[node_id],
                        turn_index=i,


turn_index should be the number of requests since initial, so branches increment turn index separately. Maybe it would be better to calculate this later from history length instead of here though.

sjmonson · 2026-07-29T14:06:32Z

+            now = time.time()
+            for state in self.turns_queue:
+                nxt = state.next_node_ready_at()
+                if nxt is not None and nxt[1] <= now:
+                    node_id, _ = nxt
+                    state.claim_node(node_id)
+                    return state, node_id
+
+            earliest: float | None = None
+            for state in self.turns_queue:
+                nxt = state.next_node_ready_at()
+                if nxt is not None and (earliest is None or nxt[1] < earliest):
+                    earliest = nxt[1]


Maybe I am missing something but why do these need to be separate loops?

Suggested change

now = time.time()

for state in self.turns_queue:

nxt = state.next_node_ready_at()

if nxt is not None and nxt[1] <= now:

node_id, _ = nxt

state.claim_node(node_id)

return state, node_id

earliest: float | None = None

for state in self.turns_queue:

nxt = state.next_node_ready_at()

if nxt is not None and (earliest is None or nxt[1] < earliest):

earliest = nxt[1]

now = time.time()

earliest: float | None = None

for state in self.turns_queue:

nxt = state.next_node_ready_at()

if nxt is not None:

if nxt[1] <= now:

node_id, _ = nxt

state.claim_node(node_id)

return state, node_id

if earliest is None or nxt[1] < earliest:

earliest = nxt[1]

sjmonson · 2026-07-29T14:23:07Z

+
+        raw = raw_values[0]
+        if isinstance(raw, str):
+            graph_data = ConversationGraphData.model_validate(json.loads(raw))


Suggested change

graph_data = ConversationGraphData.model_validate(json.loads(raw))

graph_data = ConversationGraphData.model_validate_json(raw)

Actually does this condition even make sense? Why would this ever be a string?

According to the AI, yes it can be a string.

Yes, it is a string whenever the column comes from the synthetic branched dataset (or any HF/large_string JSON column). The condition exists because that payload is serialized for storage, not because the finalizer invents a string type.

I applied your suggestion locally.

sjmonson · 2026-07-29T14:31:55Z

+            if turn.relative_timestamp is not None:
+                settings = settings.model_copy(
+                    update={"relative_timestamp": turn.relative_timestamp}
+                )
+            if turn.requeue_delay is not None:
+                settings = settings.model_copy(
+                    update={"requeue_delay": turn.requeue_delay}
+                )


I am very confused by what is going on here. Why does ConversationTurnData have a duplicate of every request setting field? And why is this code coping the model instead of just assigning the value?

I'll look into this more tomorrow.

sjmonson · 2026-07-29T14:39:52Z

I guess maybe the whole finalizer is a hack until #972 but I don't like how we have different code paths for column-wise vs single column datasets. Ideally at this point we are either all one or the other.

This is a hard problem to solve without redoing everything. Because the existing pathways always assume it's flattened, but flattened data is very inefficient for multiturn conversations with subagents. But switching away from the existing flattened format as an option would break a lot of the advantages that the current column mapper provide.

So my thought is the dual code path format until we refactor the entire column mapper and preprocessor code sections.

sjmonson · 2026-07-29T14:44:48Z

+            "For multiturn, one value is sampled per conversation and applied "
+            "to every turn (including branch turns that do not override it). "


This is a bug, it should be fixed not codified.

Maybe fixed in a separate PR but please remove this and any other reference to "one value is sampled per conversation" that is not how this should work.

I ended up fixing it, and while I was at it I switched to a way to separately specify the distribution for the first prompts, since in real world scenarios, you often include a big payload in the first message.

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

dbutenhof

Quick skim through changes (all I've got brain left for right now); one minor new comment, and I believe I've resolved all my previous comments.

dbutenhof · 2026-07-29T19:31:03Z

    )
+    merge_after: int = Field(
+        description=(
+            "How many main-chain turns after at_turn the branch merges back. "


"main-chain"? You also used this (unhyphenated) for at_turn, but I think the "Main conversation" notation used in the docstring is more clear.

(Although "main conversation" still sounds a bit awkward when we're talking about a graph of conversations, I don't really have a better concise label ... 🤔)

I switched it to main (parent) conversation.

This will allow us to better mock real world scenarios. Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

jaredoconnell force-pushed the feat/subagent-dag branch from 7c3a2bf to 0e2df94 Compare July 17, 2026 05:20

dbutenhof requested review from SkiHatDuckie, dbutenhof and sjmonson July 17, 2026 11:33

dbutenhof assigned jaredoconnell Jul 17, 2026

dbutenhof added internal filed by core contributor or associate feature Represents a new user-visible feature labels Jul 17, 2026

SkiHatDuckie reviewed Jul 20, 2026

View reviewed changes

dbutenhof reviewed Jul 20, 2026

View reviewed changes

Comment thread docs/guides/multiturn.md Outdated

Comment thread scripts/extract_conversation.py

Comment thread scripts/extract_conversation.py

Comment thread src/guidellm/data/deserializers/synthetic.py Outdated

Comment thread src/guidellm/data/deserializers/synthetic.py Outdated

SkiHatDuckie suggested changes Jul 20, 2026

View reviewed changes

Comment thread src/guidellm/scheduler/schemas/conversation.py Outdated

Comment thread src/guidellm/scheduler/schemas/conversation.py Outdated

Comment thread src/guidellm/scheduler/schemas/conversation.py Outdated

Comment thread src/guidellm/scheduler/schemas/conversation.py Outdated

dbutenhof reviewed Jul 20, 2026

View reviewed changes

Comment thread src/guidellm/scheduler/dag.py Outdated

Comment thread src/guidellm/scheduler/dag.py Outdated

Comment thread src/guidellm/scheduler/dag.py

Comment thread src/guidellm/scheduler/schemas.py Outdated

dbutenhof reviewed Jul 20, 2026

View reviewed changes

Comment thread src/guidellm/scheduler/worker_group.py Outdated

Comment thread src/guidellm/scheduler/schemas/backend.py

dbutenhof added this to the v0.8.0 milestone Jul 25, 2026

dbutenhof linked an issue Jul 27, 2026 that may be closed by this pull request

DAG support for agentic conversation flows #966

Open

jaredoconnell and others added 14 commits July 28, 2026 13:49

Initial DAG implementation

26bad64

Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Integrate DAG into full synthetic data pipeline

2599bc5

Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Address review comments

7980c20

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Implemented requeue delay to DAG, and fixed formatting

6a81f9e

Assisted-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Redesign input format from dataset deserializers for subagent convers…

d090274

…ations Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Make code compliant with all CI checks

345b2aa

Generated-by: Cursor AI Claude Opus 4.6 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Added way to make conversation extraction script less verbose

e45dd1f

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Allow setting the merge turn for synthetic subagent branches

bdcab73

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Clarify synthetic data sampling behavior

ec3d0eb

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Address review comment

d59ee9b

Assisted-by: Cursor AI Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Address review comment. Switch root_node_ids to computed field

e98decc

Generated-by: Cursor AI Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Remove sorting from DAG

49699dc

This makes the DAG function order based on the DAG node insertion order. Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Address review feedback

f64f943

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Refactor history building

c3009b1

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

jaredoconnell force-pushed the feat/subagent-dag branch from 3fc3096 to c3009b1 Compare July 28, 2026 17:57

Fix linter errors

df949c9

Generated-by: Cursor AI Signed-off-by: Jared O'Connell <joconnel@redhat.com>

sjmonson requested changes Jul 28, 2026

View reviewed changes

Comment thread src/guidellm/scheduler/worker.py Outdated

Comment thread src/guidellm/scheduler/worker.py Outdated

Comment thread src/guidellm/scheduler/worker.py Outdated

Comment thread src/guidellm/scheduler/worker.py Outdated

SkiHatDuckie mentioned this pull request Jul 28, 2026

Partial WEKA Trace Format Support (without subagents) #972

Open

3 tasks

Address review comments

a32c57d

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

jaredoconnell marked this pull request as ready for review July 29, 2026 04:40

sjmonson requested changes Jul 29, 2026

View reviewed changes

Replace turn index with history length

d0dd8ef

Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

dbutenhof reviewed Jul 29, 2026

View reviewed changes

jaredoconnell added 2 commits July 29, 2026 17:42

Fix multiturn synthetic sampling bug, and add first prompt settings

425a45e

This will allow us to better mock real world scenarios. Generated-by: Cursor AI Grok 4.5 Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Address review comments

7943bb9

Signed-off-by: Jared O'Connell <joconnel@redhat.com>

	graph_data = ConversationGraphData.model_validate(json.loads(raw))
	graph_data = ConversationGraphData.model_validate_json(raw)

		"For multiturn, one value is sampled per conversation and applied "
		"to every turn (including branch turns that do not override it). "

Uh oh!

Conversation

jaredoconnell commented Jul 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Related Issues

Use of AI

git log

Uh oh!

SkiHatDuckie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SkiHatDuckie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sjmonson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sjmonson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaredoconnell commented Jul 17, 2026 •

edited by github-actions Bot

Loading