Visualise event-based spans for tracing #52

aryanjassal · 2025-04-28T03:52:20Z

Specification

Currently, similar to OpenTracing, each span is only added to the JSONL file after it has closed. This prevents unclosed events from being visualised in the logs properly. It will also fail to yield results if the program exited unexpectedly, resulting in the spans not ending properly.

To fix this, I am converting the spans to an event-based format where each entry is a span plus an event. It could be something like 'start', 'stop', or 'log'. This will give more details than what completed spans can show.

I will add an example structure later. For now, a visualiser needs to be made which supports this new format. The old format can be updated but I'm not sure if that is a good idea as I find parsing the older visualiser difficult, but maybe just for demonstration we can just improve that.

Additional context

Related to Integrate Tracing (derived from OpenTelemetry) #15

Tasks

Use the new structure to either update or make a visualiser

linear · 2025-04-28T03:52:23Z

ENG-589

aryanjassal · 2025-04-28T04:18:11Z

This is the signature of the new spans. They are event based.

type SpanJSON = {
  spanId: string;
  name: string;
  startTime: number;
  endTime: number | undefined;
  parentSpanId: string | undefined;
  isCompleted: boolean;
  children: Array<SpanJSON>;
};

type SpanEvent = {
  type: 'start' | 'stop';
  span: SpanJSON;
}

Each span ID is expected to be reused as a span event would have a start and end event. The isCompleted field is expected to be set to false at the start and would be set to true on span end.

An example JSONL file could look like this.

{"type":"start","span":{"spanId":"abc123","name":"Span 1","startTime":123456789,"endTime":0,"parentSpanId":"abc","isCompleted":false,"children":[]}}
{"type":"stop","span":{"spanId":"abc123","name":"Span 1","startTime":123456789,"endTime":456456789,"parentSpanId":"abc","isCompleted":true,"children":[]}}

The visualiser will need to be based on this type of format.

aryanjassal · 2025-04-28T04:47:48Z

This is an actual example from a working version of the event-based spans where the existing asciinemaTest.ts was used to generate this test data.

{"type":"start","span":{"spanId":"span-1745815564887-cr4","name":"Parent-0","startTime":1745815564887,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745815565884-j24","name":"Parent-0-Fork-B","startTime":1745815565884,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745815565884-89j","name":"Parent-0-Fork-B","startTime":1745815565884,"parentSpanId":"span-1745815564887-cr4","isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745815565884-mzz","name":"Fork-B-Span-1","startTime":1745815565884,"parentSpanId":"span-1745815565884-89j","isCompleted":false,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745815565884-mzz","name":"Fork-B-Span-1","startTime":1745815565884,"endTime":1745815565884,"parentSpanId":"span-1745815565884-89j","isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745815565884-89j","name":"Parent-0-Fork-B","startTime":1745815565884,"endTime":1745815565884,"parentSpanId":"span-1745815564887-cr4","isCompleted":true,"children":[{"spanId":"span-1745815565884-mzz","name":"Fork-B-Span-1","startTime":1745815565884,"endTime":1745815565884,"parentSpanId":"span-1745815565884-89j","isCompleted":true,"children":[]}]}}
{"type":"stop","span":{"spanId":"span-1745815565884-j24","name":"Parent-0-Fork-B","startTime":1745815565884,"endTime":1745815565884,"isCompleted":true,"children":[]}}
{"type":"start","span":{"spanId":"span-1745815568385-dn7","name":"[Rejoins-Fork-0]","startTime":1745815568385,"parentSpanId":"span-1745815564887-cr4","isCompleted":false,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745815568385-dn7","name":"[Rejoins-Fork-0]","startTime":1745815568385,"endTime":1745815568385,"parentSpanId":"span-1745815564887-cr4","isCompleted":true,"children":[]}}

aryanjassal · 2025-04-29T06:09:47Z

The visualiser should be a web-based one instead of a CLI-based one. With the amount of data we are getting, we won't be able to properly show it all in the terminal. You have some example data, and I can get you some actual data after monkey-patching Polykey, so you have multiple references to build the output with.

Prioritise ease of visibility. We should be able to see which processes forked or are children of which processes. Once again, you can take reference from how git visualisers (like the one in VSCode) handle this. It might not be exactly what we want, but would be close enough for a first draft web UI.

aryanjassal · 2025-04-29T08:20:34Z

{"type":"start","span":{"spanId":"span-1745914565530-f7d","name":"span-startstop-Status","startTime":1745914565530,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566553-jri","name":"span-startstop-ClientService","startTime":1745914566553,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566553-r5d","name":"span-startstop-RPCServer","startTime":1745914566553,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566554-pwr","name":"span-startstop-WebSocketServer","startTime":1745914566554,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566595-sqr","name":"span-startstop-NodeConnectionManager","startTime":1745914566595,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566595-n27","name":"span-startstop-RPCServer","startTime":1745914566595,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566595-p64","name":"span-startstop-QUICSocket","startTime":1745914566595,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566596-9h3","name":"span-startstop-QUICServer","startTime":1745914566596,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566597-0xe","name":"span-startstop-NodeManager","startTime":1745914566597,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914566979-cic","name":"span-startstop-MDNS","startTime":1745914566979,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567144-fwz","name":"span-startstop-QUICConnection","startTime":1745914567144,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567146-w67","name":"span-startstop-QUICConnection","startTime":1745914567146,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567152-kp9","name":"span-startstop-QUICConnection","startTime":1745914567152,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567153-9ir","name":"span-startstop-QUICConnection","startTime":1745914567153,"isCompleted":false,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567146-w67","name":"span-startstop-QUICConnection","startTime":1745914567146,"endTime":1745914567209,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567153-9ir","name":"span-startstop-QUICConnection","startTime":1745914567153,"endTime":1745914567599,"isCompleted":true,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567636-l91","name":"span-startstop-QUICConnection","startTime":1745914567636,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567637-1m7","name":"span-startstop-QUICConnection","startTime":1745914567637,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567637-2ww","name":"span-startstop-QUICConnection","startTime":1745914567637,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567637-9s4","name":"span-startstop-QUICConnection","startTime":1745914567637,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567638-vc6","name":"span-startstop-QUICConnection","startTime":1745914567638,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567638-l7z","name":"span-startstop-QUICConnection","startTime":1745914567638,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914567638-ypa","name":"span-startstop-QUICConnection","startTime":1745914567638,"isCompleted":false,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567636-l91","name":"span-startstop-QUICConnection","startTime":1745914567636,"endTime":1745914569912,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567637-1m7","name":"span-startstop-QUICConnection","startTime":1745914567637,"endTime":1745914569912,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567637-2ww","name":"span-startstop-QUICConnection","startTime":1745914567637,"endTime":1745914569912,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567637-9s4","name":"span-startstop-QUICConnection","startTime":1745914567637,"endTime":1745914569912,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567638-vc6","name":"span-startstop-QUICConnection","startTime":1745914567638,"endTime":1745914569912,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567638-l7z","name":"span-startstop-QUICConnection","startTime":1745914567638,"endTime":1745914569912,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567638-ypa","name":"span-startstop-QUICConnection","startTime":1745914567638,"endTime":1745914569912,"isCompleted":true,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569921-j3f","name":"span-startstop-QUICConnection","startTime":1745914569921,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569924-27k","name":"span-startstop-QUICConnection","startTime":1745914569924,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569924-xsw","name":"span-startstop-QUICConnection","startTime":1745914569924,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569925-gvd","name":"span-startstop-QUICConnection","startTime":1745914569925,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569925-n0w","name":"span-startstop-QUICConnection","startTime":1745914569925,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569925-5ey","name":"span-startstop-QUICConnection","startTime":1745914569925,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569925-216","name":"span-startstop-QUICConnection","startTime":1745914569925,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569930-nnc","name":"span-startstop-QUICConnection","startTime":1745914569930,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569930-kwq","name":"span-startstop-QUICConnection","startTime":1745914569930,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-gv8","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-f1o","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-bjh","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-dor","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-yrj","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-p28","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-l8a","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-c72","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-8k4","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569933-9hb","name":"span-startstop-QUICConnection","startTime":1745914569933,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569936-hiu","name":"span-startstop-QUICConnection","startTime":1745914569936,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569937-0gf","name":"span-startstop-QUICConnection","startTime":1745914569937,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569937-tnx","name":"span-startstop-QUICConnection","startTime":1745914569937,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569937-2y7","name":"span-startstop-QUICConnection","startTime":1745914569937,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569940-rru","name":"span-startstop-QUICConnection","startTime":1745914569940,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569941-nk6","name":"span-startstop-QUICConnection","startTime":1745914569941,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569941-7l4","name":"span-startstop-QUICConnection","startTime":1745914569941,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569942-d0a","name":"span-startstop-QUICConnection","startTime":1745914569942,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569944-ef9","name":"span-startstop-QUICConnection","startTime":1745914569944,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569944-d2w","name":"span-startstop-QUICConnection","startTime":1745914569944,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569944-4zn","name":"span-startstop-QUICConnection","startTime":1745914569944,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569946-xg9","name":"span-startstop-QUICConnection","startTime":1745914569946,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569946-y56","name":"span-startstop-QUICConnection","startTime":1745914569946,"isCompleted":false,"children":[]}}
{"type":"start","span":{"spanId":"span-1745914569946-9ln","name":"span-startstop-QUICConnection","startTime":1745914569946,"isCompleted":false,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567144-fwz","name":"span-startstop-QUICConnection","startTime":1745914567144,"endTime":1745914571452,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567144-fwz","name":"span-startstop-QUICConnection","startTime":1745914567144,"endTime":1745914571452,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567144-fwz","name":"span-startstop-QUICConnection","startTime":1745914567144,"endTime":1745914571453,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567152-kp9","name":"span-startstop-QUICConnection","startTime":1745914567152,"endTime":1745914572595,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567152-kp9","name":"span-startstop-QUICConnection","startTime":1745914567152,"endTime":1745914572596,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914567152-kp9","name":"span-startstop-QUICConnection","startTime":1745914567152,"endTime":1745914572597,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566596-9h3","name":"span-startstop-QUICServer","startTime":1745914566596,"endTime":1745914572599,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566595-p64","name":"span-startstop-QUICSocket","startTime":1745914566595,"endTime":1745914572601,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566595-p64","name":"span-startstop-QUICSocket","startTime":1745914566595,"endTime":1745914572601,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566595-n27","name":"span-startstop-RPCServer","startTime":1745914566595,"endTime":1745914572602,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566595-sqr","name":"span-startstop-NodeConnectionManager","startTime":1745914566595,"endTime":1745914572602,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566979-cic","name":"span-startstop-MDNS","startTime":1745914566979,"endTime":1745914572610,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566597-0xe","name":"span-startstop-NodeManager","startTime":1745914566597,"endTime":1745914572918,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566554-pwr","name":"span-startstop-WebSocketServer","startTime":1745914566554,"endTime":1745914572919,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566554-pwr","name":"span-startstop-WebSocketServer","startTime":1745914566554,"endTime":1745914572919,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566553-r5d","name":"span-startstop-RPCServer","startTime":1745914566553,"endTime":1745914572919,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914566553-jri","name":"span-startstop-ClientService","startTime":1745914566553,"endTime":1745914572919,"isCompleted":true,"children":[]}}
{"type":"stop","span":{"spanId":"span-1745914565530-f7d","name":"span-startstop-Status","startTime":1745914565530,"endTime":1745914572944,"isCompleted":true,"children":[]}}

This is an example file (which only tracks opening and closing events). Note that interestingly, multiple stop events are being emitted for the same span id. This could be a fault in my code, or could be something polykey generates. Either way, your visualiser needs to handle this elegantly, showing that there were more attempts to stop an already stopped span.

I will also need to make the stop method idempotent. Currently, it re-stops a span. This means that its stopping time gets updated to the new Date.now() which is incorrect.

tegefaulkes · 2025-04-30T01:57:54Z

The format here could use a little changing.

Let's keep it flat, why do we have most of the span details in the span field? Just flatten it out.
The spanId should use the an IdSortable.
The start and end time should also be IdSortable, just as a standard way of tracking time and order of events.
4 Remember that we can't know a whole span when they happen. We need to track when the span starts and ends as separate events. So it doesn't make sense to have a startTime, EndTime and completed field as part of the span.
The dataset should be saved as an append only log, so once we write an event we can't go back and modify it since they should be immutable. This means that a children field is useless unless we break causality because we can't know its children when the span starts. So all spans should track it's parent instead.

So based on this the format should be closer to

// Will be an string encoded IdSortable
type SpanId = string;

type Span = {
  type: 'Start' | 'End',
  spanId: SpanId,
  name: string,
  parent: SpanId,
}

Since the SpanId is an IdSortable it encodes the timestamp within itself. The Parent will be the spanId of the span it spawned from. For an End span the parent will be the id of the Start span event for that span.

Fundamentally this is a very basic tree structure. If done in this way we can track spans as separate start and end events. Build a tree of these. We'll also need a way to signify that something like a promise got detached from it's original scope and finished somewhere else. So maybe the end span tracks the finish context with the parent and tracks it's start span with a separate startSpan parameter. Play with it.

So the output would look something like this.

{type: 'start', spanId: '1', name: 'span1'}
{type: 'start', spanId: '2', name: 'span2', parent: '1'}
{type: 'start', spanId: '3', name: 'span3', parent: '2'}
{type: 'stop', spanId: '4', parent: '3'}
{type: 'stop', spanId: '5', parent: '2'}
{type: 'stop', spanId: '6', parent: '1'}

Which represents this

aryanjassal · 2025-04-30T02:23:25Z

That makes sense. I was running into the children issue you described, so I planned on removing that and the isCompleted field myself as well.

For tracking the end context, perhaps we can do something like attaching the parent context as a string in the end event? So the end context would be better visible, I guess? I would need to experiment a bit with that.

For now, I'll implement the other suggestions like using idSortable and the improved, event-based span information as that is immediately applicable. In short, the idSortable, or the SpanId, is unique per event. The parentSpanId tracks the parent in case of a start event and the applicable event in case of a stop event. This is an important distinction to make compared to how the format worked so far when I proposed spanId might no longer be unique, as this will ensure unique and sortable spanIds.

aryanjassal · 2025-04-30T07:37:31Z

#15 (comment)

The new format in practice.

{"type":"end","id":"mBoEdFq/CcAC42KekLq/Ewg","spanId":"mBoEdFq+ucASS9V+qzl/new","name":"cd-DBTransaction"}
{"type":"start","id":"mBoEdFq/GcAGJgYOGo97QpQ","spanId":"mBoEdFq/GcACiPlX9p/KFiQ","name":"cdss-cd-VaultManager"}
{"type":"start","id":"mBoEdFq/GcAO0p6TRX4mrkw","spanId":"mBoEdFq/GcAKUjD2xYUtgGQ","name":"cdss-ss-VaultManager"}
{"type":"start","id":"mBoEdFq/KcACU5r/B9QcoMQ","spanId":"mBoEdFq/GcASO5U2jQoqRng","name":"cd-DBTransaction"}
{"type":"start","id":"mBoEdFq/KcAK1WTPbKFZEtA","spanId":"mBoEdFq/KcAGmouimlh4W3Q","parentSpanId":"mBoEdFq/GcAKUjD2xYUtgGQ","name":"cdss-cd-DB"}
{"type":"start","id":"mBoEdFq/OcAGmoEODkEPGjQ","spanId":"mBoEdFq/OcACHCHhNJjv+Qg","parentSpanId":"mBoEdFq/GcAKUjD2xYUtgGQ","name":"cdss-ss-DB"}
{"type":"start","id":"mBoEdFrB+cAGv49+QhawOZg","spanId":"mBoEdFrB+cACw+GyKKhaJag","parentSpanId":"mBoEdFq/GcAKUjD2xYUtgGQ","name":"cdss-cd-INodeManager"}
{"type":"start","id":"mBoEdFrCDcAGh/JM4ik764g","spanId":"mBoEdFrCDcACqIx6lJQnFZA","parentSpanId":"mBoEdFq/GcAKUjD2xYUtgGQ","name":"cdss-ss-INodeManager"}
{"type":"start","id":"mBoEdFrCDcAOa93gXayBFFg","spanId":"mBoEdFrCDcAKxcHe3UPOGEw","name":"cd-DBIterator"}
{"type":"end","id":"mBoEdFrNocACzcTji4XmxSw","spanId":"mBoEdFrCDcAKxcHe3UPOGEw","name":"cd-DBIterator"}
{"type":"start","id":"mBoEdFrNscAGszUyoqdu1GA","spanId":"mBoEdFrNscAClFXGx84X+ew","name":"cd-DBTransaction"}
{"type":"start","id":"mBoEdFrNscAOtGK5BTEaPfA","spanId":"mBoEdFrNscAKrlDL3RyWReQ","name":"cd-DBIterator"}
{"type":"end","id":"mBoEdFrNwcACN0yRVhlU4Aw","spanId":"mBoEdFrNscAKrlDL3RyWReQ","name":"cd-DBIterator"}
{"type":"end","id":"mBoEdFrOZcACZoIaabt9t8w","spanId":"mBoEdFrNscAClFXGx84X+ew","name":"cd-DBTransaction"}

(a snippet of the new format)

CMCDragonkai · 2025-05-01T03:43:28Z

Remember it is possible to have orphan spans. It's not strictly a tree. Some spans go on and start a new life.

CMCDragonkai · 2025-05-04T23:52:49Z

Visualisation target:

React Three Fibre system - WebGL
SVG - D3
Canvas?

So the easiest prototype is to use D3.

Draw lines for D3.

CMCDragonkai · 2025-05-04T23:54:48Z

@aryanjassal create a new PR link here for Polykey-CLI - explain how you are doing the monkey patching.

Then there is an index.html where the D3 code can live there and just reads from that jsonl file.

aryanjassal · 2025-05-05T05:02:55Z

MatrixAI/Polykey-CLI#403

This PR tracks the changes I have made to monkey-patch async-init to track lifecycles using spans. I have made an empty index.html file in which the D3 prototype code can go into.

I create a spanContext at the root of the file and use that to extract the parent span ID. If it exists, then I use that as the parent span ID. Otherwise, the parent span ID is empty. I have done a quick patch using gemini instead of going through the code in-depth and doing a patch myself. I will review it when I can, so I can provide a more complete usage example.

aryanjassal · 2025-05-05T05:06:38Z

Abby can also start working on the index.html in the Polykey-CLI PR. This issue can still track the visualiser status I guess?

CMCDragonkai · 2025-05-07T12:36:39Z

@aryanjassal you'll be take over all tracing related work including visualisation from now on. You can get some ideas from @shafiqihtsham to help you. If we can reuse our work in react three fibre, that would be ideal.

aryanjassal · 2025-05-15T06:37:51Z

After having a brief chat with Brian, we have concluded that with just the spans, we are still lacking context. He proposed we also load the actual logs emitted by the program on the side, so we can correlate the logs with events for time-bound render mode. This feature won't exist for logical steps. If each log event is able to emit an idSortable like spanId, then I might be able to use each span event as discrete time and cluster events near each span, but this involves more effort.

As I am increasing the features, the complexity is also increasing. It might be more maintainable if I split the scripts from index.html and move them into multiple files.

aryanjassal · 2025-05-20T00:41:11Z

We are using D3 for the prototyping phase for the visualiser, but eventually we want to reuse the work done in Polykey Enterprise with R3F. For that, once given the go-ahead to concrete the visualiser in R3F, I will collaborate with Ed to get it done.

This is the progress so far on the D3 visualiser with data monkey-patched in from a quick, 3-second PKCLI run. Note that this only tracks the lifecycles of each object by patching async-init.

2025-05-20.10-27-29.mp4

aryanjassal · 2025-05-20T07:17:19Z

This issue is getting too complicated and far-reaching now, so I will break it down into smaller issues and start tackling them one-by-one and either close this issue as completed or treat this issue as an epic.

aryanjassal · 2025-05-20T07:52:06Z

I have made two child issues to track the pending work for the visualiser: #68 #69

The visualiser work will continue on those issues. I will be closing this issue as we have a basic visualiser working.

aryanjassal added the development Standard development label Apr 28, 2025

aryanjassal assigned Abby010 Apr 28, 2025

CMCDragonkai mentioned this issue May 4, 2025

Integrate high-level observability using Tracer MatrixAI/Polykey-CLI#392

Open

aryanjassal assigned aryanjassal and unassigned Abby010 May 8, 2025

aryanjassal closed this as completed May 20, 2025

Visualise event-based spans for tracing #52

Visualise event-based spans for tracing #52

Comments

aryanjassal commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Specification

Additional context

Tasks

linear bot commented Apr 28, 2025

Uh oh!

aryanjassal commented Apr 28, 2025

Uh oh!

aryanjassal commented Apr 28, 2025

Uh oh!

aryanjassal commented Apr 29, 2025

Uh oh!

aryanjassal commented Apr 29, 2025

Uh oh!

tegefaulkes commented Apr 30, 2025

Uh oh!

aryanjassal commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aryanjassal commented Apr 30, 2025

Uh oh!

CMCDragonkai commented May 1, 2025

Uh oh!

CMCDragonkai commented May 4, 2025

Uh oh!

CMCDragonkai commented May 4, 2025

Uh oh!

aryanjassal commented May 5, 2025

Uh oh!

aryanjassal commented May 5, 2025

Uh oh!

CMCDragonkai commented May 7, 2025

Uh oh!

aryanjassal commented May 15, 2025

Uh oh!

aryanjassal commented May 20, 2025

Uh oh!

aryanjassal commented May 20, 2025

Uh oh!

aryanjassal commented May 20, 2025

Uh oh!

aryanjassal commented Apr 28, 2025 •

edited

Loading

aryanjassal commented Apr 30, 2025 •

edited

Loading