Add streaming to `profiling.sampling`

[maurycy]

Feature or enhancement

Proposal:

Right now, profiling.sampling has roughly two modes: live with the TUI, and snapshot-at-the-end (except binary? but see the note below.) There's nothing that streams the data continously as it comes. This would be ideal for long-running headless profiling.

This one is less defined than #145411, so there are more open questions:

• Should it stream raw or agreggate data? What should be the window?
• What should be the format? Unfortunately, from what I checked the current binary format is not really well-suited for streaming, as it saves the dictionaries only on finalize.
• What should be the transport layers for streaming?
• What should be the types of messages?
• What should be the configuration flags?
• How the backpressure should be handled? Should it drop the oldest? All the oldest?

My hunch is:

• support both raw and aggregate and assume that aggregate is just a different message type.
• start with something simple as JSONL
• I have mixed feelings about the transport layer. The stdout sounds great on paper but it's a mixed-use channel right now and there's a question around blocking by slower consumers. Maybe for debugging? Unix socket is good but not perfectly portable
• as for backpressure, I would just drop the oldest by default, but only the raw, agg and heartbeat messages.

For starters:

• --stream unix:/tmp/tachyon-stream.sock or --stream file:/tmp/tachyon-stream.sock (later: --stream tcp:127.0.0.1:1234)
• --stream-types meta|str_def|frame_def|raw|agg:5s|loss|heartbeat|end|error (comma separated; note the agg:5s here; we it could be extended to heartbeat:1s etc. in the future)
• --stream-format jsonl

Then, in the next phases we could think of --stream-drop-policy etc.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

[LalitMaganti]

I was chatting with @pablogsal offline about profiling.sampling and asking about the posibility of streaming. He pointed me to this issue and asked me to chime on my thoughts here.

Let me start first from the open questions above.

Should it stream raw or agreggate data? What should be the window?

I would love if we could have raw samples with individual timestamps, similar to the output of Linux's perf tool. There's some very useful things you can do if you can correlate exact timestamps with other things happening on the system which is just not possible if you are aggregating over an interval.

Keep in mind though that this means you need to have quite an efficient encoding scheme, probably by "windowing" the data within an interval.

What should be the format? Unfortunately, from what I checked the current binary format is not really well-suited for streaming, as it saves the dictionaries only on finalize.

I'm not super familiar with the binary format but from my learnings from working on Perfetto, we've found that it's best if you do batches of data with each batch having ~everything you need to decode the batch. This is really important for cases where you are streaming to a ring-buffer and keeping around only the "last X seconds of data". If you instead say "I will only emit the full function name the first time I see it", you'll end up being unable to decode the samples because ring buffer wraps.

What should be the transport layers for streaming?

It would be really nice if there was a very high performance option (i.e. ring buffer in shared memory) and using sockets as a "control channel" rather than forcing everything over sockets. profiling.sampling seems to support very high rates and so it would be a shame if we couldn't make use of that with streaming becuase we are trying to push everything over a socket.

What should be the types of messages?

jsonl seems fine for usecases where performance is not the priority but I would also love if there was some binary format for low overhead encode/deocde on both sides. Specifically, I have a usecase where I'm thinking about (see below) where I would love to stream the profiling data out of the Python process and into a ring buffer for 24/7 profiling and snapshotting when there is a "bad event" detected.

What should be the configuration flags?

Sampling rate and data format are obvious ones. If we do windowing, the time of the window would also be one. Off the top of my head, I cannot think of any others whcih I think would be critical.

How the backpressure should be handled? Should it drop the oldest? All the oldest?

If you do end up writing in a ringbuffer, the oldest first will be automatically dropped. Personally given that's how ~all OS level profiling tools work, it would be good to stay consistent with it.

---

So my concrete usecase:

I'm working on a "one-command" performance debugging tool to collecting all the performance data someone might need and visualizing it on a single timeline. This includes: CPU profiling, memory profiling, app level instrumentation, OS level scheduling and language specific features (e.g. goroutines for go, coroutines in Python/Kotlin etc). The aim is to have a very accessible experience and consistent across any language/framework/OS you are using.

For Python CPU profiling, I would love to be able to use profiling.sampling; streaming is very important for the way this tool works becuase I want to support both "long tracing" (where data is continuously streamed to disk over many minutes) and "flight recorder" (where data in a ringbuffer is snapshot to disk when a "bad event" happens in the app).

[maurycy]

@LalitMaganti Thank you for chiming in, and thank you for your work on Perfetto! These are all awesome points, and I truly appreciate you sharing them. I'm travelling right now. I will try to reply as soon as possible, likely within a day or two.

[maurycy]

@LalitMaganti

Thank you once again. This is definitely the kind of use case I was hoping for!

I definitely agree with the shape that streaming needs self-contained batches / decoding units.

Keep in mind though that this means you need to have quite an efficient encoding scheme, probably by "windowing" the data within an interval.

Do you think that windows / self-contained chunks / batches would be treated as runs? In other words, a regular run would be batch without a fixed duration / window?

What I mean here is that the idea from #146256, heavily inspired by the format, is already self-contained.

Perhaps the meta message could be extended with duration here?

Specifically: would you model these windows / chunks/ batches as units within a single run?

If you instead say "I will only emit the full function name the first time I see it", you'll end up being unable to decode the samples because ring buffer wraps.

Using the whole test run:

2026-05-03T18:39:10.253965000+0200 maurycy@gimel /Users/maurycy/work/cpython (main 1fc2b38) % sudo -E /Users/maurycy/src/github.com/maurycy/cpython/python.exe -m profiling.sampling run \
  --sampling-rate 1000000 \
  --realtime-stats \
  --binary -o /tmp/prof.bin \
  -m test

(took ~14 minutes)

Where for self-contained 5s batch:

str_def per batch:

Stat	Entries	Bytes
p50	377	22,855	22.32 KB
p90	1,153	72,619	70.92 KB
p95	1,420	89,492	87.39 KB
p99	1,982	129,605	126.57 KB
max	2,128	141,865	138.54 KB
mean	519	32,354	31.60 KB
sum	88,284	5,500,230	5.25 MB

frame_def per batch:

Stat	Entries	Bytes
p50	927	108,427	105.89 KB
p90	3,178	381,094	372.16 KB
p95	3,997	472,633	461.56 KB
p99	5,177	617,608	603.13 KB
max	5,457	647,931	632.75 KB
mean	1,304	154,188	150.57 KB
sum	221,725	26,212,124	25.00 MB

The obvious implication is that there will be many str_def and frame_def, and they always have to be buffered.

I would love if we could have raw samples with individual timestamps, similar to the output of Linux's perf tool. There's some very useful things you can do if you can correlate exact timestamps with other things happening on the system which is just not possible if you are aggregating over an interval.

...the only omissing message is raw.

Do you think (putting aside if it's JSONL or would be sent binary) something like:

{"type":"raw","v":0,"run_id":"7-Zk2_Mq","seq":123456789,"ts_usec":[481293],"threads":[{"thread_id":1535,"status":1,"stack":[1,2,3]}]}

would be good?

Where:

• run_id could actually be URL-safe Base64 or even Z85, maybe with an overflow...
• seq (in a run) could be also URL-safe Base64 or even Z85, maybe with an overflow...
• status could be a bitfield (not sure if it's guaranteed to be fixed, though),
• stack would be a list of frame_ids from the frame_def,
• ts_usec (ns?) could be delta (since meta has to be received anyway),
• seq could be ts_usec delta but it'd make it harder for the consumer to spot how many messages missed.

Of course, if we want to push further, we could abbreviate to:

{"t":"r","v":0,"r":"7-Zk2_Mq","q":"2v2B/","u":[481293],"h":[{"i":1535,"f":1,"s":[10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]}]}

That's 198 characters with 39 frames (p95 from the test suite.) With 1MHz it gives us mere 0.198 GB/s =~ 1.584 Gbit/s.

Truth be told, I don't know if 1MHz sampling rate is sustainable. I'm unable to go past 100 kHz on my MacBook Air M4. My hunch is that we're already hitting Python context switch overhead.

A bit of my feeling is that high-performance pattern like shared memory ring buffer would require rewriting Tachyon to C.

This is also interesting number because it implies how patient we could be for a slow consumer. (Not necessarily a maximum batch duration / interval, since internal encoding is more efficient.)

Also, for such messages the frame_def and str_def for 5s batches are essentially free above 100kHz:

Rate	Sample data / batch (full)	Sample data / batch (abbrev)	Dict % (v. full raw)	Dict % (v. abbreviated raw)
1 kHz	976.56 KB	732.42 KB	42.77%	49.91%
10 kHz	9.54 MB	7.15 MB	6.95%	9.06%
100 kHz	95.37 MB	71.53 MB	0.74%	0.99%
1 MHz	953.67 MB	715.26 MB	0.07%	0.10%

...and below no one would do 5s batches anyway.

Keep in mind though that this means you need to have quite an efficient encoding scheme, probably by "windowing" the data within an interval.

What the binary format does is very delta. Most records in real binary records seem to be POP_PUSH and REPEAT, but that'd be nightmarish over the wire or over the ring. :-)

It would be really nice if there was a very high performance option (i.e. ring buffer in shared memory) and using sockets as a "control channel" rather than forcing everything over sockets. profiling.sampling seems to support very high rates and so it would be a shame if we couldn't make use of that with streaming becuase we are trying to push everything over a socket.

What is the throughput you're thinking about?

My hunch is that AF_UNIX is a good enough transport layer. I like it because it's simple and still relatively cross-platform, even if there are many unnecessary copies.

Benchmarking AF_UNIX on a dockerized Linux on a MacBook Air M4:

msg_size	throughput_Mbit/s	throughput_Gbit/s (my conversion)
4096	44115.99	44.12
16384	92800.20	92.80
65536	134589.39	134.59
262144	150732.23	150.73
1048576	158313.23	158.31
4194304	151003.10	151.00

Where the Triad memory limit on this machine is 745 Gbit/s, so even with virtualization tax that's ~20% memory ceiling.

Either way, way above 1.584 Gbit/s.

I would also love if there was some binary format for low overhead encode/deocde on both sides.

My thinking is that binary format from a start would increase maintenance cost, and would be ideal if we could get away without it.

With binary format, we'd have to provide a client library. It would limit adoption to Python (realistically: cannot expect public API, and unless we commit to a documented stable spec and test vectors, binary likely implies maintaining client libraries) and Python -> binary trip would be a bit self-defeating in terms of performance.

On the other hand, SIMD JSON parsers are language-agnostic and able to process 1-4GB/s on comparable machines.

(where all data is sanpshot to disk when a "bad event" happens in the app).

What is "a bad event" in this case? A stack that matches a pattern?

What do you mean by "all data"? The current batch or just the current frame?

I'm working on a "one-command" performance debugging tool to collecting all the performance data someone might need and visualizing it on a single timeline.

https://github.com/LalitMaganti/sismo?

...wouldn't be nice if RemoteUnwinder was public Python API? ;-)

[LalitMaganti]

Do you think that windows / self-contained chunks / batches would be treated as runs? In other words, a regular run would be batch without a fixed duration / window?

I'm unclear on what "run" means in this context. I read the binary spec and I don't see any mention of it. From #146257, it seems like the run_id is set in the constructor and lives for the lifetime of the session.

If by run you just mean identifier for a batch of events, then that makes sense. It also makes sense to say the existing, non-streaming case is just a single batch. FWIW I would even be OK with making the batching an implementation detail as long as a single batch is not too big or too small.

Perhaps the meta message could be extended with duration here?

Sure seems reasonble. It might also make sense to make a size threshold rather than a duration threshold for better sampling when off-cpu. That is you will get more samples when on-cpu than off-cpu so if you do duration based batching, you would emit lots of empty samples where if you did size based batching, that problem goes away.

Specifically: would you model these windows / chunks/ batches as units within a single run?

I think I'm still not getting what run means: I'm thinking run and chunk are synonmous but this line seems like it's not so?

The obvious implication is that there will be many str_def and frame_def, and they always have to be buffered.

Agreed, you definitely need to buffer them per batch.

Do you think (putting aside if it's JSONL or would be sent binary) something like:

{"type":"raw","v":0,"run_id":"7-Zk2_Mq","seq":123456789,"ts_usec":[481293],"threads" [{"thread_id":1535,"status":1,"stack":[1,2,3]}]}
would be good?

If this is a single sample, I might be missing something but isn't this too verbose? I was thinking something more "columnar" i.e.

{"type":"batch","v":0,"run_id":"7-Zk2_Mq","seq":42,
 "start_ts_usec":1730000000123456,
 "duration_usec":5000000,
 "string_table":[...],
 "frame_table":[...],
 "threads":[
   {"tid":1535,
    "ts_delta_usec":[0,10,9,11,10,10,...],
    "status":[1,1,1,1,1,...],
    "stacks":[[1,2,3],[1,2,3],[1,2,4],[1,2,4],...]}
 ]}

A bit of my feeling is that high-performance pattern like shared memory ring buffer would require rewriting Tachyon to C.

How so? Python supports shared memory (https://docs.python.org/3/library/multiprocessing.shared_memory.html) without needing to write C, could we not use that? I think you understand the Tachyon implementations much better than me though so you are probably referring to something that I don't quite understand yet!

What the binary format does is very delta. Most records in real binary records seem to be POP_PUSH and REPEAT, but that'd be nightmarish over the wire or over the ring. :-)

It's not so bad if you do this binary encoding per-batch. That is, for each batch, you would see:

• the frame table
• the string table
• the starting timetamp

Then for each thread you would have:

• delta from the previous timestamp/start timestamp
• the stack as an encoding of the frame, run length encoding etc (same as the binary format)

What is the throughput you're thinking about?

Well I haven't dont any measurements here so take all the above with a huge grain of salt. My feeling just comes from the fact that Windows, Mac and Linux all do various ring buffer over shared-memory for commuicating between the OS and kernel for sampling.

My hunch is that AF_UNIX is a good enough transport layer. I like it because it's simple and cross-platform, even if there are many unnecessary copies.

The unnecessary copies is a bit of a bummer, especially on the read side vs just reading from an mmap but I can appreciate it makes things a lot simpler so may be a pragmatic choice

With binary format, we'd have to provide a client library. It would limit adoption to Python (realistically: cannot expect public API, and unless we commit to a documented stable spec and test vectors, binary likely implies maintaining client libraries) and Python -> binary trip would be a bit self-defeating in terms of performance.

This was actually one of the questions I asked Pablo (whether the binary format would become stable so we could write cross-language librairies) and he indicated he did want to stabilize the format. I'm not sure if that extends to any potential streaming binary format as well though

FWIW the binary format doesn't seem that bad: if I compare that with the perf binary format (which I had to implement support for in the past), it seems qutie delightful actually!

What is "a bad event" in this case? A stack that matches a pattern?

No it doesn't mean a stack: we wouldn't be directly introspecting the data flowing through, it's too expensive to do that online.

Instead, the bad event depends on the application. Some examples:

• slow processing of a request
• gui application missing a frame
• computation taking longer than the P95 for that computation
• an explicit "collect trace snapshot" or "collect bugreport" action built into the app

We take this approvach very commonly when we use Perfetto on Android system usecases. There it might be, an app being killed because of low memory, missed frames, slow app startup etc.

What do you mean by "all data"? The current batch or just the current frame?

Well downstream of the Python sampling code, I'm going to be collecting all the data from the different sources into a central Perfetto ring buffer: the snapshot will be of whatever happens to be in the ring buffer when the snapshot was requested. It doesn't really have any bearing on the profiling.sampling code beyond needing continuous streaming to work well :)

https://github.com/LalitMaganti/sismo?

Haha you sleuthed it out! I didn't want to make a big deal of it yet becuase it's very very early still, there's lots of things which I'd want to do before even a 0.1 but yes that's it :)

...wouldn't be nice if RemoteUnwinder was public Python API? ;-)

I'm always in favor of more introspection into the Python but certianly wouldn't say no! But FWIW I don't think it would help simso as much because, on the remote side, it will be a native process with no Python dependency (right now it's Zig, Rust, C and C++ all in one binary!)

[maurycy]

@LalitMaganti Ouch, fast, and I'm still travelling. :-) Thank you

To be honest, I have to sleep over re: columnar, and how to explain the runs v. chunks. I will also definitely take a look at sismo!

FWIW the binary format doesn't seem that bad: if I compare that with the perf binary format (which I had to implement support for in the past), it seems qutie delightful actually!

Never said that! It's very efficient. In this -m test run there are 81 million samples, 3 billion frames, and one frame takes 0.068 bits... 🫡 to Pablo.

I think this is a bummer:

https://github.com/python/cpython/blob/1fc2b38d631/Modules/_remote_debugging/binary_io_writer.c#L1041-L1201

...unless you just do "binary" self-contained chunks / periodical bumps / windows / whatever. Then it'd indeed actually work.

That said:

The unnecessary copies is a bit of a bummer, especially on the read side vs just reading from an mmap but I can appreciate it makes things a lot simpler so may be a pragmatic choice

My thinking goes is that less complexity we expect from the clients, the higher the adoption, and in this area consumers will be in many languages.

The above numbers seem to indicate that we can get away something that'd be parsable by mostly anyone, and still get a million samples per second on a cheap €1k laptop.

And we could also experiment with various tweaks to the JSONL format for a while... That's a bit contradictory with stabilizing the replay format.

Oh, and we get really TCP extremely easily... which would be a huge win in my opinion. The throughput is managable even as raw, without aggregates... (agg)

Generally: when I say streaming, I do not only mean in-process streaming but also dream over any network.

How so? Python supports shared memory (https://docs.python.org/3/library/multiprocessing.shared_memory.html) without needing to write C, could we not use that? I think you understand the Tachyon implementations much better than me though so you are probably referring to something that I don't quite understand yet!

So, I did not investigate this fully (yet 🙈) but my pet theory is that I'm unable to go from 100kHz -> 1Mhz (tbh I do not need for my use cases, but wanted to demonstrate if for the purpose of the benchmark) also because there's a constant overhead.

From what I noted when running it over -m test, around 60-70% the sample budget (7.4 µs per sample?) was wasted, and I assume it's because of the loop itself:

https://github.com/python/cpython/blob/1fc2b38d631/Lib/profiling/sampling/sample.py#L101-L153

Obviously, there are some allocators downstream, but I don't think they're an issue here.

So what I'm really thinking: ring-buffer is obviously the most common pattern here, but the frequency is effectively not that high because of Python -> C context switch...

EDIT: This bug (likely a race caused by probing empty stacks) indicates that the profiler, at least with 1MHz sampling rate, is indeed faster than our code...

And see the point above re: TCP.

I'm always in favor of more introspection into the Python but certianly wouldn't say no!

Above my pay grade to make this call!

[sergey-miryanov]

FWIW: GC stats now use ring buffers as well.

[superajun-wsj]

Hello @maurycy , I have a question about Tachyon. I noticed that the current Tachyon sample appears to focus on resolving Python stack frames only. I’m curious about the design decision behind this. Why does Tachyon only resolve Python stacks instead of supporting mixed Python + native stack traces, similar to tools like py-spy (when using native stack unwinding)?