Supporting multiple trace providers: Trace Events, DTrace, Lttng, ETW

[jasnell]

Node.js core supports multiple trace providers... we currently have to instrument for each individually, which increases maintenance burden. Before we consider making any changes here, however, we need to understand the critical use cases and requirements for the existing probes. Let's use this thread to discuss.

[jasnell]

Some feedback received so far:

• The ability to switch on tracing probe points on a running process is critical.
• The data collected at each probe point is also critical. We have existing probe points with existing data collections that should be maintained. Whether or not there are more that should be included needs to be determined.
• The semantics of the probe points must be maintained (e.g. gc_start must emit at the start of gc) but the specific function that triggers those is not as critical.

(will update this list as we go)

[jasnell]

One key question that needs to be answered: does a single Node.js binary need to support multiple trace mechanisms? For instance, current builds on OSX support both DTrace and v8 Trace Events independently. If we include a single instrumentation mechanism, does that need to support multiple trace backends at runtime, or is it ok to pick one backend at compile time?

[jclulow]

As long as DTrace probes are included in the OS X and SmartOS (and FreeBSD?) binaries provided by the project, I would definitely be OK with leaving out the v8 Trace Events probes.

[jasnell]

I have a distinct feeling that current users of the v8 trace event probes will say the exact same thing. I think we need to be resigned to the fact that the default node.js binary is simply going to be required to support multiple trace targets for the forseeable future. The trick will be finding a way of efficiently instrumenting once for those multiple targets.

[bnoordhuis]

Trace events are a lot more flexible, maintainable and consumable than static probes so if we had to pick one or the other, it's trace events all the way. I don't think it has to be either/or, though.

That said, the static probes do suffer from technical debt; see e.g. nodejs/node#18074. Get involved if you want to ensure their bright future.

[danielkhan]

I also thought that we agreed to think about deprecating DTrace in favor of trace events. If there is no feature gap to be expected, maybe we can initiate that process and think of EOLing DTrace with ~ v12?

[jasnell]

Given feedback I've received I don't believe we're going to be able to deprecate any time soon

[bnoordhuis]

I'm curious, feedback from whom? I can't imagine many people caring about our static probes; it might be a literal fingers-on-one-hand deal.

Data point: I quietly removed them from libuv a few years back and no one complained.

[AndreasMadsen]

Instead of supporting LTTng, could we look into using the Linux native trace events: https://lwn.net/Articles/379903/

If I remember correctly, they are also consumed by LTTng, so hopefully it will make little difference for the LTTng users. (edit: I remember correctly.)

[AndreasMadsen]

new LTTng PR: nodejs/node#18945 - it suggests there is some use.

[bnoordhuis]

Probably very little though. The build was broken for close to two years without anyone noticing.

[simark]

I opened the PR mentioned above, and was asked to provide some feedback here.

My problem is that we have a node process with an apparent memory leak (we let it run, the memory usage goes up endlessly). Inspecting using the devtools didn't show anything obvious, there wasn't a big difference before/after in the number or total size of allocated objects. But there was a difference of tens of gigabytes of mapped virtual memory (original issue: eclipse-theia/theia#1284).

I happen to know LTTng and saw that node had some LTTng instrumentation, so I wanted to try it, that's when I hit the issues addressed by my PR. I don't know if it will actually help, but I thought I'd give it a shot. I didn't look yet, but I assume there are some tracepoints in the memory allocation/deallocation functions. If there isn't, I thought I could add some. I am not familiar with the node code, so I'm in exploration mode :)

I think that properly instrumenting the binary with tracepoints placed at strategic places is useful. You won't use it often, but the day you need it you're happy it's there.

In the end I don't really mind if it's LTTng or another tracer. But one thing to keep in mind is that some tricky performance problems or rare crashes can only be reproduced in production, so it's sometimes necessary to enable tracing in production. LTTng is very efficient and designed for such use cases.

Tracing needs to have the smallest possible overhead. If it's possible to support multiple trace backends dynamically without any performance cost, then why not. But otherwise, if having to choose at compile time results in a more efficient solution, I would prefer that.

[ofrobots]

@simark lttng based tracing would probably not help find the memory leak if it happens to be in the JS heap. V8 JIT generates allocation code inline at each allocation site – something that would be hard to intercept using a tracing mechanism. You might want to check out the experimental sampling heap profiler in V8 that is designed for in-production memory leak debugging.

[simark]

@simark lttng based tracing would probably not help find the memory leak if it happens to be in the JS heap. V8 JIT generates allocation code inline at each allocation site – something that would be hard to intercept using a tracing mechanism.

And that JITed code doesn't call a common allocation function?

You might want to check out the experimental sampling heap profiler in V8 that is designed for in-production memory leak debugging.

Thanks, I'll give it a try.

[simark]

@simark in the interest of keeping the thread on topic of tracing, let's move the memory leak discussion to a separate thread, perhaps in the https://github.com/nodejs/help repo. If the leak is suspected in Node core itself, the issue can be in https://github.com/nodejs/node.

Sorry, I had both issues opened, and posted in the wrong tab. You can delete all history that this ever happened :)