Post-mortem out of memory analysis

[paulrutter]

Hi,

While reading the readme on https://github.com/nodejs/diagnostics, one of the domains is "Heap and Memory Analysis", although i cannot find anything about post-mortem out of memory analysis.

In our usecase, we run several hundred nodejs processes (in a Docker container), which are restricted to run with very low memory (+- 80MB). When these go out of memory, we want to be able to analyze what's on the heap when it happened.

For that purpose, we created "node-oom-heapdump" (https://github.com/blueconic/node-oom-heapdump), which does exactly that.

Is post-mortem out of memory analysis planned to be included in the Node.js core?

[joyeecheung]

Not in node core, but I tend to use https://github.com/nodejs/llnode to diagnose coredumps produced during OOM (it probably needs a few tweaks to generate coredumps in docker when the process crashes though). You can at least see the stacktrace (C++ and JS) when the OOM happens which tend to be useful (unless the process is not running the code that causes the problem when it crashes). You can also scan the heap and find the types with the most instances which is also helpful most of the times.

[paulrutter]

Thanks!

I have seen that one in the past but found it to be a bit too low-level for my purposes.
Of course, it might provide more insight, but the threshold to be able to use it seems to be a bit higher then just including our own module (which creates .heapsnapshot files which can be loaded in Chrome DevTools).
I would prefer something easier; just like Java does, it would be nice to have some kind of command line flag which automatically creates those heapdumps on out of memory (https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html).
Any ideas on this?

[joyeecheung]

@paulrutter There is an experimental JS API of llnode https://github.com/nodejs/llnode/blob/master/JSAPI.md which allows you to install it as a normal C++ Node.js addon and use the JS API to access info in the core dump (it's not yet completed, of course).

There is also https://github.com/nodejs/node-report which can dump a summary when the process crashes, probably a bit closer to the Java heapdump experience. The problem with heapdumps is that they only include information from the VM's managed heap, but sometimes the cause may come from the native layer, so it's sometimes inevitable to dig into there.

[joyeecheung]

Also, node-report is being integrated into core nodejs/node#22712 the current implementation allows you to use command line flags to configure when the report will be generated.

[paulrutter]

Thanks, that's very helpful.
I understand that a heapdump alone sometimes might not be enough, although for our usecases it usually is (we run pluggable JS code which usually is the cause for mem leaks).

I'll take a look into node-report to see if that's helpful 👍

[mmarchini]

Another useful tool is https://github.com/v8/sampling-heap-profiler, which shows allocation per function call. This tool is also more suited for production scenarios because taking heapdumps is expensive (high memory consumption) and slow, but it shouldn't be a problem if the process keeps memory usage around 80MB.

[paulrutter]

Thanks, that's another one not known to me until now.
Memory usage for generating heapdumps is not an issue in case of OoM, because the process is lost anyway. For on-demand heapdumps; those are created from another process, so memory consumption is not a risk for the process being analyzed (in node-oom-heapdump, other modules like v8-profiler run in the same process).

[mhdawson]

@paulrutter please let me know your experiences with node-report. Would like to hear about what worked well as well as other information that might be useful in node-report.

In my session coming up at NodeConfEU one of the demo's included is using nore-report to help when you have an OOM.

[paulrutter]

Hi @mhdawson,

I tried node-report yesterday (the fatalerror.js example included in the source), and although this is definitely helpful information in case of an OoM, i still miss information about what the heap contained at the time.
If a heapdump (v8::Isolate::GetCurrent()->GetHeapProfiler()->TakeHeapSnapshot();) was created as well, that would give more insight. Of course, as @joyeecheung mentioned, this only shows the VM managed heap and no native info.

[mhdawson]

@paulrutter taking a heap snapshot is also one of the core diagnostics capabilities that I believe should be in core. We have a PR working to get node-report into core itself (in a bit different form). After that I want to look at the best practice for getting a heapdump.

@richardlau at one point I had thought node-report was going to enable triggering a heap dump as well but maybe that was https://github.com/RuntimeTools/appmetrics instead.

You can get a heapdump with the https://www.npmjs.com/package/heapdump module and also through the inspector APIs (I've not had time to validate the later myself quite)

[paulrutter]

Thanks!

For the latter, we now created our own module called node-oom-heapdump, which works very well for us. But it would be nice to have this functionality in the core like you mentioned. Just a matter of time, i guess?

I could help on getting a pull request for node-report, if that would help?

[richardlau]

@mhdawson I have a feeling it was appmetrics and that ended up using the heapdump module.

[mhdawson]

@paulrutter you mean a PR to add heapdump trigger support in node-report or something else?

[joyeecheung]

By the way, the heap snapshot doesn't seem to work on larger heaps: https://bugs.chromium.org/p/chromium/issues/detail?id=826697

[mhdawson]

@paulrutter if you are interested we might collaborate on the path of having a heapdump solution in core. I'm quite interested in that but have just not had time to make progress. One thing people have mentioned is that it is possible through the inspector API so first step is to understand that. If its sufficient then documenting it as part of the diagnostics group's best practices and if not making the case for what we think should be in core. Adding @gireeshpunathil as another interested party.