Intermittent failure in ThreadSafe function

[mhdawson]

Failure in recent nightly run: @gabrielschulhof, @KevinEady as our threadsafe experts.

https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=fedora-latest-x64/3589/console

Running test 'threadsafe_function/threadsafe_function'


#
# Fatal error in , line 0
# Check failed: !object.IsSmi().
#
#
#
#FailureMessage Object: 0x7ffcf7a31110
 1: 0xae7a01  [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 2: 0x1b043b4 V8_Fatal(char const*, ...) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 3: 0xea7e92 v8::internal::PagedSpace::AllocateRawSlow(int, v8::internal::AllocationAlignment, v8::internal::AllocationOrigin) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 4: 0xe3d8a6 v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 5: 0xe455c8 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 6: 0xe0925a v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 7: 0xe04c72 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 8: 0xe04d24 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
 9: 0x1070bae v8::internal::Handle<v8::internal::FixedArray> v8::internal::String::CalculateLineEnds<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>, bool) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
10: 0x1031483 void v8::internal::Script::InitLineEnds<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Script>) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
11: 0x104554b v8::internal::Script::GetLineNumber(v8::internal::Handle<v8::internal::Script>, int) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
12: 0xdd5fc3 v8::internal::JSStackFrame::GetLineNumber() [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
13: 0xe13623 v8::internal::Factory::NewStackFrameInfo(v8::internal::Handle<v8::internal::FrameArray>, int) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
14: 0x1060928 v8::internal::StackTraceFrame::InitializeFrameInfo(v8::internal::Handle<v8::internal::StackTraceFrame>) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
15: 0x1060b00 v8::internal::StackTraceFrame::GetFrameInfo(v8::internal::Handle<v8::internal::StackTraceFrame>) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
16: 0x1063c85 v8::internal::SerializeStackTraceFrame(v8::internal::Isolate*, v8::internal::Handle<v8::internal::StackTraceFrame>, v8::internal::IncrementalStringBuilder*) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
17: 0x106480c v8::internal::SerializeStackTraceFrame(v8::internal::Isolate*, v8::internal::Handle<v8::internal::StackTraceFrame>) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
18: 0xcdb3d4  [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
19: 0xce32f6 v8::internal::Builtin_CallSitePrototypeToString(int, unsigned long*, v8::internal::Isolate*) [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
20: 0x1505f79  [/home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-v16.0.0-nightly20210209beee53884d-linux-x64/bin/node]
Tests aborted with SIGILL
npm ERR! code 1
npm ERR! path /home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/node-addon-api
npm ERR! command failed
npm ERR! command sh -c node test

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/iojs/build/workspace/node-test-node-addon-api-new/nodes/fedora-latest-x64/npm-cache/_logs/2021-02-10T09_48_16_307Z-debug.log
Build step 'Conditional steps (multiple)' marked build as failure
Sending e-mails to: michael_dawson@ca.ibm.com gabriel.schulhof@intel.com Arunesh.Chandra@microsoft.com
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

[mhdawson]

Also seeing occasional failure like this one: https://ci.nodejs.org/job/node-test-node-addon-api-new/nodes=ubuntu1604-64/3564/console

Running test 'threadsafe_function/threadsafe_function'
Running test 'threadsafe_function/threadsafe_function_ctx'
Running test 'threadsafe_function/threadsafe_function_existing_tsfn'
Running test 'threadsafe_function/threadsafe_function_ptr'
Running test 'threadsafe_function/threadsafe_function_sum'
Running test 'threadsafe_function/threadsafe_function_unref'
Running test 'typed_threadsafe_function/typed_threadsafe_function'
Running test 'typed_threadsafe_function/typed_threadsafe_function_ctx'
Running test 'typed_threadsafe_function/typed_threadsafe_function_existing_tsfn'
Running test 'typed_threadsafe_function/typed_threadsafe_function_ptr'
Running test 'typed_threadsafe_function/typed_threadsafe_function_sum'
Running test 'typed_threadsafe_function/typed_threadsafe_function_unref'
Running test 'typedarray-bigint'
Running test 'typedarray'
Running test 'version_management'

All tests passed!
Tests aborted with SIGSEGV
npm ERR! code 1
npm ERR! path /home/iojs/build/workspace/node-test-node-addon-api-new/nodes/ubuntu1604-64/node-addon-api
npm ERR! command failed
npm ERR! command sh -c node test

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/iojs/build/workspace/node-test-node-addon-api-new/nodes/ubuntu1604-64/npm-cache/_logs/2021-02-06T09_34_12_368Z-debug.log
Build step 'Conditional steps (multiple)' marked build as failure
Sending e-mails to: michael_dawson@ca.ibm.com gabriel.schulhof@intel.com Arunesh.Chandra@microsoft.com
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

[gabrielschulhof]

@mhdawson this looks like it's against Node.js nightly. Might it be a regression in core?

[gabrielschulhof]

The stack doesn't seem to contain any of our frames – unless I'm glossing over some.

[gabrielschulhof]

I built against nodejs/node@1c6484b and got

Running test 'threadsafe_function/threadsafe_function'
/home/nix/node/node/out/Release/node[3813]: ../src/api/callback.cc:121:void node::InternalCallbackScope::Close(): Assertion `(env_->execution_async_id()) == (0)' failed.
 1: 0xabc3c0 node::Abort() [/home/nix/node/node/out/Release/node]
 2: 0xabc43e  [/home/nix/node/node/out/Release/node]
 3: 0x9fcfe8 node::InternalCallbackScope::Close() [/home/nix/node/node/out/Release/node]
 4: 0x9fd3a3 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/home/nix/node/node/out/Release/node]
 5: 0x9fd52c node::MakeCallback(v8::Isolate*, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/home/nix/node/node/out/Release/node]
 6: 0xa3dd13 node::Environment::CheckImmediate(uv_check_s*) [/home/nix/node/node/out/Release/node]
 7: 0x155f1b9  [/home/nix/node/node/out/Release/node]
 8: 0x15578b4 uv_run [/home/nix/node/node/out/Release/node]
 9: 0x9fe64e node::SpinEventLoop(node::Environment*) [/home/nix/node/node/out/Release/node]
10: 0xb04537 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/home/nix/node/node/out/Release/node]
11: 0xa7fc50 node::Start(int, char**) [/home/nix/node/node/out/Release/node]
12: 0x7fea27c82f43 __libc_start_main [/lib64/libc.so.6]
13: 0x9faaee _start [/home/nix/node/node/out/Release/node]
Tests aborted with SIGABRT

[gabrielschulhof]

Hmmm ... I can't reproduce that crash.

[gabrielschulhof]

Trying to reproduce inside an Ubuntu 16.04 container.

[mhdawson]

The crash on shutdown with master ow seems to recreate consistently:

All tests passed!

Thread 1 "node" received signal SIGSEGV, Segmentation fault.
0x0000000000e7acb0 in v8::internal::GlobalHandles::ClearWeakness(unsigned long*) ()
Missing separate debuginfos, use: yum debuginfo-install libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) 

@gabrielschulhof I'm wondering if that might be related to the change from nodejs/node#37616. Any chance you still have the environment you had for that PR to compare before/after the change?

[mhdawson]

Cutting down the test case this is the one that is failing:

Starting program: /home/midawson/node/node-v16.0.0-nightly202103215318e53fd8-linux-x64/bin/node --expose-gc --no-concurrent-array-buffer-sweeping /home/midawson/newpull/land/node-addon-api/test/objectreference.js
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff6cbd700 (LWP 334846)]
[New Thread 0x7ffff64bc700 (LWP 334847)]
[New Thread 0x7ffff5cbb700 (LWP 334848)]
[New Thread 0x7ffff54ba700 (LWP 334849)]
[New Thread 0x7ffff4cb9700 (LWP 334850)]
[New Thread 0x7ffff7ff6700 (LWP 334851)]

Thread 1 "node" received signal SIGSEGV, Segmentation fault.
0x0000000000e7acb0 in v8::internal::GlobalHandles::ClearWeakness(unsigned long*) ()
Missing separate debuginfos, use: yum debuginfo-install libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) 

[mhdawson]

Cut down test case to these 2 and it still crashes:

  () => {
      binding.objectreference.setObjects("hello", "world", "javascript");
      const test = binding.objectreference.getFromValue();
      const test2 = binding.objectreference.getFromValue("hello");

      assert.deepEqual({ hello: "world" }, test);
      assert.deepEqual({ hello: "world" }, test2);
      assert.deepEqual(test, test2);
    },
    () => {
      binding.objectreference.setObjects(1, "hello world");
      const test = binding.objectreference.getFromValue();
      const test2 = binding.objectreference.getFromGetter(1);

      assert.deepEqual({ 1: "hello world"}, test);
      assert.equal("hello world", test2);
      assert.equal(test[1], test2);
    },

[mhdawson]

But not with just one of the 2

[mhdawson]

Just these 2 also seems to recreate

() => {
      binding.objectreference.setObjects(1, "hello world");
      const test = binding.objectreference.getFromValue();
      const test2 = binding.objectreference.getFromGetter(1);

      assert.deepEqual({ 1: "hello world"}, test);
      assert.equal("hello world", test2);
      assert.equal(test[1], test2);
    },

    () => {
      binding.objectreference.setObjects(0, "hello");
      binding.objectreference.setObjects(1, "world");
      const test = binding.objectreference.getFromValue();
      const test2 = binding.objectreference.getFromGetter(0);
      const test3 = binding.objectreference.getFromGetter(1);

      assert.deepEqual({ 1: "world"}, test);
      assert.equal(undefined, test2);
      assert.equal("world", test3);
    },

[mhdawson]

May require 2 tests so that a gc is forced after one of the tests complete ?

[mhdawson]

Cutting down the tests to just:

// info[0] is the key, which can be either a string or a number.
// info[1] is the value.
// info[2] is a flag that differentiates whether the key is a
// C string or a JavaScript string.
void SetObjects(const CallbackInfo& info) {
  Env env = info.Env();
  HandleScope scope(env);

 weak = Weak(Object::New(env));
 weak.SuppressDestruct();

 /* persistent = Persistent(Object::New(env));
  persistent.SuppressDestruct();

  reference = Reference<Object>::New(Object::New(env), 2);
  reference.SuppressDestruct();
  */

  if (info[0].IsString()) {
    if (info[2].As<String>() == String::New(env, "javascript")) {
//      weak.Set(info[0].As<String>(), info[1]);
//      persistent.Set(info[0].As<String>(), info[1]);
//      reference.Set(info[0].As<String>(), info[1]);
    } else {
      weak.Set(info[0].As<String>().Utf8Value(), info[1]);
      persistent.Set(info[0].As<String>().Utf8Value(), info[1]);
      reference.Set(info[0].As<String>().Utf8Value(), info[1]);
    }
  } else if (info[0].IsNumber()) {
    weak.Set(info[0].As<Number>(), info[1]);
    persistent.Set(info[0].As<Number>(), info[1]);
    reference.Set(info[0].As<Number>(), info[1]);
  }
}

    () => {
      binding.objectreference.setObjects("hello", "world", "javascript");
 //     const test = binding.objectreference.getFromValue();
 //     const test2 = binding.objectreference.getFromValue("hello");

   //   assert.deepEqual({ hello: "world" }, test);
//      assert.deepEqual({ hello: "world" }, test2);
  //    assert.deepEqual(test, test2);
    },
    () => {
//      binding.objectreference.setObjects(1, "hello world");
//      const test = binding.objectreference.getFromValue();
//      const test2 = binding.objectreference.getFromGetter(1);

//      assert.deepEqual({ 1: "hello world"}, test);
//      assert.equal("hello world", test2);
//      assert.equal(test[1], test2);
    },

still recreates

So it seems like just creating a Weak causes it? The strange thing is that I would have thought the C node-api tests would have caught an issue like that.

[mhdawson]

I've got to think its related to nodejs/node#37616, give the output of the crash being:Thread 1 "node" received signal SIGSEGV, Segmentation fault.
0x0000000000e7acb0 in v8::internal::GlobalHandles::ClearWeakness(unsigned long*) ()