Segmentation Fault when using a custom item

[kelvinhammond]

In libzim v8 when using a custom item and adding a lot of objects to a zim file it will throw a segmentation fault error after about 10,000 items have been inserted.

The error has been reported to nodejs/node-addon-api below.

It appears to be related to:

• the persistent object reference being lost or garbage collected before the object and function can be used
• some other unknown issue involving the global state within nodejs.
• it may also be related to the creation of ThreadSafeFunction(s) created within a ThreadSafeFunction context but this should be OK and does not crash until after about 10k or more iterations.

To reproduce the error modify tests/makeLargeZim.ts file to use the custom item and run the test, it should fail.

The issue has been reported to nodejs/node-addon-api#1174 .

Related to #72

[kelson42]

@kelvinhammond One hour ago, I have updated the build script to use libzim 8.1.0 which has been released yesterday (and fix/update the CI). I have also run (for the first time) $ npm run test-mem-leak twice on my Ubuntu 22.04 + Node.js 18.12.1 and... all works fine. It ran twice up to the end without any error message. Maybe it would be worth it you try again?

[kelvinhammond]

@kelson42 Nice, will try again. I wonder if they fixed it! Thank you.

I should have a bit more time soon (hopefully) to take a look at this and mwoffliner during the holidays.

[kelson42]

@kelvinhammond Thx, would be really helpful to know soon if this critical bug is really fixed or if I just don't run the reproduction case steps properly.

[kelvinhammond]

@kelvinhammond Thx, would be really helpful to know soon if this critical bug is really fixed or if I just don't run the reproduction case steps properly.

@kelson42 Try it with these changes
Edit test/makeLargeZim.ts
Uncomment the const customItem code and have await creator.addItem(stringItem); be replaced by await creator.addItem(customItem); and see if that works for you.

The diff looks like this.

diff --git a/test/makeLargeZim.ts b/test/makeLargeZim.ts
index b5db866..e87979e 100644
--- a/test/makeLargeZim.ts
+++ b/test/makeLargeZim.ts
@@ -43,13 +43,13 @@ console.log(`Making ZIM file with [${numArticles}] articles`);
       data,
     );

-    /*
     const customItem = { // custom item
       path: url,
       mimeType: "text/html",
       title: title,
       hints: {FRONT_ARTICLE: 1},
       getContentProvider() { // custom content provider
+        return stringItem.getContentProvider();
         let sent = false;
         return {
           size: data.length,
@@ -63,9 +63,9 @@ console.log(`Making ZIM file with [${numArticles}] articles`);
         };
       },
     };
-    */

-    await creator.addItem(stringItem);
+    //await creator.addItem(stringItem);
+    await creator.addItem(customItem);
   }

   console.log(`Finalising...`);

[kelvinhammond]

I tested it with node v19.1.0

I get

terminate called after throwing an instance of 'Napi::Error'

  what():  Illegal invocation

#8  0x00007ffff031a5c3 in Napi::Function::Call (this=0x7fffffffa0c0, recv=0x555555632990, argc=0, args=0x0) at 
/node-libzim/node_modules/node-addon-api/napi-inl.h:2341

#9  0x00007ffff031a482 in Napi::Function::Call (this=0x7fffffffa0c0, recv=0x555555632990, args=std::initializer_list of length 0) at /node-libzim/node_modules/node-addon-api/napi-inl.h:2306

#10 0x00007ffff0326f92 in ContentProviderWrapper::feed()::{lambda(Napi::Env, Napi::Function)#1}::operator()(Napi::Env, Napi::Function) const (__closure=0x7fffc40245f0, env=..., feedFunc=...) at ../src/contentProvider.h:79

Looking into this more.

[kelvinhammond]

@kelson42 It does work, just not with custom items.