Build failures with ninja

[gibfahn]

• Version: master
• Platform: Linux
• Subsystem: build, ninja

Versions

• ccache 3.3.3
• ninja 1.7.2

Worth raising an issue to cover ninja build issues in these comments, reported by @sam-github and @kenany.

Info:

(copied from above thread)

Git bisect shows breaking commit as: e03a7b2

$ ./configure && tools/gyp_node.py -f ninja && ninja -C out/Release

...

[2248/2248] LINK node
FAILED: node 
ccache g++ -pthread -rdynamic -m64 -Wl,--whole-archive,obj.target/deps/openssl/libopenssl.a -Wl,--no-whole-archive -Wl,-z,noexecstack -Wl,--whole-archive libv8_base.a -Wl,--no-whole-archive -pthread -o node -Wl,--start-group obj/src/node.debug-agent.o obj/src/node.async-wrap.o obj/src/node.env.o obj/src/node.fs_event_wrap.o obj/src/node.cares_wrap.o obj/src/node.connection_wrap.o obj/src/node.connect_wrap.o obj/src/node.handle_wrap.o obj/src/node.js_stream.o obj/src/node.node.o obj/src/node.node_buffer.o obj/src/node.node_config.o obj/src/node.node_constants.o obj/src/node.node_contextify.o obj/src/node.node_file.o obj/src/node.node_http_parser.o obj/src/node.node_javascript.o obj/src/node.node_main.o obj/src/node.node_os.o obj/src/node.node_revert.o obj/src/node.node_url.o obj/src/node.node_util.o obj/src/node.node_v8.o obj/src/node.node_stat_watcher.o obj/src/node.node_watchdog.o obj/src/node.node_zlib.o obj/src/node.node_i18n.o obj/src/node.pipe_wrap.o obj/src/node.signal_wrap.o obj/src/node.spawn_sync.o obj/src/node.string_bytes.o obj/src/node.stream_base.o obj/src/node.stream_wrap.o obj/src/node.tcp_wrap.o obj/src/node.timer_wrap.o obj/src/node.tty_wrap.o obj/src/node.process_wrap.o obj/src/node.udp_wrap.o obj/src/node.uv.o obj/src/node.util.o obj/src/node.string_search.o obj/src/node.inspector_agent.o obj/src/node.inspector_socket.o obj/src/node.node_crypto.o obj/src/node.node_crypto_bio.o obj/src/node.node_crypto_clienthello.o obj/src/node.tls_wrap.o obj/src/node.backtrace_posix.o obj/deps/v8/src/libv8_libplatform.a obj/tools/icu/libicui18n.a obj/deps/v8_inspector/src/inspector/libstandalone_inspector.a obj/deps/openssl/libopenssl.a obj/deps/zlib/libzlib.a obj/deps/http_parser/libhttp_parser.a obj/deps/cares/libcares.a obj/deps/uv/libuv.a obj/deps/v8/src/libv8_base.a obj/deps/v8/src/libv8_libbase.a obj/deps/v8/src/libv8_libsampler.a obj/tools/icu/libicuucx.a obj/tools/icu/libicudata.a obj/tools/icu/libicustubdata.a obj/deps/v8/src/libv8_snapshot.a -Wl,--end-group  -ldl -lrt -lm
g++: error: libv8_base.a: No such file or directory
ninja: build stopped: subcommand failed.

[gibfahn]

I think this may be a linux specific issue, I'm not having an issue on macOS Sierra (ccache 3.3.2, ninja 1.7.1), current master (330e63c).

[Fishrock123]

cc @bnoordhuis since you made that commit.

It still builds fine for me on OS X 10.10.5

[bnoordhuis]

ninja-on-linux probably puts the .a in a different location when standalone_static_library is on. Long shot but does this patch help?

diff --git a/common.gypi b/common.gypi
index d87205e..f9f1597 100644
--- a/common.gypi
+++ b/common.gypi
@@ -42,7 +42,7 @@
         'os_posix': 1,
         'v8_postmortem_support%': 'true',
       }],
-      ['GENERATOR == "ninja" or OS== "mac"', {
+      ['OS=="mac"', {
         'OBJ_DIR': '<(PRODUCT_DIR)/obj',
         'V8_BASE': '<(PRODUCT_DIR)/libv8_base.a',
       }, {

[kenany]

@bnoordhuis With the patch:

[2248/2248] LINK node
FAILED: node 
ccache g++ -pthread -rdynamic -m64 -Wl,--whole-archive,obj.target/deps/openssl/libopenssl.a -Wl,--no-whole-archive -Wl,-z,noexecstack -Wl,--whole-archive obj.target/deps/v8/src/libv8_base.a -Wl,--no-whole-archive -pthread -o node -Wl,--start-group obj/src/node.debug-agent.o obj/src/node.async-wrap.o obj/src/node.env.o obj/src/node.fs_event_wrap.o obj/src/node.cares_wrap.o obj/src/node.connection_wrap.o obj/src/node.connect_wrap.o obj/src/node.handle_wrap.o obj/src/node.js_stream.o obj/src/node.node.o obj/src/node.node_buffer.o obj/src/node.node_config.o obj/src/node.node_constants.o obj/src/node.node_contextify.o obj/src/node.node_file.o obj/src/node.node_http_parser.o obj/src/node.node_javascript.o obj/src/node.node_main.o obj/src/node.node_os.o obj/src/node.node_revert.o obj/src/node.node_url.o obj/src/node.node_util.o obj/src/node.node_v8.o obj/src/node.node_stat_watcher.o obj/src/node.node_watchdog.o obj/src/node.node_zlib.o obj/src/node.node_i18n.o obj/src/node.pipe_wrap.o obj/src/node.signal_wrap.o obj/src/node.spawn_sync.o obj/src/node.string_bytes.o obj/src/node.stream_base.o obj/src/node.stream_wrap.o obj/src/node.tcp_wrap.o obj/src/node.timer_wrap.o obj/src/node.tty_wrap.o obj/src/node.process_wrap.o obj/src/node.udp_wrap.o obj/src/node.uv.o obj/src/node.util.o obj/src/node.string_search.o obj/src/node.inspector_agent.o obj/src/node.inspector_socket.o obj/src/node.node_crypto.o obj/src/node.node_crypto_bio.o obj/src/node.node_crypto_clienthello.o obj/src/node.tls_wrap.o obj/src/node.backtrace_posix.o obj/deps/v8/src/libv8_libplatform.a obj/tools/icu/libicui18n.a obj/deps/v8_inspector/src/inspector/libstandalone_inspector.a obj/deps/openssl/libopenssl.a obj/deps/zlib/libzlib.a obj/deps/http_parser/libhttp_parser.a obj/deps/cares/libcares.a obj/deps/uv/libuv.a obj/deps/v8/src/libv8_base.a obj/deps/v8/src/libv8_libbase.a obj/deps/v8/src/libv8_libsampler.a obj/tools/icu/libicuucx.a obj/tools/icu/libicudata.a obj/tools/icu/libicustubdata.a obj/deps/v8/src/libv8_snapshot.a -Wl,--end-group  -ldl -lrt -lm
g++: error: obj.target/deps/v8/src/libv8_base.a: No such file or directory
ninja: build stopped: subcommand failed.

[spacekitteh]

I'm experiencing this error while trying to patch Nixos's nodejs distribution to use the OS's v8 install. I can't figure out how to change the dependency on /out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a.

[bnoordhuis]

With the patch, libv8_base.a is referenced twice in the link step with different paths: obj.target/deps/v8/src/libv8_base.a vs. obj/deps/v8/src/libv8_base.a. Second take:

diff --git a/common.gypi b/common.gypi
index d87205e..3afd4c4 100644
--- a/common.gypi
+++ b/common.gypi
@@ -42,12 +42,19 @@
         'os_posix': 1,
         'v8_postmortem_support%': 'true',
       }],
-      ['GENERATOR == "ninja" or OS== "mac"', {
+      ['OS=="mac"', {
         'OBJ_DIR': '<(PRODUCT_DIR)/obj',
         'V8_BASE': '<(PRODUCT_DIR)/libv8_base.a',
       }, {
-        'OBJ_DIR': '<(PRODUCT_DIR)/obj.target',
-        'V8_BASE': '<(PRODUCT_DIR)/obj.target/deps/v8/src/libv8_base.a',
+        'conditions': [
+          ['GENERATOR=="ninja"', {
+            'OBJ_DIR': '<(PRODUCT_DIR)/obj',
+            'V8_BASE': '<(PRODUCT_DIR)/obj/deps/v8/src/libv8_base.a',
+          }, {
+            'OBJ_DIR': '<(PRODUCT_DIR)/obj.target',
+            'V8_BASE': '<(PRODUCT_DIR)/obj.target/deps/v8/src/libv8_base.a',
+          }],
+        ],
       }],
       ['openssl_fips != ""', {
         'OPENSSL_PRODUCT': 'libcrypto.a',

Actually, I suspect this patch makes things needlessly complex. Third take:

diff --git a/common.gypi b/common.gypi
index d87205e..07a3970 100644
--- a/common.gypi
+++ b/common.gypi
@@ -43,11 +43,9 @@
         'v8_postmortem_support%': 'true',
       }],
       ['GENERATOR == "ninja" or OS== "mac"', {
-        'OBJ_DIR': '<(PRODUCT_DIR)/obj',
         'V8_BASE': '<(PRODUCT_DIR)/libv8_base.a',
       }, {
-        'OBJ_DIR': '<(PRODUCT_DIR)/obj.target',
-        'V8_BASE': '<(PRODUCT_DIR)/obj.target/deps/v8/src/libv8_base.a',
+        'V8_BASE': '<(LIB_DIR)/deps/v8/src/libv8_base.a',
       }],
       ['openssl_fips != ""', {
         'OPENSSL_PRODUCT': 'libcrypto.a',
diff --git a/node.gyp b/node.gyp
index e5f02d7..b077902 100644
--- a/node.gyp
+++ b/node.gyp
@@ -764,10 +764,10 @@
             {
               'action_name': 'node_dtrace_provider_o',
               'inputs': [
-                '<(OBJ_DIR)/node/src/node_dtrace.o',
+                '<(LIB_DIR)/node/src/node_dtrace.o',
               ],
               'outputs': [
-                '<(OBJ_DIR)/node/src/node_dtrace_provider.o'
+                '<(LIB_DIR)/node/src/node_dtrace_provider.o'
               ],
               'action': [ 'dtrace', '-G', '-xnolibs', '-s', 'src/node_provider.d',
                 '<@(_inputs)', '-o', '<@(_outputs)' ]
@@ -817,7 +817,7 @@
                 '<(SHARED_INTERMEDIATE_DIR)/v8constants.h'
               ],
               'outputs': [
-                '<(OBJ_DIR)/node/src/node_dtrace_ustack.o'
+                '<(LIB_DIR)/node/src/node_dtrace_ustack.o'
               ],
               'conditions': [
                 [ 'target_arch=="ia32"', {
@@ -978,7 +978,7 @@
             {
               'action_name': 'expfile',
               'inputs': [
-                '<(OBJ_DIR)'
+                '<(LIB_DIR)'
               ],
               'outputs': [
                 '<(PRODUCT_DIR)/node.exp'

Please let me know whether they work for you.

[kenany]

@bnoordhuis Second take:

[2251/2251] LINK node
FAILED: node 
ccache g++ -pthread -rdynamic -m64 -Wl,--whole-archive,obj.target/deps/openssl/libopenssl.a -Wl,--no-whole-archive -Wl,-z,noexecstack -Wl,--whole-archive obj/deps/v8/src/libv8_base.a -Wl,--no-whole-archive -pthread -o node -Wl,--start-group obj/src/node.debug-agent.o obj/src/node.async-wrap.o obj/src/node.env.o obj/src/node.fs_event_wrap.o obj/src/node.cares_wrap.o obj/src/node.connection_wrap.o obj/src/node.connect_wrap.o obj/src/node.handle_wrap.o obj/src/node.js_stream.o obj/src/node.node.o obj/src/node.node_buffer.o obj/src/node.node_config.o obj/src/node.node_constants.o obj/src/node.node_contextify.o obj/src/node.node_file.o obj/src/node.node_http_parser.o obj/src/node.node_javascript.o obj/src/node.node_main.o obj/src/node.node_os.o obj/src/node.node_revert.o obj/src/node.node_url.o obj/src/node.node_util.o obj/src/node.node_v8.o obj/src/node.node_stat_watcher.o obj/src/node.node_watchdog.o obj/src/node.node_zlib.o obj/src/node.node_i18n.o obj/src/node.pipe_wrap.o obj/src/node.signal_wrap.o obj/src/node.spawn_sync.o obj/src/node.string_bytes.o obj/src/node.stream_base.o obj/src/node.stream_wrap.o obj/src/node.tcp_wrap.o obj/src/node.timer_wrap.o obj/src/node.tty_wrap.o obj/src/node.process_wrap.o obj/src/node.udp_wrap.o obj/src/node.uv.o obj/src/node.util.o obj/src/node.string_search.o obj/src/node.inspector_agent.o obj/src/node.inspector_socket.o obj/src/node.inspector_socket_server.o obj/src/node.node_crypto.o obj/src/node.node_crypto_bio.o obj/src/node.node_crypto_clienthello.o obj/src/node.tls_wrap.o obj/src/node.backtrace_posix.o obj/deps/v8/src/libv8_libplatform.a obj/tools/icu/libicui18n.a obj/deps/v8_inspector/src/inspector/libstandalone_inspector.a obj/deps/openssl/libopenssl.a obj/deps/zlib/libzlib.a obj/deps/http_parser/libhttp_parser.a obj/deps/cares/libcares.a obj/deps/uv/libuv.a obj/deps/v8/src/libv8_base.a obj/deps/v8/src/libv8_libbase.a obj/deps/v8/src/libv8_libsampler.a obj/tools/icu/libicuucx.a obj/tools/icu/libicudata.a obj/tools/icu/libicustubdata.a obj/deps/v8/src/libv8_snapshot.a -Wl,--end-group  -ldl -lrt -lm
/usr/bin/ld: cannot find obj.target/deps/openssl/libopenssl.a: No such file or directory
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Third take:

[2251/2251] LINK node
FAILED: node 
ccache g++ -pthread -rdynamic -m64 -Wl,--whole-archive,obj.target/deps/openssl/libopenssl.a -Wl,--no-whole-archive -Wl,-z,noexecstack -Wl,--whole-archive libv8_base.a -Wl,--no-whole-archive -pthread -o node -Wl,--start-group obj/src/node.debug-agent.o obj/src/node.async-wrap.o obj/src/node.env.o obj/src/node.fs_event_wrap.o obj/src/node.cares_wrap.o obj/src/node.connection_wrap.o obj/src/node.connect_wrap.o obj/src/node.handle_wrap.o obj/src/node.js_stream.o obj/src/node.node.o obj/src/node.node_buffer.o obj/src/node.node_config.o obj/src/node.node_constants.o obj/src/node.node_contextify.o obj/src/node.node_file.o obj/src/node.node_http_parser.o obj/src/node.node_javascript.o obj/src/node.node_main.o obj/src/node.node_os.o obj/src/node.node_revert.o obj/src/node.node_url.o obj/src/node.node_util.o obj/src/node.node_v8.o obj/src/node.node_stat_watcher.o obj/src/node.node_watchdog.o obj/src/node.node_zlib.o obj/src/node.node_i18n.o obj/src/node.pipe_wrap.o obj/src/node.signal_wrap.o obj/src/node.spawn_sync.o obj/src/node.string_bytes.o obj/src/node.stream_base.o obj/src/node.stream_wrap.o obj/src/node.tcp_wrap.o obj/src/node.timer_wrap.o obj/src/node.tty_wrap.o obj/src/node.process_wrap.o obj/src/node.udp_wrap.o obj/src/node.uv.o obj/src/node.util.o obj/src/node.string_search.o obj/src/node.inspector_agent.o obj/src/node.inspector_socket.o obj/src/node.inspector_socket_server.o obj/src/node.node_crypto.o obj/src/node.node_crypto_bio.o obj/src/node.node_crypto_clienthello.o obj/src/node.tls_wrap.o obj/src/node.backtrace_posix.o obj/deps/v8/src/libv8_libplatform.a obj/tools/icu/libicui18n.a obj/deps/v8_inspector/src/inspector/libstandalone_inspector.a obj/deps/openssl/libopenssl.a obj/deps/zlib/libzlib.a obj/deps/http_parser/libhttp_parser.a obj/deps/cares/libcares.a obj/deps/uv/libuv.a obj/deps/v8/src/libv8_base.a obj/deps/v8/src/libv8_libbase.a obj/deps/v8/src/libv8_libsampler.a obj/tools/icu/libicuucx.a obj/tools/icu/libicudata.a obj/tools/icu/libicustubdata.a obj/deps/v8/src/libv8_snapshot.a -Wl,--end-group  -ldl -lrt -lm
g++: error: libv8_base.a: No such file or directory
ninja: build stopped: subcommand failed.

[bnoordhuis]

Okay, I'm going to leave this one for someone who cares about ninja. At least you know where to look now. :-)

[kenany]

@bnoordhuis I've worked upon your second take and have arrived at the following patch which appears to fix the issue. Worthy of a PR?:

diff --git a/common.gypi b/common.gypi
index 0fa36197be..8ce2aa72ab 100644
--- a/common.gypi
+++ b/common.gypi
@@ -42,12 +42,19 @@
         'os_posix': 1,
         'v8_postmortem_support%': 'true',
       }],
-      ['GENERATOR == "ninja" or OS== "mac"', {
+      ['OS=="mac"', {
         'OBJ_DIR': '<(PRODUCT_DIR)/obj',
         'V8_BASE': '<(PRODUCT_DIR)/libv8_base.a',
       }, {
-        'OBJ_DIR': '<(PRODUCT_DIR)/obj.target',
-        'V8_BASE': '<(PRODUCT_DIR)/obj.target/deps/v8/src/libv8_base.a',
+        'conditions': [
+          ['GENERATOR=="ninja"', {
+            'OBJ_DIR': '<(PRODUCT_DIR)/obj',
+            'V8_BASE': '<(PRODUCT_DIR)/obj/deps/v8/src/libv8_base.a',
+          }, {
+            'OBJ_DIR': '<(PRODUCT_DIR)/obj.target',
+            'V8_BASE': '<(PRODUCT_DIR)/obj.target/deps/v8/src/libv8_base.a',
+          }],
+        ],
       }],
       ['openssl_fips != ""', {
         'OPENSSL_PRODUCT': 'libcrypto.a',
diff --git a/node.gyp b/node.gyp
index e3f8aa83af..e48a5cf982 100644
--- a/node.gyp
+++ b/node.gyp
@@ -392,7 +392,7 @@
                     ['OS in "linux freebsd" and node_shared=="false"', {
                       'ldflags': [
                         '-Wl,--whole-archive,'
-                            '<(PRODUCT_DIR)/obj.target/deps/openssl/'
+                            '<(OBJ_DIR)/deps/openssl/'
                             '<(OPENSSL_PRODUCT)',
                         '-Wl,--no-whole-archive',
                       ],

[bnoordhuis]

Looks reasonable to me as long as the make-based build keeps working.

Untested, but perhaps you can even get rid of the GENERATOR=="ninja" conditional and simply define OBJ_DIR and V8_BASE like this:

'OBJ_DIR': '<(INTERMEDIATE_DIR)/../..',
'V8_BASE': '<(OBJ_DIR)/deps/v8/src/libv8_base.a',

[kenany]

Unfortunately doesn't seem to work :(

$ make -j4
/usr/bin/python2 tools/gyp_node.py -f make
gyp: Undefined variable OBJ_DIR in /home/kenan/code/node/node.gyp
make: *** [Makefile:84: out/Makefile] Error 1

Details

diff --git a/common.gypi b/common.gypi
index 0fa36197be..65b05112fd 100644
--- a/common.gypi
+++ b/common.gypi
@@ -42,12 +42,12 @@
         'os_posix': 1,
         'v8_postmortem_support%': 'true',
       }],
-      ['GENERATOR == "ninja" or OS== "mac"', {
+      ['OS=="mac"', {
         'OBJ_DIR': '<(PRODUCT_DIR)/obj',
         'V8_BASE': '<(PRODUCT_DIR)/libv8_base.a',
       }, {
-        'OBJ_DIR': '<(PRODUCT_DIR)/obj.target',
-        'V8_BASE': '<(PRODUCT_DIR)/obj.target/deps/v8/src/libv8_base.a',
+        'OBJ_DIR': '<(INTERMEDIATE_DIR)/../..',
+        'V8_BASE': '<(OBJ_DIR)/deps/v8/src/libv8_base.a',
       }],
       ['openssl_fips != ""', {
         'OPENSSL_PRODUCT': 'libcrypto.a',
diff --git a/node.gyp b/node.gyp
index e3f8aa83af..e48a5cf982 100644
--- a/node.gyp
+++ b/node.gyp
@@ -392,7 +392,7 @@
                     ['OS in "linux freebsd" and node_shared=="false"', {
                       'ldflags': [
                         '-Wl,--whole-archive,'
-                            '<(PRODUCT_DIR)/obj.target/deps/openssl/'
+                            '<(OBJ_DIR)/deps/openssl/'
                             '<(OPENSSL_PRODUCT)',
                         '-Wl,--no-whole-archive',
                       ],

But yeah, the diff I posted still passes make clean && make && make test.