Proposal: fix shared library linking on macOS

[haberman]

Problem Statement

It is currently impossible to use Zig to build shared (dynamic) libraries on macOS where some symbols are left undefined. For example, consider the program:

extern fn bar() void;

export fn foo() void {
    bar();
}

On Linux we can build a shared library from this without error:

$ zig build-lib -dynamic test.zig
$ ls
libtest.so  test.zig  zig-cache

However the same program compiled with the same command line fails on macOS:

$ zig build-lib -dynamic test.zig
warning(link): unexpected LLD stderr:
Undefined symbol: zig-cache/o/c4e95092e3ee25bb54ad9a24fb2ee9a5/test.o: _bar
symbol(s) not found

error: FileNotFound

This has previously been reported in #3085 and #8016.

Motivation

This is a blocker for creating loadable extension modules for Python, Ruby, Lua, Postgres, etc. -- basically any program that loads modules via dlopen() and expects that the symbols will be resolved at load time. If you only need your module to work on Linux, then Zig will work. But if you need macOS support, there appears to be no way to make this work, and no workaround that I could find.

Background

Both macOS and Linux have command line options that control whether symbols may be undefined at link time. However the options are different and the two platforms have opposite defaults. By default, Linux is lax and macOS is strict.

Linux

Linux has two different options:

• --no-undefined: causes the link to fail if unresolved symbols are present in the library currently being built. This is unset by default, so Linux allows undefined references by default, as we observed in the problem statement.
• --no-allow-shlib-undefined: causes the link to fail if unresolved symbols are present in any shared libraries that we are linking against (eg. pre-existing shared libraries). This defaults to unset (ie. allowing undefined references) when you are building a shared library, but the default is the opposite (failing for undefined references) if you are building an executable. This is a somewhat odd option that I don't know a real use case for, and macOS doesn't have an analogue to this that I am aware of.

(There are also a few other flags such as --unresolved-symbols=method and -z defs, which are just different porcelain around the same underlying options, AFAICT).

More info in this StackOverflow thread

macOS

macOS has just one option for this:

• -undefined treatment, where treatment is one of: error, warning, suppress, or dynamic_lookup. The default is error.

Existing practice for loadable modules on macOS is to prefer -undefined dynamic_lookup. For example, see: Building Python extension modules on OS X with cmake or autotools

Prior Art

zig cc has an existing option --allow-shlib-undefined. It appears to have the following semantics:

• This flag is only available for zig cc, not zig build-lib.
• On macOS, it adds -undefined dynamic_lookup to the link, which is just what we want. 👍👍👍
• On ELF targets, it adds --allow-shlib-undefined to the link. This seems wrong and possibly confused, since it is not the analogue of the -undefined dynamic_lookup flag on macOS (see the "Linux" section above).

Proposed Solution

The only hard requirement is that we can somehow pass -undefined dynamic_lookup to the linker on macOS. Anything that makes this possible, even an escape hatch like -Wl,arbitrary_linker_flag would solve the problem.

That said, here is my proposed solution:

1. Make Zig pass -undefined dynamic_lookup to the macOS linker by default for zig build-lib -dynamic. That way, anybody porting their code from Linux to macOS will find that it "just works" and no platform-specific tweaks are needed.
2. Do the same for zig cc, and deprecate the zig cc --allow-shlib-undefined option (since this behavior is the default now).
3. (If desired/requested): Add a zig build-lib --no-undefined option for cases where users do truly want to force the shared library to define all symbols. This option would remove -undefined dynamic_lookup on macOS and add --no-undefined on Linux.

I think the use case for (3) is somewhat rare: I personally have never used ld --no-undefined or heard of anybody using it. I'm sure there are some users for it out there, but in terms of bang for the buck this seems rather niche.

[mikdusan]

If a workaround is immediately needed you can minimally specify a target (at least 1 element has to be explicit). Here the macos component is explicit. This puts zig into a "cross-compile" mentality and it adds -undefined dynamic_lookup. Note we are in a state of flux with macos linking, so this should work now in master but no guarantees how long it will work.

$ zig build-lib -dynamic test.zig -target native-macos

[haberman]

Thanks @mikdusan. That appears to work!

Do you have any feedback on the proposal?

[mikdusan]

@haberman,

my initial thoughts are, with respect to creating dynamic-libs:

• keep zig {cc,c++,build-lib} undef treatment consistent with each target format native linker
• no changes required to zig {cc,c++} as it already exposes cli options to change undef treatment
• add cli options for build-lib (dynamic-lib) to set undef treatment

[haberman]

Thanks for the feedback @mikdusan!

keep zig {cc,c++,build-lib} undef treatment consistent with each target format native linker

This seems unfortunate. It forces each user has to go through the trouble of adding this flag when they port to Mac. What is the benefit?

I had thought the zig command-line driver was trying to provide relatively consistent semantics across platforms.

no changes required to zig {cc,c++} as it already exposes cli options to change undef treatment

In my proposal I concluded that the existing flag (--allow-shlib-undefined) has inconsistent semantics across ELF and Mach-O. Do you disagree with that assessment?

[mikdusan]

@haberman,

I'm sure there are more but these benefits come to mind when resolving symbols:

• what can be done at build-time to avoid run-time problems is a positive; wrong named symbol? c++ wrong typed? missing? if build-time linking fails something is really wrong
• dependencies on other libraries. consider:

gcc -shared -o libBar.so bar.c
gcc -shared -o libFoo.so foo.c -L. -lBar

• libFoo.so is encoded with libBar.so pathname and if done right will only require end-user to have -lFoo instead of both. Consider more frameworks with half dozen or more dependent libs
• on macos more information is embedded into libFoo.dylib, it supports two-level namespaces and associates symbols to libraries which has search/loadtime benefits, and a big help in diagnosing missing symbols: you know which lib it should be in

had thought the zig command-line driver was trying to provide relatively consistent semantics across platforms.

zig {cc,c++} drivers are very different from zig {build-obj,build-lib,build-exe}. One of the primary things is to make zig {cc,c++} work "as expected". This includes cmake and autotools style probes, which are complex and full of assumptions. So here's an example which may or may not hold true for this particular case about undef symbols:

1. detect compiler zig cc → cang version 11.0.1
2. detect kind of compiler: clang
3. detect os → macos
4. detect object format → macho

at this point is a detection test going to run to detect if undefined is allowed? The test isn't so simple. You can't just key off "try a simple clang -shared ..." and see if it fails. A real test has to make sure the failure is for the reason it's supposed to fail. So it's at this point that assumptions come in. If those 4 hypothetical steps were taken, the project config logic could simply be "symbols required" and conversely for gcc/linux/elf "undef is ok" and if we geared zig cc do do something different then things start to break everywhere and this stuff is difficult to diagnose.

has inconsistent semantics across ELF and Mach-O. Do you disagree with that assessment?

currently whether or not zig is in "cross-compilation" mode has an impact on --allow-shlib-undefined. It's something like "if cross-compiling then allow undefined" and I have a feeling it's too simplistic.

[haberman]

@mikdusan,

I take your point that macOS has a two level namespace, which confers extra benefits if a dylib is available for resolution at link time. But I disagree when you say "if build-time linking fails something is really wrong" -- this is a common case when libraries are intended for dlopen() (eg. native extensions for language interpreters).

If build errors for undefined symbols were solving lots of problems, I would expect to see more users opting into these checks on Linux by using -Wl,--no-undefined. But I don't think this is a common practice.

zig {cc,c++} drivers are very different from zig {build-obj,build-lib,build-exe}.

I totally agree: zig cc is trying to mimic the interface of existing cc command-line drivers. I agree that it should do whatever cc does now.

But for zig build-lib, which doesn't have this backward-compatibility constraint, would it make sense to make it more consistent across platforms?

[haberman]

This is a somewhat more subtle and complicated point, so I'm putting it in a separate comment:

currently whether or not zig is in "cross-compilation" mode has an impact on --allow-shlib-undefined.

I'm making a slightly different point, which is that zig cc --allow-shlib-undefined seems to be treating macOS -undefined dynamic_lookup and Linux --allow-shlib-undefined as equivalent, but they aren't quite the same.

Take a command like gcc -shared -o libFoo.so foo.c libBar.so. There are two separate checks you can do:

1. Do we throw errors if foo.c has unresolved symbols (ie. direct refs)?
2. Do we throw errors if libBar.so has unresolved symbols (ie. shlib refs)?

On macOS, -undefined dynamic_lookup affects both direct refs and shlib refs. It only offers the user one actual bit of state that controls both checks. Whereas the Linux linker has two bits of state that the user can control independently. I tried to gather some more information on exactly how these flags work, and this is what I came up with:

platform	build type	flags	checks direct refs	checks shlib refs
macOS	-shared		✅
macOS	-shared	-flat_namespace	✅	✅*
macOS	-shared	-undefined dynamic_lookup
macOS	binary		✅
macOS	binary	-flat_namespace	✅	✅*
macOS	binary	-undefined dynamic_lookup
Linux	-shared
Linux	-shared	--no-undefined or -z defs	✅
Linux	-shared	--no-allow-shlib-undefined		✅
Linux	binary		✅	✅
Linux	binary	--allow-shlib-undefined	✅
Linux	binary	z undefs		✅

*) It will check these if the dylib was also built with -flat_namespace

If zig cc is trying to behave more like regular cc, it seems like it shouldn't try to translate between --allow-shlib-undefined and -undefined dynamic_lookup since these two have different semantics.

[mikdusan]

• on macos:

mode	command	linker arg of significance
host	zig cc -shared -o foo foo.o -v
host	zig cc -shared -o foo foo.o -v -allow-shlib-undefined
cross	zig cc -shared -o foo foo.o -v -target x86_64-macos	-undefined dynamic_lookup
cross	zig cc -shared -o foo foo.o -v -allow-shlib-undefined -target x86_64-macos	-undefined dynamic_lookup

• on linux:

mode	command	linker arg of significance
host	zig cc -shared -o foo foo.o -v
host	zig cc -shared -o foo foo.o -v -allow-shlib-undefined
cross	zig cc -shared -o foo foo.o -v -target x86_64-linux	--allow-shlib-undefined
cross	zig cc -shared -o foo foo.o -v -allow-shlib-undefined -target x86_64-linux	--allow-shlib-undefined

[wojtekmach]

Hey everyone, I tried ignoring missing symbols on latest master but couldn't get it to work.

% uname -srm
Darwin 20.5.0 x86_64
% zig version
0.9.0-dev.339+81bf05bf6

// hello.c
extern int external_hello();

int hello() {
  return external_hello();
}

% zig cc hello.c -o libhello.dylib -shared -allow-shlib-undefined
zig: warning: argument unused during compilation: '-allow-shlib-undefined' [-Wunused-command-line-argument]
error(zld): undefined reference to symbol '_external_hello'
error(zld):     | referenced in /Users/wojtek/.cache/zig/o/778bb55ecf5c94c3ec9ed6c002690aea/hello.o
error: UndefinedSymbolReference

% zig build-lib hello.c -dynamic -fallow-shlib-undefined
error(zld): undefined reference to symbol '_external_hello'
error(zld):     | referenced in /Users/wojtek/.cache/zig/o/2bfd50fea2d903d80e9b2c1223d3a654/hello.o
error: UndefinedSymbolReference

I assume it's because zld doesn't support this flag yet. Any help, workarounds, etc, would be really appreciated.