[SR-14008] "Library not loaded: /usr/lib/swift/libswift_Differentiation.dylib" issue for `swift test`

[dan-zheng]

Previous ID	SR-14008
Radar	None
Original Reporter	@dan-zheng
Type	Bug

Attachment: Download

Environment

https://swift.org/builds/development/xcode/swift-DEVELOPMENT-SNAPSHOT-2020-12-23-a/swift-DEVELOPMENT-SNAPSHOT-2020-12-23-a-osx.pkg

Apple Swift version 5.3-dev (LLVM bcee1b98f3b26c5, Swift dbcf7fe1bf76226)
Target: x86_64-apple-darwin19.6.0

Additional Detail from JIRA

Votes	2
Component/s	Package Manager
Labels	Bug, AutoDiff
Assignee	None
Priority	Medium

md5: 4ecdf44eee77ebefd73035c76ce61be5

Issue Description:

With development snapshot toolchains, swift build/swift run for executable/library targets importing _Differentiation has no problem.

However, running swift test for test targets importing _Differentiation fails with a dynamic linker issue:

$ swift test
[7/7] Linking SwiftDifferentiationPackageTests

* Build Completed!
2020-12-30 14:28:23.465 xctest[20739:26994456] The bundle “SwiftDifferentiationPackageTests.xctest” couldn’t be loaded because it is damaged or missing necessary resources. Try reinstalling the bundle.
2020-12-30 14:28:23.465 xctest[20739:26994456] (dlopen_preflight(/Users/danielzheng/Differentiation/.build/x86_64-apple-macosx/debug/SwiftDifferentiationPackageTests.xctest/Contents/MacOS/SwiftDifferentiationPackageTests): Library not loaded: /usr/lib/swift/libswift_Differentiation.dylib
  Referenced from: /Users/danielzheng/Differentiation/.build/x86_64-apple-macosx/debug/SwiftDifferentiationPackageTests.xctest/Contents/MacOS/SwiftDifferentiationPackageTests
  Reason: image not found)

Setting DYLD_LIBRARY_PATH to the toolchain library directory containing libswift_Differentiation.dylib does not help:

$ DYLD_LIBRARY_PATH=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2020-12-23-a.xctoolchain/usr/lib/swift/macosx swift test
2020-12-30 14:29:55.586 xctest[20925:26997481] The bundle “SwiftDifferentiationPackageTests.xctest” couldn’t be loaded because it is damaged or missing necessary resources. Try reinstalling the bundle.
2020-12-30 14:29:55.586 xctest[20925:26997481] (dlopen_preflight(/Users/danielzheng/Differentiation/.build/x86_64-apple-macosx/debug/SwiftDifferentiationPackageTests.xctest/Contents/MacOS/SwiftDifferentiationPackageTests): Library not loaded: /usr/lib/swift/libswift_Differentiation.dylib
  Referenced from: /Users/danielzheng/Differentiation/.build/x86_64-apple-macosx/debug/SwiftDifferentiationPackageTests.xctest/Contents/MacOS/SwiftDifferentiationPackageTests
  Reason: image not found)

However, manually invoking the xctest command from swift test -v with DYLD_LIBRARY_PATH does work:

$ DYLD_LIBRARY_PATH=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2020-12-23-a.xctoolchain/usr/lib/swift/macosx /Applications/Xcode-beta12_3.app/Contents/Developer/usr/bin/xctest /Users/danielzheng/Differentiation/.build/x86_64-apple-macosx/debug/SwiftDifferentiationPackageTests.xctest
Test Suite 'All tests' started at 2020-12-30 14:35:43.472
Test Suite 'SwiftDifferentiationPackageTests.xctest' started at 2020-12-30 14:35:43.473
Test Suite 'DifferentiationTests' started at 2020-12-30 14:35:43.473
Test Case '-[DifferentiationTests.DifferentiationTests testExample]' started.
Test Case '-[DifferentiationTests.DifferentiationTests testExample]' passed (0.112 seconds).
Test Suite 'DifferentiationTests' passed at 2020-12-30 14:35:43.585.
     Executed 1 test, with 0 failures (0 unexpected) in 0.112 (0.112) seconds
Test Suite 'SwiftDifferentiationPackageTests.xctest' passed at 2020-12-30 14:35:43.585.
     Executed 1 test, with 0 failures (0 unexpected) in 0.112 (0.112) seconds
Test Suite 'All tests' passed at 2020-12-30 14:35:43.585.
     Executed 1 test, with 0 failures (0 unexpected) in 0.112 (0.113) seconds

Maybe this is because swift test does not pass the DYLD_LIBRARY_PATH environment variable to the xctest invocation?

[dan-zheng]

Potentially related PR: swiftlang/swift#34216 The PR changes the install name directory (LC_ID_DYLIB) of libswift_Differentiation.dylib from @rpath/libswift_Differentiation.dylib to /usr/lib/swift/libswift_Differentiation.dylib.

[compnerd]

With SIP enabled, `DYLD_*` variables will be scrubbed, which means that `DYLD_LIBRARY_PATH` would not make it to the process AIUI.

A quick sanity check:

$ codesign --display --entitlements :- /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest
Executable=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.get-task-allow</key>
    <true/>
</dict>
</plist>

This seems to not have the required com.apple.security.cs.allow-dyld-environment-variables entitlement as per https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-dyld-environment-variables . On systems with SIP enabled, this would disallow execution with a custom toolchain (or a snapshot).

[compnerd]

CC: @lorentey @edymtt

[swift-ci]

Comment by Alexey Novakov (JIRA)

it seems my issue is related to this:

DYLD_LIBRARY_PATH=/Users/Alexey_Novakov/Library/Developer/Toolchains/swift-5.4-DEVELOPMENT-SNAPSHOT-2021-03-25-a.xctoolchain/usr/lib/swift/macosx xcrun --toolchain "Swift Development Snapshot" swift run
[1/1] Planning build

* Build Completed!dyld: lazy symbol binding failed: Symbol not found: _swift_autoDiffCreateLinearMapContext
 Referenced from: /Users/Alexey_Novakov/dev/git/swift-linear-regression/.build/x86_64-apple-macosx/debug/swift-linear-regression
 Expected in: /usr/lib/swift/libswiftCore.dylib

dyld: Symbol not found: _swift_autoDiffCreateLinearMapContext
 Referenced from: /Users/Alexey_Novakov/dev/git/swift-linear-regression/.build/x86_64-apple-macosx/debug/swift-linear-regression
 Expected in: /usr/lib/swift/libswiftCore.dylib

zsh: abort DYLD_LIBRARY_PATH= xcrun --toolchain "Swift Development Snapshot" swift run

otool paths:

% otool -L .build/x86_64-apple-macosx/debug/swift-linear-regression
.build/x86_64-apple-macosx/debug/swift-linear-regression:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
    @rpath/libswiftCore.dylib (compatibility version 1.0.0, current version 0.0.0)
    @rpath/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftCoreGraphics.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftDispatch.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftFoundation.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
    @rpath/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftSwiftOnoneSupport.dylib (compatibility version 1.0.0, current version 0.0.0)
    @rpath/libswiftXPC.dylib (compatibility version 1.0.0, current version 1.1.0, weak)
    @rpath/libswift_Differentiation.dylib (compatibility version 1.0.0, current version 0.0.0)

[philipturner]

I have a reproducer for this. The Swift package tests usually run just fine, but encountering this edge case stops them from happening. To reproduce:

1. Make an empty Swift package.
2. In the Sources/PACKAGE_NAME.swift file, paste the following:

import _Differentiation

extension Array.DifferentiableView: RandomAccessCollection
where Element: Differentiable {
  public typealias Element = Array<Element>.Element
  public typealias Index = Array<Element>.Index
  public typealias SubSequence = Array<Element>.SubSequence

  public subscript(position: Array<Element>.Index) -> Element {
    get { fatalError() }
    set { fatalError() }
  }

  public subscript(bounds: Range<Array<Element>.Index>) -> SubSequence {
    get { fatalError() }
    set { fatalError() }
  }

  public var startIndex: Index { fatalError() }
  public var endIndex: Index { fatalError() }
}

3. You can have anything in the Tests directory, as long as there's a file. It can even be blank.
4. Run swift test from the command line, and you get the following output. Notice the very last line.

(base) philipturner@m1-max-mbp sr14008 % swift test
objc[92004]: Class _TtCs25CheckedContinuationCanary is implemented in both /usr/lib/swift/libswift_Concurrency.dylib (0x218cb4450) and /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2022-05-04-a.xctoolchain/usr/lib/swift/macosx/libswift_Concurrency.dylib (0x106cfc8c8). One of the two will be used. Which one is undefined.
objc[92004]: Class _TtCs17DispatchQueueShim is implemented in both /usr/lib/swift/libswift_Concurrency.dylib (0x218cb44e0) and /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2022-05-04-a.xctoolchain/usr/lib/swift/macosx/libswift_Concurrency.dylib (0x106cfc958). One of the two will be used. Which one is undefined.
objc[92004]: Class _TtCs9MainActor is implemented in both /usr/lib/swift/libswift_Concurrency.dylib (0x218cb4570) and /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2022-05-04-a.xctoolchain/usr/lib/swift/macosx/libswift_Concurrency.dylib (0x106cfc9f0). One of the two will be used. Which one is undefined.
warning: Usage of /Users/philipturner/Library/org.swift.swiftpm/collections.json has been deprecated. Please delete it and use the new /Users/philipturner/Library/org.swift.swiftpm/configuration/collections.json instead.
Building for debugging...
[3/3] Linking sr14008PackageTests
Build complete! (0.36s)
2022-05-14 15:30:31.658 xctest[92072:1569176] The bundle “sr14008PackageTests.xctest” couldn’t be loaded. Try reinstalling the bundle.
2022-05-14 15:30:31.658 xctest[92072:1569176] (dlopen(/Users/philipturner/Desktop/sr14008/.build/arm64-apple-macosx/debug/sr14008PackageTests.xctest/Contents/MacOS/sr14008PackageTests, 0x0109): Symbol not found: _$sSa16_DifferentiationAA14DifferentiableRzlE0B4ViewVMn
  Referenced from: /Users/philipturner/Desktop/sr14008/.build/arm64-apple-macosx/debug/sr14008PackageTests.xctest/Contents/MacOS/sr14008PackageTests
  Expected in: /usr/lib/swift/libswift_Differentiation.dylib)

@novakov-alexey I wasn't able to reproduce your compile failure from #4454 (comment). From your recollection, did it involve something reminiscent of my reproducer?

[philipturner]

Related: tensorflow/swift-models#727

Note as a result of this, the build process for testing s4tf/s4tf (formerly swift-apis) becomes greatly more complex. But it's still possible (this is an example of a workaround).

# prepare for building S4TF
export TOOLCHAINS="org.swift.57202205041a"
tensorflow_version="2.8.0"
DESTDIR=${PWD}/Library/tensorflow-${tensorflow_version}

# build S4TF
cd s4tf
swift build -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN \
  -Xcc -I${DESTDIR}/usr/include -Xlinker -L${DESTDIR}/usr/lib

cp ${DESTDIR}/usr/lib/libx10.dylib .build/$(uname -m)-apple-macosx/debug/libx10.dylib

# fails, but that's okay
swift test -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN \
  -Xcc -I${DESTDIR}/usr/include -Xlinker -L${DESTDIR}/usr/lib

export DYLD_LIBRARY_PATH="/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2022-05-04-a.xctoolchain/usr/lib/swift/macosx"

# actually runs the tests
/Applications/Xcode.app/Contents/Developer/usr/bin/xctest \
  .build/$(uname -m)-apple-macosx/debug/TensorFlowPackageTests.xctest

[philipturner]

Perhaps I could remove this crash from S4TF by taking the Array.DifferentiableView extension out of S4TF and into the Swift Standard Library. The code moved there may or may not conflict with the intentions of DifferentiableCollection, if that protocol ever makes it into the stdlib. After that, we may need a second reproducer because the original one won't fail anymore.

Would my idea be permissible?

[tomerd]

@abertelrud @neonichu @lorentey - this echoes some of our discussion, correct?

[lorentey]

The contents of the _Differentiation module have not gone through Swift Evolution. It is not an ABI stable component of the Standard Library and therefore it isn't shipping as a public SDK component on ABI stable platforms. Moving Array.DifferentiableView to the core stdlib is not an option until it has been proposed & accepted as public API.

(Beware: while /usr/lib/swift/libswift_Differentation.dylib does ship in some versions of macOS, it is not a public library on any Apple platform. Accordingly, it may disappear or its contents may change in any OS release, including point releases. Do not link to it from any production binary -- such code will break.)

% otool -L .build/x86_64-apple-macosx/debug/swift-linear-regression
.build/x86_64-apple-macosx/debug/swift-linear-regression:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
    @rpath/libswiftCore.dylib (compatibility version 1.0.0, current version 0.0.0)
    @rpath/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftCoreGraphics.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftDispatch.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftFoundation.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
    @rpath/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
    @rpath/libswiftSwiftOnoneSupport.dylib (compatibility version 1.0.0, current version 0.0.0)
    @rpath/libswiftXPC.dylib (compatibility version 1.0.0, current version 1.1.0, weak)
    @rpath/libswift_Differentiation.dylib (compatibility version 1.0.0, current version 0.0.0)

The version of dyld in macOS Monterey (and later) does not support using DYLD_LIBRARY_PATH to override libraries loaded via @rpath. To avoid duplicate loading of these modules (and subsequent crashes), arrange for the binary to pick up its dependencies directly from /usr/lib/swift. One way to do this is to set the minimum deployment target to macOS 10.15 or later -- this stops the compiler from using rpath-relative install names for these libraries. (As it can safely assume that /usr/lib/libswiftCore.dylib etc do exist.)

(If this dyld limitation is causing you trouble, please do file a feedback report describing the problem -- such reports do help prioritizing the issue.)

[philipturner]

Moving Array.DifferentiableView to the core stdlib is not an option until it has been proposed & accepted as public API.

Array.DifferentiableView is already in the core stdlib. It is just removed on release toolchains, where I have to depend on philipturner/differentiation to add it back in.

[lorentey]

Also beware -- custom builds of the Swift Standard Library (whether they were produced by manually building the stdlib from the apple/swift repo, or by downloading one of the toolchain snapshots from swift.org) aren't drop-in replacements for the stdlib that ships in any OS version.

Such stdlib builds are not guaranteed to interoperate with OS components that load the Swift stdlib, including Foundation.