[Python] macOS Segfault on Import, Both arm64 and x86_64

[essandess]

Describe the bug, including details regarding any error messages, version, and platform.

The pyarrow packages create a segfault on import when compiled on macOS boxes. This happens on macOS arm64 and x86_64, using (at least) version 12.0.0 and 13.0.0.

Context: I am trying to update the MacPorts apache-arrow ports at macports/macports-ports#19664 and discovered this issue. MacPorts uses the apache-arrow build instructions and approach in python.rst#build-and-test and python_wheel_macos_build.sh.

Behavior:

$ port installed py310-pyarrow
The following ports are currently installed:
  py310-pyarrow @12.0.0_3 (active)

$ PYTHONFAULTHANDLER=1 python3.10 -c 'import pyarrow'
Fatal Python error: Segmentation fault

Current thread 0x00007ff84d123700 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1176 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 674 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/__init__.py", line 65 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<string>", line 1 in <module>
Segmentation fault: 11

Component(s)

Python

[kou]

Could you get backtrace by lldb?

[essandess]

Here's the macOS Crash Reporter output. What's the lldb command to capture the trace for python3.10 -c 'import pyarrow'?

Also, I've tried setting a few more cmake flags to be more consistent with the build script at python_wheel_macos_build.sh, but keep hitting the same issue.

Process:               Python [51251]
Path:                  /opt/local/Library/Frameworks/Python.framework/Versions/3.10/Resources/Python.app/Contents/MacOS/Python
Identifier:            org.python.python
Version:               3.10.12 (3.10.12)
Code Type:             X86-64 (Native)
Parent Process:        bash [15799]
Responsible:           Terminal [15797]
User ID:               502

Date/Time:             2023-08-03 16:08:12.4564 -0400
OS Version:            macOS 13.5 (22G74)
Report Version:        12
Bridge OS Version:     7.6 (20P6072)

Time Awake Since Boot: 810000 seconds
Time Since Wake:       607027 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000012340
Exception Codes:       0x0000000000000001, 0x0000000000012340

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [51251]

VM Region Info: 0x12340 is not in any region.  Bytes before following region: 4352785600
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                      103736000-10373a000    [   16K] r-x/r-x SM=COW  .../MacOS/Python

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libjemalloc.2.dylib           	       0x1059ec3df je_free_default + 193
1   libprotobuf.3.21.12.0.dylib   	       0x105688477 google::protobuf::internal::ArenaStringPtr::Destroy() + 39
2   libprotobuf.3.21.12.0.dylib   	       0x10571d5a5 google::protobuf::FileDescriptorProto::SharedDtor() + 149
3   libprotobuf.3.21.12.0.dylib   	       0x10571d6bd google::protobuf::FileDescriptorProto::~FileDescriptorProto() + 45
4   libprotobuf.3.21.12.0.dylib   	       0x105738657 google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) + 167
5   libprotobuf.3.21.12.0.dylib   	       0x1056e34f4 google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) + 36
6   libprotobuf.3.21.12.0.dylib   	       0x1057544d9 google::protobuf::(anonymous namespace)::AddDescriptors(google::protobuf::internal::DescriptorTable const*) + 105
7   dyld                          	    0x7ff81b4f93fb invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 175
8   dyld                          	    0x7ff81b537b7a invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 242
9   dyld                          	    0x7ff81b52bf22 invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 577
10  dyld                          	    0x7ff81b4dc0af dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 245
11  dyld                          	    0x7ff81b52b0bf dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 175
12  dyld                          	    0x7ff81b53773a dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 470
13  dyld                          	    0x7ff81b4f666c dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 220
14  dyld                          	    0x7ff81b4f685a dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 178
…

[essandess]

Here's the lldb trace:

lldb trace

lldb trace

/opt/local/bin/lldb-mp-16
(lldb) command script import pyarrow
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /opt/local/libexec/llvm-16/bin/lldb
1.	HandleCommand(command = "command script import pyarrow")
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libLLVM.dylib               0x0000000113a91c6c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  libLLVM.dylib               0x0000000113a90c38 llvm::sys::RunSignalHandlers() + 112
2  libLLVM.dylib               0x0000000113a922f8 SignalHandler(int) + 344
3  libsystem_platform.dylib    0x000000018d386a24 _sigtramp + 56
4  libjemalloc.2.dylib         0x000000010f5940e0 je_free_default + 1104
5  libprotobuf.3.21.12.0.dylib 0x000000010f2471f0 google::protobuf::internal::ArenaStringPtr::Destroy() + 56
6  libprotobuf.3.21.12.0.dylib 0x000000010f2d0000 google::protobuf::FileDescriptorProto::SharedDtor() + 144
7  libprotobuf.3.21.12.0.dylib 0x000000010f2d00e4 google::protobuf::FileDescriptorProto::~FileDescriptorProto() + 48
8  libprotobuf.3.21.12.0.dylib 0x000000010f2ea634 google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) + 168
9  libprotobuf.3.21.12.0.dylib 0x000000010f29e878 google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) + 40
10 libprotobuf.3.21.12.0.dylib 0x000000010f3056a8 google::protobuf::(anonymous namespace)::AddDescriptors(google::protobuf::internal::DescriptorTable const*) + 136
11 libprotobuf.3.21.12.0.dylib 0x000000010f3056dc google::protobuf::internal::AddDescriptorsRunner::AddDescriptorsRunner(google::protobuf::internal::DescriptorTable const*) + 24
12 dyld                        0x000000018d01c1d8 invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
13 dyld                        0x000000018d05de94 invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 340
14 dyld                        0x000000018d0511a4 invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 528
15 dyld                        0x000000018cffc2d8 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 296
16 dyld                        0x000000018d0501cc dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
17 dyld                        0x000000018d05d958 dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516
18 dyld                        0x000000018d01885c dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 448
19 dyld                        0x000000018d018c10 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
20 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
21 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
22 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
23 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
24 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
25 dyld                        0x000000018d01c264 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
26 dyld                        0x000000018d018d90 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 304
27 dyld                        0x000000018d036d58 dyld4::APIs::dlopen_from(char const*, int, void*) + 1440
28 Python                      0x000000010378ae64 _imp_create_dynamic + 616
29 Python                      0x00000001036ce5d0 cfunction_vectorcall_FASTCALL + 80
30 Python                      0x0000000103760d94 _PyEval_EvalFrameDefault + 50396
31 Python                      0x0000000103763424 _PyEval_Vector + 116
32 Python                      0x000000010368859c object_vacall + 224
33 Python                      0x0000000103688458 PyObject_CallMethodObjArgs + 92
34 Python                      0x00000001037873a8 PyImport_ImportModuleLevelObject + 1224
35 Python                      0x000000010375b8f0 _PyEval_EvalFrameDefault + 28728
36 Python                      0x0000000103753d74 PyEval_EvalCode + 168
37 Python                      0x0000000103750174 builtin_exec + 332
38 Python                      0x00000001036ce6cc cfunction_vectorcall_FASTCALL_KEYWORDS + 76
39 Python                      0x0000000103760d94 _PyEval_EvalFrameDefault + 50396
40 Python                      0x0000000103763424 _PyEval_Vector + 116
41 Python                      0x000000010368859c object_vacall + 224
42 Python                      0x0000000103688458 PyObject_CallMethodObjArgs + 92
43 Python                      0x00000001037873a8 PyImport_ImportModuleLevelObject + 1224
44 Python                      0x000000010375b8f0 _PyEval_EvalFrameDefault + 28728
45 Python                      0x0000000103753d74 PyEval_EvalCode + 168
46 Python                      0x00000001037a5d38 run_eval_code_obj + 84
47 Python                      0x00000001037a5c9c run_mod + 112
48 Python                      0x00000001037a8084 PyRun_StringFlags + 112
49 liblldb.16.0.6.dylib        0x000000010469effc lldb_private::python::runStringMultiLine(llvm::Twine const&, lldb_private::python::PythonDictionary const&, lldb_private::python::PythonDictionary const&) + 120
50 liblldb.16.0.6.dylib        0x00000001046a5838 lldb_private::ScriptInterpreterPythonImpl::ExecuteMultipleLines(char const*, lldb_private::ExecuteScriptOptions const&) + 960
51 liblldb.16.0.6.dylib        0x00000001046ab314 lldb_private::ScriptInterpreterPythonImpl::LoadScriptingModule(char const*, lldb_private::LoadScriptOptions const&, lldb_private::Status&, std::__1::shared_ptr<lldb_private::StructuredData::Object>*, lldb_private::FileSpec) + 2092
52 liblldb.16.0.6.dylib        0x0000000104777158 CommandObjectCommandsScriptImport::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) + 328
53 liblldb.16.0.6.dylib        0x00000001043469c0 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) + 484
54 liblldb.16.0.6.dylib        0x000000010433d2e8 lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&) + 2136
55 liblldb.16.0.6.dylib        0x00000001043409a0 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) + 852
56 liblldb.16.0.6.dylib        0x00000001042731a0 lldb_private::IOHandlerEditline::Run() + 304
57 liblldb.16.0.6.dylib        0x0000000104256968 lldb_private::Debugger::RunIOHandlers() + 140
58 liblldb.16.0.6.dylib        0x0000000104341c14 lldb_private::CommandInterpreter::RunCommandInterpreter(lldb_private::CommandInterpreterRunOptions&) + 156
59 liblldb.16.0.6.dylib        0x00000001040a0614 lldb::SBDebugger::RunCommandInterpreter(bool, bool) + 124
60 lldb                        0x0000000102ff48e0 Driver::MainLoop() + 2784
61 lldb                        0x0000000102ff541c main + 2168
62 dyld                        0x000000018cffff28 start + 2236
LLDB diagnostics will be written to /var/folders/2s/r9z6gsps7gn5wd89c0sqs8k80000gq/T/diagnostics-5f4976
Please include the directory content when filing a bug report
Segmentation fault: 11

[kou]

Thanks.
It seems that jemalloc is related.

Could you try disabling jemalloc when you build Apache Arrow C++?
You can disable jemalloc by specifying the -DARROW_JEMALLOC=OFF CMake option.

[essandess]

Thank you, that fixes this issue. I'll bump this upstream to jemalloc.

[kou]

In general, jemalloc is faster than system memory allocator.
So you may want to use jemalloc instead of disabling jemalloc.

Apache Arrow C++ uses bundled jemalloc instead of system jemalloc by default. It means that Apache Arrow C++ and Protobuf use different jemalloc in the same process. It may cause this problem. This problem may be solved by using system jemalloc by specifying -Djemalloc_SOURCE=SYSTEM instead of -DARROW_JEMALLOC=OFF.

[essandess]

Thanks. MacPorts’s apache-arrow already uses these system build flags, so this looks like an issue with jemalloc on macOS.

https://github.com/macports/macports-ports/blob/11afc500de4b609a3ed51465915c8102c8550aac/devel/apache-arrow/Portfile#L156

[essandess]

Several people have pointed out that jemalloc may not be the cause of this issue, so I am reopening the issue here. See:

• macOS Segfault with apache-arrow jemalloc/jemalloc#2511 (comment)
• https://trac.macports.org/ticket/67928#comment:3

[pitrou]

Several people have pointed out that jemalloc may not be the cause of this issue, so I am reopening the issue here.

Well, it depends. Does this issue still happen if you disable Arrow's jemalloc integration with -DARROW_JEMALLOC=OFF?

[prniii]

Whether this is the same issue or a different one, I am seeing a crash on 'import pyarrow'. But it is a BAD INSTRUCTION crash. Seeing this on both Apple Silicon M1, and Apple Silicon M1 Max. Big Sur and Ventura using the public wheels downloaded via pip. 12.0.1 is the last version that works, 13.0.0, 14.0.0, 14.0.1 and 14.0.2 all crash with the BAD INSTRUCTION. This is not being seen on a M2 Sonoma machine... Are the wheels being built on a M2 machine picking up a higher optimization or processor instruction flag?