Link-time-optimization with clang is broken

[matthiasgoergens]

I tried to enable link time optimizations for clang.

export CC="clang"
export LD="clang"

nice "${here}/configure" \
    --enable-optimizations \
    --with-lto=thin

nice make -j8

Eventually building reaches this step:

clang -fprofile-instr-generate -o _bootstrap_python Modules/getbuildinfo.o Parser/token.o Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Objects/perf_trampoline.o Objects/asm_trampoline.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/ceval_gil.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o Modules/config.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o Modules/faulthandler.o Modules/posixmodule.o Modules/signalmodule.o Modules/_tracemalloc.o Modules/_codecsmodule.o Modules/_collectionsmodule.o Modules/errnomodule.o Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o Modules/itertoolsmodule.o Modules/_sre/sre.o Modules/_threadmodule.o Modules/timemodule.o Modules/_weakref.o Modules/_abc.o Modules/_functoolsmodule.o Modules/_localemodule.o Modules/_operator.o Modules/_stat.o Modules/symtablemodule.o Modules/pwdmodule.o Programs/_bootstrap_python.o Modules/getpath.o -ldl -lm

And fails:

Modules/getbuildinfo.o: file not recognized: file format not recognized
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

Your environment

I'm on Archlinux x86-64.

$ clang --version
clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

I tested against main.

[corona10]

-no-lto flag was first introduced at #29859 by @tiran
Would you like to take a look at this issue please?

[matthiasgoergens]

@corona10 Thanks! That seems to be mostly about speeding up the GCC build. LLVM is already pretty fast with thin-lto.

[corona10]

I will try to reproduce the issue in my Linux environment with clang14 before reviewing the PR.
cc @tiran

[corona10]

Okay, I am able to reproduce the issue even with the lld.

[matthiasgoergens]

Thanks for confirming that it's not just some idiosyncratic weirdness of my environment.

[corona10]

For @matthiasgoergens, @tiran @vstinner cc @pablogsal as release manager.

My conclusion

• At this moment Modules/*.o are actually LLVM bitcode not object files.
• so the LTO processor needs to transform the LLVM bitcode into the object file for the traditional linker but looks like without -flto it doesn't transform to object files for the traditional linker.
• This issue not only happens for ThinLTO but also in full LTO because both processes have the phase for transforming the LLVM bitcode into the object file.
• see http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html
• IMHO we can add the transformation phase for -fno-lto but it could be complicated, I prefer using gh-96761: Fix lto-build for clang #96762 as a workaround approach and we need to backport to Python 3.11 too.
• But I am still curious about the fact why apple clang does not raise this issue.

Reference

file format

corona10@python-dev:~/cpython$ file  Modules/getbuildinfo.o
Modules/getbuildinfo.o: LLVM IR bitcode

clang full LTO process

clang ThinLTO process

[pablogsal]

Why is this only affecting 3.11 and not previous version?

[corona10]

Why is this only affecting 3.11 and not previous version?

because #29859 was first introduced in Python 3.11 for _bootstrap_python.

[corona10]

When building the final binary for CPython, we actually pass the -flto flags that the process will transform the LLVM bitcode into object files, so this issue only affects while building the _bootstrap_python.
I didn't check if the issue will affect to old versions (< 3.10)

[corona10]

Manual transformation example:

corona10@python-dev:~/cpython$ llc-14 -filetype=obj Modules/getbuildinfo.o -o Modules/getbuildinfo.o
corona10@python-dev:~/cpython$ file Modules/getbuildinfo.o
Modules/getbuildinfo.o: ELF 64-bit LSB relocatable, x86-64, version 1 (GNU/Linux), with debug_info, not stripped