Add cross-platform compilation support to nimporter CLI

[scuffi]

Issue

I've been doing some development with nimpy & nimporter on an M1 Max (ARM64) and have had quite a few obstacles. Currently, I'm using nimporter to compile on demand in an amd64 Docker container to ensure everything works.

If I tried to precompile the nim files using nimporter build/compile I would run into an error when running the Docker container as the binaries were built for arm64, but the container is amd64, which looks like this:

Traceback (most recent call last):
  File "main.py", line 7, in <module>
    import nimporter, nim_math
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "/usr/local/lib/python3.8/site-packages/nimporter.py", line 1300, in find_spec
    return Nimporter.import_nim_code(fullname, path, library=True)
  File "/usr/local/lib/python3.8/site-packages/nimporter.py", line 957, in import_nim_code
    cls.__validate_spec(spec)
  File "/usr/local/lib/python3.8/site-packages/nimporter.py", line 1025, in __validate_spec
    raise NimporterException(error_message) from import_error
nimporter.NimporterException: Error importing /nim_math/__pycache__/nim_math.so
Error Message:

    /nim_math/__pycache__/nim_math.so: invalid ELF header

Which I have assumed from testing is because the platforms do not match.

Fix

My suggestion is to add a --cpu flag onto the build & compile command, mimicking the flag from the nim compile command found here

This issue isn't pressing as it is still possible to run nimporter and numpy, however, runtime compilation tends to be quite slow.
If you would like, I would be able to add this functionality and create a PR to implement it.

For other context, I've been building exclusively in a Docker container for testing purposes and due to arm64 installation issues with nim (I will be attempting to solve these as I go and will provide any updates here for anyone that is struggling with a similar issue)

[Pebaz]

Hi @scuffi , yes a PR would really help!

[scuffi]

@Pebaz Just a quick question before I get going with this, just a quick question about the folder structure, I assume you'd like me to expand the 2.0 version as that structure is pretty different to master? Also was going add some more functionality when it comes to cross-platform use, with the idea of compiling multiple different platforms then storing the outputs in the pycache, just allowing for a library to hold precompiled binaries for multiple platforms, so the question was if that was already possible or that functionality could be used?

[Pebaz]

Yeah probably best to go with 2.0rc, although, full disclosure, I'm not sure about the exact timeline for merging it (could be sooner, could be later). I'm just not going to be able to provide support for people after the breaking changes land in 2.0 (good docs only go so far) so I'm kind of just waiting until my current situation allows for more free time.

For the cross-platform ideas, could you go over what you had in mind? That kind of sounds like what Nimporter 2.0 already does for packaging but I could be wrong.

[scuffi]

Yeah I'll go with 2.0, no stress on timing though of course.

The idea was to precompile multiple different binaries for different platforms to be used. Thinking about it I'm not sure how it would be implemented in the current methods in my head, so I'll leave it for now and maybe create something in the future and come back to it. Definitely going to add that precompilation platform support though, as it would aid my own development!

[Pebaz]

Awesome, sounds good! 👍

[Pebaz]

@scuffi Cross-compilation might be as simple as adding ARM to this table:

https://github.com/Pebaz/nimporter/blob/master/nimporter/lib.py#L37C24-L37C24

ARCH_TABLE: Dict[str, str] = {  # Keys are known to Python and values are Nim-understood
    'x86_32': 'i386',
    'x86_64': 'amd64',

    # ? Are these needed
    # 'arm_8': 'arm64',
    # 'arm_7': 'arm',
    # 'ppc_32': 'powerpc',
    # 'ppc_64': 'powerpc64',
    # 'sparc_32': 'sparc',
    # 'sparc_64': 'sparc64',
    # 'mips_32': 'mips',
    # 'mips_64': 'mips64',
    # 'riscv_32': 'riscv32',
    # 'riscv_64': 'riscv64',
}

The way this would work is that you'd add ARM to that table and then produce a source distribution (docs, python setup.py sdist). From there, the Docker container would install the Python package containing the ARM C code produced from Nim. I have no idea if that will work though. Cross-compilation is a famously difficult problem with C compilers but it might be this simple. 🤷