Build binary packages using clang-cl on Windows

[TkTech]

Support for clang-cl is coming. As part of the PR that allows CPython to build against clang-cl, distutils is updated to build with clang-cl (python/cpython#18371). Once this PR is merged and a new CPython release includes it we can start using it for our binary releases.

Clang has reached a point where it's safe enough for us to use with CPython's built with MSVC2015 or newer. https://clang.llvm.org/docs/MSVCCompatibility.html

This would alleviate poor windows performance caused by MSVC issues (simdjson/simdjson#847, but not entirely, simdjson/simdjson#848).

We only need to do this if upstream simdjson doesn't figure out what's up with MSVC. @lemire

[lemire]

We only need to do this if upstream simdjson doesn't figure out what's up with MSVC.

MSVC lacks some features that clang has, such as branch likelihood hints, computed gotos, architecture-specific function attributes...

[lemire]

I have reported some of the issue at https://developercommunity.visualstudio.com/content/problem/1106837/unexpected-volume-of-256-bit-loads-and-stores-in-h.html

It may be hard for Visual Studio to have parity, however. I would like to submit the following code example as an illustration...

struct go  {
    __m128i array[4];
};

__m128i orme(go & x) {
    __m128i z = x.array[0];
    for(size_t i = 1; i < 4; i++)
      z = _mm_or_si128(z, x.array[i]);
    return z;
}

Clang compiles it to...

orme(go&):
        movdqa  xmm0, XMMWORD PTR [rdi]
        movdqa  xmm1, XMMWORD PTR [rdi+32]
        por     xmm0, XMMWORD PTR [rdi+16]
        por     xmm1, XMMWORD PTR [rdi+48]
        por     xmm0, xmm1
        ret

while GCC goes for...

orme(go&):
        movdqa  xmm0, XMMWORD PTR [rdi]
        movdqa  xmm1, XMMWORD PTR [rdi+32]
        por     xmm0, XMMWORD PTR [rdi+16]
        por     xmm1, XMMWORD PTR [rdi+48]
        por     xmm0, xmm1
        ret

Yet Visual Studio does the following...

__m128i orme(go &) PROC                    ; orme, COMDAT
        movdqu  xmm1, XMMWORD PTR [rcx]
        lea     rax, QWORD PTR [rcx+16]
        mov     ecx, 3
        npad    3
$LL4@orme:
        movdqu  xmm0, XMMWORD PTR [rax]
        lea     rax, QWORD PTR [rax+16]
        por     xmm0, xmm1
        movdqa  xmm1, xmm0
        sub     rcx, 1
        jne     SHORT $LL4@orme
        ret     0

So Visual Studio does a loop. It is going to be many times slower.

So, at least for the kind of code that simdjson deals with, it might be that the Visual Studio compiler is just flat out inferior to clang.

(I would rather not say this, but it looks bad when I look at the assembly that Visual Studio produces.)

[lemire]

It looks like numpy can be built using clangcl...

https://github.com/numpy/numpy/pull/13816/files

There is some Python glue needed, but nothing terrible by the look of it.

Of course, this probably assumes that Visual Studio was installed along with clang, but that's an easy requirement.

[TkTech]

That "glue" is far more terrible than it looks. It's including the entire distutils project with a bunch of hacks for ICC and such.

Like I mentioned above Cpython is getting clang-cl support which does the same changes upstream.

[lemire]

I was not suggesting that anyone goes the numpy route, I was just pointing out that it is obviously viable.

All around, this is very good news for Windows support.

[TkTech]

The upstream performance drop on MSVC is acceptable now. When clang-cl support lands we'll support it, but I no longer consider it an issue.