Substantial drop in performance with new version, especially on ARM

[JMartinon]

Hello again,

I've been playing around with the new version of TBLIS and, to my surprise, observed a substantial performance drop between its old version and the new one.

Here's the piece of code I've been using for my benchmarks:

#include <iostream>
#include <chrono>
#include <random>

#include "omp.h"

#define TBLIS_ENABLE_COMPAT 1

#include "tblis/frame/base/aligned_allocator.hpp"

#include "tblis/tblis.h"

//Function to set TBLIS thread count
extern "C" {
    void tblis_set_num_threads(unsigned num_threads);
    unsigned tblis_get_num_threads();
}

using Clock = std::chrono::steady_clock;
using Seconds = std::chrono::duration<double>;

// Aligned allocator for better SIMD/cacheline behavior
using Al64 = tblis::aligned_allocator<double, 64>;

// Convenient alias
using Tensor = tblis::tensor<double, Al64>;


int main() {
    
    // Force TBLIS to use all threads ( if needed ?)
    //tblis_set_num_threads(omp_get_max_threads());
    std::cout << "TBLIS using " << tblis_get_num_threads() << " threads\n";
    
    // Dimensions
    const long Bn = 40;   // batch b
    const long I  = 80;   // i
    const long J  = 100;  // j
    const long K  = 60;   // k
    const long P  = 60;   // p
    const long Q  = 80;   // q
    const long R  = 40;   // r
    
    // Declare the tensors
    Tensor A({Bn, I, K, P}, MArray::COLUMN_MAJOR);                              // A[b,i,k,p]
    Tensor B({Bn, K, J, Q, R}, MArray::COLUMN_MAJOR);                           // B[b,k,j,q,r]
    Tensor C({P,  Q, R}, MArray::COLUMN_MAJOR);                                 // C[p,q,r]
    Tensor BC({Bn, K, J, P}, MArray::uninitialized, MArray::COLUMN_MAJOR);      // temp: BC[b,k,j,p]
    Tensor D ({Bn, I, J, P}, MArray::uninitialized, MArray::COLUMN_MAJOR);      // result: D[b,i,j,p]
    
    
    // Fill A,B,C with random doubles
#pragma omp parallel
    {
        std::mt19937_64 rng(42 + omp_get_thread_num()*1337);
        std::uniform_real_distribution<double> dist(0.0,1.0);
#pragma omp for
        for (std::size_t t=0; t<A.size(); ++t) A.data()[t]=dist(rng);
#pragma omp for
        for (std::size_t t=0; t<B.size(); ++t) B.data()[t]=dist(rng);
#pragma omp for
        for (std::size_t t=0; t<C.size(); ++t) C.data()[t]=dist(rng);
    }
    
    double total_time_TBLIS = 0;
    
    int MaxIter = 40;
    
    // Loop for average evaluation time
    for(int i=0; i<MaxIter;i++){
        
        auto t0 = Clock::now();
        
        // Contract B and C over (q,r):  BC[b,k,j,p] = sum_{q,r} B[b,k,j,q,r] * C[p,q,r]
        tblis::mult<double>(1.0,
                            B,  "bkjqr",
                            C,  "pqr",
                            0.0,
                            BC, "bkjp");
        
        //Contract A and BC over (k):   D[b,i,j,p]  = sum_{k}   A[b,i,k,p]   * BC[b,k,j,p]
        tblis::mult<double>(1.0,
                            A,  "bikp",
                            BC, "bkjp",
                            0.0,
                            D,  "bijp");
        
        auto t1 = Clock::now();
        
        total_time_TBLIS += Seconds(t1 - t0).count();
    }
    
    double time_TBLIS = total_time_TBLIS/MaxIter;
    
    std::cout << "TBLIS time average: " << time_TBLIS << " s\n";
    
    return 0;
}

I've been running this code on two different machines:

Mac

OS: macOS Sequoia 15.5
Chip: Apple M3 Pro
Total Number of Cores: 11 (5 performance and 6 efficiency)
Memory: 18 GB

Linux

OS: Ubuntu 24.04. 3 LTS
Chip: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
Total Number of Cores: 8
Memory: 16 GB

So one ARM and one x86 architecture. Here's the result of my benchmarks:

Version	Arch	Threads	Average Time (s)
old	ARM	11	0.734512
new	ARM	11	2.92892
old	x86	4	1.75475
new	x86	4	2.21599

From my understanding, the former version of TBLIS didn't come with microkernels for ARM architectures. While one of the advantages of the newer version is to use BLIS' plugins to generates those microkernels. I was therefore expecting an interesting performance gain on my Mac machine but the actual opposite seems to happen.
On my Mac the new version is about 4 times slower while on my linux machine it's still 1.3 times slower.

Is this an abnormal behavior or a misunderstanding from me ?

Both version have been install without specifying dependancies (letting the ./configure download and build BLIS), with GCC-14 on Linux and GCC-15 on Mac.

Old version:

commit 4de1919dfe194f5e47dfc93660ce4206d8e12c4e (HEAD -> master, origin/master, origin/HEAD)
Merge: 7d979b6 7436875
Author: Devin Matthews <damatthews@smu.edu>
Date:   Sat Apr 22 14:57:23 2023 -0500

New version:

commit 9b95712966cb8804be51c62bfd6207957f62bc6f (HEAD -> develop, origin/develop, origin/HEAD)
Author: Christopher Hillenbrand <chillenb.lists@gmail.com>
Date:   Mon Aug 18 22:16:35 2025 -0400

Quick word on threading

I observed that by default TBLIS only uses 4 threads out of the 8 available on my Intel chip. If I try to manually constraint the library to use all the threads: tblis_set_num_threads(omp_get_max_threads()); the average evaluation time remains the same.

Version	Arch	Threads	Average Time (s)
new	x86	4	2.21599
new	x86	8	2.16684

I tried to do the same analysis with smaller dimensions (to stay away from memory-bandwidth issues) and observed the same phenomenon. Can someone explain to me what is happening here ?

Thank you again for all your work !

[devinamatthews]

Hi @JMartinon I will definitely take a look at performance with your example and some others. Some quick observations for now:

1. The i7-4770K does actually have 4 cores, which appears as 8 "cores" due to hyperthreading. However for HPC applications hyperthreads are typically a performance loss.
2. Similarly, on the M3 and other big.LITTLE machines, the efficiency cores are essentially useless for HPC. I recommend explicitly setting OMP_NUM_THREADS=5 or TBLIS_NUM_THREADS=5.

[devinamatthews]

Another quick note:

        tblis::mult<double>(1.0,
                            A,  "bikp",
                            BC, "bkjp",
                            0.0,
                            D,  "bijp");

For this contraction, placing the Hadamard (appearing in all 3 tensors) index first in column-major ordering is the worst possible performance situation. You want your tensors to all have a unit stride index remaining after taking out the Hadamard indices (this is implemented as an explicit loop over contractions with the remaining indices). For example this would probably be much better:

        tblis::mult<double>(1.0,
                            A,  "ikbp",
                            BC, "kjbp",
                            0.0,
                            D,  "ijbp");

[devinamatthews]

What is the commit for your "old" version, and how was it configured?

[JMartinon]

Another quick note:
For example this would probably be much better:

    tblis::mult<double>(1.0,
                        A,  "ikbp",
                        BC, "kjbp",
                        0.0,
                        D,  "ijbp");

Thank you for the quick reply and suggestion. Indeed I didn't pay much attention to the indices ordering since my main focus was comparing the two versions of the library. But, definitely, an optimal index ordering when you have a cascade of contractions (e.g. tensor networks) is something to consider.

What is the commit for your "old" version, and how was it configured?

As stated in my initial message the old version I used is the following:

commit 4de1919dfe194f5e47dfc93660ce4206d8e12c4e (HEAD -> master, origin/master, origin/HEAD)
Merge: 7d979b6 7436875
Author: Devin Matthews <damatthews@smu.edu>
Date:   Sat Apr 22 14:57:23 2023 -0500

Which was, I believe, until recently, the master branch.

I simply configured both libraries as ./configure --prefix=/path/to/installdir CC=gcc-15 CXX=g++-15

[devinamatthews]

The good news is that the new version can indeed be a little faster than the old one if the thread distribution among the various loops is "fixed". The old version was getting very lucky because the compiler actually did a good job vectorizing the generic microkernel.

I'll have to dig into the thread distribution logic now to see what changed.

[devinamatthews]

Well, now I can't reproduce the performance degradation with the new version that I saw. macOS is absolutely terrible for performance measurements... In any case, here are some observations and notes:

1. Calling tblis::mult once before the measurement loop helps a lot because the first call does substantial initialization. I went ahead and called both contractions before measuring to also try and pre-fault pages in the output tensors.
2. Recording the minimum time provides a more stable baseline, although of course average is more meaningful for real-world performance.
3. Splitting the timings for the two contractions shows that the second one takes only ~10% of the time, and the slowdown was (when I did see it) concentrated in the first contraction.
4. The way that old and new TBLIS assign numbers of threads to various loops is different, but NOT because of a change in logic, but because the microkernels have different "preferences" for column- or row-major matrices. Because the output tensor is highly non-square (when viewed as a matrix), there is a potential performance difference between computing a matrix A which is m*n and the matrix A^T which is n*m.

The good news is that this change is about equally likely to result in a speedup, depending on the shapes of the tensors passed in. Getting the "best of both worlds" is a difficult problem which will need to be solved in BLIS first. For now, I recommend testing different orders of indices to get the best performance. Note that the Apple M series kernel and the Intel/AMD kernels have the same row/column preference, so the same orderings should work well on both.

[JMartinon]

1. Calling tblis::mult once before the measurement loop helps a lot because the first call does substantial initialization. I went ahead and called both contractions before measuring to also try and pre-fault pages in the output tensors.

My apologies, I'm not sure to quite understand what you mean here. You're saying that in our sampling loop over the contractions (where we measure the evaluation time and later take the average) the first iteration is always significantly slower than the other ones ? If so, is my method of a simple loop over tblis::mult a good approach to benchmark real life performances ?

As you suggested I did few new benchmarks on my Mac fixing tblis_set_num_threads(5); (without knowing exactly which cores are picked tho) and increasing the sampling MaxIter=100. Using C1 for the first contraction and C2 for the second:

Version	Arch	Threads	Avr Total Time (s)	Min Total Time (s)	Avr C1 Time (s)	Min C1 Time (s)	Avr C2 Time (s)	Min C2 Time (s)
old	ARM	11	0.769774	0.72498	0.622687	0.590996	0.147087	0.129283
old	ARM	5	0.822876	0.786168	0.784531	0.749371	0.0383454	0.036663
new	ARM	11	2.49889	1.83115	0.653405	0.607351	1.84549	1.17281
new	ARM	5	0.888229	0.804252	0.813835	0.736644	0.0743936	0.064987

Several things to notice here:

• Switching from 11 to 5 threads indeed leads to a massive speed up for the average evaluation time in the case of the new library. For the old one the opposite actually happens (although the difference might not be significative)
• For the new library and 11 threads we see that the second contraction is the actual bottleneck. Where for the 5 threads case the second contraction now only represent ~10% of the total evaluation time, as you previously observed. However, the first contraction seems to be faster in the 11 threads case.
• Almost the same behavior seems to appear in the old library: C1 fast for 11 threads but C2 faster for 5 threads.

Finally, my overall understanding here is quite limited but, switching from a generic microkernel to an architecture specific one, I expected a consequent leap in performance, even for clunky contractions. However, so far, I observed that the old library is, at best, still slightly faster. Is it because the new version of TBLIS still needs some polishing, or because I’m not using it correctly?
Thanks again for your feedback !