[BUG] Mixed Precision Gemm Correctness Regression in Cutlass 3.7/3.8

[jwfromm]

Describe the bug
Since Cutlass 3.7, mixed input dtype GEMMs are producing less accurate outputs than they were in Cutlass 3.6. The loss of accuracy is substantial and makes using mixed input impractical for real use-cases.

Specifically, we have a collection of mixed input GEMMs in FBGEMM that work well on Cutlass 3.6. While these kernels compile fine with newer versions of cutlass (after small api updates), they produce garbage outputs.

Directly copying example 55's BF16 x I4 Gemm example produces slightly better results, but the outputs are still much less accurate than the 3.6 equivalents.

Steps/Code to reproduce bug
We use this benchmarking script to measure the performance and accuracy of kernels. The script can be run with these sample arguments:

python quantize_bench.py --kernels=bf16_baseline,cutlass_bf16i4_rowwise --M=128 --N=2048 --K=2048

This will produce an output like this:

bf16_baseline sim: 0.000.
bf16_baseline ms: 0.007.
bf16_baseline TFLOPS: 150.635.
bf16_baseline GB/s: 1323.942.
cutlass_bf16i4_rowwise sim: 28.375.
cutlass_bf16i4_rowwise ms: 0.013.
cutlass_bf16i4_rowwise TFLOPS: 79.561.
cutlass_bf16i4_rowwise GB/s: 233.089.

The sim metric is an L1 distance from the BF16 output. After updating to cutlass 3.7, copying example 55, and running the same script we get:

bf16_baseline sim: 0.000.
bf16_baseline ms: 0.007.
bf16_baseline TFLOPS: 150.563.
bf16_baseline GB/s: 1323.308.
cutlass_bf16i4_rowwise sim: 328.000.
cutlass_bf16i4_rowwise ms: 0.013.
cutlass_bf16i4_rowwise TFLOPS: 80.016.
cutlass_bf16i4_rowwise GB/s: 234.421.

Which has a clearly less correct output. The updated version of the kernel can be found at this PR

Expected behavior
The accuracy of mixed input kernels should not have changed due to updates.

Environment details (please complete the following information):
cuda 12.4 driver version 535.154.05 on Linux system with 8X H100 GPUs.

[hwu36]

@IwakuraRein