Discord Server

Join the discord server if you are interested in LLM architecture or distributed training/inference research.

Efficient GPU kernels written in both CUDA and Triton

Modules

Module	Triton	CUDA
GRU	✅	❌
MoE	✅	✅
RNN	✅	❌

Ops

Module	Triton	CUDA
bmm	✅	❌
continuous_count	❌	✅
cross_entropy	✅	❌
fused_linear_cross_entropy	✅	❌
fused_residual_add_rmsnorm	✅	❌
gemm	✅	❌
grouped_gemm	❌	✅
matrix_transpose	✅	❌
rmsnorm	✅	❌
pack_sequence	✅	✅
softmax	✅	❌
swiglu	✅	✅
swiglu_packed	✅	❌
unpack_sequence	✅	✅
zeros	✅	✅