How to implement accelerator support as a backend?

[taronaeo]

Hi everyone,

I'm trying to implement a zDNN (z-Deep Neural Network) backend for the IBM NNPA Accelerator in the Telum processors but I am unsure of how to start. This accelerator is a synchronous compute device that requires it's own ztensor data structure to run computation from/to.

I've tried taking references from the CPU and CUDA implementations but I am confused by which methods/functions are required to be implemented for an accelerator device.

It would be really helpful if someone could provide a "shopping cart" list of items to do, or maybe point me to a documentation that I may have missed. Thank you!

[zhouwg]

hello, glad to see you here. I think the following links can provide a useful reference for your needs:

https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-blas/ggml-blas.cpp

https://github.com/ggml-org/ggml/blob/master/examples/simple/simple-backend.cpp

https://github.com/ggml-org/ggml/tree/master/examples/simple

accordingly, there is a ggml-qnn backend might be another useful reference for your zDNN backend: when I started this ggml-qnn backend I had nothing knowledge about qnn and limited knowledge about ggml internal, so I'm sure you will be success with your zDNN backend.

at the same time, you can utilize a command line program to understand how a specified backend works(offload a specified ggml op to hardware accelerator or NPU through the highly-well designed ggml backend subsystem) and verify your zDNN backend accordingly:

https://github.com/kantv-ai/llama.cpp/blob/kantvai-ggmlqnn-npurpc/tests/ggml-qnn-ut.cpp

one more important thing: slaren is the author of the highly-well designed ggml backend subsystem, maybe you can ask him any related technical questions.

[taronaeo]

Hey @zhouwg, thanks for providing that useful list of references that I could use! It has been helpful referencing ggml-blas.cpp so far but I have a follow-up implementation question because zDNN is a little different where it requires it's down zTensor data format.

Is there a specific function that I should be calling/putting the tensor initialisation implementation such that ggml_tensor's data can be mapped to ztensor? There are a whole bunch of steps required to prepare the ztensor for computation and I can't seem to find a suitable function to have all these done.

The preparation steps include:

1. Keeping track of all pre-transform descriptors, descriptors, the actual zTensor itself and allocating the tensor storage i.e.,

zdnn_tensor_desc pre_tfm_desc, tfm_desc;
zdnn_tensor z_input1, z_input2, z_result;

void * z_input_storage1 = malloc(...);
void * z_input_storage2 = malloc(...);
void * z_result_stoage = malloc(...);

2. Load tensor data i.e.,

// set input tensor data from 0 to 127 sequentially and repeat
for (uint64_t i = 0; i < num_elements; i++) {
    ((float *)z_input_storage1)[i] = (float)(i & 0x7F);
    ((float *)z_input_storage2)[i] = (float)(i & 0x7F);
}

3. Initialize tensor descriptor with pre-transformed (original) shape information and transform i.e.,

zdnn_init_pre_transformed_desc(ZDNN_NHWC, type, &pre_tfm_desc, dim_n, dim_h, dim_w, dim_c);

zdnn_generate_transformed_desc(&pre_tfm_desc, &tfm_desc);

4. Initialize ztensors with the tensor descriptors i.e.,

zdnn_init_ztensor_with_malloc(&pre_tfm_desc, &tfm_desc, &z_input1);
zdnn_init_ztensor_with_malloc(&pre_tfm_desc, &tfm_desc, &z_input2);
zdnn_init_ztensor_with_malloc(&pre_tfm_desc, &tfm_desc, &z_result);

5. Convert tensor to supported format for zDNN operation i.e.,

zdnn_transform_ztensor(&z_input1, z_input_storage1);
zdnn_transform_ztensor(&z_input2, z_input_storage2);

6. Finally do the zDNN operation i.e.,

zdnn_add(&z_input_storage1, &z_input_storage2, &z_result);

I've investigated the .init_tensor mapping from ggml_backend_buffer_i (CUDA example) and it appears that it is just padding the tensor and zero-ing out the empty spaces to prevent NaN. But it doesn't seem to be the correct place to initialize the actual tensor.

Any pointers would be very much appreciated! :)

[zhouwg]

hello @taronaeo,

glad to see your technical discussion.

when I see this:

Is there a specific function that I should be calling/putting the tensor initialisation implementation such that ggml_tensor's data can be mapped to ztensor? There are a whole bunch of steps required to prepare the ztensor for computation and I can't seem to find a suitable function to have all these done.

I think following code snippet in function ggml_qnn_add() would be helpful for your zDNN backend:

QNN_VER_PTR(*tensor_0)->clientBuf = {src0->data,
                                             qnn_get_ggml_tensor_data_size(src0)};
        QNN_VER_PTR(*tensor_1)->clientBuf = {src1->data,
                                             qnn_get_ggml_tensor_data_size(src1)};
        QNN_VER_PTR(*tensor_2)->clientBuf = {dst->data,
                                             qnn_get_ggml_tensor_data_size(dst)};

        QNN_VER_PTR(*tensor_0)->dimensions = dimensions_input_0;
        QNN_VER_PTR(*tensor_0)->rank       = qnn_get_ggml_tensor_rank(src0);
        QNN_VER_PTR(*tensor_0)->dataType   = src0_qnn_type;
        QNN_VER_PTR(*tensor_1)->dimensions = dimensions_input_1;
        QNN_VER_PTR(*tensor_1)->rank       = qnn_get_ggml_tensor_rank(src1);
        QNN_VER_PTR(*tensor_1)->dataType   = src1_qnn_type;
        QNN_VER_PTR(*tensor_2)->dimensions = dimensions_output;
        QNN_VER_PTR(*tensor_2)->rank       = qnn_get_ggml_tensor_rank(dst);
        QNN_VER_PTR(*tensor_2)->dataType   = dst_qnn_type;

or you might be get some useful hints from function ggml_qnn_general_node in project ggml-qnn.

from my personal point of view, Qualcomm QNN SDK is a highly-well designed AI SDK and it matches well with highly-well designed and compact ggml machine learning library. I guess IBM's zDNN SDK also provide similar mechanism.

in the fact, we are trying to implement a concise(without much complex encapsulation) QNN backend for llama.cpp through the C API provided by Qualcomm QNN SDK, so domain developers can clearly know how data path works in this specified backend, how ggml tensors mapping to the QNN tensors and how QNN SDK handles the hardware acceleration.

I've investigated the .init_tensor mapping from ggml_backend_buffer_i (CUDA example) and it appears that it is just padding the tensor and zero-ing out the empty spaces to prevent NaN. But it doesn't seem to be the correct place to initialize the actual tensor.

.init_tensor is not actually used in a specified backend(I was also a bit confused in the early stage of ggml-qnn), we can also see this in Intel's sycl backend. generally speaking, we'll create/initialize a hardware-relative tensor in a specified hardware accelerated op function, pls refer to code snippet in function ggml_qnn_general_node in project ggml-qnn, this approach can also be found in Huawei's CANN backend. one more important thing, as a backend developer, we should provide a well-designed data structure to hold/save these hardware-relative resources and ensure no memory leak in a specified cleanup function. from my personal point of view, I recommend use a simple and concise data structure rather than complex/complicated data structure(e.g. very complex encapsulation through C++).

summary as following for a specified ggml backend:

• follow the principle of highly-well designed ggml backend subsystem(this is template which can be found at specified backend from Intel/Huawei/Qualcomm or our ggml-qnn backend)
• mapping ggml tensors to hardware-relative tensors(key-point)
• provide a simple and functional data structure to manage hardware-relative resources(key-point)
• implement hardware accelerated op functions one by one(key-point)
• mapping the entire ggml's cgraph(computational graph) to hardware accelerated computational graph(this is suitable for Qualcomm's QNN backend because Qualcomm has a dedicated graph optimization tech for AI computation in a specified QNN hardware device)
• simple is beautiful rather than complex C++ encapsulation although developer's C++ skill might-be/is really good(this is my personal point of view, in the fact, we can find this philosophy in the great ggml machine learning library, I personally think that's one of the key-reasons why ggml/whisper.cpp/llama.cpp is so popular and so successful, especially we can see that there are already TensorflowLite from Google, Executorch from Meta, MACE from Xiaomi... they are both good but many programmers and IT giants and research institutions prefer ggml for device-AI scenario)

hope this a bit useful for you and pls correct me if I made incorrect expression.