How to let executorch export intput output int8

[kris-himax]

Hi,

I use https://github.com/pytorch/executorch/blob/main/examples/arm/ethos_u_minimal_example.ipynb to export example model and run on FVP.
The output is 2.0(float).

But I modify the code and that it output and intput to be int8. But the output at FVP show 1(char).
I think that it is wrong. How can I fix it?

from executorch.backends.arm.ethosu import EthosUPartitioner
from executorch.exir import (
    EdgeCompileConfig,
    ExecutorchBackendConfig,
    to_edge_transform_and_lower,
)
from executorch.extension.export_util.utils import save_pte_program
from executorch.exir.passes.quantize_io_pass import QuantizeInputs, QuantizeOutputs



# Create partitioner from compile spec
partitioner = EthosUPartitioner(compile_spec)

# Lower the exported program to the Ethos-U backend
edge_program_manager = to_edge_transform_and_lower(
            quantized_exported_program,
            partitioner=[partitioner],
            compile_config=EdgeCompileConfig(
                _check_ir_validity=False,
            ),
        )
edge_program_manager.transform(passes=[QuantizeInputs(edge_program_manager, [0, 1]), QuantizeOutputs(edge_program_manager, [0])])
# Convert edge program to executorch
executorch_program_manager = edge_program_manager.to_executorch(
            config=ExecutorchBackendConfig(extract_delegate_segments=False)
        )

_ = executorch_program_manager.exported_program().graph_module.print_readable()

# Save pte file
save_pte_program(executorch_program_manager, "ethos_u_minimal_example_test_inout_int8.pte")

By the way, according to #7590
How can the embedded application access the quantisation scale & zero point finally?

Thanks,
Kris

cc @freddan80 @per @zingo @oscarandersson8218 @digantdesai

[gggekov]

Hi @kris-himax ,
Thanks for reaching out again!

When you apply the QuantizeInputs, QuantizeOutputs passes, the pte contains multiple methods(as opposed to just containing the forward method when the input is float32 and you do Q/DQ within the pte). There are separate methods for reading the input quantization parameters and output quantization parameters(e.g. one method for reading the input zero point, another method for reading the quantization scale and so on). As per #7590, you can read access the return values of these methods(meaning the zero point, quant scale,etc) by doing

std::vector<EValue> outputs(method->outputs_size());
EValue* dataPtr = outputs.data();
ET_LOG(Info,"zero point = %d",dataPtr->toInt());  //Assuming you have executed the method for the zero point from the pte

in the runtime application.

Are you trying to access the quantization parameters via the GUI ? How do you display the graph with the EthosUBackend operator - do you use model explorer?

One final comment - if you do 1+1 in fp32 and calibrate with random data, i would expect that the final result in the fp32 domain will be a bit different than 2 due to the quantization scale & zp.

[kris-himax]

Hi @gggekov ,
Thank you for the reply.
I add the code to print the result about zero point at here.
But got the value is not int error at FVP.
I also try transform to double. I still face the error. How can I fix it?

void print_outputs(RunnerContext& ctx) {
  std::vector<EValue> outputs(ctx.method.value()->outputs_size());
  ET_LOG(Info, "%zu outputs: ", outputs.size());
  Error status =
      ctx.method.value()->get_outputs(outputs.data(), outputs.size());
  ET_CHECK(status == Error::Ok);
EValue* dataPtr = outputs.data();
ET_LOG(Info,"zero point = %d",dataPtr->toInt());  //Assuming you have executed the method for the zero point from the pte
  // Print the outputs.

Also,there is many [method](https://github.com/pytorch/executorch/blob/82e37dfaac747ecf5b861f68acb152de2468c091/exir/passes/quantize_io_pass.py#L231) to quantize input and output from floating to int8, and I want to know about what is different with each other ("scale", "zp", "quant_min", "quant_max", "dtype"). If I did not set any method to do the quantize input and output, is it will not use any algorithm and just let input to be in8?

Are you trying to access the quantization parameters via the GUI ? How do you display the graph with the EthosUBackend operator - do you use model explorer?

Yes.I use https://netron.app/ from https://github.com/lutzroeder/netron to see the graph about .pte file.

One final comment - if you do 1+1 in fp32 and calibrate with random data, i would expect that the final result in the fp32 domain will be a bit different than 2 due to the quantization scale & zp.

If I got the final result from model is 1, then I do the dequantize by zero point and scale, it should be almost close to 2?