adding onnx support for Funnel Transformer #1231
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
regisss
reviewed
Jul 27, 2023
There was a problem hiding this comment.
@sidistic Nice PR! I left one comment that should be easily addressed.
There are also a few more things to do:
- Could you rebase your branch on main? That way the PR documentation will be generated without error. Edit: not sure if this is linked to this actually, let's see.
- The code style check failed. Here is what you need to run from the root of the repo to solve it:
pip install .[quality] make style
- Also, could you add "Funnel" right after "Flaubert" in this list of supported models?
Finally, to test your PR, you can try to export a model with Optimum CLI as follows:
optimum-cli export onnx --model funnel-transformer/large --task question-answering funnel_onnx/You can of course try with other checkpoints and tasks.
Don't hesitate if you have any questions 🙂
Comment on lines
1249
to
1263
| class FunnelTransformerOnnxConfig(TextEncoderOnnxConfig): | ||
| NORMALIZED_CONFIG_CLASS = NormalizedTextConfig | ||
| ATOL_FOR_VALIDATION = 1e-4 | ||
|
|
||
| @property | ||
| def inputs(self) -> Dict[str, Dict[int, str]]: | ||
| if self.task == "multiple-choice": | ||
| dynamic_axis = {0: "batch_size", 1: "num_choices", 2: "sequence_length"} | ||
| else: | ||
| dynamic_axis = {0: "batch_size", 1: "sequence_length"} | ||
| return { | ||
| "input_ids": dynamic_axis, | ||
| "attention_mask": dynamic_axis, | ||
| "token_type_ids": dynamic_axis, | ||
| } |
There was a problem hiding this comment.
Suggested change
| class FunnelTransformerOnnxConfig(TextEncoderOnnxConfig): | |
| NORMALIZED_CONFIG_CLASS = NormalizedTextConfig | |
| ATOL_FOR_VALIDATION = 1e-4 | |
| @property | |
| def inputs(self) -> Dict[str, Dict[int, str]]: | |
| if self.task == "multiple-choice": | |
| dynamic_axis = {0: "batch_size", 1: "num_choices", 2: "sequence_length"} | |
| else: | |
| dynamic_axis = {0: "batch_size", 1: "sequence_length"} | |
| return { | |
| "input_ids": dynamic_axis, | |
| "attention_mask": dynamic_axis, | |
| "token_type_ids": dynamic_axis, | |
| } | |
| class FunnelTransformerOnnxConfig(BertOnnxConfig): | |
| pass |
If I'm not mistaken, it's the exact same config as BERT. So you can just make this class inherit from BertOnnxConfig and it will behave the same. There are actually quite a lot of models with such configs: https://github.com/huggingface/optimum/blob/bdc3ecc88b0796fbb7ae051ace55d74bb2edb81d/optimum/exporters/onnx/model_configs.py#L86C24-L86C38
There was a problem hiding this comment.
I've added this suggestion and there was an error with the opset to i changed to default opset to 12. But now there's a different error and there's no exporting of the model itself. Before i changed the default opset I had this error
Framework not specified. Using pt to export to ONNX.
Some weights of FunnelForQuestionAnswering were not initialized from the model checkpoint at funnel-transformer/large and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 2.0.1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:322: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
num_remove = shift * len(pooled_pos)
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:668: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
pooling_flag = pooling_flag and block_index > 0
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:497: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
shift = 2 if q_head.shape[1] != context_len else 1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/_internal/jit_utils.py:306: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/jit/passes/onnx/constant_fold.cpp:181.)
_C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
Traceback (most recent call last):
File "/Users/siddharthsagar/Desktop/Open Source/hf/bin/optimum-cli", line 10, in <module>
sys.exit(main())
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/optimum_cli.py", line 163, in main
service.run()
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/export/onnx.py", line 219, in run
main_export(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/__main__.py", line 442, in main_export
_, onnx_outputs = export_models(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 758, in export_models
export(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 861, in export
export_output = export_pytorch(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 578, in export_pytorch
onnx_export(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 506, in export
_export(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1548, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1117, in _model_to_graph
graph = _optimize_graph(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 665, in _optimize_graph
graph = _C._jit_pass_onnx(graph, operator_export_type)
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1891, in _run_symbolic_function
return symbolic_fn(graph_context, *inputs, **attrs)
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 392, in wrapper
return fn(g, *args, **kwargs)
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 306, in wrapper
return fn(g, *args, **kwargs)
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_opset11.py", line 625, in symbolic_fn
padding_ceil = opset9.get_pool_ceil_padding(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_opset9.py", line 1501, in get_pool_ceil_padding
return symbolic_helper._unimplemented(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 607, in _unimplemented
_onnx_unsupported(f"{op}, {msg}", value)
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 618, in _onnx_unsupported
raise errors.SymbolicValueError(
torch.onnx.errors.SymbolicValueError: Unsupported: ONNX export of operator get_pool_ceil_padding, input size not accessible. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues [Caused by the value '2491 defined in (%2491 : Float(*, 1, *, 1024, device=cpu) = onnx::Pad[mode="constant"](%2487, %2490), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder
)' (type 'Tensor') in the TorchScript graph. The containing node has kind 'onnx::Pad'.]
Inputs:
#0: 2487 defined in (%2487 : Float(*, 1, *, 1024, strides=[16384, 16384, 1024, 1], requires_grad=0, device=cpu) = onnx::Unsqueeze[axes=[1]](%2486), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder # /Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:380:0
) (type 'Tensor')
#1: 2490 defined in (%2490 : Long(8, strides=[1], device=cpu) = onnx::Constant[value= 0 0 0 0 0 0 0 0 [ CPULongType{8} ]](), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder
) (type 'Tensor')
Outputs:
#0: 2491 defined in (%2491 : Float(*, 1, *, 1024, device=cpu) = onnx::Pad[mode="constant"](%2487, %2490), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder
) (type 'Tensor')
This is what i get before adding DEFAULT_ONNX_OPSET = 12, This was the console log :
Framework not specified. Using pt to export to ONNX.
Downloading model.safetensors: 100%|█████████████████████████████████████████████████████████| 1.54G/1.54G [01:09<00:00, 22.2MB/s]
Some weights of FunnelForQuestionAnswering were not initialized from the model checkpoint at funnel-transformer/large and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 2.0.1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:322: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
num_remove = shift * len(pooled_pos)
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:668: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
pooling_flag = pooling_flag and block_index > 0
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:497: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
shift = 2 if q_head.shape[1] != context_len else 1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/_internal/jit_utils.py:306: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/jit/passes/onnx/constant_fold.cpp:181.)
_C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 1 ERROR ========================
ERROR: operator-supported-in-newer-opset-version
================================================
Exporting the operator 'aten::einsum' to ONNX opset version 11 is not supported. Support for this operator was added in version 12, try exporting with this version.
None
<Set verbose=True to see more details>
Traceback (most recent call last):
File "/Users/siddharthsagar/Desktop/Open Source/hf/bin/optimum-cli", line 10, in <module>
sys.exit(main())
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/optimum_cli.py", line 163, in main
service.run()
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/export/onnx.py", line 219, in run
main_export(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/__main__.py", line 442, in main_export
_, onnx_outputs = export_models(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 758, in export_models
export(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 861, in export
export_output = export_pytorch(
File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 578, in export_pytorch
onnx_export(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 506, in export
_export(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1548, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1117, in _model_to_graph
graph = _optimize_graph(
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 665, in _optimize_graph
graph = _C._jit_pass_onnx(graph, operator_export_type)
File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1901, in _run_symbolic_function
raise errors.UnsupportedOperatorError(
torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 'aten::einsum' to ONNX opset version 11 is not supported. Support for this operator was added in version 12, try exporting with this version.
Could you point me toward what files I have to look at next to fix this issue or where I could have gone wrong in my implementation?
There was a problem hiding this comment.
It seems to be an issue with this line: https://github.com/huggingface/transformers/blob/4692d2619433f1eb064f3da4f3573f060a115eac/src/transformers/models/funnel/modeling_funnel.py#L385
According to what I see here, the issue has not been solved yet 🙁
There was a problem hiding this comment.
Hello @regisss , Could I work on the bug? I would love to help in any way I can.
There was a problem hiding this comment.
Unfortunately there is not much we can do I think. We basically need:
- This PR to be merged in ONNXRuntime
- Some change in Torch too to align on this
Maybe you can reach out in this issue: pytorch/pytorch#101397
There was a problem hiding this comment.
Sure I will look into that! thank you for your help.
sidistic
force-pushed
the
funnel-transformer
branch
from
be80e0a to
ee28c09
Compare
|
This PR has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Hello this is my first PR ever for an Open source project. I would love to hear your thoughts on how to test the code on my machine before committing any changes ! I've added a class for input and I've added a class in class. I've also added a few lines of code specifying the models to be used for tiny and large model tests in
optimum/tests/exporters/exporters_utils.py
and a NormalizedTextConfig in normalized_config.py. I've also added the mapping in
optimum/exporters/tasks.py
. I have been reading up the model specification in tranformer/models for funnel and would love to hear your advice in the next steps and how to test my code and if my class definition for onnxconfig is correct. I use Hugging face models for a lot of my projects and I would love to keep contributing to Hugging Face in the future. Looking forward to working with the Hugging face community.
Fixes # (issue)
Before submitting