feat(transformers): add Florence2 (v4.57.1)

[alien-0119]

What does this PR do?

Adds # (feature)
Add Florence2 model and fast ut.

Usage Example:

import mindspore as ms
import requests
from PIL import Image
from transformers import AutoProcessor
from mindone.transformers import Florence2ForConditionalGeneration

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

model = Florence2ForConditionalGeneration.from_pretrained("florence-community/Florence-2-base")
processor = AutoProcessor.from_pretrained("florence-community/Florence-2-base")

task_prompt = "<OD>"
inputs = processor(text=task_prompt, images=image, return_tensors="np")
inputs = {k: ms.tensor(v) for k, v in inputs.items()}

generated_ids = model.generate(
    **inputs,
    max_new_tokens=1024,
    num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

image_size = image.size
parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=image_size)
print(parsed_answer)
# {'<od>': {'bboxes': [[34, 160, 597, 371], [272, 241, 303, 247], [454, 276, 553, 370], [96, 280, 198, 371]], 'labels': ['car', 'door handle', 'wheel', 'wheel']}}</od>

Performance:
Experiments were tested on Ascend Atlas 800T A2 machines with mindspore 2.7.0 pynative mode.

model	precision	weight load(s)	s/step
microsoft/Florence-2-base	fp32	15.662	0.064
microsoft/Florence-2-base	fp16	15.974	0.067
microsoft/Florence-2-base	bf16	18.089	0.070

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
  documentation guidelines
• Did you build and run the code without any errors?
• Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@xxx

[gemini-code-assist]

Summary of Changes

Hello @alien-0119, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the capabilities of the mindone/transformers library by introducing the Florence2 model, a versatile vision-language model. Concurrently, it brings the Bart model up to date with version 4.57.1, which includes a comprehensive overhaul of its attention mechanisms and caching strategies, enhancing its efficiency and aligning it with modern best practices in model architecture.

Highlights

• New Model Addition: The Florence2 model, a powerful vision model capable of tasks like captioning, detection, and segmentation, has been integrated into the mindone/transformers library.
• Bart Model Update: The existing Bart model implementation has been updated to version 4.57.1, featuring significant refactoring of its attention mechanisms, cache handling, and general code structure for improved performance and maintainability.
• Attention Mechanism Refactoring: The Bart model's attention logic has been modernized, introducing a new eager_attention_forward function and transitioning to a more robust Cache system for managing past key-value states, replacing older FlashAttention2 specific implementations.
• Enhanced Docstring Generation: The Bart model now leverages the @auto_docstring decorator for automated documentation generation, streamlining the process and ensuring consistency across the codebase.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the Florence-2 model and includes a significant refactoring of the underlying BART model to align with newer features from the Hugging Face Transformers library, such as improved cache management and attention mechanisms. The changes are extensive and mostly look good.

I've identified a critical issue in the BartForCausalLM implementation where the configuration object is modified in-place, which could lead to bugs if the config is reused. I've also found a couple of medium-severity issues: a misleading error message in the BART decoder and an unused parameter in the new Florence-2 model code.

Overall, this is a great contribution. Addressing these points will improve the robustness and clarity of the code.

[gemini-code-assist]

The config object is modified in-place. This is dangerous as it can lead to unexpected behavior if the config object is reused for instantiating other models. A deep copy of the configuration should be made to avoid side effects, as was done in the previous version.

Please also add import copy back to the top of the file.

Suggested change

	def __init__(self, config):
	config = copy.deepcopy(config)
	config.is_decoder = True
	config.is_encoder_decoder = False
	super().__init__(config)
	def __init__(self, config):
	config = copy.deepcopy(config)
	config.is_decoder = True
	config.is_encoder_decoder = False
	super().__init__(config)

[gemini-code-assist]

The error message is misleading. It refers to decoder_input_ids and decoder_inputs_embeds, but the check is performed on input_ids and inputs_embeds. Additionally, the message only mentions one of the failure conditions (specifying both), while the logic also fails if neither is specified. A more accurate error message would improve clarity.

Suggested change

	if (input_ids is None) ^ (inputs_embeds is not None):
	raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")
	if (input_ids is None) == (inputs_embeds is None):
	raise ValueError("You have to specify either `input_ids` or `inputs_embeds`, but not both and not neither.")

[gemini-code-assist]

The pixel_mask parameter is defined but not used within the construct method. It should be removed to improve code clarity.

Suggested change

	def construct(self, pixel_values, pixel_mask=None):
	def construct(self, pixel_values):