🚀 The feature, motivation and pitch

We want to be able to measure the performance for multimodal encoder alone quantitatively, which requires a benchmark that focuses on encoder forward pass.

Ideally this benchmark should do:

1. Import encoder modules from model file and initialize it with dummy weights & vLLM config
2. Convert dummy images to inputs of the encoder via models's HF processor
3. Measure latency with processing a batch of X images with Y sizes.

X and Y should be configurable.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.