Has Gemini-Exp-1206 been tested?

[Consistos]

If not, I can volunteer.

[cpldcpu]

I have not tested it, because the access is very instable via openrouter.

If you want to give it a try, that would be much appreciated. However, I am currently updating the eval scripts and dataset (new branch), it would make sense to wait until that is completed.

[cpldcpu]

I added support for the google API now and evaluated some of the models on the long set.

It was very painful due to all the rate limits. Had to update the script to support restarts and collect data over multiple days.

https://github.com/cpldcpu/MisguidedAttention/tree/main/eval