LLM Response Consistancy Across Langauge and Scale

[puffy310]

Hello! I'm working on a project involving bilingual language models(specifically the InternLM2 Series). The questions that come out of this are as followed: How far apart are models in responses to the same question in different languages, is it conceptually different in some cases or simply semantically different, for certain questions, will both languages give different answers, how does the effect of China’s web culture versus United States web culture affect LLM training, and the biggest question is if models start blurring meaning at larger sizes.
I am still working on much of the details but here's a somewhat simplified plan.

Software to be Used:
InternLM2 Series, LMDeploy
BAAI Bge-m3

Basic Algorithm:
Send Common QA Questions, with one in English a translated version.
Test similarity.
Sort them by approximate category
Average out results.
Try again at a higher scale.

Requested Hardware:
A100 1x for a prolonged period of time.

[puffy310]

Update: I have created a dataset of questions to ask, I have to sleep unfortunately so I will be working on this project tomorrow morning. I have realized how tough it is to make open source experiments. Hopefully I can make a reproducible version of the project.