Autoreload with patch."ssh://git@..."

[apetrovic-indata]

I have a directory with several repos under development, N lib repos an M bin repos that use the lib repos. In order to cover my development, there is a sibling .cargo directory with config.toml that declares patches for all the development repos.
The patches are in the form of

[patch."ssh://git@myforgejo/my_lib"]
my_lib = { path = "/workspace/my-lib" }

Now, when rust analyzer starts working, the ssh urls will get updated (Updating git repository ssh://git@...), this obviously touches the repos (I'm not exactly sure which files, but I suspect it's the Cargo.toml) and Autoreload gets triggered. This brings rust analyzer to an infinite loop.
Currently, the only remedy to the infinite loop I'm aware of is to disable Autoreload.
But, with Autoreload disabled, any change I make in any repo of mine triggers a warning
Auto-reloading is disabled and the workspace has changed, a manual workspace reload is required.
and rust analyzer stops being a reliable helper in my development.

Is there a more elegant approach to get rust analyzer out of the infinite Autoreload loop with ssh url patches?