When I disable Sonarr using the toggle, it automatically re-enables itself after a while. This doesn't happen immediately; it appears that some background timer is switching it back to "enabled." To work around this and temporarily disable it, I had to remove the API key instead.
I suspect this behavior might affect other apps as well, but I haven't tested them to confirm.