pipeline breaks libcudnn_ops_infer library if model is not used

[jzju]

Tested versions

pyannote.audio==3.3.0
pyannote.pipeline==3.0.1

System information

Ubuntu 22.04

Issue description

from faster_whisper import WhisperModel
from pyannote.audio import Pipeline
import torch

annote = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1", use_auth_token="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"
)
annote.to(torch.device("cuda"))
annote("a.wav") # without this line Whisper crashes
trans = WhisperModel("large-v3", device="cuda", compute_type="float16")
trans.transcribe("a.wav")

If you don't call annote after annote.to(torch.device("cuda")) then whisper crash with this error Could not load library libcudnn_cnn_infer.so.8. Error: /lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8: undefined symbol: _ZN5cudnn14cublasSaxpy_v2EP13cublasContextiPKfS3_iPfi, version libcudnn_ops_infer.so.8 but it works fine if you call annote("a.wav")

Minimal reproduction example (MRE)

https://colab.research.google.com/see_description