llm-lab/CLASP

CLASP (Contrastive Language-Speech Pretraining) is a novel, lightweight, multilingual, multimodal representation designed for audio-text retrieval.
To learn more about our proposed model, please refer to this paper. All code is available on this GitHub page.
The newly introduced dataset, SpeechBrown, which we created for training this model, can be found on this page

CLASP creates powerful and meaningful semantic embeddings for raw speech in a 768-dimensional multilingual representation space. These embeddings can be used in various tasks such as speech retrieval or classification.

Here, we have uploaded different versions of the CLASP and LASP models we trained:

• CLASP_Concat_Final_Fusion_Encoder.pt: The best model we trained based on retrieval and classification metrics. It uses the concatenation fusion encoder strategy and is trained with contrastive loss.
• CLASP_Gating.pt: Trained with contrastive loss and employs a gating algorithm.
• LASP_Concat.pt: Trained with Huber loss and employs the concatenation strategy.
• LASP_Gating.pt: Trained with Huber loss and employs the gating algorithm.

To use these models or train your own on custom datasets, please refer to our GitHub page. The clasp-inference.ipynb notebook provides an example of loading and using the model.

Steps for Inference with CLASP:

1. Load our model with the specified architecture.
2. Load the EfficientNet encoder for spectrogram encoding.
3. Load the HuBERT encoder for self-supervised speech encoding.
4. Use our scripts to generate embeddings for your audio files.
5. Use these embeddings for tasks like classification or speech retrieval. For speech retrieval, you can load the LaBSE sentence transformer to compute the cosine similarity between query and speech embeddings.

Architecture Overview

Contributions

1. We introduce CLASP (Contrastive Language-Speech Pretraining), a novel, lightweight, multilingual, multimodal representation designed for audio-text retrieval.
2. We present a diverse paired speech-text dataset (Speech Brown) in 15 categories, covering a wide range of topics from fiction to religion.
3. We demonstrate that combining audio spectrograms with a pre-trained self-supervised speech model enhances audio encoding in retrieval applications.
4. Evaluations in multiple languages show that CLASP achieves new benchmarks in HITS@1, Mean Reciprocal Rank (MRR), and Mean Rank (meanR) metrics.

Citations

If you find our paper, code, data, or models useful, please cite the paper:

@misc{abootorabi2024claspcontrastivelanguagespeechpretraining,
      title={CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval}, 
      author={Mohammad Mahdi Abootorabi and Ehsaneddin Asgari},
      year={2024},
      eprint={2412.13071},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.13071}, 
}

Contact

If you have questions, please email [email protected] or [email protected].