[Refactor 2] ECG-Language Model building pipeline

[willxxy]

Background:

We currently use the ECG-Language Model building pipeline implemented in src/elms. The top-level class is BuildELM in src/elms/build_elm.py, which instantiates BuildLLM, BuildEncoder, and ConnectNN. As their names suggest, BuildLLM builds the LLM, BuildEncoder builds an encoder if specified, and ConnectNN connects the LLM with the encoder or other intermediate module if needed.

Problem:

1. The main challenge is that it is difficult to define what exactly counts as an LLM, an encoder, or a connector. For example, in encoder-free models like ELF (src/elms/llm_encoders/base_elf.py), we do not treat the projection layer as an encoder. Instead, we treat it as a connector. Therefore, when building ELF, we simply connect the LLM with the connector (src/elms/connect_nns.py). However, this becomes less clear when the connector becomes more complex, such as in src/elms/connectors/mlp_proj.py. Most people would likely argue that this MLP “connector” is still not enough to be considered an encoder, since encoders are usually more architecturally complex and are often pretrained on a large corpus of data.

2. However, I feel like the boundary between encoder and connector may become increasingly blurry over time. It may be better to think at a higher level and separate the system into the LLM and all other attachments, where attachments include both encoders and connectors. This may simplify the building process: given an ELM name, we first load the pretrained LLM, then build whatever encoder, connector, or attachment that specific ELM requires.

3. This also relates to how we currently separate encoders, such as vision encoders and ECG encoders. Should these remain separated by modality, or should we place all encoders under a single encoders directory?

4. I also do not like the current src/elms/llm_encoders directory. The naming feels unclear, especially because some of these models are not really LLM encoders in the conventional sense.

5. Overall, this is mainly an organization problem. The current build pipeline works, but it does not feel clean.

6. Lastly, I do not like how much we separate HuggingFace and non-HuggingFace models. Right now, we have HF-specific code for building HF models. I feel like it may be cleaner to keep the source code for these models in the repository and have users download the pretrained weights separately, similar to the HF workflow. However, I am not fully sure what the cleanest abstraction should be here.

Solution:

1. No solution yet.

Additional Notes:

1. None.