update

.gitignore

@@ -11,7 +11,9 @@ test-trainer
 bert_dataset
 trained_model
 results
+results_qlora
 output
+qlora_final_model
 
 src
 tmp

README.md

@@ -16,4 +16,4 @@ A sandbox environment to experiment with large language models and various NLP t
 - **Fine-tuning Series**: Develop a series of proof-of-concept (POC) repositories for fine-tuning LLMs, hosted on GitHub (e.g.,[ft2](https://github.com/locchh/ft2), [ft3](https://github.com/locchh/ft3), etc.). 🔥
 - **LLM Blog**: Write and publish insightful blog posts on LinkedIn, sharing perspectives and research related to LLMs (e.g.,[Artificial Intelligence For Programming](https://www.linkedin.com/posts/chuong-loc_artificialintelligence-softwaredevelopment-activity-7278039943480856576-2o3E?utm_source=share), [The Limitation of Context](https://www.linkedin.com/posts/chuong-loc_%F0%9D%90%93%F0%9D%90%A1%F0%9D%90%9E-%F0%9D%90%8B%F0%9D%90%A2%F0%9D%90%A6%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%AC-%F0%9D%90%A8%F0%9D%90%9F-%F0%9D%90%86%F0%9D%90%A2%F0%9D%90%AF%F0%9D%90%A2%F0%9D%90%A7%F0%9D%90%A0-%F0%9D%90%82%F0%9D%90%A8%F0%9D%90%A7%F0%9D%90%AD%F0%9D%90%9E%F0%9D%90%B1%F0%9D%90%AD-activity-7274037081000034304-tTY4?utm_source=share), etc.). 📝🌐
 - **POC Repositories**: Create and share repositories showcasing new technologies and quick demonstrations for easy exploration (e.g.,[lr4lg](https://github.com/locchh/lr4lg), [debugger](https://github.com/locchh/debugger), etc.). 📂
-- **Build Continuously Growing Repositories**: Develop and maintain long-term, evolving repositories for ongoing projects (e.g., [Synthflow](https://github.com/locchh/synthflow), [Antflow](https://github.com/locchh/antflow), [Antelligence](https://github.com/locchh/antelligence), etc.). 🔄
+- **Build Continuously Growing Repositories**: Develop and maintain long-term, evolving repositories for ongoing projects (e.g., [Synthflow](https://github.com/locchh/synthflow), [Antflow](https://github.com/locchh/antflow), [Antelligence](https://github.com/locchh/antelligence), etc.). 🔄

docs/instruction_tuning_best_practices.md

@@ -0,0 +1,43 @@
+
+# Best Practices for Instruction-Tuning Large Language Models
+
+Instruction-tuning is a powerful fine-tuning approach that adapts large language models (LLMs) to follow specific instructions more effectively, enhancing their usefulness in practical applications. Below, we outline best practices for optimizing instruction-tuning in LLMs.
+
+## Data Selection for Instruction-Tuning
+
+High-quality data is crucial for effective instruction-tuning. The selected data should reflect diverse instructions and responses to help the model generalize and respond accurately across varied scenarios.
+
+- **Diverse Dataset Collection**: Use datasets that cover a wide range of topics, contexts, and instructions. Including different prompt types and response styles helps the model handle a broader set of instructions.
+- **Balance of Specialized and General Data**: While it's beneficial to include domain-specific instructions, balancing this with general data improves versatility, allowing the model to perform well across various domains.
+
+## Optimize Prompt Engineering
+
+Effective prompt engineering enables the model to understand and respond appropriately to different instructions.
+
+- **Contextual Prompt Design**: Design prompts that reflect real-world use cases and specific contexts the model might encounter. For instance, instructions could vary in formality, complexity, or specificity, helping the model adapt to different audiences.
+- **Testing Prompt Variability**: Experiment with different prompts to assess how well the model generalizes to unseen instructions. This helps ensure that the model doesn't overly rely on specific patterns or structures.
+
+## Measure Response Consistency
+
+Consistency in response quality is key to creating a reliable model.
+
+- **Evaluate Accuracy and Consistency**: Regularly test the model with similar instructions to measure consistency. Consistent and accurate responses to repeated instructions indicate a well-tuned model.
+- **Monitor Task-Specific Performance**: If the model is tuned for a specialized application, evaluate its performance across task-specific scenarios to ensure consistency within that context.
+
+## Limit Overfitting on Instruction Style
+
+Overfitting on specific instruction styles or tones can reduce the model’s adaptability.
+
+- **Style Variety in Instructions**: Include a variety of tones and structures in the instruction dataset to avoid making the model too reliant on specific formats.
+- **Balance Precision and Flexibility**: Fine-tune the model to be precise in its responses without limiting its ability to adapt to different instruction types. This balance helps create a model that is accurate yet flexible in understanding various instructions.
+
+## Implement Regular Evaluation Metrics
+
+Regular evaluation of the fine-tuned model ensures it meets the desired quality standards.
+
+- **Use Metrics for Instruction Adherence**: Implement metrics that evaluate how closely the model's responses align with provided instructions.
+- **Human Review and Quality Checks**: Regular human review of model responses provides insights that are difficult to capture with automated metrics, adding another layer of evaluation for adherence and appropriateness.
+
+## Conclusion
+
+Following these best practices for instruction-tuning can significantly enhance an LLM's performance, enabling it to respond more accurately and flexibly to a wide array of instructions. By focusing on quality data, diverse prompt engineering, and regular evaluation, you can create an instruction-tuned model that is both effective and reliable in real-world applications.

docs/peft.md

@@ -0,0 +1,136 @@
+
+### **LoRA (Low-Rank Adaptation):**
+
+- **Core Idea:** 
+  LoRA introduces trainable low-rank matrices into the model, targeting specific weight matrices (e.g., attention layers). Instead of fine-tuning the full model, LoRA adds a low-rank decomposition to certain layers and fine-tunes only these low-rank matrices.
+
+- **Implementation:**
+  - Adds two small low-rank matrices \( A \) and \( B \) to the original weights \( W \) of the model, where \( \Delta W = AB^T \).
+  - The original pre-trained weights \( W \) remain frozen during training, and only \( A \) and \( B \) are updated.
+
+- **Advantages:**
+  - Memory efficient: Fewer parameters are updated.
+  - Doesn't require modifying the model architecture significantly.
+  - Easy to integrate into transformer-based models.
+
+- **Use Case:**
+  - Primarily used for fine-tuning language models (e.g., LLaMA, GPT).
+  - Common in NLP and generative tasks.
+
+---
+
+### **Adapters:**
+
+- **Core Idea:** 
+  Adapters are small trainable modules inserted into the layers of a pre-trained model. These modules are trained while keeping the original model weights frozen.
+
+- **Implementation:**
+  - Adapters are typically lightweight neural network modules (e.g., feedforward layers) inserted between layers of a model, such as attention or feedforward blocks.
+  - During training, only the parameters of the adapters are updated, while the rest of the model remains fixed.
+
+- **Advantages:**
+  - Modular: Different tasks can have separate adapters without modifying the original model.
+  - Memory efficient: Reduces the need to fine-tune the entire model.
+  - Easy task-switching by replacing adapters.
+
+- **Use Case:**
+  - Popular in multi-task learning and scenarios requiring task-specific fine-tuning.
+  - Useful in both NLP and multimodal applications.
+
+---
+
+### **Key Differences:**
+
+| Feature               | LoRA                          | Adapters                       |
+|-----------------------|-------------------------------|--------------------------------|
+| **Architecture**      | Adds low-rank matrices to weight updates. | Inserts small modules between layers. |
+| **Frozen Parameters** | Original model weights are frozen. | Original model weights are frozen. |
+| **Parameter Updates** | Updates low-rank matrices \( A, B \). | Updates adapter parameters only. |
+| **Overhead**          | Minimal: modifies specific weight matrices. | Moderate: introduces new modules into the model. |
+| **Use Cases**         | NLP fine-tuning, generative tasks. | Multi-task learning, task-specific adaptation. |
+
+---
+
+Both methods are highly efficient and suitable for scenarios where training large models directly is infeasible. However, the choice depends on your use case, such as the number of tasks, modularity requirements, and computational constraints.
+
+
+### **1. LoRA for Fine-Tuning**
+
+#### **Advantages:**
+
+1. **Efficiency:** 
+   - LoRA modifies only a small subset of trainable parameters (the low-rank matrices), making it highly parameter-efficient.
+2. **Minimal Overhead:** 
+   - No additional modules are inserted into the architecture; only the weight matrices of certain layers are modified.
+3. **Better for Single Tasks:** 
+   - Works well for scenarios where you want to fine-tune on a single or specific domain/task without modularity concerns.
+4. **Speed:** 
+   - Training and inference are faster compared to adapters due to the lightweight nature of LoRA’s modifications.
+
+#### **Use Cases:**
+- Single-task fine-tuning.
+- Resource-constrained environments (e.g., limited GPU memory).
+- Generative tasks such as text generation or summarization.
+
+#### **Limitations:**
+- Less modular: Difficult to manage multiple tasks or transfer fine-tuned components across models.
+- Potentially less effective when task-switching is required.
+
+---
+
+### **2. Adapters for Fine-Tuning**
+
+#### **Advantages:**
+
+1. **Modularity:** 
+   - Adapters are ideal for multi-task setups since each task can have its own adapter, enabling seamless task switching.
+2. **Transfer Learning:** 
+   - Adapters trained on one task can be reused or adapted for related tasks, making them versatile.
+3. **Isolation:** 
+   - Fine-tuning with adapters avoids interference between tasks, which is especially useful in multi-task or federated learning setups.
+
+#### **Use Cases:**
+- Multi-task learning or scenarios requiring task switching.
+- Incremental fine-tuning on new tasks/domains.
+- Applications where modularity and reusability of components are important.
+
+#### **Limitations:**
+- Higher computational overhead compared to LoRA due to added modules.
+- Slightly more complex integration into existing architectures.
+
+---
+
+### **Comparison Table:**
+
+| Feature                  | LoRA                           | Adapters                        |
+|--------------------------|---------------------------------|---------------------------------|
+| **Parameter Efficiency** | High                           | Moderate                       |
+| **Modularity**           | Low                            | High                           |
+| **Fine-Tuning Overhead** | Minimal                        | Moderate                       |
+| **Use Case**             | Single-task fine-tuning        | Multi-task or modular setups   |
+| **Scalability**          | Limited for multiple tasks     | Scalable for multi-task setups |
+| **Inference Efficiency** | Higher                         | Lower due to added modules     |
+
+---
+
+### **What’s Better?**
+
+- **LoRA** is generally better if:
+  - You are working on a single task or domain.
+  - You prioritize efficiency and minimal computational overhead.
+  - You have limited resources (e.g., GPU memory).
+
+- **Adapters** are better if:
+  - You need to fine-tune on multiple tasks or domains.
+  - Task modularity and reusability are important.
+  - You want a framework that supports incremental learning.
+
+---
+
+### **Best Practices:**
+- **Experimentation:** 
+   If resources allow, experiment with both methods to see which performs better for your specific task.
+- **Hybrid Approaches:** 
+   Recent work combines both methods for more efficient and effective fine-tuning.
+- **Task-Specific Considerations:** 
+   Consider the complexity of the task, the expected level of generalization, and memory constraints.