🌱 GreenFormer

A Multi-Architecture Model Compression Framework for Sustainable AI

GreenFormer is a generalized pipeline that reduces the energy consumption, carbon footprint, and inference latency of any Transformer-based NLP model through a sequential compression pipeline: Knowledge Distillation → Structured Pruning → INT8 Quantization (PTQ & QAT).

---

🎯 Key Features

• 🔄 Multi-Architecture Support — Works across BERT, RoBERTa, ALBERT, and ELECTRA via a pluggable Model Registry
• ⚡ 4-Stage Compression Pipeline — Knowledge Distillation → Structured Pruning → Post-Training Quantization → Quantization-Aware Training
• 📊 Real-Time Energy Tracking — CodeCarbon integration for measuring kWh and CO₂ emissions per training run
• 🖥️ Dual-Platform Benchmarking — Compare algorithmic vs. datacenter efficiency on Apple M2 CPU and NVIDIA T4 GPU
• 📈 Interactive Dashboard — Streamlit-based research visualization with comparative tables and charts
• 📄 Auto-Generated Reports — One-click Word document generation for academic submissions

---

🏗️ Architecture

                    ┌──────────────────────────────────────────┐
                    │         MODEL REGISTRY                   │
                    │  bert | roberta | albert | electra       │
                    └──────────────┬───────────────────────────┘
                                   │
                    ┌──────────────▼───────────────────────────┐
                    │       GENERALIZED PIPELINE               │
                    │                                          │
                    │  Stage 1: Baseline Teacher Training      │
                    │            ↓                             │
                    │  Stage 2: Knowledge Distillation (KD)    │
                    │            ↓                             │
                    │  Stage 3: Structured Pruning (30%)       │
                    │            ↓                             │
                    │  Stage 4a: Post-Training Quantization    │
                    │  Stage 4b: Quantization-Aware Training   │
                    │                                          │
                    └──────────────┬───────────────────────────┘
                                   │
                    ┌──────────────▼───────────────────────────┐
                    │         OUTPUT                           │
                    │  • Compressed INT8 model                 │
                    │  • Energy & CO₂ metrics (JSON)           │
                    │  • ONNX export for deployment            │
                    │  • Streamlit dashboard                   │
                    │  • Auto-generated DOCX report            │
                    └──────────────────────────────────────────┘

---

📊 Results Summary (SST-2, NVIDIA T4 GPU)

Stage	Model	Accuracy	Size (MB)	Speedup	Energy Reduction
Baseline	BERT-base (Teacher)	93.1%	418	1.0×	—
Knowledge Distillation	DistilBERT-6L (Student)	81.0%	255	1.9×	-36%
Structured Pruning	Pruned DistilBERT (30%)	81.4%	255	2.1×	-41%
INT8 QAT	Quantized DistilBERT	81.3%	91	4.7×	-68%

Key Finding: INT8 quantization performance is highly hardware-dependent — Apple M2 (CPU) shows latency increases due to lack of native INT8 support, while NVIDIA T4 (GPU) achieves significant speedups.

---

🚀 Quick Start

Prerequisites

# Clone the repository
git clone https://github.com/ATULSHARMA1234/GreenFormer.git
cd GreenFormer

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # macOS/Linux

# Install dependencies
pip install -r requirements.txt

Run the Original Pipeline (BERT only)

python run_fast_pipeline.py --task sst2

Run the Generalized Pipeline (Any Model)

# Single model
python run_generalized_pipeline.py --model bert --task sst2
python run_generalized_pipeline.py --model roberta --task cola
python run_generalized_pipeline.py --model albert --task sst2

# All models × all tasks
python run_generalized_pipeline.py --model all --task both

Launch the Dashboard

streamlit run dashboard/unified_paper_dashboard.py

Generate Research Report

python generate_report.py

---

📁 Project Structure

GreenFormer/
├── src/
│   ├── model_registry.py       # 🔌 Pluggable model family registry
│   ├── train_baseline.py       # Stage 1: Teacher training
│   ├── distillation.py         # Stage 2: Knowledge Distillation
│   ├── pruning.py              # Stage 3: Structured Pruning
│   ├── quantization.py         # Stage 4: PTQ & QAT
│   ├── evaluation.py           # Metrics & benchmarking
│   ├── energy_tracker.py       # CodeCarbon wrapper
│   ├── export_onnx.py          # ONNX model export
│   ├── evaluate_onnx.py        # ONNX Runtime benchmarking
│   └── utils.py                # Shared utilities
├── dashboard/
│   ├── unified_paper_dashboard.py  # Main Streamlit dashboard
│   ├── graph_dashboard.py          # Chart visualizations
│   └── tables_dashboard.py         # Comparative tables
├── run_generalized_pipeline.py # 🌱 Generalized multi-model pipeline
├── run_fast_pipeline.py        # Original BERT-specific pipeline
├── generate_report.py          # Auto-generate Word report
├── requirements.txt
└── README.md

---

🔌 Supported Model Families

Family	Teacher	Student	Parameters
BERT	bert-base-uncased (110M)	DistilBERT-6L (66M)	40% reduction
RoBERTa	roberta-base (125M)	DistilRoBERTa (82M)	34% reduction
ALBERT	albert-base-v2 (11.8M)	ALBERT-tiny 4L (~4M)	66% reduction
ELECTRA	electra-base (110M)	ELECTRA-small (14M)	87% reduction

Adding a new model requires only a new entry in src/model_registry.py — zero pipeline code changes.

---

🧪 Benchmarking Platforms

Platform	Role	Hardware
Apple MacBook Air M2	Algorithmic efficiency (CPU)	8-core CPU, 8GB RAM
Google Colab / RunPod	Datacenter efficiency (GPU)	NVIDIA Tesla T4, 16GB VRAM

---

📚 Techniques Implemented

Knowledge Distillation (Hinton et al., 2015)

L = α · T² · KL(student_soft ‖ teacher_soft) + (1 - α) · CE(student, labels)

• Temperature T = 4.0, Alpha α = 0.7

Structured Pruning

• L1-norm global unstructured pruning targeting all nn.Linear layers
• 30% sparsity with post-pruning fine-tuning for accuracy recovery

Quantization

• PTQ: Dynamic INT8 quantization via torch.quantization.quantize_dynamic
• QAT: Quantization-aware training with fake quantization nodes during forward pass

Energy Measurement

• CodeCarbon for real-time energy (kWh) and CO₂ (kg) tracking
• PUE scaling: E_total = E_IT × PUE (industry avg 1.58 → optimized 1.09)

---

🛠️ Tools & Technologies

Frameworks: PyTorch, HuggingFace Transformers, ONNX Runtime, Streamlit

Libraries: CodeCarbon, Scikit-learn, Plotly, Matplotlib, Pandas, NumPy, TensorBoard

Platforms: Python 3.10+, Google Colab (NVIDIA T4 GPU), Apple M2 CPU

Benchmarks: GLUE (SST-2 — Sentiment, CoLA — Linguistic Acceptability)

---

👨‍💻 Author

Atul Sharma , Sarthak Goyal , Panya Kapoor

---

📝 License

This project is licensed under the MIT License.

---

🌟 Citation

If you use GreenFormer in your research, please cite:

@software{greenformer2026,
  author = {Sharma, Atul},
  title = {GreenFormer: A Multi-Architecture Model Compression Framework for Sustainable AI},
  year = {2026},
  url = {https://github.com/ATULSHARMA1234/GreenFormer}
}