ViT Transfer Baseline

[lukhsaankumar]

Issue Type

• Model: ML model bug, training issue, or architecture problem
• Data: Dataset issue, preprocessing bug, or data pipeline problem
• Web: Frontend bug or UI issue in the Next.js dashboard
• API: Backend API bug or FastAPI endpoint issue
• Research: Research question or experimental feature request
• Documentation: Documentation bug or improvement needed
• Bug: General bug fix needed
• Enhancement: New feature or improvement request

Description

• Use a pretrained Vision Transformer (ViT) and adapt it to our preprocessing pipeline (vit tensor from preprocess_pipeline.py). Test both feature-extraction (frozen backbone) and fine-tuning (unfrozen) modes for binary classification.

Deliverable

• Notebook: notebooks/vit_transfer_baseline.ipynb
• Includes model modification, training results, and comparison of frozen vs fine-tuned performance

Additional Context

Pretrained Model Options:

• ViT-Base Patch16 224 (ImageNet-21k pretrained)

from timm import create_model
model = create_model("vit_base_patch16_224.orig_in21k", pretrained=True)

• DeiT-Small Patch16 224 (lighter variant fine-tuned for smaller datasets)

model = create_model("deit_small_patch16_224", pretrained=True)