Font-Robust Character Recognition with Multi-Task Learning

Aurelie Mei Yin Ng Yin Hing 101246374

This project explores Optical Character Recognition (OCR) using deep learning, focusing on whether multi-task learning (character + font classification) improves robustness to unseen fonts.

A convolutional neural network based on a pretrained ResNet18 backbone is trained to recognize alphanumeric characters across multiple font styles. The system compares a baseline model (character-only training) against a multi-task model that also predicts font identity.

Objectives

• Build a CNN-based OCR system for 36 classes (A–Z, 0–9)
• Investigate whether font supervision improves character recognition
• Evaluate generalization to unseen fonts
• Analyze representation learning under multi-task learning

Project Structure

project/
├── main.py                  # Main script: dataset generation, training, evaluation, plots
├── fonts/                   # TrueType/OpenType font files
├── dataset/
│   └── images/              # Generated character images
├── labels.csv               # Dataset manifest
├── results/                 # Output figures and saved models
│   ├── loss_curves.png
│   ├── Baseline_Seen Fonts.png
│   ├── Baseline_Unseen Fonts.png
│   ├── Multitask_Seen Fonts.png
│   ├── Multitask_Unseen Fonts.png
│   ├── per_character_accuracy_seen.png
│   ├── per_character_accuracy_unseen.png
│   ├── baseline.pth
│   └── multitask.pth
└── README.md

Requirements

pip install torch torchvision pillow numpy pandas scikit-learn matplotlib

Python 3.8+ is recommended.

Setup

1. Add Fonts

Download TrueType (.ttf) or OpenType (.otf) fonts and place them in the fonts/ directory.

Update the FONTS_DIR path in main.py to point to fonts folder.

2. Generate Dataset

The dataset is generated automatically on first run if labels.csv does not exist. To regenerate (e.g. after changing font size or augmentation settings), delete dataset/ and labels.csv before running:

rm -rf dataset/ labels.csv

Running the Project

python main.py

This will:

1. Generate the dataset (if not already generated)
2. Split fonts 80/20 into seen/unseen sets
3. Train the baseline CNN
4. Train the multi-task CNN
5. Evaluate both models on seen and unseen font test sets
6. Save all plots and model weights to results/