Real-time Italian Sign Language (LIS — Lingua Italiana dei Segni) recognition system. Recognizes static letter signs and dynamic word signs from a webcam using MediaPipe keypoints and deep learning (Dense / CNN-LSTM / LSTM / Transformer), with an optional LLM post-processing layer for grammatical conversion into fluent Italian.
For over 70,000 deaf people in Italy who use LIS (Volterra, 2004), everyday communication with hearing people almost always requires a human interpreter. LISai is a research prototype exploring whether an accessible, locally-run pipeline can lower that barrier.
- Overview
- System Architecture
- Two Independent Subsystems
⚠️ Real-World Performance (WORDS model)- Project Structure
- Installation
- How to Reproduce
- Running the GUI
- Configuration
- Model Performance
- Optional Components
- Future Work
- License
The system has two fully independent recognition pipelines:
| LETTERS | WORDS | |
|---|---|---|
| Input | Static images .jpg |
Video clips .mp4 |
| MediaPipe | Hands (static, 1 hand) | Holistic (body + hands + face, 543 landmarks) |
| Features | 63 (21 kp × 3) | 309 per frame (sparse-sampled) |
| Sequence | single image | 80–90 frames (padded) |
| Model | Dense [256,128,64] | CNN-LSTM / LSTM / Transformer |
| Classes | 28 (26 LIS letters + del + space) |
10 (best documented configuration) |
| Test accuracy | ~87% |
88.67% (LSTM, 10 classes) |
| Environment | Local Python | Google Colab (A100 GPU) |
⚠️ Accuracy note (letters): the thesis documents ~87% test accuracy for the Dense/MLP letter model (Chapter 5.9). A separate figure of 98.6% has appeared in earlier project notes but is not yet corroborated by any evaluation report in this repository — if you have a more recent retraining that confirms it, update this table and remove this note; otherwise treat 87% as the validated number. Regardless of the exact figure, the letters model is reported to perform reliably in real, live use — unlike the words model (see the dedicated warning below).
A unified real-time GUI (src/gui/lis_interpreter_gui_v2_4_5.py) runs both modes and optionally connects to a local LLM (Ollama / LM Studio) or cloud API (Anthropic / OpenAI) to build grammatical context from recognized signs.
graph TD
Input[Webcam / Video] --> MP[MediaPipe: Hands or Holistic]
MP -->|Static hand pose| LM[Letter Model: Dense / MLP]
MP -->|Temporal landmark sequence| WM[Word Model: LSTM / CNN-LSTM / Transformer]
LM --> CE[Context Engine]
WM --> CE
CE --> LLM[Local LLM: Gemma 3 — grammatical post-processing]
LLM --> GUI[GUI Display: subtitle overlay]
The Letter and Word models never run the same architecture: letters are a single static frame (no temporal dimension), words are a sequence — this is why the pipelines and the neural architectures are kept fully separate rather than sharing a model.
data/letters/{class}/*.jpg
└─► 1_preprocessing_letters_LOCAL.py [local, Python]
└─► preprocessed keypoints (63 features)
└─► train_letters_unified.py OR 2_training_letters_COLAB.ipynb
└─► model.keras → GUI (Letters mode)
data/words/{class}/*.mp4
└─► [optional] Augmenter_Video_FFmpeg_v4.0.py (15×)
└─► 1_preprocessing_words_colab_A100_FINAL.ipynb [Colab]
└─► words_dataset_*.npz + label_encoder.pkl
└─► 2_training_words_colab_A100_COMPLETE.ipynb [Colab, A100]
└─► model.keras + confidence_thresholds.json
└─► GUI (Words mode)
The notebook 2_training_words_colab_A100_COMPLETE.ipynb contains separate cell groups for CNN-LSTM, LSTM, and Transformer — run only the cells for the desired architecture. LSTM is the recommended default (best accuracy, lowest overfitting — see Model Performance).
This note applies specifically to the WORDS (dynamic sign) model. The letters model is reported to perform reliably in live, real-world use — the static single-frame nature of fingerspelling makes it far less sensitive to the issues described below.
The word-recognition test accuracy (88.67%) and the cross-signer validation (95.65% on 23 curated videos from two external signers) were both measured under controlled conditions: cooperative signers performing isolated, known words, recorded specifically for evaluation.
In open, everyday use, real-world accuracy is substantially lower than these numbers suggest. This is a known and documented limitation, not a training bug, and it comes from two compounding causes:
-
Dataset limitations. The training set is small (~100 samples/word before augmentation) and comes almost entirely from a single primary signer. Augmentation was applied before the train/val/test split, so even the 88.67% same-signer test score is optimistic (documented data leakage). The model has limited exposure to how the same word looks when signed by someone else.
-
Inherent CV difficulty with signer variability. Sign language execution varies significantly from person to person — speed, amplitude, hand shape precision, and personal style all differ even for the "same" sign. A landmark-based sequence classifier trained on one signer's style has to generalize across all of this variation, which is a fundamentally harder problem than the curated test set reflects.
The practical consequence, observed during live deployment: only a minority of the trained classes are reliably recognized in casual use, while a few dominant classes tend to "capture" ambiguous input (class concentration collapse). There is also no dedicated "no-sign" class, so idle periods can be misclassified as a sign rather than rejected.
Bottom line: treat the reported metrics as an upper bound achieved under favorable, controlled conditions — not as a guarantee of live performance. See Future Work for the concrete steps (multi-signer dataset, idle-class rejection) that would close this gap.
LISai/
├── src/
│ ├── gui/ [CORE] GUI real-time interpreter
│ │ ├── lis_interpreter_gui_v2_4_5.py ← main app
│ │ ├── context_engine.py
│ │ ├── subtitle_overlay.py
│ │ └── tooltip_system.py
│ │
│ ├── processing/ [CORE] Preprocessing scripts
│ │ ├── 1_preprocessing_letters_LOCAL.py ← step 1 letters (local)
│ │ ├── 1_preprocessing_words_colab_A100_FINAL.ipynb ← step 1 words (Colab)
│ │ ├── preprocessing_unified.py ← shared preprocessing utilities
│ │ └── preprocessing_words_unified.py ← words preprocessing utilities
│ │
│ ├── training/ [CORE] Training scripts
│ │ ├── train_letters_unified.py ← letters training (local)
│ │ ├── 2_training_letters_COLAB.ipynb ← letters training (Colab)
│ │ ├── train_words_unified.py ← words training utilities
│ │ └── 2_training_words_colab_A100_COMPLETE.ipynb ← words training (Colab A100)
│ │
│ └── utils/ [OPTIONAL] Augmentation and analysis tools
│ ├── Augmenter_Video_FFmpeg_v4.0.py ← video augmentation 15× (FFmpeg)
│ └── lis_thesis_plots_styles.py ← report charts generator
│
├── configs/ [CORE] Configuration files
│ ├── lis_config_unified.py ← unified config (letters + words, switch via `mode`)
│ ├── lis_config_creator_gui_v2_8_0.py ← GUI to generate/edit the config
│ └── augmentation_config.json ← FFmpeg augmentation parameters
│
├── models/ Placeholder — model weights are NOT included
│ ├── letters_dense_256-128-64/MODEL_CARD.md
│ ├── words_cnn_lstm_256-128-64/MODEL_CARD.md
│ ├── words_lstm_256-128-64/MODEL_CARD.md
│ └── words_transformer_h8/MODEL_CARD.md
│
├── data/ Placeholder — dataset is NOT included
│ ├── letters/ README.md + .gitkeep
│ ├── words/ README.md + .gitkeep
│ └── README.md
│
├── results/ Light reports and charts
│ ├── class_distribution.png
│ ├── exclusion_report.txt
│ └── word_analysis.json
│
├── docs/ Technical documentation
│
├── .gitignore
├── requirements.txt
├── LICENSE
└── README.md
CORE = required to reproduce and run the project. OPTIONAL = accessory tools; not needed for basic usage.
git clone https://github.com/<YOUR_USERNAME>/LISai.git
cd LISai
python -m venv .venv
# Windows:
.venv\Scripts\activate
# Linux/macOS:
source .venv/bin/activate
pip install -r requirements.txtFFmpeg (required only for video augmentation — src/utils/):
# Windows: winget install FFmpeg
# Linux: sudo apt install ffmpeg
# macOS: brew install ffmpegEdit configs/lis_config_unified.py and set:
mode = "letters"ormode = "words"dataset_pathpointing to your local/Drive data foldermodels_output_dirfor saving trained weights
Or use the GUI config creator: python configs/lis_config_creator_gui_v2_8_0.py
Input: .jpg images organized as data/letters/{class}/
Step 1 — Preprocessing (local machine)
python src/processing/1_preprocessing_letters_LOCAL.py \
--dataset data/letters/ \
--output <preprocessed_output_dir>Produces a .npz file with shape (N, 63).
Step 2 — Training
# Option A: local Python
python src/training/train_letters_unified.py
# Option B: Google Colab
# Upload src/training/2_training_letters_COLAB.ipynb to Colab and runProduces model.keras (~774 KB).
Step 3 — GUI
python src/gui/lis_interpreter_gui_v2_4_5.pySelect Letters mode and point to model.keras.
Input: .mp4 video clips organized as data/words/{class}/
Step 1 (optional) — Video augmentation (local, requires FFmpeg)
python src/utils/Augmenter_Video_FFmpeg_v4.0.py \
--input data/words/ \
--output data/words_augmented/ \
--factor 15Step 2 — Preprocessing (Google Colab A100)
Upload src/processing/1_preprocessing_words_colab_A100_FINAL.ipynb to Colab.
Set <YOUR_GDRIVE_PATH> at the top of the notebook to your Google Drive folder.
Run all cells. Produces words_dataset_*.npz + label_encoder.pkl.
Step 3 — Training (Google Colab A100)
Upload src/training/2_training_words_colab_A100_COMPLETE.ipynb to Colab.
The notebook contains separate cell groups for each architecture:
- LSTM (recommended — best accuracy, lowest overfitting): run cells under
# === LSTM === - CNN-LSTM: run cells under
# === CNN-LSTM === - Transformer: run cells under
# === Transformer ===
Produces model.keras + confidence_thresholds.json + label_encoder.pkl.
Step 4 — GUI
python src/gui/lis_interpreter_gui_v2_4_5.pySelect Words mode and point to model.keras + label_encoder.pkl.
python src/gui/lis_interpreter_gui_v2_4_5.pyThe GUI (CustomTkinter, v2.4.5) supports:
- Dual mode: Letters and Words can run simultaneously
- Auto model discovery: scans
models/for.kerasfiles - Confidence thresholds: per-class, auto-calibrated
- LLM integration (optional): Ollama (
localhost:11434), LM Studio (127.0.0.1:1234), Anthropic API, OpenAI API - Subtitle overlay: floating transcript window
For LLM usage, set environment variables:
export ANTHROPIC_API_KEY="sk-..."
export OPENAI_API_KEY="sk-..."All parameters (paths, architecture, classes, features) are centralized in one file:
configs/lis_config_unified.py
Switch subsystem with:
mode: Literal["letters", "words"] = "words"The file contains both letters and words sections — do not split them.
Use configs/lis_config_creator_gui_v2_8_0.py (GUI) to generate or edit it interactively.
| Component | Original samples | Post-augmentation | Classes | Format |
|---|---|---|---|---|
| LIS Alphabet (letters) | ~4,000/letter | ~112,000 total (not augmented) | 26 | Images (JPG) |
| LIS Vocabulary (words) | ~100/word | ~1,000 total (15×) | 10 | Video (MP4) |
| Metric | Value |
|---|---|
| Classes | 28 (26 LIS letters + del + space) |
| Test accuracy | ~87% |
| Input | 63 features (21 kp × 3) |
| Model size | ~774 KB |
| Metric | Value |
|---|---|
| Classes | 10 |
| Test accuracy | 88.67% |
| Cross-signer accuracy | 95.65% (22/23, controlled/curated videos) |
| F1 macro | 88.83% |
| Overfitting gap | 9.95% |
| Input | (80, 309) — 80 frames × 309 features |
| Training | Colab A100/T4, AdamW, early stopping |
See
⚠️ Real-World Performance (WORDS model) — these numbers were measured under controlled conditions and do not reflect open, everyday use. The letters model above is not affected by this gap.
Applies 15× augmentation to video clips using FFmpeg (speed variation, flip, brightness, noise…). Requires FFmpeg installed on the system.
Generates training curves, confusion matrices, per-class metrics.
Requires matplotlib and seaborn.
- Multi-signer dataset expansion — the single most impactful fix for the real-world performance gap described above.
- Dedicated "no-sign" / idle class — to stop the model from classifying idle periods as a sign.
- Split before augmentation — to remove the same-signer/same-session data leakage documented in the current test accuracy.
- Mobile deployment — convert the recognition model to TensorFlow Lite (MediaPipe Holistic already ships a mobile SDK); the LLM constraint is the harder part, but Gemma 3n (Google DeepMind's on-device variant of Gemma 3, designed to run on under 2 GB of RAM) is the most realistic path to keeping grammatical post-processing on-device. (This is a proposed direction, not yet implemented.)
- Continuous sign recognition — moving from isolated-sign classification to sentence-level, boundary-aware recognition.
- SignGemma integration — Google DeepMind's SignGemma (2025) is a multimodal model processing video directly, currently in limited developer preview for ASL. If LIS support is added to its roadmap, fine-tuning it (e.g. via LoRA) on a custom LIS dataset could outperform the current LSTM-based approach.
MIT — see LICENSE.
Made with ❤️ for the Italian LIS community.