Cyber Shield — Intelligent Behavioral Threat Detection System

A dual-model ML-powered threat detection system with an integrated honeypot decoy website.

---

Architecture Overview

[Falco / Live Telemetry]
        |
        v
[FastAPI Inference Server] ──── [SHAP Explainer]
        |                             |
   [Host RF]   [Network RF]     [Dashboard]
        \           /
         [Meta-Classifier]
                |
         score > 0.85?
                |
         [nginx redirect]
                |
         [Decoy Portal] ──── [Redis Logger]
                                     |
                          [Retrain Worker] ──► [partial_fit]

---

Directory Structure

threat-detection/
├── data/
│   ├── adfa_ld/                 # Raw ADFA-LD dataset (download separately)
│   ├── cicids2017/              # CIC-IDS 2017 CSVs (download separately)
│   ├── adfa_features.csv        # Generated by preprocess_adfa.py
│   └── cicids_features.csv      # Generated by preprocess_cicids.py
├── ml/
│   ├── preprocess_adfa.py       # ADFA-LD → feature vectors
│   ├── preprocess_cicids.py     # CIC-IDS 2017 preprocessing
│   ├── train_host_model.py      # Host Random Forest
│   ├── train_network_model.py   # Network Random Forest
│   ├── train_meta_classifier.py # Ensemble meta-layer
│   ├── explain.py               # SHAP explainability module
│   ├── model_store.py           # Singleton model loader + predict()
│   └── retrain_worker.py        # Honeypot feedback retraining loop
├── models/                      # Generated after training (commit these!)
│   ├── host_rf.pkl
│   ├── host_scaler.pkl
│   ├── host_feature_cols.pkl
│   ├── net_rf.pkl
│   ├── net_scaler.pkl
│   ├── net_feature_cols.pkl
│   └── meta_clf.pkl
├── api/
│   └── main.py                  # FastAPI inference server
├── decoy/                       # Honeypot decoy website
│   ├── app.py
│   ├── logger.py
│   ├── .env                     # Honeypot bait file
│   └── templates/
└── requirements.txt

Dataset Downloads

Dataset	URL	Place in
ADFA-LD	https://research.unsw.edu.au/projects/adfa-ld	data/adfa_ld/
CIC-IDS 2017	https://www.unb.ca/cic/datasets/ids-2017.html	data/cicids2017/

Note: CIC-IDS 2017 requires registration.

Step	Command	Wait for
1	python ml/preprocess_adfa.py	data/adfa_features.csv created
2	python ml/preprocess_cicids.py	data/cicids_features.csv created
3	python ml/train_host_model.py	F1 ≥ 0.88 printed
4	python ml/train_network_model.py	F1 ≥ 0.88 printed
5	python ml/train_meta_classifier.py	models/meta_clf.pkl created
6	uvicorn api.main:app --host 0.0.0.0 --port 8000	API startup message
7	python ml/retrain_worker.py (background)	Worker polling message

Running the Decoy

cd decoy
pip install -r requirements.txt
flask run --host 0.0.0.0 --port 5001

Honeypot Trap Endpoints

Endpoint	Event Logged	Severity
GET /.env	credential_harvest	CRITICAL
GET /backup/	backup_access	HIGH
POST /api/v1/exec	rce_attempt	CRITICAL
POST /api/v1/upload	file_upload_attempt	HIGH
GET /api/v1/config	config_access	MEDIUM
GET /api/v1/users	api_enumeration	MEDIUM
/../etc/passwd	path_traversal	HIGH
POST /login	credential_attempt	HIGH

FastAPI Endpoints

Method	Path	Purpose
POST	/predict	Run full ensemble inference
GET	/alerts/history?limit=N	Fetch recent alerts
GET	/metrics	Honeypot and alert counts
POST	/feedback	Manual label + online retrain
GET	/honeypot/log?limit=N	Fetch honeypot events
WS	/alerts/live	WebSocket live alert stream

---

Model Performance

Model	Metric	Target
Host RF	F1 (attack class)	≥ 0.88
Network RF	F1 (attack class)	≥ 0.88
Meta-Classifier	ROC-AUC	≥ 0.92