Soft Voting Ensembled Malware Image Classification

A three-track feature engineering and ensemble learning framework for malware family classification using image-based representations of binary executables.

Author: Sushant Rakesh Lokhande

Overview

Malware binaries are converted into grayscale images using eight conversion strategies, then three complementary feature representations are extracted from each image:

1. Handcrafted descriptors — Histogram of Oriented Gradients, Haralick texture features, and 38 statistical features (ADV38), fused and reduced via two-stage Random Forest selection
2. Pretrained CNN embeddings — frozen VGG16, ResNet50, and ViT-B/16 used as feature extractors
3. Custom CNN embeddings — a domain-specific CNN trained from scratch on malware images, producing 512-dim embeddings

The top 5 configurations from each track (15 voters total) are combined in a soft voting ensemble.

Headline Result

Method	Best configuration	Test accuracy
Handcrafted	hilbert_byteclass + RF (83 features)	0.7794
Custom CNN	grayscale + RF (512 dims)	0.7518
Pretrained CNN	VGG16 / grayscale + RF (4,096 dims)	0.7394
Soft voting ensemble (15 voters)	All three tracks	0.9424

Evaluated on 1,700 held-out test samples across 17 malware families.

Repository Structure

common/                    Stratified train/val/test split (run once)
track1_handcrafted/        HOG + Haralick + ADV38 extraction, RFE, classifiers
track2_pretrained_cnn/     VGG16, ResNet50, ViT embedding extraction and 144 classifier experiments
track3_custom_cnn/         MalwareCNN architecture, training, and 40 classifier experiments
ensemble/                  15-voter soft voting and pool ablations
data/                      Splits and extracted features (gitignored — regenerated by notebooks)
models/                    Trained checkpoints and selected feature indices
results/                   Tables and figures

Each track directory contains numbered notebooks intended to run in order (01_*.ipynb first, then 02_*.ipynb, etc.).

Reproduction

The dataset of malware images is from Agrawal et al. (ICISSP 2026, arXiv:2509.10838): 17 families × 1,000 images per family × 8 conversion types = 136,000 images at 224×224.

After obtaining the raw dataset and placing it under data/raw/<conversion>/<family>/, the pipeline runs in this order:

1. common/01_split_dataset.ipynb — produces the 80/10/10 stratified split
2. track1_handcrafted/01_* through 04_* — handcrafted features and reduction
3. track2_pretrained_cnn/01_* and 02_* — CNN embedding extraction and classifier training
4. track3_custom_cnn/01_* and 02_* — custom CNN training and classifier training
5. ensemble/01_soft_voting.ipynb — the final soft voting ensemble

Several notebooks were originally run on Google Colab (paths inside such notebooks point to /content/drive/...); the first markdown cell of each Colab-run notebook flags this. All produced outputs (CSVs, figures, selected feature JSONs) are committed to the repo. The intermediate features and trained models are not — they're reproducible by re-running the relevant notebooks.

License

MIT License — see LICENSE.