Automated YARA Rule Generation using LLMs

Final Project for Introduction to Malware Analysis

Students: KABLY Malak - MANSOUB Yahya
Instructor: LEKSSAYS Ahmed

Repository: https://github.com/YahyaMansoub/Automated_YARA_Rule_Generation_using_LLMs.git

Date: March 2026
Status: Complete with APIARY Comparison & Visualizations

---

Project Overview

This project implements LLM-based YARA rule generation using both static and dynamic malware analysis approaches:

Component	Approach	Technology	Status
Static Pipeline	API-only features from PE imports	Groq LLM + Voting Classifier	Complete
Dynamic Pipeline	Behavioral features from CAPE sandbox	ML models + LLM refinement	Complete

Key Results:

• Static pipeline: 76.0% accuracy on EMBER dataset
• 4 publication-quality comparison figures vs APIARY paper
• Both pipelines generate human-readable YARA rules

---

Pipelines

Static Analysis Pipeline

Location: src/staticdatagen/yara_rule_generation_pipeline.ipynb

Approach: Extracts API imports from PE headers, generates YARA rules using LLM (Groq), evaluates with voting-based classification.

Paper Reference: APIARY: An API-based automatic rule generator for YARA to enhance malware detection

Documentation: See src/staticdatagen/README.md for detailed pipeline walkthrough.

---

Dynamic Analysis Pipeline

Location: src/dynamicdatagen/

Approach: Loads behavioral reports from the CAPE sandbox, extracts features from API calls, registry modifications, and network activity, and then uses a machine learning model to identify key family-specific behaviors. An LLM refines these behaviors into a human-readable YARA rule. This approach focuses on capturing the actual behavior of malware, making the rules robust against packing and obfuscation.

Feature Engineering: The core of this pipeline is its feature engineering process. It parses detailed JSON reports from the CAPE sandbox to extract sequences of actions, such as:

• API Calls: Critical functions called by the malware.
• Registry Keys: Keys created, modified, or deleted.
• Network Traffic: Domains contacted and protocols used. These features provide a rich, behavioral fingerprint of the malware.

How to Use: The dynamic pipeline is a two-stage process: first, you train the ML models, and then you use them to generate YARA rules.

1. Train the Models:
   • Navigate to src/dynamicdatagen/feature_extraction/notebooks/training/.
   • Open training_pipeline.ipynb and run the cells to train the classifiers.
2. Generate YARA Rules:
   • Navigate to src/dynamicdatagen/feature_extraction/notebooks/project/.
   • Open 02_dynamic_to_yara_generation.ipynb and run the cells to generate rules using the trained models.

Documentation: See src/dynamicdatagen/README.md for a detailed pipeline walkthrough.

---

Project Structure

Automated_YARA_Rule_Generation_using_LLMs/
|
+-- README.md                              (Main project guide)
+-- docker-compose.yml                     (Docker orchestration)
+-- Dockerfile                             (Container definition)
+-- .env.example                           (Configuration template)
|
+-- data/                                  (Input datasets)
|   +-- raw/
|       +-- public_labels.csv
|
+-- src/                                   (Source code)
|   |
|   +-- staticdatagen/                     (STATIC ANALYSIS PIPELINE)
|   |   +-- yara_rule_generation_pipeline.ipynb  (Main notebook)
|   |   +-- README.md                      (Detailed guide)
|   |   +-- analyzer_walkthrough.ipynb     (PE analysis tutorial)
|   |   +-- code/                          (PE analysis library)
|   |   +-- results/                       (Generated outputs)
|   |
|   +-- dynamicdatagen/                    (DYNAMIC ANALYSIS PIPELINE)
|   |   +-- feature extraction/
|   |   |   +-- notebooks/
|   |   |   |   +-- training_pipeline.ipynb     (Train on CAPE)
|   |   |   |   +-- inference_pipeline.ipynb    (Inference)
|   |   |   +-- outputs/                        (Rules & models)
|   |   +-- scripts/
|   |       +-- generate_yara_from_hash.py     (Single sample)
|   |
|   +-- LLMs/                              (LLM integration)
|   +-- utils/                             (Utility functions)
|
+-- outputs/                               (Generated artifacts)
|   +-- yara_pipeline/                     (Rules, logs, figures)
|
+-- research/                              (Reference materials)
    +-- Resources.md                       (Papers & references)
    +-- What's a YARA Rule.md              (YARA primer)

---

Setup Guide for Professors

Step 1: Install Docker

Windows/Mac:

1. Visit https://www.docker.com/products/docker-desktop
2. Download Docker Desktop for your OS
3. Install and follow the setup wizard
4. Verify installation: Open terminal and run:

   docker --version

Linux:

sudo apt-get install docker.io docker-compose

Step 2: Get GROQ API Key (Free - Required for Static Pipeline)

1. Visit https://console.groq.com
2. Sign up with email (no credit card required)
3. Navigate to "API Keys" section
4. Click "Create New API Key"
5. Copy the key starting with gsk_
6. Save it for later use

Note: Dynamic pipeline does NOT require an API key.

Step 3: Clone Repository

git clone https://github.com/YahyaMansoub/Automated_YARA_Rule_Generation_using_LLMs.git
cd Automated_YARA_Rule_Generation_using_LLMs

Step 4: Setup Configuration

cp .env.example .env

Edit .env with your text editor (Notepad, VSCode, nano, etc.):

GROQ_API_KEY=gsk_<your_actual_key_from_step_2>

Note: Only required if running static pipeline.

Step 5: Download Datasets

For Static Pipeline (EMBER Dataset):

• Auto-downloads on first run in cell 3
• Size: ~300MB for 600 samples
• Source: https://www.unb.ca/cic/datasets/ember-dataset.html (optional manual download)
• Time: 2-5 minutes on first run

For Dynamic Pipeline (CAPE Reports):

• This project uses a pre-processed subset of the CAPE v2 Dataset. You can find the full dataset at https://github.com/avast/avast-ctu-cape-dataset.
• The dataset comes in two versions: a "mini" set with a few hundred samples and a "large" set with over 400,000. For this project, we use a small, pre-processed portion of the "mini" dataset.
• The required reports are already included in the repository at src/dynamicdatagen/feature_extraction/data/.
• No additional download is needed to run the main pipeline.
• These reports are JSON files generated by the CAPE sandbox, detailing the behavior of malware samples.
• Size: ~50MB

Optional Manual Dataset Download:

If you want to manually download before running:

1. EMBER Dataset:

   • Download from: https://www.unb.ca/cic/datasets/ember-dataset.html
   • After download, extract files to: src/staticdatagen/ember/
   • Directory structure should look like:

     src/staticdatagen/ember/
     ├── ember_model_2017.txt
     ├── test_features.jsonl
     ├── train_features_0.jsonl
     ├── train_features_1.jsonl
     ├── train_features_2.jsonl
     ├── train_features_3.jsonl
     ├── train_features_4.jsonl
     ├── train_features_5.jsonl
     ├── goodware_samples/
     └── malware_samples/
     

   • Note: Files will be placed in the Docker container automatically during setup

2. CAPE Sandbox Reports:

   • Included in repo at: src/dynamicdatagen/feature extraction/data/
   • No action needed - already in correct location

Step 6: Build Docker Environment

docker-compose build

This process:

• Downloads Python 3.11 slim base image (~100MB)
• Installs all Python dependencies (~1.5GB total)
• Creates Docker image (~2GB)
• Takes 5-15 minutes on first run

Success indicator:

Successfully built <image-id>
Successfully tagged automated_yara_llm_pipeline:latest

Step 7: Start Jupyter Server

docker-compose up

Wait for this output:

Jupyter starts at http://127.0.0.1:8888
Copy and paste URL with token in your browser:
http://127.0.0.1:8888/?token=<your-token>

Step 8: Open in Browser

Copy and paste the URL from Step 7 into your browser. You'll see the Jupyter file browser.

Step 9: Run Your Pipeline

For Static Analysis Pipeline:

1. Navigate to: src/staticdatagen/
2. Click: yara_rule_generation_pipeline.ipynb
3. Run cells sequentially using Shift+Enter or the Run button

For Dynamic Analysis Pipeline:

1. Navigate to: src/dynamicdatagen/feature extraction/notebooks/
2. Click: training_pipeline.ipynb
3. Run cells sequentially using Shift+Enter or the Run button

---

Documentation Files

Pipeline-Specific Guides:

• src/staticdatagen/README.md - Static pipeline detailed walkthrough
• src/dynamicdatagen/feature_extraction/README.md - Dynamic pipeline detailed walkthrough

Reference Materials:

• research/Resources.md - Papers and academic references
• [research/What's a YARA Rule.md](research/What's a YARA Rule.md) - Introduction to YARA rules

In Notebooks:

• Cell markdown sections explain each pipeline phase
• Code comments explain key logic
• Output cells display expected results

---

Key Concepts

APIARY Paper:

• Automated rule generation using API-only features
• Achieves 89.6% average accuracy across multiple datasets
• Combines algorithmic scoring with C-based YARA rule synthesis
• This project recreates parts of APIARY using LLMs instead of algorithmic scoring

Voting-Based Classification:

• Rules are separated by their source (malware vs goodware training data)
• For each test sample: count how many malware rules match vs goodware rules
• Prediction = which vote wins (majority vote classifier)
• Improves accuracy by leveraging semantic information about rule source

Why LLMs for YARA Generation?

• Generate interpretable, human-readable rules
• Quick adaptation to different feature sets
• Few-shot learning with example-based prompting
• Multiple prompt techniques tested to find optimal style

---

Repository Information

GitHub: https://github.com/YahyaMansoub/Automated_YARA_Rule_Generation_using_LLMs.git

Project Initialization:

git clone https://github.com/YahyaMansoub/Automated_YARA_Rule_Generation_using_LLMs.git

---

Citation

APIARY Paper Reference: [See research/Resources.md for full citations]

This Project:

Automated YARA Rule Generation using LLMs
Kably Malak and Mansoub Yahya
Introduction to Malware Analysis Course, 2026
College of Computing - UM6P