Can AI-text detectors survive a change of decoding knobs or a paraphrase attack? This project pits Binoculars (zero-shot, cross-perplexity) against supervised detectors (TF-IDF + LR, Kim CNN, RoBERTa) on matched human/LLM text pairs, then shifts the generation distribution and watches what breaks.
Georgia Tech graduate NLP team project.
Supervised detectors look unbeatable when the machine text at test time comes from the same generator settings they trained on. The interesting question is what happens under distribution shift:
- Decoding shifts: temperature sweeps (0.1 to 1.5), top-p sweeps (0.8 to 0.99), greedy, beam search, top-k
- Paraphrase attacks: back-translation through German (MarianMT), applied to the machine side only (an evasion attack) or to both sides
- Held-out generators: machine text from models never seen in training
Every experiment reads from one master table: 9,458 human texts from TuringBench, each paired with a Llama-2-7b-chat continuation of the same 50-token prompt. Splits are made at the prompt level, so a prompt and all of its variants live in exactly one of train/val/test and nothing leaks.
flowchart LR
TB[(TuringBench<br/>9,458 human texts)] --> MT[master table<br/>id, prompt, human_text]
MT --> GEN[Llama-2-7b-chat<br/>on GT PACE GPUs]
GEN --> MD[machine_default<br/>T=0.7, top-p=0.95]
GEN --> KN[decoding variants<br/>T and top-p sweeps,<br/>greedy, beam, top-k]
MT --> BT[MarianMT<br/>back-translation]
MD --> SPL[prompt-level splits<br/>60 / 20 / 20]
SPL --> SUP[TF-IDF + LR<br/>Kim CNN, RoBERTa]
KN & BT --> TEST[shifted test sets]
SUP --> EV[AUROC, F1,<br/>TPR at low FPR]
TEST --> EV
MD & KN & BT --> BINO[Binoculars<br/>Falcon-7B pair]
BINO --> EV
The base dataset (data/master_table.jsonl, all 9,458 human/machine pairs) is bundled in this repo, so the supervised baselines run out of the box on a laptop CPU. Only variant generation and Binoculars scoring need a GPU.
git clone https://github.com/abhi25072002/NLP-Project.git
cd NLP-Project
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
# 1. Build train/val/test splits from the bundled master table (seconds)
python scripts/create_supervised_datasets.py
# 2. Train and evaluate the TF-IDF + LR baseline (about 15 s on CPU)
python scripts/train_tfidf.py
python scripts/eval_tfidf.py --model_path checkpoints/tfidf_lr/model.joblib --data_dir data/supervised/base
# 3. Train and evaluate the Kim CNN (about 10 min on CPU)
python scripts/train_cnn.py
python scripts/eval_cnn.py --model_path checkpoints/cnn/best_model.pt --data_dir data/supervised/base
# RoBERTa fine-tuning (GPU recommended)
python scripts/train_roberta.pySupervised detectors score near-perfect in-distribution; that ceiling is the point. The shifted test sets (data/supervised/knobs/, data/supervised/backtrans/) exist to measure how far each detector falls from it, and how Binoculars, which never trains at all, compares.
Zero-shot detection with the Binoculars method: score = observer perplexity / cross-perplexity, computed with the Falcon-7B / Falcon-7B-Instruct pair. The runner evaluates every machine_* column of the master table against the human side:
python run_zero_shot_master_table.py \
--data_path data/master_table.jsonl \
--output_path results/full_standard.csv--compute_token_metrics adds KL and Jensen-Shannon divergence per token; merge_results.py aggregates multiple runs into a single Markdown report. See EXPERIMENTS.md for all scenarios, flags, and output formats.
The bundled master table was generated on Georgia Tech's PACE cluster. To regenerate or extend it:
# Requires a HuggingFace token with access to meta-llama/Llama-2-7b-hf
python scripts/prepare_local_turingbench.py # consolidate TuringBench human texts
python scripts/generate_default_dataset.py # 9,458 baseline continuations
python scripts/generate_variants.py --mode all # temperature/top-p/decoding variants
python scripts/backtranslate.py # MarianMT paraphrase attack sets
python scripts/create_supervised_datasets.py # final train/val/test JSONL filesOn a single PACE GPU, one decoding variant over a 148-prompt subset takes about 38 minutes; the full 9,458-prompt baseline is an overnight job.
PACE / HPC environment notes (cache paths, gated models)
# Point HuggingFace caches at scratch to avoid home-dir quota errors
export HF_HOME=~/scratch/NLP-Project/hf/hf_home
export TRANSFORMERS_CACHE=~/scratch/NLP-Project/hf/hf_cache
export HF_DATASETS_CACHE=~/scratch/NLP-Project/hf/data_cache
export XDG_CACHE_HOME=~/scratch/NLP-Project/hf/hf_cache
mkdir -p $HF_HOME $TRANSFORMERS_CACHE $HF_DATASETS_CACHE $XDG_CACHE_HOME
# Llama-2 is gated: accept the license on its model page, then
export HF_TOKEN="your_hf_token_here" # or: huggingface-cli loginNLP-Project/
├── data/
│ ├── master_table.jsonl # 9,458 human/Llama-2 pairs (bundled)
│ └── human_turingbench.csv # consolidated TuringBench human texts
├── configs/ # generation, model, eval, data YAMLs
├── scripts/
│ ├── generate_default_dataset.py, generate_variants.py, backtranslate.py
│ ├── create_supervised_datasets.py
│ └── train_/eval_ x {tfidf, cnn, roberta}
├── src/
│ ├── binoculars/ # detector, PPL/X-PPL/KL/JSD metrics
│ ├── models/ # TF-IDF+LR, Kim CNN, RoBERTa
│ ├── generation/ # Llama-2 variant generator, MarianMT back-translation
│ └── data/ # master-table builder, prompt-level splitter
├── run_zero_shot_master_table.py # Binoculars runner (all machine_* columns)
├── merge_results.py # aggregate runs into one report
└── EXPERIMENTS.md # Binoculars experiment cookbook
Full generated artifacts (variant datasets, checkpoints) live on Drive: datasets · checkpoints.
- Abhishek Dharmadhikari
- Neel Shah
- Vishrut Goel
- Aditya Pandit
- Hans et al., Spotting LLMs with Binoculars: Zero-Shot Detection of Machine-Generated Text (ICML 2024)
- Uchendu et al., TuringBench: A Benchmark Environment for Turing Test in the Age of Neural Text Generation (EMNLP Findings 2021)