**updated (3/4/2024)** Our paper and repo are updated: DS-1000 benchmark is included, a new baseline (EXP-edit) is included for reproducing the main results. Experiments of using surrogate model, variable renaming, and detectability@T will be added.
Official repository of the paper:
"Who Wrote this Code? Watermarking for Code Generation" by Taehyun Lee*, Seokhee Hong*, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin', Gunhee Kim'
We conducted our (main) experiments by separating them into generation and detection phases. However, anyone wanting to run both phases with a single command removes the --generation_only
argument.
For EXP-edit with a high entropy setting, please set top_p=1.0
and temperature=1.0
.
bash scripts/main/run_{MODEL}_generation.sh
bash scripts/main/run_{MODEL}_detection.sh
bash scripts/main/run_{MODEL}_detection_human.sh
With both metric output files from machine-generated and human-written codes, we calculate metrics including AUROC and TPR and update the results to OUTPUT_DIRECTORY
.
python calculate_auroc_tpr.py \
--task {humaneval,mbpp} \
--human_fname OUTPUT_DIRECTORY_HUMAN \
--machine_fname OUTPUT_DIRECTORY
This repository is based on the codes of bigcode-evaluation-harness in BigCode Project.
If you have any questions about our codes, feel free to ask us: Taehyun Lee ([email protected]) or Seokhee Hong ([email protected])