Issue with Llama-3-8b and Llama-3.1-8b-Instruct

[NamburiSrinath]

Hi @asillycat

Llama-3-8B (ASR-GPT, ASR KW)

• GCG - 1, 4 (vs reported 42, 50)
• AttnGCG - 1, 15 (vs reported 45, 51)

Llama-3.1-8B-Instruct (ASR-GPT, ASR-KW)

• GCG - 4, 64 (vs reported 32, 40)
• AttnGCG - 10, 85 (vs reported 34, 45)

I am unable to reproduce any of the values from the table.

Here, in this line https://github.com/UCSC-VLAA/AttnGCG-attack/blob/main/AttnGCG/base/attack_manager.py#L190, I changed it to
elif self.conv_template.name == 'llama-3' or self.conv_template.name == "meta-llama-3.1":

Llama-3-8B (this was provided in configs/)

import os

os.sys.path.append("..")
from configs.template import get_config as default_config

def get_config():
    
    config = default_config()

    config.tokenizer_paths=["meta-llama/Meta-Llama-3-8B-Instruct"]
    config.model_paths=["meta-llama/Meta-Llama-3-8B-Instruct"]
    config.conversation_templates=['llama-3-'] ### Make a note of **-**
    config.control_init = "a a a a a a a a a a a a a a a a a a a a"
    
    config.attention_weight = 50.0

    return config

Llama-3.1-8B (this was not provided in configs/, replicated from Llama-3-8B)

import os

os.sys.path.append("..")
from configs.template import get_config as default_config

def get_config():
    
    config = default_config()

    config.tokenizer_paths=["meta-llama/Llama-3.1-8B-Instruct"] ## Changed to 3.1
    config.model_paths=["meta-llama/Llama-3.1-8B-Instruct"] ## Changed to 3.1
    config.conversation_templates=['llama-3.1']
    config.control_init = "a a a a a a a a a a a a a a a a a a a a"
    
    config.attention_weight = 50.0

    return config

[NamburiSrinath]

Hi @asillycat ,

Is it possible to share the config script and necessary changes you made to run for Llama-3.1 models? The current repo doesn't have these scripts and changes.

Really appreciate your time and help, looking forward.

[NamburiSrinath]

Hi @asillycat

I am following up to see if you can provide the configs/scripts for Llama-3.1-8B for reproducability. Because the control init or other hyperparameters might be different for different models. (P.S: I am still unable to figure out why the results for Llama-3-8B are not reproducible even though the current repo has all the details for this model, so any thoughts here is also much appreciated.)

I can see from the logs that the above scripts didnot produce any errors, so I believe the mismatch in numbers is due to some changes in settings. Also, I ran it multiple times just to make sure.

Here're few details that might help to debug further in the meantime.

From experiments/results_individual_direct_gcg_llama31_8b_it folder, few lines

...
"test_goals": [],
"test_targets": [],
"control_init": "a a a a a a a a a a a a a a a a a a a a",
...
 "models": [
     {
         "model_path": "meta-llama/Llama-3.1-8B-Instruct",
         "tokenizer_path": "meta-llama/Llama-3.1-8B-Instruct",
         "conv_template": "meta-llama-3.1"
     }
       "test_models": [],
       "n_steps": 500,
       "test_steps": 20,
       "batch_size": 256,
       "topk": 128,
       "temp": 1,
       "allow_non_ascii": false,
       "target_weight": 1.0,
       "control_weight": 0.0,
       "anneal": false,
       "incr_control": false,
       "stop_on_success": false
   },
...

Some output details from bash_scripts/out_llama31_8b_it_gcg_direct. Please make a note of the ConversationTemplate, control_init, attention_weight. I believe these play an important role and might be helpful.

Folder ../results_individual_direct_gcg_llama31_8b_it already exists.
allow_non_ascii: false
anneal: false
attack: gcg
attention_pooling_method: mean
attention_weight: 50.0
attention_weight_dict:
  control: -1.0
  goal: 0.0
  sys_role: 0.0
batch_size: 256
control_init: a a a a a a a a a a a a a a a a a a a a
control_weight: 0.0
conversation_templates:
- llama-3.1
data_offset: 0
devices:
- cuda:0
enable_prefix_sharing: true
filter_cand: true
incr_control: false
model: llama31_8b_it
...
model_paths:
- meta-llama/Llama-3.1-8B-Instruct
n_steps: 500
n_test_data: 0
n_train_data: 100
num_train_models: 1
progressive_goals: false
progressive_models: false
result_prefix: ../results_individual_direct_gcg_llama31_8b_it/individual_direct_llama31_8b_it_offset0
setup: ''
stop_on_success: false
target_weight: 1.0
temp: 1
test_case_path: ../testcase/llama31_8b_it_gcg_direct.json
test_data: ''
test_steps: 20
tokenizer_kwargs:
- {}
tokenizer_paths:
- meta-llama/Llama-3.1-8B-Instruct
topk: 128
train_data: ../../data/advbench/harmful_behaviors_direct.csv
transfer: false
verbose: true

Loaded 100 train goals
Loaded 0 test goals
Loaded 1 tokenizers
Conversational template: [Conversation(name='meta-llama-3.1', system_template='{system_message}', system_message='Cutting Knowledge Date: December 2023\nToday Date: {{currentDateTimev2}}', system_message_vision='', roles=('user', 'assistant'), messages=[], offset=0, sep_style=<SeparatorStyle.DEFAULT: 23>, sep=None, sep2=None, stop_str=None, stop_token_ids=None, max_image_size_mb=None)]
Loaded 1 conversation templates

Providing the configs and other details for Llama-3.1-8B will be super helpful for my research and I can rerun to see if I can reproduce. And please do update the attack_manager.py accordingly as the changes I made might not be the right ones.

Please let me know if you need more details, happy to share.

Looking forward for your help in this regard :)