PR Breaking MAINT: Moving Role Play Attack to Technique#2160
Merged
rlundeen2 merged 10 commits intoJul 11, 2026
Merged
Conversation
Remove RolePlayAttack/RolePlayPaths and reimplement all five role-play personas as with_simulated_conversation core techniques (role_play_movie_script, role_play_video_game, role_play_trivia_game, role_play_persuasion, role_play_persuasion_written). Update scenario defaults, inline constructors (jailbreak/scam/psychosocial), CLI help, tests, and docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Group the five role_play_* adversarial system-prompt YAMLs under a
dedicated red_teaming/role_play/ subdirectory and pass an explicit
adversarial_chat_system_prompt_path everywhere the simulated-conversation
factory is built (core catalog + the jailbreak/scam/psychosocial inline
constructors), since the factory default resolves to red_teaming/{name}.yaml.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…essage prompt The shared direct_next_message prompt broke the role-play frame on the final turn (asking for the raw objective / real-life instructions), which robust targets refused. Add a role-play-aware next-message system prompt that stays in the established fiction, delivers the same objective as the next in-world beat, and never breaks character. Wire the 5 role_play core factories and the 3 inline scenario constructors (scam, jailbreak, psychosocial) to it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… create Add the same non-None objective_target guard used by scam and jailbreak so the migrated role-play simulated-conversation .create() call type-checks (ty). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…-migration Regenerate the 0_attack_techniques catalog table so it lists the 5 role_play_* simulated-conversation techniques (PromptSendingAttack) instead of the removed RolePlayAttack, and correct the scam scenario RolePlay docstring to describe the simulated role-play conversation and current YAML name. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jbolor21
approved these changes
Jul 10, 2026
jbolor21
left a comment
jbolor21
left a comment
Contributor
There was a problem hiding this comment.
tiny comment but otherwise lgtm!
romanlutz
approved these changes
Jul 10, 2026
Reword the parenthetical so it clearly states the anti-pattern: an executor assembling its own prompt scaffolding is attack-technique work bleeding into the executor (per jbolor21 review feedback). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…onversation # Conflicts: # pyrit/executor/attack/single_turn/role_play.py # pyrit/scenario/scenarios/airt/jailbreak.py # pyrit/scenario/scenarios/airt/psychosocial.py # tests/unit/executor/attack/single_turn/test_role_play.py
…onversation # Conflicts: # doc/code/scenarios/0_attack_techniques.ipynb
…onversation # Conflicts: # pyrit/executor/attack/single_turn/role_play.py # tests/unit/executor/attack/single_turn/test_role_play.py
varunj-msft
pushed a commit
to varunj-msft/PyRIT
that referenced
this pull request
Jul 13, 2026
…hods Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/ microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique` dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable technique, not a converter. Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True) that the base AtomicAttack merges into every objective via with_technique() at run time, without touching the SeedObjective. The technique enum is now pure delivery methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques. - Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent). - num_templates / num_attempts / jailbreak_names are now declared run parameters (additional_parameters), so they are settable from the CLI / config, resolved from the parameter bag at build time. - Small curated default template set (aim, dan_11); full catalog opt-in via num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation. - Results group by jailbreak template (display_group); atomic-attack names are unique per (delivery method x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - VERSION 1 -> 2. Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective sent as the user turn), which is the proven pattern. The `user`, `variation`, and `translation` delivery methods, and the baseline-skip optimization, are deferred: `user` currently drops the objective because a single user-role technique seed becomes the send's next_message (PromptSendingAttack sends next_message over the objective), so faithful user delivery needs objective-after-user-framing sequencing in the send path. Note: the default delivery semantics change from "objective embedded in the template as one user message" to "template as system prompt + objective as user turn".
varunj-msft
pushed a commit
to varunj-msft/PyRIT
that referenced
this pull request
Jul 14, 2026
…hods Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/ microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique` dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable technique, not a converter. Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True) that the base AtomicAttack merges into every objective via with_technique() at run time, without touching the SeedObjective. The technique enum is now pure delivery methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques. - Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent). - num_templates / num_attempts / jailbreak_names are now declared run parameters (additional_parameters), so they are settable from the CLI / config, resolved from the parameter bag at build time. - Small curated default template set (aim, dan_11); full catalog opt-in via num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation. - Results group by jailbreak template (display_group); atomic-attack names are unique per (delivery method x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - VERSION 1 -> 2. Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective sent as the user turn), which is the proven pattern. The `user`, `variation`, and `translation` delivery methods, and the baseline-skip optimization, are deferred: `user` currently drops the objective because a single user-role technique seed becomes the send's next_message (PromptSendingAttack sends next_message over the objective), so faithful user delivery needs objective-after-user-framing sequencing in the send path. Note: the default delivery semantics change from "objective embedded in the template as one user message" to "template as system prompt + objective as user turn".
varunj-msft
pushed a commit
to varunj-msft/PyRIT
that referenced
this pull request
Jul 14, 2026
…hods Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/ microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique` dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable technique, not a converter. Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True) that the base AtomicAttack merges into every objective via with_technique() at run time, without touching the SeedObjective. The technique enum is now pure delivery methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques. - Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent). - num_templates / num_attempts / jailbreak_names are now declared run parameters (additional_parameters), so they are settable from the CLI / config, resolved from the parameter bag at build time. - Small curated default template set (aim, dan_11); full catalog opt-in via num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation. - Results group by jailbreak template (display_group); atomic-attack names are unique per (delivery method x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - VERSION 1 -> 2. Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective sent as the user turn), which is the proven pattern. The `user`, `variation`, and `translation` delivery methods, and the baseline-skip optimization, are deferred: `user` currently drops the objective because a single user-role technique seed becomes the send's next_message (PromptSendingAttack sends next_message over the objective), so faithful user delivery needs objective-after-user-framing sequencing in the send path. Note: the default delivery semantics change from "objective embedded in the template as one user message" to "template as system prompt + objective as user turn".
varunj-msft
pushed a commit
to varunj-msft/PyRIT
that referenced
this pull request
Jul 14, 2026
…hods Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/ microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique` dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable technique, not a converter. Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True) that the base AtomicAttack merges into every objective via with_technique() at run time, without touching the SeedObjective. The technique enum is now pure delivery methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques. - Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent). - num_templates / num_attempts / jailbreak_names are now declared run parameters (additional_parameters), so they are settable from the CLI / config, resolved from the parameter bag at build time. - Small curated default template set (aim, dan_11); full catalog opt-in via num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation. - Results group by jailbreak template (display_group); atomic-attack names are unique per (delivery method x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - VERSION 1 -> 2. Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective sent as the user turn), which is the proven pattern. The `user`, `variation`, and `translation` delivery methods, and the baseline-skip optimization, are deferred: `user` currently drops the objective because a single user-role technique seed becomes the send's next_message (PromptSendingAttack sends next_message over the objective), so faithful user delivery needs objective-after-user-framing sequencing in the send path. Note: the default delivery semantics change from "objective embedded in the template as one user message" to "template as system prompt + objective as user turn".
rlundeen2
added a commit
to rlundeen2/PyRIT
that referenced
this pull request
Jul 14, 2026
Reconciles the context_compliance technique migration with main's role_play (microsoft#2160) and FlipAttack (microsoft#2178) technique migrations and the seed-group renames (microsoft#2165: SeedAttackGroup/AttackTechniqueSeedGroup). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: aeb8c8c6-5c23-490c-b451-a59c096f54f7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #2155. Migrates the role-play attacks (all 5 personas) from the standalone
RolePlayAttackexecutor to core simulated-conversation techniques, and tunes the final-turn prompt so the attacks actually land against robust targets.What changed
RolePlayAttackexecutor (single_turn/role_play.py) and its tests.AttackTechniqueFactory.with_simulated_conversation, registered in the core technique catalog.datasets/executors/red_teaming/role_play/.simulated_target/role_play_next_message.yaml). The shareddirect_next_messageprompt broke the fiction on the final turn (asking for the raw / real-life objective), which targets refused. The new prompt keeps the last message in character and delivers the same objective as the next in-world beat.objective_targetguard in psychosocial to match scam/jailbreak.Validation
pre-commit(ruff, ty, yaml, etc.): all pass.Note: this PR is intended to be a breaking, standalone change from the squashed #2155.