Skip to content

MAINT: Technique Organization#2155

Merged
rlundeen2 merged 3 commits into
microsoft:mainfrom
rlundeen2:rlundeen2-organizing-techniques
Jul 10, 2026
Merged

MAINT: Technique Organization#2155
rlundeen2 merged 3 commits into
microsoft:mainfrom
rlundeen2:rlundeen2-organizing-techniques

Conversation

@rlundeen2

@rlundeen2 rlundeen2 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Reorganizes how attack techniques are cataloged, tagged, and selected so the catalog can scale to hundreds of techniques, and makes a bare initialize_pyrit_async produce a usable environment by default.

Decisions

  • Catalog layout (pyrit/setup/initializers/techniques/):
    • core.py — a small, curated standard set (auto-loaded).
    • extra.py — the broader, opt-in collection, and the default home for new general-purpose contributions.
    • Per-source modules (e.g. airt.py) — techniques owned by a scenario/source but still reusable; tagged with their owner and kept out of the default pool.
  • No global default tag. What runs by default is scenario-relative, so each scenario declares its own defaults via build_strategy_class_from_factories. The three roles are tag queries over the pool and relate as strict subsets: available (the pool), aggregates (named presets like single_turn), and default / default_technique_names (what runs when nothing is selected).
  • Scenario-local techniques are legitimate. A technique pinned to one scenario (e.g. Doctor's Policy Puppetry factories, pinned to a specific template) can stay local; promote to a catalog module only when another scenario could reuse it.
  • Doctor migration. Doctor's hand-written strategy enum now uses the shared factory generator like the other registry-driven scenarios. Only externally visible change: technique enum attribute names go PascalCase → snake_case (e.g. DoctorStrategy.PolicyPuppetry → DoctorStrategy("policy_puppetry")); enum values are unchanged.

Default loading

  • A bare initialize_pyrit_async(...) now auto-runs TechniqueInitializer + TargetInitializer only when the caller supplies no initializers/scripts, so core techniques and the available default targets are registered with no config.
  • Supplying any initializer or script means the caller owns setup and the defaults are skipped; a new load_defaults=False flag skips them even on a bare call (e.g. to start from an empty state).
  • Only core techniques and default targets are auto-loaded — extra / per-source technique groups and scorer target variants stay opt-in (via initializer tags core/extra/all, a custom initializer, or the CLI).

Docs

  • Expanded the Attack Techniques section of doc/code/framework.md to describe the catalog, the tagging model, and the available/aggregate/default selection roles.

Tests

  • tests/unit/scenario/ + tests/unit/registry/: 1140 passed. Full pre-commit (ruff, format, ty, doc-structure, async-suffix, etc.) is green.

@rlundeen2
MAINT: Technique Organization
89ce00d 
Reorganize how attack techniques are cataloged, tagged, and selected so the
catalog can scale to hundreds of techniques.

Decisions:
- Catalog lives under pyrit/setup/initializers/techniques/. `core.py` is a
  small, curated standard set; `extra.py` is the broader opt-in collection and
  the default home for new general-purpose contributions. Per-source modules
  (e.g. `airt.py`) hold techniques owned by a scenario/source but still
  reusable, tagged with their owner and kept out of the default pool.
- Retire the global `default` tag. What runs by default is scenario-relative,
  so each scenario declares its own defaults via
  build_strategy_class_from_factories (available selects the pool, aggregates
  are named presets, default/default_technique_names set what runs when nothing
  is chosen). available/aggregate/default relate as strict subsets.
- Techniques pinned to one scenario may stay local to it (e.g. Doctor's Policy
  Puppetry factories); promote to a catalog module only when reusable.
- Migrate Doctor's hand-written strategy enum onto the shared factory generator
  (technique enum attr names go PascalCase -> snake_case; values unchanged).

Default loading:
- A bare initialize_pyrit_async now auto-runs TechniqueInitializer (core) and
  TargetInitializer when none are supplied, so `core` techniques and default
  targets are registered without any config. Users select other subsets by
  passing initializer tags (core/extra/all) or their own initializer, including
  from the CLI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
behnam-o
behnam-o approved these changes Jul 9, 2026
View reviewed changes

@behnam-o behnam-o left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread pyrit/scenario/scenarios/garak/doctor.py Outdated
@rlundeen2
MAINT: build DoctorStrategy in __init__ for consistency
5cfd1a9 
Address review feedback: build the Doctor strategy class the same way the other
dynamically-generated scenarios do (rapid_response/leakage/cyber) instead of at
module level.

- `_build_doctor_strategy()` is now `@cache`d and called in `Doctor.__init__`
  (`strategy_class = _build_doctor_strategy()`; `default_strategy=strategy_class("default")`).
- The public `DoctorStrategy` symbol is resolved lazily via a `__getattr__` in the
  garak package `__init__` (mirroring the airt package), so `@cache` keeps every
  instance sharing one enum class.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@rlundeen2
Merge remote-tracking branch 'origin/main' into rlundeen2-organizing-…
e434826 
…techniques

# Conflicts:
#	pyrit/registry/components/attack_technique_registry.py
#	pyrit/scenario/scenarios/airt/cyber.py
#	pyrit/scenario/scenarios/airt/leakage.py
#	pyrit/scenario/scenarios/airt/rapid_response.py
#	pyrit/scenario/scenarios/garak/__init__.py
#	pyrit/scenario/scenarios/garak/doctor.py
#	pyrit/setup/initializers/techniques/core.py
#	tests/unit/scenario/garak/test_doctor.py
@rlundeen2 rlundeen2 enabled auto-merge July 9, 2026 23:52
@rlundeen2 rlundeen2 added this pull request to the merge queue Jul 10, 2026
Merged via the queue into microsoft:main with commit 40cecc7 Jul 10, 2026
53 checks passed
@rlundeen2 rlundeen2 deleted the rlundeen2-organizing-techniques branch July 10, 2026 00:21
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 10, 2026
MAINT: Standardize garak.encoding defaults and fix atomic-attack name…
d0b29a3 
… collisions

Re-migrated onto current main (post technique rename microsoft#2153 / technique
organization microsoft#2155). garak.encoding was already seam-migrated on main to
EncodingTechnique(ScenarioTechnique); this layers the remaining #8380
standardization on top.

Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across
base-N, cipher, and symbolic families) with a get_aggregate_tags() override,
make it the default technique, and bump VERSION to 2. Follows the hand-written
enum DEFAULT pattern used by scam/web_injection (not the factory
default_technique_names path, which applies only to the core catalog).

Fix atomic-attack name collisions by naming each converter variant
f"{variant_slug}_{config_suffix}" and grouping variants under
display_group=encoding_name so per-encoding results aggregate. Trim
near-duplicate base64 variants (standard_b64encode is byte-identical to the
default; b2a_base64 only appends a newline).

Keep the default per-dataset cap at main's max_dataset_size=3 (no timing change).
u7k4rs6 added a commit to u7k4rs6/PyRIT that referenced this pull request Jul 10, 2026
@u7k4rs6
MAINT port code_attack registration to TechniqueInitializer / techniq…
b500382 
…ues/core.py

Upstream (microsoft#2155) replaced initializers/components/scenario_techniques.py
with initializers/techniques/core.py and renamed strategy_tags to
technique_tags. The core group tag is now injected by build_technique_factories
rather than stored in the factory. Port the code_attack entry and its test
list update to the new layout.
@rlundeen2 rlundeen2 mentioned this pull request Jul 10, 2026
Merged
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 13, 2026
MAINT: Standardize garak.encoding defaults and fix atomic-attack name…
967f703 
… collisions

Re-migrated onto current main (post technique rename microsoft#2153, technique
organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165).
garak.encoding was already seam/Technique-migrated on main; this layers the
remaining #8380 standardization on top.

Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across
base-N, cipher, and symbolic families) with a get_aggregate_tags() override, and
make it the default technique (was the exhaustive ALL). Follows the hand-written
enum DEFAULT pattern used by scam/web_injection (not the factory
default_technique_names path, which applies only to the core catalog).

Fix atomic-attack name collisions: previously every converter variant and decode
config shared atomic_attack_name=encoding_name, so base64's variants collided and
their result buckets (keyed by atomic_attack_name) collapsed. Each variant now
gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}",
with display_group=encoding_name so variants still group per encoding in output.

Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the
default; b2a_base64 only appends a newline). Bump VERSION 1->2. Keep main's
per-dataset max_dataset_size=3 (no timing change).
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 13, 2026
MAINT: Redesign Jailbreak scenario around technique-seed delivery met…
4bfab9c 
…hods

Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/
microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique`
dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable
technique, not a converter.

Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True)
that the base AtomicAttack merges into every objective via with_technique() at run
time, without touching the SeedObjective. The technique enum is now pure delivery
methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the
role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques.

- Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent).
- num_templates / num_attempts / jailbreak_names are now declared run parameters
  (additional_parameters), so they are settable from the CLI / config, resolved from
  the parameter bag at build time.
- Small curated default template set (aim, dan_11); full catalog opt-in via
  num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation.
- Results group by jailbreak template (display_group); atomic-attack names are unique
  per (delivery method x template x attempt).
- Resume-stable: the resolved template set is persisted to ScenarioResult metadata and
  replayed on resume so a random num_templates sample does not diverge.
- VERSION 1 -> 2.

Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective
sent as the user turn), which is the proven pattern. The `user`, `variation`, and
`translation` delivery methods, and the baseline-skip optimization, are deferred:
`user` currently drops the objective because a single user-role technique seed becomes
the send's next_message (PromptSendingAttack sends next_message over the objective), so
faithful user delivery needs objective-after-user-framing sequencing in the send path.
Note: the default delivery semantics change from "objective embedded in the template as
one user message" to "template as system prompt + objective as user turn".
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
MAINT: Standardize garak.encoding defaults and fix atomic-attack name…
8bd4cf9 
… collisions

Re-migrated onto current main (post technique rename microsoft#2153, technique
organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165).
garak.encoding was already seam/Technique-migrated on main; this layers the
remaining #8380 standardization on top.

Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across
base-N, cipher, and symbolic families) with a get_aggregate_tags() override, and
make it the default technique (was the exhaustive ALL). Follows the hand-written
enum DEFAULT pattern used by scam/web_injection (not the factory
default_technique_names path, which applies only to the core catalog).

Fix atomic-attack name collisions: previously every converter variant and decode
config shared atomic_attack_name=encoding_name, so base64's variants collided and
their result buckets (keyed by atomic_attack_name) collapsed. Each variant now
gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}",
with display_group=encoding_name so variants still group per encoding in output.

Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the
default; b2a_base64 only appends a newline). Bump VERSION 1->2. Keep main's
per-dataset max_dataset_size=3 (no timing change).
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
MAINT: Redesign Jailbreak scenario around technique-seed delivery met…
1fa4bac 
…hods

Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/
microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique`
dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable
technique, not a converter.

Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True)
that the base AtomicAttack merges into every objective via with_technique() at run
time, without touching the SeedObjective. The technique enum is now pure delivery
methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the
role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques.

- Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent).
- num_templates / num_attempts / jailbreak_names are now declared run parameters
  (additional_parameters), so they are settable from the CLI / config, resolved from
  the parameter bag at build time.
- Small curated default template set (aim, dan_11); full catalog opt-in via
  num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation.
- Results group by jailbreak template (display_group); atomic-attack names are unique
  per (delivery method x template x attempt).
- Resume-stable: the resolved template set is persisted to ScenarioResult metadata and
  replayed on resume so a random num_templates sample does not diverge.
- VERSION 1 -> 2.

Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective
sent as the user turn), which is the proven pattern. The `user`, `variation`, and
`translation` delivery methods, and the baseline-skip optimization, are deferred:
`user` currently drops the objective because a single user-role technique seed becomes
the send's next_message (PromptSendingAttack sends next_message over the objective), so
faithful user delivery needs objective-after-user-framing sequencing in the send path.
Note: the default delivery semantics change from "objective embedded in the template as
one user message" to "template as system prompt + objective as user turn".
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
MAINT: Standardize garak.encoding defaults and fix atomic-attack name…
369dbfe 
… collisions

Re-migrated onto current main (post technique rename microsoft#2153, technique
organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165).
garak.encoding was already seam/Technique-migrated on main; this layers the
remaining #8380 standardization on top.

Add an EncodingTechnique.DEFAULT aggregate (curated 8-scheme spread across
base-N, cipher, and symbolic families) with a get_aggregate_tags() override, and
make it the default technique (was the exhaustive ALL). Follows the hand-written
enum DEFAULT pattern used by scam/web_injection (not the factory
default_technique_names path, which applies only to the core catalog).

Fix atomic-attack name collisions: previously every converter variant and decode
config shared atomic_attack_name=encoding_name, so base64's variants collided and
their result buckets (keyed by atomic_attack_name) collapsed. Each variant now
gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}",
with display_group=encoding_name so variants still group per encoding in output.

Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the
default; b2a_base64 only appends a newline). Bump VERSION 1->2. Keep main's
per-dataset max_dataset_size=3 (no timing change).
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
MAINT: Redesign Jailbreak scenario around technique-seed delivery met…
5016e4e 
…hods

Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/
microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique`
dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable
technique, not a converter.

Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True)
that the base AtomicAttack merges into every objective via with_technique() at run
time, without touching the SeedObjective. The technique enum is now pure delivery
methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the
role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques.

- Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent).
- num_templates / num_attempts / jailbreak_names are now declared run parameters
  (additional_parameters), so they are settable from the CLI / config, resolved from
  the parameter bag at build time.
- Small curated default template set (aim, dan_11); full catalog opt-in via
  num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation.
- Results group by jailbreak template (display_group); atomic-attack names are unique
  per (delivery method x template x attempt).
- Resume-stable: the resolved template set is persisted to ScenarioResult metadata and
  replayed on resume so a random num_templates sample does not diverge.
- VERSION 1 -> 2.

Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective
sent as the user turn), which is the proven pattern. The `user`, `variation`, and
`translation` delivery methods, and the baseline-skip optimization, are deferred:
`user` currently drops the objective because a single user-role technique seed becomes
the send's next_message (PromptSendingAttack sends next_message over the objective), so
faithful user delivery needs objective-after-user-framing sequencing in the send path.
Note: the default delivery semantics change from "objective embedded in the template as
one user message" to "template as system prompt + objective as user turn".
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
MAINT: Redesign Jailbreak scenario around technique-seed delivery met…
135d7c3 
…hods

Rebuild airt.jailbreak on the mergeable-technique contract (post microsoft#2153/microsoft#2155/microsoft#2160/
microsoft#2161/microsoft#2165), replacing the bespoke TextJailbreakConverter + `match technique`
dispatch with the pattern Rich described in microsoft#2045: a jailbreak template is a reusable
technique, not a converter.

Each jailbreak template becomes an AttackTechniqueSeedGroup (is_general_technique=True)
that the base AtomicAttack merges into every objective via with_technique() at run
time, without touching the SeedObjective. The technique enum is now pure delivery
methods (JailbreakTechnique.System), each an AttackTechniqueFactory carrying the
role-tagged framing seed. This mirrors the migrated core `flip`/role_play techniques.

- Default objectives switched airt_harms -> harmbench (mirrors adversarial/red_team_agent).
- num_templates / num_attempts / jailbreak_names are now declared run parameters
  (additional_parameters), so they are settable from the CLI / config, resolved from
  the parameter bag at build time.
- Small curated default template set (aim, dan_11); full catalog opt-in via
  num_templates (random) or jailbreak_names (explicit), with mutual-exclusion validation.
- Results group by jailbreak template (display_group); atomic-attack names are unique
  per (delivery method x template x attempt).
- Resume-stable: the resolved template set is persisted to ScenarioResult metadata and
  replayed on resume so a random num_templates sample does not diverge.
- VERSION 1 -> 2.

Scope: ships the `system` delivery method (jailbreak framing as system prompt, objective
sent as the user turn), which is the proven pattern. The `user`, `variation`, and
`translation` delivery methods, and the baseline-skip optimization, are deferred:
`user` currently drops the objective because a single user-role technique seed becomes
the send's next_message (PromptSendingAttack sends next_message over the objective), so
faithful user delivery needs objective-after-user-framing sequencing in the send path.
Note: the default delivery semantics change from "objective embedded in the template as
one user message" to "template as system prompt + objective as user turn".
varunj-msft pushed a commit to varunj-msft/PyRIT that referenced this pull request Jul 14, 2026
MAINT: Standardize garak.encoding defaults and fix atomic-attack name…
a28388c 
… collisions

Re-migrated onto current main (post technique rename microsoft#2153, technique
organization microsoft#2155, PromptConverter->Converter microsoft#2161, seed-group rename microsoft#2165).
garak.encoding was already seam/Technique-migrated on main; this layers the
remaining #8380 standardization on top.

Add an EncodingTechnique.DEFAULT aggregate (curated 13-of-17-scheme spread across
base-N, byte-encoding, cipher, and symbolic families) with a get_aggregate_tags()
override, and make it the default technique (was the exhaustive ALL). The niche/
lossy schemes (braille, ecoji, zalgo, ascii_smuggler) stay ALL-only. Follows the
hand-written enum DEFAULT pattern used by scam/web_injection (not the factory
default_technique_names path, which applies only to the core catalog).

Fix atomic-attack name collisions: previously every converter variant and decode
config shared atomic_attack_name=encoding_name, so base64's variants collided and
their result buckets (keyed by atomic_attack_name) collapsed. Each variant now
gets a unique variant_slug -> atomic_attack_name=f"{variant_slug}_{config_suffix}",
with display_group=encoding_name so variants still group per encoding in output.

Trim near-duplicate base64 variants (standard_b64encode is byte-identical to the
default; b2a_base64 only appends a newline). Bump VERSION 1->2.

Make the default run meatier per Rich's microsoft#2058 asks (06-20 'target of 10-20 minutes
... may finish too quickly'; 07-08 'change the dataset size a bit and adding more
defaults'): raise per-dataset max_dataset_size 3->10 and expand DEFAULT from 8 to
13 schemes. ALL remains the exhaustive ceiling (adds the 4 niche/lossy schemes).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@behnam-o behnam-o behnam-o approved these changes
@varunj-msft varunj-msft varunj-msft approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rlundeen2 @behnam-o @varunj-msft