Skip to content

Commit 9919557

Browse files
committed
Merge remote-tracking branch 'origin/main' into rlundeen2-cleanup-initializers
# Conflicts: # pyrit/scenario/core/scenario.py # pyrit/scenario/scenarios/adaptive/adaptive_scenario.py # pyrit/scenario/scenarios/adaptive/text_adaptive.py # tests/unit/scenario/airt/test_cyber.py # tests/unit/scenario/airt/test_rapid_response.py # tests/unit/scenario/core/test_baseline_deprecation.py
2 parents 374ca1a + 9e4549c commit 9919557

69 files changed

Lines changed: 3520 additions & 1077 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/instructions/scenarios.instructions.md

Lines changed: 71 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,9 @@ For scenarios whose strategy enum is built dynamically (RapidResponse pattern),
3939
strategy class in a module-level `@cache`-decorated function and pass the result through
4040
the constructor — no classmethod indirection required.
4141

42-
4. **Optionally override `_get_atomic_attacks_async()`**the base class provides a default
43-
that uses the factory/registry pattern (see "AtomicAttack Construction" below).
44-
Only override if your scenario needs custom attack construction logic.
42+
4. **Implement `_build_atomic_attacks_async(self, *, context)`**this is the single
43+
abstract extension point every scenario must define (see "AtomicAttack Construction" below).
44+
Matrix-shaped scenarios delegate to `build_matrix_atomic_attacks(context=...)` in one line.
4545

4646
## Constructor Pattern
4747

@@ -58,7 +58,7 @@ def __init__(
5858
if not objective_scorer:
5959
objective_scorer = self._get_default_scorer()
6060

61-
# 2. Store config objects for _get_atomic_attacks_async
61+
# 2. Store config objects for _build_atomic_attacks_async
6262
self._scorer_config = AttackScoringConfig(objective_scorer=objective_scorer)
6363

6464
# 3. Call super().__init__ — required args: version, strategy_class, objective_scorer
@@ -80,7 +80,7 @@ Requirements:
8080
may opt into a one-release grace period via the class attribute
8181
`_brick_legacy_init = True`, which downgrades the error to a
8282
`DeprecationWarning(removed_in="0.16.0")`. The opt-out is removed in 0.16.0.
83-
- **All constructor parameters must be optional** (default to `None`) so the registry can instantiate the scenario with no arguments for metadata introspection. Defer required-input validation to `initialize_async()` or `_get_atomic_attacks_async()`. `ScenarioRegistry._build_metadata` raises `TypeError` if `scenario_class()` cannot be called with no arguments.
83+
- **All constructor parameters must be optional** (default to `None`) so the registry can instantiate the scenario with no arguments for metadata introspection. Defer required-input validation to `initialize_async()` or `_build_atomic_attacks_async()`. `ScenarioRegistry._build_metadata` raises `TypeError` if `scenario_class()` cannot be called with no arguments.
8484
- `super().__init__()` called with `version`, `strategy_class`, `default_strategy`, `default_dataset_config`, `objective_scorer`
8585
- complex objects like `adversarial_chat` or `objective_scorer` should be passed into the constructor.
8686

@@ -134,39 +134,70 @@ class MyStrategy(ScenarioStrategy):
134134
- Each member: `NAME = ("string_value", {tag_set})`
135135
- Aggregates expand to all strategies matching their tag
136136

137-
### `_build_display_group()`Result Grouping
137+
### Result grouping (`display_group`)
138138

139-
Override `_build_display_group()` on the `Scenario` base class to control how attack results are grouped for display:
139+
`display_group` controls how attack results are aggregated for display. It is set per
140+
`AtomicAttack` at construction time — there is no `_build_display_group` hook. When you build
141+
via `build_matrix_atomic_attacks`/`MatrixAtomicAttackBuilder`, pass a `display_group_fn`
142+
callback that maps each `MatrixCombo` to a group string:
140143

141144
```python
142-
def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:
143-
# Default: group by technique name (most common)
144-
return technique_name
145-
146-
# Override examples:
147-
# Group by dataset/harm category: return seed_group_name
148-
# Cross-product: return f"{technique_name}_{seed_group_name}"
145+
build_matrix_atomic_attacks(
146+
context=context,
147+
objective_scorer=self._objective_scorer,
148+
display_group_fn=lambda combo: combo.technique_name, # default: group by technique
149+
# Group by dataset/harm category: lambda combo: combo.dataset_name
150+
# Cross-product: lambda combo: f"{combo.technique_name}_{combo.dataset_name}"
151+
)
149152
```
150153

151154
Note: `atomic_attack_name` must remain unique per `AtomicAttack` for correct resume behaviour.
152155
`display_group` controls user-facing aggregation only.
153156

154-
## AtomicAttack Construction — Default Base Class Behaviour
157+
## AtomicAttack Construction — `_build_atomic_attacks_async(context)`
158+
159+
Every scenario implements the single abstract extension point:
160+
161+
```python
162+
async def _build_atomic_attacks_async(self, *, context: ScenarioContext) -> list[AtomicAttack]:
163+
...
164+
```
165+
166+
`initialize_async` resolves the run's inputs once (objective target, strategies, dataset
167+
config, memory labels, baseline flag, and seed groups), snapshots them into an immutable
168+
`ScenarioContext`, calls this method, and then inserts the baseline centrally. Scenario authors
169+
never read half-initialized `self._*` state to build attacks — read everything from `context`.
155170

156-
The `Scenario` base class provides a default `_get_atomic_attacks_async()` that uses the
157-
factory/registry pattern. Scenarios that register their techniques via `_get_attack_technique_factories()`
158-
get atomic-attack construction **for free** — no override needed.
171+
### Zero-boilerplate matrix scenarios
159172

160-
The default implementation:
161-
1. Calls `self._get_attack_technique_factories()` to get name→factory mapping
162-
(defaults to reading every `AttackTechniqueFactory` registered in the
163-
`AttackTechniqueRegistry` singleton)
164-
2. Iterates over every (technique × dataset) pair from `self._dataset_config`
165-
3. Calls `factory.create()` with `objective_target` and conditional scorer override
166-
(also forwards any per-technique converters from `self._strategy_converters`, populated
167-
from the CLI `--strategies <technique>:converter.<name>` modifier, as `extra_request_converters`)
168-
4. Uses `self._build_display_group()` for user-facing grouping
169-
5. Builds `AtomicAttack` with unique `atomic_attack_name` = `"{technique}_{dataset}"`
173+
Scenarios whose construction is the plain technique × dataset cross-product delegate to the
174+
`build_matrix_atomic_attacks` helper in one line (see `Cyber`, `RapidResponse`):
175+
176+
```python
177+
from pyrit.scenario.core.matrix_atomic_attack_builder import build_matrix_atomic_attacks
178+
179+
async def _build_atomic_attacks_async(self, *, context: ScenarioContext) -> list[AtomicAttack]:
180+
return build_matrix_atomic_attacks(
181+
context=context,
182+
objective_scorer=self._objective_scorer,
183+
strategy_converters=self._strategy_converters, # optional CLI converter stacks
184+
)
185+
```
186+
187+
`build_matrix_atomic_attacks`:
188+
1. Calls `resolve_technique_factories(context=context)` to map the selected strategies to their
189+
registered `AttackTechniqueFactory` instances (reads the `AttackTechniqueRegistry` singleton;
190+
strategies with no registered factory are dropped).
191+
2. Iterates every (technique × dataset) pair from `context.seed_groups_by_dataset`.
192+
3. Calls `factory.create()` with the objective target, conditional scorer override, and any
193+
per-technique converters (from `--strategies <technique>:converter.<name>`) as
194+
`extra_request_converters`.
195+
4. Builds each `AtomicAttack` with a unique `atomic_attack_name` and a `display_group`
196+
(customizable via `display_group_fn`).
197+
198+
Scenarios needing extra axes (adversarial targets, caching, converter stacks) call
199+
`MatrixAtomicAttackBuilder` directly; scenarios whose construction is composite or
200+
per-objective build the `AtomicAttack` list themselves (see "Manual AtomicAttack construction").
170201

171202
### AttackTechniqueFactory
172203

@@ -213,29 +244,28 @@ by the registry. Tests that exercise scenarios should reset both `AttackTechniqu
213244
and `TargetRegistry` and re-register a mock `adversarial_chat` so the catalog builder
214245
resolves without falling back to `OpenAIChatTarget`.
215246

216-
### Customization hooks (no need to override `_get_atomic_attacks_async`):
217-
- **`_get_attack_technique_factories()`** — override to add/remove/replace factories
218-
- **`_build_display_group()`** — override to change grouping (default: by technique)
219-
220-
### When to override `_get_atomic_attacks_async`:
221-
Only override when the scenario **cannot** use the factory/registry pattern — e.g., scenarios
222-
with custom composite logic, per-strategy converter stacks, or non-standard attack construction.
247+
### Baseline
223248

224-
Overrides that want baseline support must emit it themselves by calling `self._build_baseline_atomic_attack(seed_groups=...)` with the same seeds used for the strategy attacks and prepending the result. The base implementation emits baseline automatically; passing freshly resolved seeds reintroduces ADO 9012 (baseline-vs-strategy population divergence under `max_dataset_size`).
249+
The baseline (a `PromptSendingAttack` over the run's seeds) is inserted **centrally** by
250+
`Scenario.initialize_async` according to the scenario's `BASELINE_ATTACK_POLICY` class var and
251+
the runtime `include_baseline` flag. `_build_atomic_attacks_async` must **never** prepend its own
252+
baseline — doing so double-emits it and reintroduces baseline-vs-strategy population divergence
253+
under `max_dataset_size`.
225254

226-
### Manual AtomicAttack construction (for overrides):
255+
### Manual AtomicAttack construction:
227256

228257
```python
229258
AtomicAttack(
230-
atomic_attack_name=strategy_name, # groups related attacks
259+
atomic_attack_name=strategy_name, # must be unique per AtomicAttack
231260
attack=attack_instance, # AttackStrategy implementation
232261
seed_groups=list(seed_groups), # must be non-empty
233-
memory_labels=self._memory_labels, # from base class
262+
memory_labels=context.memory_labels, # from the context snapshot
234263
)
235264
```
236265

237266
- `seed_groups` must be non-empty — validate before constructing
238-
- `self._objective_target` is only available after `initialize_async()` — don't access in `__init__`
267+
- Read runtime inputs from `context`, not `self._*``self._objective_target` and
268+
`self._scenario_strategies` are only populated after `initialize_async()`
239269
- Pass `memory_labels` to every AtomicAttack
240270

241271
## Exports
@@ -248,4 +278,4 @@ New scenarios must be registered in `pyrit/scenario/__init__.py` as virtual pack
248278
- Forgetting `@apply_defaults` on `__init__`
249279
- Empty `seed_groups` passed to `AtomicAttack`
250280
- Missing `VERSION` class constant
251-
- Missing `_async` suffix on `_get_atomic_attacks_async`
281+
- Missing `_async` suffix on `_build_atomic_attacks_async`

doc/code/executor/0_executor.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ as it goes.
4444
- **[Prompt Generator](7_promptgen.ipynb)** — produces attack prompts (e.g. fuzzing, Anecdoctor) to
4545
augment datasets; some generate from a model alone, others probe a target to evolve effective
4646
prompts.
47+
- **[Modality Feedback](8_modality_feedback.ipynb)** — shows how `TargetCapabilities` determine
48+
whether media is forwarded between objective and adversarial targets in multi-turn attacks, with a
49+
two-seed Crescendo image-edit example.
4750

4851
**[Attack Configuration](3_attack_configuration.ipynb)** isn't an executor — it's the cross-cutting
4952
inputs every attack accepts (objective vs. adversarial target, prepended conversations, multimodal

0 commit comments

Comments
 (0)