You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
may opt into a one-release grace period via the class attribute
81
81
`_brick_legacy_init = True`, which downgrades the error to a
82
82
`DeprecationWarning(removed_in="0.16.0")`. The opt-out is removed in 0.16.0.
83
-
-**All constructor parameters must be optional** (default to `None`) so the registry can instantiate the scenario with no arguments for metadata introspection. Defer required-input validation to `initialize_async()` or `_get_atomic_attacks_async()`. `ScenarioRegistry._build_metadata` raises `TypeError` if `scenario_class()` cannot be called with no arguments.
83
+
-**All constructor parameters must be optional** (default to `None`) so the registry can instantiate the scenario with no arguments for metadata introspection. Defer required-input validation to `initialize_async()` or `_build_atomic_attacks_async()`. `ScenarioRegistry._build_metadata` raises `TypeError` if `scenario_class()` cannot be called with no arguments.
84
84
-`super().__init__()` called with `version`, `strategy_class`, `default_strategy`, `default_dataset_config`, `objective_scorer`
85
85
- complex objects like `adversarial_chat` or `objective_scorer` should be passed into the constructor.
86
86
@@ -134,39 +134,70 @@ class MyStrategy(ScenarioStrategy):
134
134
- Each member: `NAME = ("string_value", {tag_set})`
135
135
- Aggregates expand to all strategies matching their tag
136
136
137
-
### `_build_display_group()` — Result Grouping
137
+
### Result grouping (`display_group`)
138
138
139
-
Override `_build_display_group()` on the `Scenario` base class to control how attack results are grouped for display:
139
+
`display_group` controls how attack results are aggregated for display. It is set per
140
+
`AtomicAttack` at construction time — there is no `_build_display_group` hook. When you build
141
+
via `build_matrix_atomic_attacks`/`MatrixAtomicAttackBuilder`, pass a `display_group_fn`
142
+
callback that maps each `MatrixCombo` to a group string:
1. Calls `resolve_technique_factories(context=context)` to map the selected strategies to their
189
+
registered `AttackTechniqueFactory` instances (reads the `AttackTechniqueRegistry` singleton;
190
+
strategies with no registered factory are dropped).
191
+
2. Iterates every (technique × dataset) pair from `context.seed_groups_by_dataset`.
192
+
3. Calls `factory.create()` with the objective target, conditional scorer override, and any
193
+
per-technique converters (from `--strategies <technique>:converter.<name>`) as
194
+
`extra_request_converters`.
195
+
4. Builds each `AtomicAttack` with a unique `atomic_attack_name` and a `display_group`
196
+
(customizable via `display_group_fn`).
197
+
198
+
Scenarios needing extra axes (adversarial targets, caching, converter stacks) call
199
+
`MatrixAtomicAttackBuilder` directly; scenarios whose construction is composite or
200
+
per-objective build the `AtomicAttack` list themselves (see "Manual AtomicAttack construction").
170
201
171
202
### AttackTechniqueFactory
172
203
@@ -213,29 +244,28 @@ by the registry. Tests that exercise scenarios should reset both `AttackTechniqu
213
244
and `TargetRegistry` and re-register a mock `adversarial_chat` so the catalog builder
214
245
resolves without falling back to `OpenAIChatTarget`.
215
246
216
-
### Customization hooks (no need to override `_get_atomic_attacks_async`):
217
-
-**`_get_attack_technique_factories()`** — override to add/remove/replace factories
218
-
-**`_build_display_group()`** — override to change grouping (default: by technique)
219
-
220
-
### When to override `_get_atomic_attacks_async`:
221
-
Only override when the scenario **cannot** use the factory/registry pattern — e.g., scenarios
222
-
with custom composite logic, per-strategy converter stacks, or non-standard attack construction.
247
+
### Baseline
223
248
224
-
Overrides that want baseline support must emit it themselves by calling `self._build_baseline_atomic_attack(seed_groups=...)` with the same seeds used for the strategy attacks and prepending the result. The base implementation emits baseline automatically; passing freshly resolved seeds reintroduces ADO 9012 (baseline-vs-strategy population divergence under `max_dataset_size`).
249
+
The baseline (a `PromptSendingAttack` over the run's seeds) is inserted **centrally** by
250
+
`Scenario.initialize_async` according to the scenario's `BASELINE_ATTACK_POLICY` class var and
251
+
the runtime `include_baseline` flag. `_build_atomic_attacks_async` must **never** prepend its own
252
+
baseline — doing so double-emits it and reintroduces baseline-vs-strategy population divergence
253
+
under `max_dataset_size`.
225
254
226
-
### Manual AtomicAttack construction (for overrides):
255
+
### Manual AtomicAttack construction:
227
256
228
257
```python
229
258
AtomicAttack(
230
-
atomic_attack_name=strategy_name, #groups related attacks
259
+
atomic_attack_name=strategy_name, #must be unique per AtomicAttack
0 commit comments