Merge branch 'main' into hawestra/scenario_blog

Author: romanlutz

.github/instructions/scenarios.instructions.md

@@ -39,9 +39,9 @@ For scenarios whose strategy enum is built dynamically (RapidResponse pattern),
 strategy class in a module-level `@cache`-decorated function and pass the result through
 the constructor — no classmethod indirection required.
 
-4. **Optionally override `_get_atomic_attacks_async()`** — the base class provides a default
-   that uses the factory/registry pattern (see "AtomicAttack Construction" below).
-   Only override if your scenario needs custom attack construction logic.
+4. **Implement `_build_atomic_attacks_async(self, *, context)`** — this is the single
+   abstract extension point every scenario must define (see "AtomicAttack Construction" below).
+   Matrix-shaped scenarios delegate to `build_matrix_atomic_attacks(context=...)` in one line.
 
 ## Constructor Pattern
 
@@ -58,7 +58,7 @@ def __init__(
     if not objective_scorer:
         objective_scorer = self._get_default_scorer()
 
-    # 2. Store config objects for _get_atomic_attacks_async
+    # 2. Store config objects for _build_atomic_attacks_async
     self._scorer_config = AttackScoringConfig(objective_scorer=objective_scorer)
 
     # 3. Call super().__init__ — required args: version, strategy_class, objective_scorer
@@ -80,7 +80,7 @@ Requirements:
   may opt into a one-release grace period via the class attribute
   `_brick_legacy_init = True`, which downgrades the error to a
   `DeprecationWarning(removed_in="0.16.0")`. The opt-out is removed in 0.16.0.
-- **All constructor parameters must be optional** (default to `None`) so the registry can instantiate the scenario with no arguments for metadata introspection. Defer required-input validation to `initialize_async()` or `_get_atomic_attacks_async()`. `ScenarioRegistry._build_metadata` raises `TypeError` if `scenario_class()` cannot be called with no arguments.
+- **All constructor parameters must be optional** (default to `None`) so the registry can instantiate the scenario with no arguments for metadata introspection. Defer required-input validation to `initialize_async()` or `_build_atomic_attacks_async()`. `ScenarioRegistry._build_metadata` raises `TypeError` if `scenario_class()` cannot be called with no arguments.
 - `super().__init__()` called with `version`, `strategy_class`, `default_strategy`, `default_dataset_config`, `objective_scorer`
 - complex objects like `adversarial_chat` or `objective_scorer` should be passed into the constructor.
 
@@ -134,39 +134,70 @@ class MyStrategy(ScenarioStrategy):
 - Each member: `NAME = ("string_value", {tag_set})`
 - Aggregates expand to all strategies matching their tag
 
-### `_build_display_group()` — Result Grouping
+### Result grouping (`display_group`)
 
-Override `_build_display_group()` on the `Scenario` base class to control how attack results are grouped for display:
+`display_group` controls how attack results are aggregated for display. It is set per
+`AtomicAttack` at construction time — there is no `_build_display_group` hook. When you build
+via `build_matrix_atomic_attacks`/`MatrixAtomicAttackBuilder`, pass a `display_group_fn`
+callback that maps each `MatrixCombo` to a group string:
 
 ```python
-def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:
-    # Default: group by technique name (most common)
-    return technique_name
-
-    # Override examples:
-    # Group by dataset/harm category: return seed_group_name
-    # Cross-product: return f"{technique_name}_{seed_group_name}"
+build_matrix_atomic_attacks(
+    context=context,
+    objective_scorer=self._objective_scorer,
+    display_group_fn=lambda combo: combo.technique_name,  # default: group by technique
+    # Group by dataset/harm category: lambda combo: combo.dataset_name
+    # Cross-product:                   lambda combo: f"{combo.technique_name}_{combo.dataset_name}"
+)
 ```
 
 Note: `atomic_attack_name` must remain unique per `AtomicAttack` for correct resume behaviour.
 `display_group` controls user-facing aggregation only.
 
-## AtomicAttack Construction — Default Base Class Behaviour
+## AtomicAttack Construction — `_build_atomic_attacks_async(context)`
+
+Every scenario implements the single abstract extension point:
+
+```python
+async def _build_atomic_attacks_async(self, *, context: ScenarioContext) -> list[AtomicAttack]:
+    ...
+```
+
+`initialize_async` resolves the run's inputs once (objective target, strategies, dataset
+config, memory labels, baseline flag, and seed groups), snapshots them into an immutable
+`ScenarioContext`, calls this method, and then inserts the baseline centrally. Scenario authors
+never read half-initialized `self._*` state to build attacks — read everything from `context`.
 
-The `Scenario` base class provides a default `_get_atomic_attacks_async()` that uses the
-factory/registry pattern.  Scenarios that register their techniques via `_get_attack_technique_factories()`
-get atomic-attack construction **for free** — no override needed.
+### Zero-boilerplate matrix scenarios
 
-The default implementation:
-1. Calls `self._get_attack_technique_factories()` to get name→factory mapping
-   (defaults to reading every `AttackTechniqueFactory` registered in the
-   `AttackTechniqueRegistry` singleton)
-2. Iterates over every (technique × dataset) pair from `self._dataset_config`
-3. Calls `factory.create()` with `objective_target` and conditional scorer override
-   (also forwards any per-technique converters from `self._strategy_converters`, populated
-   from the CLI `--strategies <technique>:converter.<name>` modifier, as `extra_request_converters`)
-4. Uses `self._build_display_group()` for user-facing grouping
-5. Builds `AtomicAttack` with unique `atomic_attack_name` = `"{technique}_{dataset}"`
+Scenarios whose construction is the plain technique × dataset cross-product delegate to the
+`build_matrix_atomic_attacks` helper in one line (see `Cyber`, `RapidResponse`):
+
+```python
+from pyrit.scenario.core.matrix_atomic_attack_builder import build_matrix_atomic_attacks
+
+async def _build_atomic_attacks_async(self, *, context: ScenarioContext) -> list[AtomicAttack]:
+    return build_matrix_atomic_attacks(
+        context=context,
+        objective_scorer=self._objective_scorer,
+        strategy_converters=self._strategy_converters,  # optional CLI converter stacks
+    )
+```
+
+`build_matrix_atomic_attacks`:
+1. Calls `resolve_technique_factories(context=context)` to map the selected strategies to their
+   registered `AttackTechniqueFactory` instances (reads the `AttackTechniqueRegistry` singleton;
+   strategies with no registered factory are dropped).
+2. Iterates every (technique × dataset) pair from `context.seed_groups_by_dataset`.
+3. Calls `factory.create()` with the objective target, conditional scorer override, and any
+   per-technique converters (from `--strategies <technique>:converter.<name>`) as
+   `extra_request_converters`.
+4. Builds each `AtomicAttack` with a unique `atomic_attack_name` and a `display_group`
+   (customizable via `display_group_fn`).
+
+Scenarios needing extra axes (adversarial targets, caching, converter stacks) call
+`MatrixAtomicAttackBuilder` directly; scenarios whose construction is composite or
+per-objective build the `AtomicAttack` list themselves (see "Manual AtomicAttack construction").
 
 ### AttackTechniqueFactory
 
@@ -213,29 +244,28 @@ by the registry. Tests that exercise scenarios should reset both `AttackTechniqu
 and `TargetRegistry` and re-register a mock `adversarial_chat` so the catalog builder
 resolves without falling back to `OpenAIChatTarget`.
 
-### Customization hooks (no need to override `_get_atomic_attacks_async`):
-- **`_get_attack_technique_factories()`** — override to add/remove/replace factories
-- **`_build_display_group()`** — override to change grouping (default: by technique)
-
-### When to override `_get_atomic_attacks_async`:
-Only override when the scenario **cannot** use the factory/registry pattern — e.g., scenarios
-with custom composite logic, per-strategy converter stacks, or non-standard attack construction.
+### Baseline
 
-Overrides that want baseline support must emit it themselves by calling `self._build_baseline_atomic_attack(seed_groups=...)` with the same seeds used for the strategy attacks and prepending the result. The base implementation emits baseline automatically; passing freshly resolved seeds reintroduces ADO 9012 (baseline-vs-strategy population divergence under `max_dataset_size`).
+The baseline (a `PromptSendingAttack` over the run's seeds) is inserted **centrally** by
+`Scenario.initialize_async` according to the scenario's `BASELINE_ATTACK_POLICY` class var and
+the runtime `include_baseline` flag. `_build_atomic_attacks_async` must **never** prepend its own
+baseline — doing so double-emits it and reintroduces baseline-vs-strategy population divergence
+under `max_dataset_size`.
 
-### Manual AtomicAttack construction (for overrides):
+### Manual AtomicAttack construction:
 
 ```python
 AtomicAttack(
-    atomic_attack_name=strategy_name,   # groups related attacks
+    atomic_attack_name=strategy_name,   # must be unique per AtomicAttack
     attack=attack_instance,             # AttackStrategy implementation
     seed_groups=list(seed_groups),       # must be non-empty
-    memory_labels=self._memory_labels,   # from base class
+    memory_labels=context.memory_labels, # from the context snapshot
 )
 ```
 
 - `seed_groups` must be non-empty — validate before constructing
-- `self._objective_target` is only available after `initialize_async()` — don't access in `__init__`
+- Read runtime inputs from `context`, not `self._*` — `self._objective_target` and
+  `self._scenario_strategies` are only populated after `initialize_async()`
 - Pass `memory_labels` to every AtomicAttack
 
 ## Exports
@@ -248,4 +278,4 @@ New scenarios must be registered in `pyrit/scenario/__init__.py` as virtual pack
 - Forgetting `@apply_defaults` on `__init__`
 - Empty `seed_groups` passed to `AtomicAttack`
 - Missing `VERSION` class constant
-- Missing `_async` suffix on `_get_atomic_attacks_async`
+- Missing `_async` suffix on `_build_atomic_attacks_async`

doc/code/scenarios/0_scenarios.ipynb

@@ -57,13 +57,12 @@
     "   - Each enum member represents an **attack technique** (the *how* of an attack)\n",
     "   - Each member is defined as `(value, tags)` where value is a string and tags is a set of strings\n",
     "   - Include an `ALL` aggregate strategy that expands to all available strategies\n",
-    "   - Optionally override `_prepare_strategies()` for custom composition logic (see `FoundryComposite`)\n",
     "\n",
     "2. **Scenario Class**: Extend `Scenario` and pass these to `super().__init__()`:\n",
     "   - `strategy_class`: Your strategy enum class\n",
     "   - `default_strategy`: The default strategy (typically `YourStrategy.ALL` or `YourStrategy.DEFAULT`)\n",
-    "   - The base class provides a default `_get_atomic_attacks_async()` that uses the factory/registry\n",
-    "     pattern. Override it only if your scenario needs custom attack construction logic.\n",
+    "   - Implement `_build_atomic_attacks_async(context)` — the single abstract extension point.\n",
+    "     Matrix-shaped scenarios delegate to `build_matrix_atomic_attacks(context=...)` in one line.\n",
     "\n",
     "3. **Default Dataset**: Pass `default_dataset_config=` to `super().__init__()` to specify the datasets your scenario uses out of the box.\n",
     "   - Returns a `DatasetConfiguration` with one or more named datasets (e.g., `DatasetConfiguration(dataset_names=[\"my_dataset\"])`)\n",
@@ -89,9 +88,10 @@
     "\n",
     "### Example Structure\n",
     "\n",
-    "The simplest approach uses the **factory/registry pattern**: define your strategy,\n",
-    "dataset config, and constructor — the base class handles building atomic attacks\n",
-    "automatically from registered attack techniques."
+    "The construction path: define your strategy, dataset config, and constructor, then\n",
+    "implement `_build_atomic_attacks_async(context)`. Matrix-shaped scenarios delegate to the\n",
+    "`build_matrix_atomic_attacks` helper, which builds atomic attacks automatically from the\n",
+    "registered attack techniques."
    ]
   },
   {
@@ -124,6 +124,7 @@
     "    Scenario,\n",
     "    ScenarioStrategy,\n",
     ")\n",
+    "from pyrit.scenario.core.matrix_atomic_attack_builder import build_matrix_atomic_attacks\n",
     "from pyrit.score.true_false.true_false_scorer import TrueFalseScorer\n",
     "from pyrit.setup import initialize_pyrit_async\n",
     "from pyrit.setup.initializers.components import ScenarioTechniqueInitializer\n",
@@ -166,14 +167,15 @@
     "            scenario_result_id=scenario_result_id,\n",
     "        )\n",
     "\n",
-    "    # Optional: override _build_display_group to customize result grouping.\n",
-    "    # Default groups by technique name; override to group by dataset instead:\n",
-    "    def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:\n",
-    "        return seed_group_name\n",
-    "\n",
-    "    # No _get_atomic_attacks_async override needed!\n",
-    "    # The base class builds attacks from the (technique x dataset) cross-product\n",
-    "    # using the factory/registry pattern automatically."
+    "    # Implement the single abstract extension point. Matrix-shaped scenarios delegate\n",
+    "    # to build_matrix_atomic_attacks; pass display_group_fn to customize result grouping\n",
+    "    # (default groups by technique; here we group by dataset instead).\n",
+    "    async def _build_atomic_attacks_async(self, *, context):\n",
+    "        return build_matrix_atomic_attacks(\n",
+    "            context=context,\n",
+    "            objective_scorer=self._objective_scorer,\n",
+    "            display_group_fn=lambda combo: combo.dataset_name,\n",
+    "        )"
    ]
   },
   {
@@ -337,7 +339,7 @@
       "      ``TargetRegistry`` — typically by ``TargetInitializer`` from\n",
       "      ``ADVERSARIAL_CHAT_*`` env vars, or programmatically via\n",
       "      ``TargetRegistry.get_registry_singleton().instances.register``. At run\n",
-      "      time, ``_get_atomic_attacks_async`` performs the ``(technique ×\n",
+      "      time, ``_build_atomic_attacks_async`` performs the ``(technique ×\n",
       "      adversarial_target × dataset)`` cross-product: for each selected\n",
       "      adversarial-capable ``core`` factory in the ``AttackTechniqueRegistry``\n",
       "      and each requested target, it calls\n",

doc/code/scenarios/0_scenarios.py

@@ -59,13 +59,12 @@
 #    - Each enum member represents an **attack technique** (the *how* of an attack)
 #    - Each member is defined as `(value, tags)` where value is a string and tags is a set of strings
 #    - Include an `ALL` aggregate strategy that expands to all available strategies
-#    - Optionally override `_prepare_strategies()` for custom composition logic (see `FoundryComposite`)
 #
 # 2. **Scenario Class**: Extend `Scenario` and pass these to `super().__init__()`:
 #    - `strategy_class`: Your strategy enum class
 #    - `default_strategy`: The default strategy (typically `YourStrategy.ALL` or `YourStrategy.DEFAULT`)
-#    - The base class provides a default `_get_atomic_attacks_async()` that uses the factory/registry
-#      pattern. Override it only if your scenario needs custom attack construction logic.
+#    - Implement `_build_atomic_attacks_async(context)` — the single abstract extension point.
+#      Matrix-shaped scenarios delegate to `build_matrix_atomic_attacks(context=...)` in one line.
 #
 # 3. **Default Dataset**: Pass `default_dataset_config=` to `super().__init__()` to specify the datasets your scenario uses out of the box.
 #    - Returns a `DatasetConfiguration` with one or more named datasets (e.g., `DatasetConfiguration(dataset_names=["my_dataset"])`)
@@ -91,9 +90,10 @@
 #
 # ### Example Structure
 #
-# The simplest approach uses the **factory/registry pattern**: define your strategy,
-# dataset config, and constructor — the base class handles building atomic attacks
-# automatically from registered attack techniques.
+# The construction path: define your strategy, dataset config, and constructor, then
+# implement `_build_atomic_attacks_async(context)`. Matrix-shaped scenarios delegate to the
+# `build_matrix_atomic_attacks` helper, which builds atomic attacks automatically from the
+# registered attack techniques.
 # %%
 
 from pyrit.common import apply_defaults
@@ -102,6 +102,7 @@
     Scenario,
     ScenarioStrategy,
 )
+from pyrit.scenario.core.matrix_atomic_attack_builder import build_matrix_atomic_attacks
 from pyrit.score.true_false.true_false_scorer import TrueFalseScorer
 from pyrit.setup import initialize_pyrit_async
 from pyrit.setup.initializers.components import ScenarioTechniqueInitializer
@@ -144,14 +145,15 @@ def __init__(
             scenario_result_id=scenario_result_id,
         )
 
-    # Optional: override _build_display_group to customize result grouping.
-    # Default groups by technique name; override to group by dataset instead:
-    def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:
-        return seed_group_name
-
-    # No _get_atomic_attacks_async override needed!
-    # The base class builds attacks from the (technique x dataset) cross-product
-    # using the factory/registry pattern automatically.
+    # Implement the single abstract extension point. Matrix-shaped scenarios delegate
+    # to build_matrix_atomic_attacks; pass display_group_fn to customize result grouping
+    # (default groups by technique; here we group by dataset instead).
+    async def _build_atomic_attacks_async(self, *, context):
+        return build_matrix_atomic_attacks(
+            context=context,
+            objective_scorer=self._objective_scorer,
+            display_group_fn=lambda combo: combo.dataset_name,
+        )
 
 
 # %% [markdown]

doc/scanner/1_pyrit_scan.ipynb

@@ -242,9 +242,10 @@
     "        )\n",
     "        # ... your scenario-specific initialization code\n",
     "\n",
-    "    async def _get_atomic_attacks_async(self):\n",
-    "        # Override only if your scenario needs custom attack construction.\n",
-    "        # The base class provides a default that uses the factory/registry pattern.\n",
+    "    async def _build_atomic_attacks_async(self, *, context):\n",
+    "        # The single abstract extension point every scenario implements.\n",
+    "        # Read runtime inputs from `context`; return the list of AtomicAttack to run.\n",
+    "        # Matrix-shaped scenarios can delegate to build_matrix_atomic_attacks(context=...).\n",
     "        # Example: create attacks for each strategy composite\n",
     "        return []\n",
     "\n",

doc/scanner/1_pyrit_scan.py

@@ -180,9 +180,10 @@ def __init__(self, *, scenario_result_id=None, **kwargs):
         )
         # ... your scenario-specific initialization code
 
-    async def _get_atomic_attacks_async(self):
-        # Override only if your scenario needs custom attack construction.
-        # The base class provides a default that uses the factory/registry pattern.
+    async def _build_atomic_attacks_async(self, *, context):
+        # The single abstract extension point every scenario implements.
+        # Read runtime inputs from `context`; return the list of AtomicAttack to run.
+        # Matrix-shaped scenarios can delegate to build_matrix_atomic_attacks(context=...).
         # Example: create attacks for each strategy composite
         return []