Skip to content

Commit e0df568

Browse files
committed
Merge remote-tracking branch 'origin/main' into rlundeen2-cleanup-initializers
# Conflicts: # .pyrit_conf_example # doc/getting_started/pyrit_conf.md # doc/scanner/pyrit_conf.yaml # pyrit/registry/class_registries/initializer_registry.py # pyrit/setup/initializers/components/scenario_techniques.py # pyrit/setup/initializers/scenarios/load_default_datasets.py # tests/unit/registry/test_attack_technique_registry.py # tests/unit/scenario/airt/test_cyber.py # tests/unit/scenario/airt/test_rapid_response.py # tests/unit/scenario/core/test_scenario_strategy_invariants.py # tests/unit/setup/test_load_default_datasets.py # tests/unit/setup/test_preload_scenario_metadata.py # tests/unit/setup/test_technique_initializer.py
2 parents e0117a6 + 6f1aa95 commit e0df568

271 files changed

Lines changed: 13313 additions & 5680 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/copilot-instructions.md

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,16 @@ PyRIT (Python Risk Identification Tool for generative AI) is an open-source fram
44

55
## Architecture
66

7-
PyRIT uses a modular pluggable-brick design. The main extensibility points are:
7+
PyRIT uses a modular pluggable-brick design.
88

9-
- **Prompt Converters** (`pyrit/prompt_converter/`) — Transform prompts (70+ implementations). Base: `PromptConverter`.
10-
- **Scorers** (`pyrit/score/`) — Evaluate responses. Base: `Scorer`.
11-
- **Prompt Targets** (`pyrit/prompt_target/`) — Send prompts to LLMs/APIs. Base: `PromptTarget`.
12-
- **Executors / Scenarios** (`pyrit/executor/`, `pyrit/scenario/`) — Orchestrate multi-turn attacks.
13-
- **Memory** (`pyrit/memory/`) — `CentralMemory` for prompt/response persistence.
9+
**[`doc/code/framework.md`](../doc/code/framework.md) is the canonical reference for how these pieces fit together.** It defines each component's responsibilities — what it owns and, critically, what it *does not* own — and how scenarios, attack techniques, executors, and the core/shared layers relate. Read it before adding or reviewing components so new code lands in the right place.
1410

1511
## Code Review Guidelines
1612

1713
When performing a code review, be selective. Only leave comments for issues that genuinely matter:
1814

19-
- Bugs, logic errors, or security concerns
15+
- Bugs, correctness, logic errors, or security concerns
16+
- **Component responsibilities** — Each component should do its job and *only* its job, per [`doc/code/framework.md`](../doc/code/framework.md). Flag responsibility bleed: e.g. an executor assembling prepended/system prompts or role-play framing (that's an attack technique), a converter or target making branching decisions (that's an attack/scorer), a scorer acting on its own result (the attack branches), or business logic living in memory/output. If logic belongs in a different brick, say so.
2017
- Unclear code that would benefit from refactoring for readability
2118
- Violations of the critical coding conventions above (async suffix, keyword-only args, type annotations)
2219

.github/instructions/attacks.instructions.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ applyTo: "pyrit/executor/attack/**"
66

77
`AttackStrategy` subclasses (single-turn attacks like `PromptSendingAttack`, multi-turn attacks like `RedTeamingAttack`, etc.) are pluggable bricks orchestrated by `AttackExecutor` and the `Scenario` framework. Style rules from `style-guide.instructions.md` (async `_async` suffix, keyword-only args, type hints, enums-over-Literals) still apply and are not repeated here.
88

9+
**Does not own** (see [framework.md](../../doc/code/framework.md)): packaging the attack. Prepended/system prompts, role-play framing, the converter stack, and dataset selection are passed in as configuration by the **attack technique** — an attack must accept them as parameters, not assemble them itself (e.g. `RolePlayAttack` building its own prompt scaffolding is attack-technique work bleeding into the executor). It also must not branch on raw responses (use a scorer), construct its own components (use the registry), or format/persist results itself (output/memory). Flag such bleed in review.
10+
911
## Constructor contract
1012

1113
`AttackStrategy` subclasses MUST follow the keyword-only constructor shape:

.github/instructions/converters.instructions.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ applyTo: "pyrit/prompt_converter/**"
44

55
# Prompt Converter Development Guidelines
66

7+
**Responsibility**: A converter transforms a prompt into something else (rephrasing, encoding, translating to a Word document, overlaying text on an image, ...). Converters can be stacked and combined, and any converter may also be a NoOp.
8+
9+
**Does not own** (see [framework.md](../../doc/code/framework.md)): conversation state or attack decisions. A converter transforms input into output (and may call a target to do so); it must not branch on results, score, persist to memory itself, or decide when it runs — the attack/technique configures the stack. Flag such bleed in review.
10+
711
## Base Class Contract
812

913
All converters MUST inherit from `PromptConverter` and implement:

.github/instructions/datasets.instructions.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ applyTo: "pyrit/datasets/seed_datasets/**"
44

55
# Seed Dataset Loader Guidelines
66

7+
**Responsibility**: Seed dataset loaders (`SeedDatasetProvider` subclasses) are the single place to manage the prompts/objectives for a source. They load seeds into `CentralMemory`; components then retrieve seeds from memory — components never read from a loader directly.
8+
9+
**Does not own** (see [framework.md](../../doc/code/framework.md)): a loader defines and holds seeds; it must not select or combine which seeds an attack uses (that's a scenario/attack technique) or render/parameterize prompts at send time (converters/normalizers). Flag such bleed in review.
10+
711
These rules apply when adding or modifying loaders under `pyrit/datasets/seed_datasets/`.
812
Style rules from `style-guide.instructions.md` (async `_async` suffix, keyword-only args, type hints, enums-over-Literals) still apply and are not repeated here.
913

.github/instructions/models.instructions.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ applyTo: "pyrit/models/**"
44

55
# `pyrit.models` Guidelines
66

7+
**Responsibility**: `pyrit.models` is the lightweight, canonical data layer — the core types shared across components (and preferred in REST) so representations don't drift. It depends only on lightweight Python (the standard library and pydantic) and `pyrit.common`.
8+
79
## Import Boundary
810

911
PyRIT enforces a two-layer rule for its foundational packages. `pyrit.common`

.github/instructions/output.instructions.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ For full architecture documentation, usage examples, and extension guides, see [
88

99
This file covers the rules for **writing and reviewing** code in `pyrit/output/`.
1010

11+
**Does not own** (see [framework.md](../../doc/code/framework.md)): deciding *what* to render or *when*. Components hand results to output; format classes only turn data into strings and must never fetch data, touch `CentralMemory`, or call `print()` directly (that's isolated to leaf printer classes). Flag such bleed in review.
12+
1113
## Critical Rules
1214

1315
### Output goes through the sink — never call `print()` directly

.github/instructions/scenarios.instructions.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ applyTo: "pyrit/scenario/**"
66

77
Scenarios orchestrate multi-attack security testing campaigns. Each scenario groups `AtomicAttack` instances and executes them sequentially against a target.
88

9+
**Does not own** (see [framework.md](../../doc/code/framework.md)): the per-objective conversation logic. Branching, turn-by-turn adaptation, and scoring-based decisions belong to the attack — a scenario selects and packages existing attack techniques and owns parallelism/resiliency, not new attack algorithms or datasets. Flag such bleed in review.
10+
911
## Base Class Contract
1012

1113
All scenarios inherit from `Scenario` (ABC) and must:
@@ -161,6 +163,8 @@ The default implementation:
161163
`AttackTechniqueRegistry` singleton)
162164
2. Iterates over every (technique × dataset) pair from `self._dataset_config`
163165
3. Calls `factory.create()` with `objective_target` and conditional scorer override
166+
(also forwards any per-technique converters from `self._strategy_converters`, populated
167+
from the CLI `--strategies <technique>:converter.<name>` modifier, as `extra_request_converters`)
164168
4. Uses `self._build_display_group()` for user-facing grouping
165169
5. Builds `AtomicAttack` with unique `atomic_attack_name` = `"{technique}_{dataset}"`
166170

.github/instructions/scorers.instructions.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ applyTo: "pyrit/score/**"
66

77
Scorers evaluate model responses against an objective and live under `pyrit/score/`. Style rules from `style-guide.instructions.md` (async `_async` suffix, keyword-only args, type hints, enums-over-Literals) still apply and are not repeated here.
88

9+
**Does not own** (see [framework.md](../../doc/code/framework.md)): acting on its own result. A scorer evaluates a response and returns a score; branching on that score is the attack's job and aggregating scores across runs is analytics'. It may call a target to evaluate, but must not send the attack's objective prompt or manage the conversation. Flag such bleed in review.
10+
911
## Constructor contract
1012

1113
`Scorer` subclasses MUST use the keyword-only constructor shape:

.github/instructions/style-guide.instructions.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -481,10 +481,19 @@ async def temporary_config(self, **kwargs):
481481
### Property Decorators
482482
- Use @property for simple computed attributes
483483
- Use explicit getter/setter methods for complex logic
484+
- Property docstrings must be **noun phrases** describing the value (e.g.
485+
`"""The display name."""`), not verb phrases (e.g. `"""Return the display
486+
name."""`). This is enforced by Ruff `D421` (property-docstring-starts-with-verb).
484487

485488
```python
486489
# CORRECT
487490
@property
491+
def is_complete(self) -> bool:
492+
"""Whether the attack is complete."""
493+
return self._status == AttackStatus.COMPLETE
494+
495+
# INCORRECT - verb-phrase docstring, flagged by Ruff D421
496+
@property
488497
def is_complete(self) -> bool:
489498
"""Check if the attack is complete."""
490499
return self._status == AttackStatus.COMPLETE

.github/instructions/targets.instructions.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ applyTo: "pyrit/prompt_target/**"
44

55
# Prompt Target Development Guidelines
66

7+
**Responsibility**: A prompt target is "the thing we're sending the prompt to" — often an LLM, but it can be any endpoint (e.g. a storage account for cross-domain prompt injection). Targets use `message_normalizer` together with `TargetConfiguration` to transform `Message`s into the format the target supports.
8+
9+
**Does not own** (see [framework.md](../../doc/code/framework.md)): what to send or what to do with the response. A target sends a prepared `Message` and returns a response; it must not convert prompts (converters), score (scorers), or manage the conversation / decide the next turn (attacks). Flag such bleed in review.
10+
711
## Base Class Contract
812

913
All targets MUST inherit from ``PromptTarget`` (or one of its public

0 commit comments

Comments
 (0)