diff --git a/reflexio/server/README.md b/reflexio/server/README.md index 677be741..e1f1841a 100644 --- a/reflexio/server/README.md +++ b/reflexio/server/README.md @@ -424,13 +424,14 @@ Key files: - `agent_success_evaluation_constants.py`: Output schema (`AgentSuccessEvaluationOutput`) - `agent_success_evaluation_utils.py`: Message construction utilities - `scheduler.py`: `GroupEvaluationScheduler` singleton - min-heap priority queue with daemon thread, defers evaluation until 10 min after last request in session +- `sampling.py`: deterministic per-session sampling for the session-success and retrieved-learning judge families; the publish scheduler samples once and passes explicit booleans to the runner - `runner.py`: `run_group_evaluation()` - fetches all requests/interactions for a session, builds `RequestInteractionDataModel` list, runs `service.py`, then the retrieved-learning phase; returns `GroupEvaluationOutcome` (per-family statuses) - `components/retrieved_learning_evaluator.py`: `RetrievedLearningEvaluator` - per-learning relevance/impact judges over `Interaction.retrieved_learnings`; results replace the session's `retrieved_learning_evaluation` snapshot atomically (generation + session-fingerprint fenced, see `services/storage/storage_base/retrieved_learning_state.py`) - `services/storage/storage_base/evaluation_state_keys.py`: single source of truth for the three evaluation `_operation_state` key formats (agent-success marker, grade-on-demand cache, retrieved-learning state) shared by producers and governance erasure **Flow**: Interactions → `agent_success_evaluation/scheduler.py` → `agent_success_evaluation/runner.py` → `agent_success_evaluation/service.py` → `agent_success_evaluation/components/evaluator.py` → `AgentSuccessEvaluationResult` → Storage -**Session-Level Evaluation**: Evaluator treats one user's `request_interaction_data_models` in a session as a single conversation. Sampling rate checked once per session (not per-request). Results are keyed by `(user_id, session_id, evaluation_name)` so reused session IDs across users do not clobber each other. +**Session-Level Evaluation**: Evaluator treats one user's `request_interaction_data_models` in a session as a single conversation. Sampling is checked once per session (not per-request): `sampling_rate` controls session-success, `evaluation_only_sampling_rate` can override evaluation-only publishes, and `retrieved_learning_sampling_rate` lets retrieved-learning judges run at a separate density while inheriting `sampling_rate` when unset. Results are keyed by `(user_id, session_id, evaluation_name)` so reused session IDs across users do not clobber each other. **Tool Context**: Reads `tool_can_use` from root `Config` level (shared with playbook extraction). diff --git a/reflexio/server/services/README.md b/reflexio/server/services/README.md index 002a90b4..e9c8c08f 100644 --- a/reflexio/server/services/README.md +++ b/reflexio/server/services/README.md @@ -35,7 +35,7 @@ strings before deleting old import paths in the same PR. |-----------|-------------|-----------| | `profile/` | `ProfileGenerationService` | `service.py`, `components/extractor.py`, `components/consolidator.py` | | `playbook/` | `PlaybookGenerationService` | `components/extractor.py`, `components/consolidator.py`, `components/aggregator.py` (cluster-fingerprint change detection) — has its own [README](playbook/README.md) | -| `agent_success_evaluation/` | `AgentSuccessEvaluationService` | `service.py` (session-level service), `runner.py` (`run_group_evaluation`), `scheduler.py` (`GroupEvaluationScheduler`, 10-min defer), `regen_jobs.py`, `components/evaluator.py` | +| `agent_success_evaluation/` | `AgentSuccessEvaluationService` | `service.py` (session-level service), `runner.py` (`run_group_evaluation`), `scheduler.py` (`GroupEvaluationScheduler`, 10-min defer), `sampling.py` (per-family session gates), `regen_jobs.py`, `components/evaluator.py` | ## Durable and Async Extraction diff --git a/reflexio/server/services/agent_success_evaluation/README.md b/reflexio/server/services/agent_success_evaluation/README.md index 7844868d..12ded390 100644 --- a/reflexio/server/services/agent_success_evaluation/README.md +++ b/reflexio/server/services/agent_success_evaluation/README.md @@ -7,6 +7,7 @@ Session-level agent success evaluation module. - `service.py`: `AgentSuccessEvaluationService`, the request-path service that runs configured evaluators and saves result rows. - `runner.py`: `run_group_evaluation(...)`, the background/manual workflow entry point that loads a session, runs the service, and marks operation state. After agent-success work it also runs the retrieved-learning evaluation (independent completion; generation + session-fingerprint fenced) and returns a `GroupEvaluationOutcome` carrying both statuses. - `scheduler.py`: `GroupEvaluationScheduler`, the deferred inactivity scheduler. +- `sampling.py`: deterministic per-session sampling gates for the session-success and retrieved-learning judge families. `generation_service.py` is the only scheduled-publish caller; it samples once, admits the session when either family is selected, and passes booleans to the runner. - `components/evaluator.py`: `AgentSuccessEvaluator`, the LLM evaluator component. - `components/retrieved_learning_evaluator.py`: `RetrievedLearningEvaluator`, per-learning relevance/impact judges over `Interaction.retrieved_learnings`; results go to the `retrieved_learning_evaluation` table. - `regen_jobs.py`: regeneration job planning and execution; remains root-level because API/admin regenerate flows import it directly. @@ -14,6 +15,12 @@ Session-level agent success evaluation module. - `agent_success_evaluation_constants.py`: prompt/model output constants. - `agent_success_evaluation_utils.py`: request DTO and prompt-message construction helpers. +## Sampling Contract + +- `AgentSuccessConfig.sampling_rate` controls the session-success judge; `evaluation_only_sampling_rate` can override it for evaluation-only publishes. +- `AgentSuccessConfig.retrieved_learning_sampling_rate` controls the retrieved-learning relevance/impact judges and inherits `sampling_rate` when unset. +- Direct runner callers (regen jobs and on-demand grade routes) leave the sampling booleans at their defaults and run both families; sampling remains a scheduler/publish decision. + ## Prompt IDs - Owns `agent_success_evaluation`, `retrieved_learning_relevance`, and `retrieved_learning_impact`.