Skip to content

signals: follow-ups deferred from PR #903 #909

@syedhashmi

Description

@syedhashmi

Title

signals: follow-ups deferred from PR #903

Summary

PR #903 landed the Rust port of the Signals paper taxonomy into brightstaff
with 100% parity against the Python reference on 2,000 lmsys-chat-1m samples.
Three items came up during review that were intentionally scoped out of that
PR. This issue tracks them so they don't drop on the floor.

Follow-ups

1. Documentation for the new OTel attribute set — P1

The layered attribute keys (signals.interaction.misalignment.count,
signals.execution.failure.severity, etc.) and per-instance span events
(signal.<type> with confidence/snippet/metadata) added in #903 are not yet
documented. Legacy keys (signals.frustration.*, signals.escalation.requested,
etc.) are being kept in dual-emit for backward compatibility and should be
flagged as deprecated in the docs.

Acceptance:

  • docs/source/guides/observability/ has a signals attributes reference
    listing all layered keys with descriptions and example values
  • Legacy keys section marks each as deprecated with a migration pointer
  • docs/source/observability/monitoring.rst lead example updated to show
    layered keys

2. Upstream Python reference: grouped summary format — P3, optional

In #903 the Rust generate_summary was refactored to produce a
taxonomy-grouped output. The Python reference at katanemo/signals
still emits the flat format. The parity comparator classifies summary
as Tier-C so this is not a correctness issue, but it's the one known
intentional divergence and worth aligning upstream.

Acceptance:

  • Open a PR against katanemo/signals porting the grouped-summary
    format (same layer headers, same severity notation, same alert tail)
  • Re-run the parity harness and confirm Tier-C divergence drops to zero

3. Legacy OTel key sunset — P3

Decided offline during #903 review to keep signals.frustration.*,
signals.escalation.requested, signals.follow_up.repair.*,
signals.positive_feedback.count, signals.repetition.count for one
release for backward compatibility. This issue tracks the eventual removal.

Acceptance:

  • Identify all internal consumers of legacy keys (dashboards, alerts,
    samplers, any downstream ingest)
  • Migrate each consumer to the layered keys
  • Remove emit_legacy_attributes from crates/brightstaff/src/signals/otel.rs
  • Remove legacy constants from crates/brightstaff/src/tracing/constants.rs
  • Bump version per the release process in CLAUDE.md

Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions