Implement DORA Metrics and Engineering Effectiveness Tracking

[jwilger]

Priority: 🟡 HIGH

Problem

No capability to measure DORA metrics for the Union Square development team's effectiveness. This creates a blind spot in understanding and improving our open source project's development velocity.

Clarification

This is about measuring the effectiveness of the Union Square development team, NOT about features for end users. Union Square is an open source proxy server - these metrics help us improve our development and release process.

Expert Analysis (Nicole Forsgren - Engineering Effectiveness Expert)

Current state assessment:

• Release Frequency: ❌ Not measurable - no crates.io release tracking
• Lead Time for Changes: ⚠️ Partially measurable - PR lifecycle only
• Mean Time to Recovery: ❌ Not measurable - no issue resolution tracking
• Change Failure Rate: ⚠️ Partially measurable - CI failures only

Missing infrastructure for measuring developer productivity and release effectiveness.

Required Implementation

1. Release Metrics Collection

# Track crates.io releases and GitHub binary releases
- name: Track Release
  uses: your-org/release-metrics-action@v1
  with:
    release_version: ${{ github.ref }}
    release_type: crates_io
    
# Track time from PR merge to release
- name: Calculate Lead Time
  run: |
    MERGE_TIME=$(gh pr view $PR --json mergedAt -q .mergedAt)
    RELEASE_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
    # Calculate and report lead time

2. Development Velocity Tracking

• PR cycle time (creation to merge)
• Issue resolution time
• Release frequency to crates.io
• Version adoption metrics

3. CI/CD Performance Optimization

Quick wins identified:

• Pre-build CI container images (3-5 min saved)
• Enable test caching with cargo-nextest
• Run only affected tests on PRs
• Implement incremental coverage

4. Developer Experience Metrics

# .cargo/config.toml
[alias]
precommit = "run --package precommit-runner --"
quick-check = "clippy --all-targets --no-deps -- -W clippy::all"
test-changed = "nextest run --changed-since HEAD~1"

5. Project Health Dashboard

Create dashboards showing:

• Release frequency to crates.io
• Time from PR to release
• CI pipeline performance
• Test execution times
• Issue resolution velocity
• Community contribution metrics

Implementation Priority

Phase 1 (Days):

• Add release tracking for crates.io
• Implement basic metrics collection
• Create CI performance baseline

Phase 2 (Weeks):

• Full DORA implementation for the project
• Incremental test/coverage system
• Remote build caching setup

Phase 3 (Month):

• Contributor experience tracking
• Automated project health reports
• Release cadence optimization

Success Criteria

• Track releases to crates.io and binaries
• Measure time from PR merge to release
• CI pipeline < 10 minutes for PRs
• Automated release process
• Project health dashboards available

SLOs for Union Square Development

• PR checks complete < 10 minutes
• Local test suite < 30 seconds
• Release to crates.io < 1 hour after merge
• Binary builds complete < 30 minutes

References

• Accelerate (book by Nicole Forsgren, Jez Humble, Gene Kim)
• DORA State of DevOps reports
• Open source project best practices