Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions worker_plan/worker_plan_internal/self_audit/self_audit.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,26 @@

Value: It serves as a prioritized "fix-it" list. It tells the project manager which fires are the biggest. They don't need to worry about the team size (a ⚠️ Medium risk) if the entire project is a 🛑 High "Legal Minefield." It focuses their attention on the foundational, existential threats to the project's success.


The "violates known physics" detection is often gets triggered, freaking out about "faster than light travel" on documents that have nothing to do with FTL.
The "elephant-alpha" model especially struggle with detecting that, when it otherwise does an ok job at everything else.
I'm considering making a dedicated system prompt only for VIOLATES_KNOWN_PHYSICS, that is have lower rate of false positives.

Adding a new checklist item:
Find fabricated evidence. Where things aren't true.
When it hallucinates laws or articles that are non-existing.

Adding a new checklist item:
Calling out fake confidence in the plan, where there isn’t sufficient evidence.

Adding a new checklist item:
attack on vague filler language.
It's filler language when the document repeatedly says
- “develop a robust strategy”
- “implement a communication strategy,”
Without specify the actual strategy.
that is placeholder language.

Calibation. Skewed distribution: 16 “High” flags out of 20 reads like alarm fatigue.
Yes, the majority may be red. Holding the items up against each other is not the job for this checklist. It's further downstream.
I'm not going to reserve High for the most significant issues.
Expand Down