feat: add Agent Reliability Score & Trust Insights - #820
Conversation
Adds a lightweight, client-side reliability score (0-100) for each agent, computed from existing local signals: user ratings (up/down feedback), usage frequency (run count via analytics events), and recency of last use. No ML or external calls required. - useReliabilityScore: heuristic scoring hook with diminishing-returns normalization for activity and a recency decay curve - ReliabilityBadge: compact badge (agent cards) and full breakdown card (agent detail page) with score, trust level, progress bar, and rating/run counts - Wired into AgentCard (grid) and AgentRunner (detail page header) Closes AditthyaSS#617 Signed-off-by: aaniya22 <aaniyaatomar@gmail.com>
|
@aaniya22 is attempting to deploy a commit to the aditthyass' projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
Note
|
| Layer / File(s) | Summary |
|---|---|
Reliability score computation src/lib/useReliabilityScore.js |
Computes a memoized 0–100 score from rating feedback, run activity, and recency. It derives High, Medium, or Low trust levels. |
Trust badge presentation and agent card wiring src/components/ReliabilityBadge.jsx, src/components/AgentCard.jsx |
Adds compact and full badge variants. Agent cards display the compact badge with provider information. |
Runner state and scheduling updates src/components/AgentRunner.jsx |
Adds version input snapshots, input sanitization, updated error handling, sanitized history restoration, and expanded scheduled-job payload data. |
Runner reliability and input UI src/components/AgentRunner.jsx |
Adds the reliability section, character-limit enforcement, and reorganized runner controls and output rendering. |
Estimated code review effort: 4 (Complex) | ~60 minutes
Sequence Diagram(s)
sequenceDiagram
participant AgentSurface
participant ReliabilityBadge
participant useReliabilityScore
participant AgentRatings
participant Analytics
AgentSurface->>ReliabilityBadge: render with agentId
ReliabilityBadge->>useReliabilityScore: compute reliability data
useReliabilityScore->>AgentRatings: read rating information
useReliabilityScore->>Analytics: read agent events
AgentRatings-->>useReliabilityScore: rating information
Analytics-->>useReliabilityScore: run count and timestamps
useReliabilityScore-->>ReliabilityBadge: score and trust level
ReliabilityBadge-->>AgentSurface: render compact or full badge
Possibly related PRs
- AditthyaSS/iloveAgents#781 — Provides the agent-rating persistence consumed by
useReliabilityScore. - AditthyaSS/iloveAgents#704 — Also modifies
AgentRunner.jsxto enforce a 4,000-character textarea limit. - AditthyaSS/iloveAgents#744 — Also modifies version-history snapshot and restoration logic in
AgentRunner.jsx.
Suggested labels: level:intermediate, type:feature
Suggested reviewers: aditthyass
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Out of Scope Changes check | AgentRunner removes prompt-history controls and includes a broad reorganization unrelated to the reliability feature. | Move prompt-history removal and unrelated AgentRunner refactoring to a separate PR; limit this PR to reliability scoring and UI integration. |
✅ Passed checks (4 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | The title clearly summarizes the added Agent Reliability Score and Trust Insights feature. |
| Linked Issues check | ✅ Passed | The PR implements the core objectives in issue #617 with heuristic scoring, trust badges, metrics, and homepage and detail-page integration. |
| Docstring Coverage | ✅ Passed | Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. |
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
- Create stacked PR
- Commit on current branch
🧪 Generate unit tests (beta)
- Create PR with unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
|
Hey @aaniya22! 👋
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
src/components/AgentRunner.jsx (1)
250-262: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
newVersionis constructed but never used; logic is duplicated inline.Lines 250-254 build
newVersion, butsetVersionHistoryon lines 255-262 re-creates an equivalent object literal, leavingnewVersiondead. Also notetimestampis captured twice via separatenew Date().toLocaleTimeString()calls. Reuse the object and take a single timestamp.Separately, verify version history is intended to persist across agent switches: the
[agent.id]reset effect (lines 117-153) resets inputs/output/error but never clearsversionHistory, so ifAgentRunnerisn't remounted per route the previous agent's versions remain visible.♻️ Reuse the constructed object
const newVersion = { versionNumber: versionHistory.length + 1, timestamp: new Date().toLocaleTimeString(), configSnapshot: { ...inputs }, }; - setVersionHistory((prevHistory) => [ - { - versionNumber: prevHistory.length + 1, - timestamp: new Date().toLocaleTimeString(), - configSnapshot: { ...inputs }, - }, - ...prevHistory, - ]); + setVersionHistory((prevHistory) => [newVersion, ...prevHistory]);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/components/AgentRunner.jsx` around lines 250 - 262, Update the version-history creation flow in AgentRunner to use the constructed newVersion object rather than duplicating its literal inside setVersionHistory, and capture one timestamp for that object. Also inspect the [agent.id] reset effect and clear versionHistory when switching agents so prior agent versions do not remain visible.src/lib/useReliabilityScore.js (1)
41-51: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoffPer-badge analytics subscription and full-event scan may not scale on the agent grid.
Each
ReliabilityBadgerenders this hook, and every instance callsuseAnalytics('all')(its own state +windowlisteners) and then filters the entire events array byagentId. On the homepage grid with many agent cards, this means N independent subscriptions plus N full-array scans re-running on everyila_analytics_update/storageevent. Consider lifting analytics to a shared context/provider or precomputing a per-agent index once, then passing counts down to the badges.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/useReliabilityScore.js` around lines 41 - 51, The useReliabilityScore hook creates one useAnalytics('all') subscription and full events-array scan per badge; centralize analytics state and indexing for the grid. Update useReliabilityScore and its consumers to reuse a shared provider or memoized per-agent event index, and pass agent-specific counts/data to badges instead of filtering all events independently on each update.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/components/AgentRunner.jsx`:
- Around line 1012-1017: Update the button label in the AgentRunner reload
button to use the action-oriented text “Reload page” instead of describing the
behavior. Keep the existing click handler and styling unchanged.
- Around line 592-599: Update the text input and voice onChange handlers in
AgentRunner so values exceeding MAX_CHAR_LIMIT are truncated to that limit
before calling updateInput, rather than rejecting the entire update. Preserve
the existing height recalculation and ensure the character counter reflects the
truncated value.
---
Nitpick comments:
In `@src/components/AgentRunner.jsx`:
- Around line 250-262: Update the version-history creation flow in AgentRunner
to use the constructed newVersion object rather than duplicating its literal
inside setVersionHistory, and capture one timestamp for that object. Also
inspect the [agent.id] reset effect and clear versionHistory when switching
agents so prior agent versions do not remain visible.
In `@src/lib/useReliabilityScore.js`:
- Around line 41-51: The useReliabilityScore hook creates one
useAnalytics('all') subscription and full events-array scan per badge;
centralize analytics state and indexing for the grid. Update useReliabilityScore
and its consumers to reuse a shared provider or memoized per-agent event index,
and pass agent-specific counts/data to badges instead of filtering all events
independently on each update.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e6c95121-64ee-4db3-83e0-53da3500e608
📒 Files selected for processing (4)
src/components/AgentCard.jsxsrc/components/AgentRunner.jsxsrc/components/ReliabilityBadge.jsxsrc/lib/useReliabilityScore.js
| onChange={(e) => { | ||
| if (e.target.value.length <= MAX_CHAR_LIMIT) { | ||
| updateInput(input.id, e.target.value); | ||
|
|
||
| e.target.style.height = "auto"; | ||
| e.target.style.height = `${e.target.scrollHeight}px`; | ||
| } | ||
| }} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Over-limit input is silently dropped instead of truncated.
When pasted/voice text exceeds MAX_CHAR_LIMIT, the guard rejects the whole update, so the field stays unchanged with no feedback. Truncating to the limit is friendlier and keeps the counter accurate. Same applies to the voice onChange at lines 609-612.
✂️ Truncate to limit
onChange={(e) => {
- if (e.target.value.length <= MAX_CHAR_LIMIT) {
- updateInput(input.id, e.target.value);
-
- e.target.style.height = "auto";
- e.target.style.height = `${e.target.scrollHeight}px`;
- }
+ updateInput(input.id, e.target.value.slice(0, MAX_CHAR_LIMIT));
+ e.target.style.height = "auto";
+ e.target.style.height = `${e.target.scrollHeight}px`;
}}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| onChange={(e) => { | |
| if (e.target.value.length <= MAX_CHAR_LIMIT) { | |
| updateInput(input.id, e.target.value); | |
| e.target.style.height = "auto"; | |
| e.target.style.height = `${e.target.scrollHeight}px`; | |
| } | |
| }} | |
| onChange={(e) => { | |
| updateInput(input.id, e.target.value.slice(0, MAX_CHAR_LIMIT)); | |
| e.target.style.height = "auto"; | |
| e.target.style.height = `${e.target.scrollHeight}px`; | |
| }} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/components/AgentRunner.jsx` around lines 592 - 599, Update the text input
and voice onChange handlers in AgentRunner so values exceeding MAX_CHAR_LIMIT
are truncated to that limit before calling updateInput, rather than rejecting
the entire update. Preserve the existing height recalculation and ensure the
character counter reflects the truncated value.
✅ Addressed in commit bb6a9a4
There was a problem hiding this comment.
Updated both the textarea and voice-input onChange handlers to truncate to MAX_CHAR_LIMIT instead of rejecting the whole update. The counter now stays accurate even when pasting text beyond the limit. Thanks!
There was a problem hiding this comment.
@aaniya22, confirmed — both handlers now truncate values to MAX_CHAR_LIMIT, so pasted or dictated over-limit text is retained up to the limit and the counter remains consistent. Thanks!
🐇✨
If you found this review helpful, would you consider giving us a shout-out on X?
Thank you for using CodeRabbit!
| <button | ||
| className="underline text-accent" | ||
| onClick={() => window.location.reload()} | ||
| > | ||
| Reloads page after an invalid API key error | ||
| </button> |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Button label reads as a description, not an action.
The reload button's text "Reloads page after an invalid API key error" is a behavior description rather than a label. Use an action label such as "Reload page".
✏️ Fix label
- Reloads page after an invalid API key error
+ Reload page📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| <button | |
| className="underline text-accent" | |
| onClick={() => window.location.reload()} | |
| > | |
| Reloads page after an invalid API key error | |
| </button> | |
| <button | |
| className="underline text-accent" | |
| onClick={() => window.location.reload()} | |
| > | |
| Reload page | |
| </button> |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/components/AgentRunner.jsx` around lines 1012 - 1017, Update the button
label in the AgentRunner reload button to use the action-oriented text “Reload
page” instead of describing the behavior. Keep the existing click handler and
styling unchanged.
✅ Addressed in commit bb6a9a4
There was a problem hiding this comment.
Already using "Reload page" as the label in the latest commit. Resolving.
There was a problem hiding this comment.
@aaniya22, confirmed—the button now uses the action-oriented label “Reload page”. Thanks for fixing it.
If you found this review helpful, would you consider giving us a shout-out on X?
Thank you for using CodeRabbit!
- Textarea onChange and VoiceInput onChange now truncate to MAX_CHAR_LIMIT instead of rejecting the entire update when the input exceeds the limit - Keeps the character counter accurate and gives the user visible feedback
|
|
|
|
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Build is failing on this PR.
Please fix before merging:
- Run
npm run buildlocally - Fix any errors shown
- Push your fix — the check will re-run automatically
Most common issue: broken registry import.
Replace:
import agents from '../agents/registry'
With:
import { useAgents } from '../lib/useAgents'
const { agents } = useAgents()
See CONTRIBUTING.md for help. 🙏
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/components/AgentRunner.jsx (1)
593-631: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winKeep the character limit consistent across all input-state writes.
The new truncation applies to direct textarea and voice changes, but
setInputsis also used by example filling and version restoration. Those paths can reintroduce values aboveMAX_CHAR_LIMIT, making the counter exceed the cap and allowing over-limit prompts to run. Centralize normalization or truncate restored/example values before storing them.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/components/AgentRunner.jsx` around lines 593 - 631, Ensure every input-state write enforces MAX_CHAR_LIMIT, including example-filling and version-restoration paths that call setInputs, not only the textarea and VoiceInput handlers. Centralize truncation in the shared update/setInputs flow or truncate values before storing them, while preserving the existing counter and prompt behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/components/AgentRunner.jsx`:
- Around line 593-631: Ensure every input-state write enforces MAX_CHAR_LIMIT,
including example-filling and version-restoration paths that call setInputs, not
only the textarea and VoiceInput handlers. Centralize truncation in the shared
update/setInputs flow or truncate values before storing them, while preserving
the existing counter and prompt behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: b0ed3bfe-e4d7-4da7-b0bc-01bf2258968d
📒 Files selected for processing (1)
src/components/AgentRunner.jsx
|
The "Build failed" comment on this PR is misleading — the actual build ( The real failure is in the workflow itself: the The "broken registry import" text in the comment is just static boilerplate baked into the workflow — not an actual diagnosis, since none of the affected files import from Fix would be adding explicit permissions to the workflow yaml: permissions:
pull-requests: write
issues: writeor switching to Can you take a look at the workflow config? Happy to help however's useful. |
|
@aaniya22 i will look into the workflow....and update |
|
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
src/components/AgentRunner.jsx (5)
1188-1189: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick winRemove the unmatched JSX closing tags.
AgentRunneropens a<div>at Line 418. ThebatchModeconditional is already closed at Line 1160, and the schedule conditional closes at Line 1187. Lines 1188-1189 add</>and)}without matching openers. The root<div>is also not closed. JSX parsing fails. Close the root element once and remove the stray fragment and conditional closers.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/components/AgentRunner.jsx` around lines 1188 - 1189, Fix the JSX closing structure in AgentRunner by removing the unmatched fragment and conditional closers after the schedule conditional, then closing the root div opened near the component’s render start exactly once.
335-346: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winClear streaming state after a failed run.
If
streamAgentemits a chunk and then rejects,handleChunkleavesisStreamingset totrue. This catch path only setserror, andfinallydoes not reset the streaming state. The UI then displaysStreaming....and partial output after the request has failed. ClearisStreamingandstreamingOutput, or explicitly mark the partial output as failed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/components/AgentRunner.jsx` around lines 335 - 346, Update the catch/finally handling in the AgentRunner run flow so a rejected stream clears the state set by handleChunk: reset isStreaming and streamingOutput when streamAgent fails, while preserving the existing error handling and loading cleanup.
35-42: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick winRestore or remove the prompt-history integration.
The import changes leave
usePromptHistoryandPromptHistoryPanelreferenced but not imported.handleUsePromptis also passed at Line 1166, but no declaration appears inAgentRunner. The component will fail when it renders.savePromptis never called, so this runner cannot add entries to the panel. Restore the imports and complete the handler/save path, or remove the history state, button, hook, and panel together.Also applies to: 100-102, 1162-1167
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/components/AgentRunner.jsx` around lines 35 - 42, Fix the prompt-history integration in AgentRunner by either restoring the usePromptHistory and PromptHistoryPanel imports and declaring handleUsePrompt so it calls savePrompt, or removing all related state, hook, button, panel, and handleUsePrompt usage. Ensure no undefined prompt-history symbols remain and the selected approach is internally complete.
1177-1183: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftPreserve
customPromptin scheduled jobs.
AgentRunnerlets the user editcustomPrompt, but the schedule payload contains no custom prompt.src/lib/useScheduler.jslater sendsagent.systemPromptunconditionally. Scheduled runs therefore use the default prompt instead of the prompt shown in the runner. Add the prompt to the job payload and make the scheduler use it with a fallback.Preserve the scheduled system prompt
// src/components/AgentRunner.jsx addJob({ agentId: agent.id, agentName: agent.name, agentDefinition: agent, inputs: { ...inputs }, + systemPrompt: customPrompt, ...scheduleData, }) // src/lib/useScheduler.js - systemPrompt: agent.systemPrompt, + systemPrompt: job.systemPrompt ?? agent.systemPrompt,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/components/AgentRunner.jsx` around lines 1177 - 1183, Update the scheduled job payload in AgentRunner to include the currently edited customPrompt, then update the scheduler logic in useScheduler to send that job prompt as the system prompt and fall back to agent.systemPrompt when it is absent. Preserve existing scheduling behavior for jobs without a custom prompt.
660-682: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winEnforce the limits shown by the character counters.
The code editor and custom prompt display
maxLength={5000}, but neither textarea nor itsVoiceInputenforces that limit. Users can exceed 5,000 characters, andcustomPromptis sent tostreamAgentwithout a bound. Apply the same limit to direct and voice updates, or label 5,000 as a soft warning.Also applies to: 817-831
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/components/AgentRunner.jsx` around lines 660 - 682, Enforce the 5,000-character limit consistently for the code input and custom prompt flows identified by their CharCounter/maxLength settings. Apply the bound to both textarea edits and VoiceInput updates before calling updateInput or updating customPrompt, and ensure the value passed to streamAgent cannot exceed 5,000 characters; alternatively, change the counters to clearly indicate a soft warning.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/components/AgentRunner.jsx`:
- Around line 1188-1189: Fix the JSX closing structure in AgentRunner by
removing the unmatched fragment and conditional closers after the schedule
conditional, then closing the root div opened near the component’s render start
exactly once.
- Around line 335-346: Update the catch/finally handling in the AgentRunner run
flow so a rejected stream clears the state set by handleChunk: reset isStreaming
and streamingOutput when streamAgent fails, while preserving the existing error
handling and loading cleanup.
- Around line 35-42: Fix the prompt-history integration in AgentRunner by either
restoring the usePromptHistory and PromptHistoryPanel imports and declaring
handleUsePrompt so it calls savePrompt, or removing all related state, hook,
button, panel, and handleUsePrompt usage. Ensure no undefined prompt-history
symbols remain and the selected approach is internally complete.
- Around line 1177-1183: Update the scheduled job payload in AgentRunner to
include the currently edited customPrompt, then update the scheduler logic in
useScheduler to send that job prompt as the system prompt and fall back to
agent.systemPrompt when it is absent. Preserve existing scheduling behavior for
jobs without a custom prompt.
- Around line 660-682: Enforce the 5,000-character limit consistently for the
code input and custom prompt flows identified by their CharCounter/maxLength
settings. Apply the bound to both textarea edits and VoiceInput updates before
calling updateInput or updating customPrompt, and ensure the value passed to
streamAgent cannot exceed 5,000 characters; alternatively, change the counters
to clearly indicate a soft warning.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: a0ed8206-1184-4cb3-bcd8-a54b276a646d
📒 Files selected for processing (1)
src/components/AgentRunner.jsx
- Remove stray unmatched </> and )} left over from a removed conditional block. These broke JSX parsing entirely and prevented the app from building. - Clear isStreaming/streamingOutput when a run fails (not just on abort), so the UI doesn't get stuck showing 'Streaming...' with stale partial output after a genuine error. - Preserve the user's edited system prompt in scheduled jobs: pass customPrompt through to addJob, and have the scheduler fall back to agent.systemPrompt only when no custom prompt was saved. - Enforce the 5000-char limit shown by CharCounter on the code input and custom prompt textareas, including via VoiceInput, which previously bypassed the display-only limit entirely. Signed-off-by: aaniya22 <aaniyaatomar@gmail.com>
|
pls review and merge if ready |
|
|
2 participants
Adds a lightweight, client-side reliability score (0-100) for each agent, computed from existing local signals: user ratings (up/down feedback), usage frequency (run count via analytics events), and recency of last use. No ML or external calls required.
Closes #617
What does this PR do?
Implements the Agent Reliability Score & Trust Insights feature requested in #617, using the "lightweight heuristic scoring without ML" approach the issue calls out as acceptable for an initial version.
The score (0–100) is computed per agent from three weighted signals, all already available locally:
Each agent gets a Trust Level badge (High / Medium / Low) derived from the score. The compact badge appears on agent cards in the grid; a full breakdown card (score, progress bar, trust badge, rating counts, run count) appears on the agent detail page. The scoring logic is modular (
useReliabilityScore.js) so it can be extended later without touching the display components.Note: I noticed PR #673 is also linked to this issue — happy to have this reviewed as an alternative approach or coordinate with the maintainers on which to move forward with.
Type of change
Checklist
npm run buildlocally and it passed ✅import agents from '../agents/registry'✅Summary by CodeRabbit