feat: add reliability score - #673
Conversation
|
@avanibapna06 is attempting to deploy a commit to the aditthyass' projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Note
|
| Layer / File(s) | Summary |
|---|---|
Reliability scoring and badge component src/lib/reliabilityScorer.js, src/components/TrustBadge.jsx |
Adds calculateReliabilityScore(agent), which computes a deterministic or randomized score clamped to 0–100 and classifies it into Low/Medium/High badges, and TrustBadge, a component that renders the badge label and score with badge-specific styling. |
HomePage agent grid integration src/pages/HomePage.jsx |
Imports the new scorer and badge component, and renders a top-right TrustBadge overlay on each AgentCard in the filtered agent grid, computing score/badge per agent and logging the agent to console during that computation. |
Sequence Diagram(s)
sequenceDiagram
participant HomePage
participant calculateReliabilityScore
participant TrustBadge
HomePage->>calculateReliabilityScore: agent data
calculateReliabilityScore-->>HomePage: score, badge
HomePage->>TrustBadge: score, badge props
TrustBadge-->>HomePage: rendered badge overlay
Estimated code review effort: 2 (Simple) | ~12 minutes
Suggested reviewers: AditthyaSS
🚥 Pre-merge checks | ✅ 2 | ❌ 3
❌ Failed checks (3 warnings)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Linked Issues check | The PR adds a score and trust badge, but it omits the broader insights dashboard and supporting metrics requested in #617. |
Add the remaining trust insights UI and populate it from richer agent metadata such as activity, ratings, updates, and community signals. | |
| Out of Scope Changes check | The HomePage console.log is a debug change unrelated to the requested reliability scoring UI. | Remove the console.log or keep it behind a temporary development-only debug flag. | |
| Docstring Coverage | Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (2 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | The title is concise and accurately reflects the main addition: a reliability score feature. |
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
|
hey @avanibapna06! 👋 |
|
hey @avanibapna06! 👋 |
|
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
src/lib/reliabilityScorer.js (1)
4-29: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueNon-English code comments.
Comments mix Hindi/Romanized text ("Agar agent object hi nahi mila", "taaki dashboard bhara hua lage") with English code. For a public OSS project, consistent English comments improve readability for all contributors.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/reliabilityScorer.js` around lines 4 - 29, The comments in reliabilityScorer’s scoring logic are mixed Hindi/Romanized and English, so standardize them to clear English throughout the calculate/score path. Update the inline comments around the agent guard, data extraction, scoring fallback, and badge assignment in reliabilityScorer.js so they are consistent and readable for all contributors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/components/TrustBadge.jsx`:
- Around line 5-11: The badge color mapping in getBadgeColor is light-theme
only, so add matching dark-mode Tailwind variants for each badge state to keep
TrustBadge consistent with the app’s dark UI. Update the returned class strings
for the High, Medium, and default cases in TrustBadge.jsx so they include
appropriate dark: background, text, and border colors similar to the patterns
used in AgentCard.jsx.
In `@src/lib/reliabilityScorer.js`:
- Around line 11-20: The fallback in reliabilityScorer’s scoring logic should
not generate a random “Trust Score” when usageCount and rating are missing,
because that fabricates reliability data. Update the scorer to return an
explicit neutral/unknown state (for example, “Unrated” or “Not enough data”)
from the reliabilityScorer.js logic instead of a random number, and make sure
any downstream label/formatting that currently assumes a numeric score handles
that state cleanly.
In `@src/pages/HomePage.jsx`:
- Line 522: Remove the debug console.log from HomePage’s agent card rendering
path; it is logging the full agent object on every render. Update the relevant
render logic in HomePage to eliminate the Agent Data Check statement so no debug
output ships in production.
- Around line 518-526: The inline IIFE inside the AgentCard render is
recomputing calculateReliabilityScore on every re-render, which causes the
TrustBadge to change unexpectedly for agents that fall back to random scoring.
Move the score calculation out of the JSX and memoize it once per agent, using
useMemo in HomePage keyed by the agent list (or by agent identity) or by storing
the computed score when agents are loaded. Then render TrustBadge from the
cached score/badge instead of recalculating inside the AgentCard block.
---
Nitpick comments:
In `@src/lib/reliabilityScorer.js`:
- Around line 4-29: The comments in reliabilityScorer’s scoring logic are mixed
Hindi/Romanized and English, so standardize them to clear English throughout the
calculate/score path. Update the inline comments around the agent guard, data
extraction, scoring fallback, and badge assignment in reliabilityScorer.js so
they are consistent and readable for all contributors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1ca2090a-e1ed-4d6b-980e-95c736bd5566
📒 Files selected for processing (3)
src/components/TrustBadge.jsxsrc/lib/reliabilityScorer.jssrc/pages/HomePage.jsx
| const getBadgeColor = (badge) => { | ||
| switch (badge) { | ||
| case 'High': return 'bg-green-100 text-green-700 border-green-200'; | ||
| case 'Medium': return 'bg-yellow-100 text-yellow-700 border-yellow-200'; | ||
| default: return 'bg-red-100 text-red-700 border-red-200'; | ||
| } | ||
| }; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Missing dark-mode color variants.
Colors (bg-green-100 text-green-700, etc.) are light-theme only. Per the PR screenshots, the app is dark-themed throughout (AgentCard.jsx consistently pairs classes with dark: variants, e.g. dark:bg-surface-input dark:text-text-muted). Without dark variants, the light pastel backgrounds will look out of place / have poor contrast against the dark card background.
♻️ Proposed fix
const getBadgeColor = (badge) => {
switch (badge) {
- case 'High': return 'bg-green-100 text-green-700 border-green-200';
- case 'Medium': return 'bg-yellow-100 text-yellow-700 border-yellow-200';
- default: return 'bg-red-100 text-red-700 border-red-200';
+ case 'High': return 'bg-green-100 text-green-700 border-green-200 dark:bg-green-500/10 dark:text-green-400 dark:border-green-500/20';
+ case 'Medium': return 'bg-yellow-100 text-yellow-700 border-yellow-200 dark:bg-yellow-500/10 dark:text-yellow-400 dark:border-yellow-500/20';
+ default: return 'bg-red-100 text-red-700 border-red-200 dark:bg-red-500/10 dark:text-red-400 dark:border-red-500/20';
}
};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const getBadgeColor = (badge) => { | |
| switch (badge) { | |
| case 'High': return 'bg-green-100 text-green-700 border-green-200'; | |
| case 'Medium': return 'bg-yellow-100 text-yellow-700 border-yellow-200'; | |
| default: return 'bg-red-100 text-red-700 border-red-200'; | |
| } | |
| }; | |
| const getBadgeColor = (badge) => { | |
| switch (badge) { | |
| case 'High': return 'bg-green-100 text-green-700 border-green-200 dark:bg-green-500/10 dark:text-green-400 dark:border-green-500/20'; | |
| case 'Medium': return 'bg-yellow-100 text-yellow-700 border-yellow-200 dark:bg-yellow-500/10 dark:text-yellow-400 dark:border-yellow-500/20'; | |
| default: return 'bg-red-100 text-red-700 border-red-200 dark:bg-red-500/10 dark:text-red-400 dark:border-red-500/20'; | |
| } | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/components/TrustBadge.jsx` around lines 5 - 11, The badge color mapping
in getBadgeColor is light-theme only, so add matching dark-mode Tailwind
variants for each badge state to keep TrustBadge consistent with the app’s dark
UI. Update the returned class strings for the High, Medium, and default cases in
TrustBadge.jsx so they include appropriate dark: background, text, and border
colors similar to the patterns used in AgentCard.jsx.
| // 3. Scoring Logic: | ||
| // Agar real data hai toh use karo, agar nahi hai toh random score do (UI testing ke liye) | ||
| let score = 0; | ||
|
|
||
| if (usage > 0 || rating > 0) { | ||
| score = (rating * 15) + (Math.min(usage, 500) / 10); | ||
| } else { | ||
| // Fallback: Random score between 40 and 95 taaki dashboard bhara hua lage | ||
| score = Math.floor(Math.random() * (95 - 40 + 1) + 40); | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Random fallback fabricates a "Trust Score" — undermines the feature's purpose.
When usageCount and rating are both absent/zero, the function returns a random score between 40–95 instead of a neutral/unknown state. This is presented to users as a real reliability metric ("Low/Medium/High Trust • N/100"), but for agents without usage data it's pure noise that changes on every call. This actively misleads users — the opposite of what a "Trust Insights" feature should do — and conflicts with the PR's stated goal of "helping users identify trustworthy agents."
Consider returning an explicit "Unrated"/"Not enough data" state instead of a fabricated number.
♻️ Proposed fix
let score = 0;
-
if (usage > 0 || rating > 0) {
score = (rating * 15) + (Math.min(usage, 500) / 10);
} else {
- // Fallback: Random score between 40 and 95 taaki dashboard bhara hua lage
- score = Math.floor(Math.random() * (95 - 40 + 1) + 40);
+ // No usage/rating data available; surface this explicitly rather than fabricating a score.
+ return { score: null, badge: 'Unrated' };
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // 3. Scoring Logic: | |
| // Agar real data hai toh use karo, agar nahi hai toh random score do (UI testing ke liye) | |
| let score = 0; | |
| if (usage > 0 || rating > 0) { | |
| score = (rating * 15) + (Math.min(usage, 500) / 10); | |
| } else { | |
| // Fallback: Random score between 40 and 95 taaki dashboard bhara hua lage | |
| score = Math.floor(Math.random() * (95 - 40 + 1) + 40); | |
| } | |
| // 3. Scoring Logic: | |
| // Agar real data hai toh use karo, agar nahi hai toh random score do (UI testing ke liye) | |
| let score = 0; | |
| if (usage > 0 || rating > 0) { | |
| score = (rating * 15) + (Math.min(usage, 500) / 10); | |
| } else { | |
| // No usage/rating data available; surface this explicitly rather than fabricating a score. | |
| return { score: null, badge: 'Unrated' }; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lib/reliabilityScorer.js` around lines 11 - 20, The fallback in
reliabilityScorer’s scoring logic should not generate a random “Trust Score”
when usageCount and rating are missing, because that fabricates reliability
data. Update the scorer to return an explicit neutral/unknown state (for
example, “Unrated” or “Not enough data”) from the reliabilityScorer.js logic
instead of a random number, and make sure any downstream label/formatting that
currently assumes a numeric score handles that state cleanly.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
2 participants
What does this PR do?
This PR introduces an Agent Reliability Score & Trust Insights system for the platform. It adds a visual TrustBadge component that displays a reliability score (0-100) on each agent card. The scoring engine calculates reliability using a heuristic-based approach, providing feedback on agent performance and maintenance status. This helps users quickly identify trustworthy agents and improves overall platform transparency.
Type of change
[x] UI improvement
[x] New feature (Reliability Scoring Engine)
Checklist
[x] I ran npm run build locally and it passed ✅
[x] I tested my changes in the browser ✅
[x] I did not break any existing agents ✅
[x] I did not use import agents from '../agents/registry' ✅
[x] My PR has a clear description above ✅
Screenshots
closes #617
Summary by CodeRabbit