feat: Signal Quality Oracle — Bradbury Hackathon Submission#36
Open
codenan42 wants to merge 1 commit into
Open
feat: Signal Quality Oracle — Bradbury Hackathon Submission#36codenan42 wants to merge 1 commit into
codenan42 wants to merge 1 commit into
Conversation
AI-powered news signal quality scoring using GenLayer's LLM and web capabilities. - submit_signal: LLM evaluates signals against editorial standards - verify_web_signal: On-chain fact-checking by reading source URLs - Reputation tracking and configurable per-beat standards - Demonstrates GenLayer's unique differentiator: AI-native smart contracts
Contributor
📝 WalkthroughWalkthroughThe PR replaces the GenLayer football betting project with a "Signal Quality Oracle" application that uses AI to evaluate and score news signals. It introduces a new smart contract for signal submission and verification, updates the deployment script to target the new contract, and rewrites the README with project architecture and hackathon details. Changes
Sequence DiagramsequenceDiagram
participant Client as Client/User
participant Contract as SignalQualityOracle
participant AI as GenLayer AI Runtime
participant Storage as Contract Storage
Client->>Contract: submit_signal(headline, body, beat)
Contract->>Contract: Generate signal_id
Contract->>Contract: Select EditorialStandard
Contract->>AI: gl.nondet.exec_prompt(evaluation)
AI-->>Contract: {score, approved, feedback}
Contract->>Storage: Persist SignalSubmission
Contract->>Storage: Update submitter_scores
Contract->>Storage: Increment total_submissions
Contract-->>Client: {signal_id, score, approved, feedback}
Client->>Contract: verify_web_signal(headline, source_url)
Contract->>AI: gl.nondet.web.render(source_url)
AI-->>Contract: Web content
Contract->>AI: gl.nondet.exec_prompt(verify headline vs content)
AI-->>Contract: Verification result
Contract-->>Client: Verification JSON
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
README.md (1)
25-32: Add language specifier to code block.The architecture diagram code block should specify a language (e.g.,
textorplaintext) for better rendering and to satisfy markdown linting.Proposed fix
-``` +```text contracts/ signal_oracle.py # Main intelligent contract deploy/ deployScript.ts # Deployment script frontend/ # Next.js web interface test/ # Integration tests ```🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@README.md` around lines 25 - 32, In the README.md code block that lists the project layout (the triple-backtick block containing "contracts/ signal_oracle.py deploy/ deployScript.ts frontend/ test/"), add a language specifier after the opening backticks (e.g., change ``` to ```text or ```plaintext) so the block is explicitly marked as plain text for markdown linting and proper rendering.deploy/deployScript.ts (1)
11-11: Missing validation for PRIVATE_KEY environment variable.If
PRIVATE_KEYis not set, the script will passundefinedtoprivateKeyToAccount, causing a cryptic runtime error. Consider adding explicit validation.Proposed fix with validation
-const account = privateKeyToAccount(process.env.PRIVATE_KEY as `0x${string}`); +const privateKey = process.env.PRIVATE_KEY; +if (!privateKey) { + throw new Error("PRIVATE_KEY environment variable is required"); +} +const account = privateKeyToAccount(privateKey as `0x${string}`);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@deploy/deployScript.ts` at line 11, Validate process.env.PRIVATE_KEY before calling privateKeyToAccount: check that PRIVATE_KEY is defined and matches expected hex format (starts with "0x" and correct length) and if not, throw or log a clear error and exit; then call privateKeyToAccount(process.env.PRIVATE_KEY as `0x${string}`) to assign account (referencing the PRIVATE_KEY env var, privateKeyToAccount function, and account variable in deployScript.ts).contracts/signal_oracle.py (1)
205-205: No error handling for web fetch failures.
gl.nondet.web.render(source_url, mode="text")could fail if the URL is invalid, unreachable, or returns an error. Consider wrapping this in error handling or validating the URL format before fetching.Proposed defensive handling
def verify_against_source() -> str: - web_content = gl.nondet.web.render(source_url, mode="text") + try: + web_content = gl.nondet.web.render(source_url, mode="text") + except Exception as e: + return json.dumps({ + "accuracy": 0, + "supported": False, + "source_quality": "error", + "discrepancies": [f"Failed to fetch source: {str(e)}"] + }, sort_keys=True)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@contracts/signal_oracle.py` at line 205, Wrap the call to gl.nondet.web.render(source_url, mode="text") in a defensive block: validate source_url (e.g., check scheme/host via URL parse) and catch exceptions around gl.nondet.web.render so failures (network errors, invalid URLs, non-200 responses) don’t propagate; on error return a safe fallback or raise a clear, typed exception and log the failure (reference the web_content variable, gl.nondet.web.render call, and source_url so you change the code at that exact call site).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@contracts/signal_oracle.py`:
- Around line 204-227: The current verify_against_source flow uses
gl.eq_principle.strict_eq on LLM output and relies on gl.nondet.web.render,
causing non-determinism; replace the single strict_eq call with a deterministic
aggregation: call verify_against_source (or directly invoke the prompt) multiple
times (e.g., 3 runs), catch and handle failures from
gl.nondet.web.render/exec_prompt, parse each JSON result, compute a consensus
(majority for "supported"/"source_quality", average or median for "accuracy",
merge "discrepancies"), and return that aggregated JSON instead of using
gl.eq_principle.strict_eq; ensure you add retries/timeouts and a cache or
snapshot for web_content to avoid differing renders between runs.
- Around line 141-153: get_submission currently directly indexes
self.submissions[signal_id] which will raise an uncontrolled KeyError for
missing IDs; modify get_submission to first check if signal_id exists in
self.submissions (using "if signal_id not in self.submissions") and handle the
missing case by returning a clear error/None or raising a controlled exception
(e.g., ValueError or a custom exception) with a descriptive message, otherwise
proceed to build and return the dict as before; update callers/tests if they
expect the old behavior.
- Line 110: The current use of gl.eq_principle.strict_eq on evaluate_signal is
too brittle for LLM outputs; replace the strict equality call with a semantic
comparison such as gl.eq_principle.prompt_comparative (or
gl.eq_principle.prompt_non_comparative for subjective judgments) when computing
eval_result, i.e., call gl.eq_principle.prompt_comparative(evaluate_signal)
instead of strict_eq and then parse its result the same way (ensure json.loads
is applied to the returned value if prompt_comparative returns a JSON string) so
that evaluate_signal comparisons tolerate semantic/formatting variance across
validators.
In `@deploy/deployScript.ts`:
- Line 4: Remove the unused import of abi from "./SignalOracle.json" in
deployScript.ts: delete the line "import { abi } from \"./SignalOracle.json\"";
ensure there are no other references to the symbol abi or to SignalOracle.json
elsewhere in this file and remove or replace any remaining code that expected
that JSON to avoid module resolution errors.
---
Nitpick comments:
In `@contracts/signal_oracle.py`:
- Line 205: Wrap the call to gl.nondet.web.render(source_url, mode="text") in a
defensive block: validate source_url (e.g., check scheme/host via URL parse) and
catch exceptions around gl.nondet.web.render so failures (network errors,
invalid URLs, non-200 responses) don’t propagate; on error return a safe
fallback or raise a clear, typed exception and log the failure (reference the
web_content variable, gl.nondet.web.render call, and source_url so you change
the code at that exact call site).
In `@deploy/deployScript.ts`:
- Line 11: Validate process.env.PRIVATE_KEY before calling privateKeyToAccount:
check that PRIVATE_KEY is defined and matches expected hex format (starts with
"0x" and correct length) and if not, throw or log a clear error and exit; then
call privateKeyToAccount(process.env.PRIVATE_KEY as `0x${string}`) to assign
account (referencing the PRIVATE_KEY env var, privateKeyToAccount function, and
account variable in deployScript.ts).
In `@README.md`:
- Around line 25-32: In the README.md code block that lists the project layout
(the triple-backtick block containing "contracts/ signal_oracle.py deploy/
deployScript.ts frontend/ test/"), add a language specifier after the opening
backticks (e.g., change ``` to ```text or ```plaintext) so the block is
explicitly marked as plain text for markdown linting and proper rendering.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: de276ae0-065d-49bd-b6d3-d42ddfd7081d
📒 Files selected for processing (3)
README.mdcontracts/signal_oracle.pydeploy/deployScript.ts
| result = gl.nondet.exec_prompt(prompt, response_format="json") | ||
| return json.dumps(result, sort_keys=True) | ||
|
|
||
| eval_result = json.loads(gl.eq_principle.strict_eq(evaluate_signal)) |
Contributor
There was a problem hiding this comment.
strict_eq may cause consensus failures with LLM outputs.
LLM responses are inherently non-deterministic—even with the same prompt, different validators may receive different scores, feedback text, or JSON formatting. Using gl.eq_principle.strict_eq() requires byte-exact matches across all validators, which is unlikely to succeed with LLM-generated content.
Consider using gl.eq_principle.prompt_comparative() for outputs where semantic similarity is acceptable, or gl.eq_principle.prompt_non_comparative() for subjective assessments. As per coding guidelines: "Use appropriate equivalence principle validation: gl.eq_principle.strict_eq() for exact outputs, gl.eq_principle.prompt_comparative() for similar outputs, gl.eq_principle.prompt_non_comparative() for subjective assessments."
Proposed fix using prompt_comparative
- eval_result = json.loads(gl.eq_principle.strict_eq(evaluate_signal))
+ eval_result = json.loads(gl.eq_principle.prompt_comparative(
+ evaluate_signal,
+ "Compare these two signal evaluations. Are the scores within 10 points, approval decisions the same, and feedback conveying similar issues?"
+ ))🧰 Tools
🪛 Ruff (0.15.7)
[error] 110-110: gl may be undefined, or defined from star imports
(F405)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@contracts/signal_oracle.py` at line 110, The current use of
gl.eq_principle.strict_eq on evaluate_signal is too brittle for LLM outputs;
replace the strict equality call with a semantic comparison such as
gl.eq_principle.prompt_comparative (or gl.eq_principle.prompt_non_comparative
for subjective judgments) when computing eval_result, i.e., call
gl.eq_principle.prompt_comparative(evaluate_signal) instead of strict_eq and
then parse its result the same way (ensure json.loads is applied to the returned
value if prompt_comparative returns a JSON string) so that evaluate_signal
comparisons tolerate semantic/formatting variance across validators.
Comment on lines
+141
to
+153
| def get_submission(self, signal_id: str) -> dict: | ||
| """Get details of a signal submission.""" | ||
| sub = self.submissions[signal_id] | ||
| return { | ||
| "signal_id": sub.signal_id, | ||
| "submitter": sub.submitter.as_hex, | ||
| "headline": sub.headline, | ||
| "beat": sub.beat, | ||
| "quality_score": sub.quality_score, | ||
| "is_approved": sub.is_approved, | ||
| "feedback": sub.feedback, | ||
| "timestamp": int(sub.timestamp), | ||
| } |
Contributor
There was a problem hiding this comment.
Missing error handling for non-existent signal_id.
Accessing self.submissions[signal_id] will raise a KeyError if the signal doesn't exist. This could cause unexpected transaction failures for callers.
Proposed fix to handle missing submissions
`@gl.public.view`
def get_submission(self, signal_id: str) -> dict:
"""Get details of a signal submission."""
+ if signal_id not in self.submissions:
+ raise Exception(f"Signal {signal_id} not found")
sub = self.submissions[signal_id]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@contracts/signal_oracle.py` around lines 141 - 153, get_submission currently
directly indexes self.submissions[signal_id] which will raise an uncontrolled
KeyError for missing IDs; modify get_submission to first check if signal_id
exists in self.submissions (using "if signal_id not in self.submissions") and
handle the missing case by returning a clear error/None or raising a controlled
exception (e.g., ValueError or a custom exception) with a descriptive message,
otherwise proceed to build and return the dict as before; update callers/tests
if they expect the old behavior.
Comment on lines
+204
to
+227
| def verify_against_source() -> str: | ||
| web_content = gl.nondet.web.render(source_url, mode="text") | ||
|
|
||
| prompt = f"""A news correspondent claims: "{headline}" | ||
|
|
||
| Source content from {source_url}: | ||
| {web_content[:3000]} | ||
|
|
||
| Verify: Does the source content support this headline? | ||
| Score accuracy 0-100. | ||
|
|
||
| Respond in JSON only: | ||
| {{ | ||
| "accuracy": int, | ||
| "supported": bool, | ||
| "source_quality": str, | ||
| "discrepancies": [str] | ||
| }} | ||
| It is mandatory that you respond only using the JSON format above, nothing else.""" | ||
|
|
||
| result = gl.nondet.exec_prompt(prompt, response_format="json") | ||
| return json.dumps(result, sort_keys=True) | ||
|
|
||
| verification = json.loads(gl.eq_principle.strict_eq(verify_against_source)) |
Contributor
There was a problem hiding this comment.
Same strict_eq issue applies to web verification.
The verify_web_signal method also uses strict_eq with LLM output (Line 227), which will face the same consensus challenges as submit_signal. Additionally, gl.nondet.web.render() may return different content at different times or fail entirely, compounding the non-determinism.
Proposed fix
- verification = json.loads(gl.eq_principle.strict_eq(verify_against_source))
+ verification = json.loads(gl.eq_principle.prompt_comparative(
+ verify_against_source,
+ "Compare these two verification results. Are the accuracy scores within 15 points, the supported boolean the same, and key discrepancies similar?"
+ ))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def verify_against_source() -> str: | |
| web_content = gl.nondet.web.render(source_url, mode="text") | |
| prompt = f"""A news correspondent claims: "{headline}" | |
| Source content from {source_url}: | |
| {web_content[:3000]} | |
| Verify: Does the source content support this headline? | |
| Score accuracy 0-100. | |
| Respond in JSON only: | |
| {{ | |
| "accuracy": int, | |
| "supported": bool, | |
| "source_quality": str, | |
| "discrepancies": [str] | |
| }} | |
| It is mandatory that you respond only using the JSON format above, nothing else.""" | |
| result = gl.nondet.exec_prompt(prompt, response_format="json") | |
| return json.dumps(result, sort_keys=True) | |
| verification = json.loads(gl.eq_principle.strict_eq(verify_against_source)) | |
| def verify_against_source() -> str: | |
| web_content = gl.nondet.web.render(source_url, mode="text") | |
| prompt = f"""A news correspondent claims: "{headline}" | |
| Source content from {source_url}: | |
| {web_content[:3000]} | |
| Verify: Does the source content support this headline? | |
| Score accuracy 0-100. | |
| Respond in JSON only: | |
| {{ | |
| "accuracy": int, | |
| "supported": bool, | |
| "source_quality": str, | |
| "discrepancies": [str] | |
| }} | |
| It is mandatory that you respond only using the JSON format above, nothing else.""" | |
| result = gl.nondet.exec_prompt(prompt, response_format="json") | |
| return json.dumps(result, sort_keys=True) | |
| verification = json.loads(gl.eq_principle.prompt_comparative( | |
| verify_against_source, | |
| "Compare these two verification results. Are the accuracy scores within 15 points, the supported boolean the same, and key discrepancies similar?" | |
| )) |
🧰 Tools
🪛 Ruff (0.15.7)
[error] 205-205: gl may be undefined, or defined from star imports
(F405)
[error] 224-224: gl may be undefined, or defined from star imports
(F405)
[error] 227-227: gl may be undefined, or defined from star imports
(F405)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@contracts/signal_oracle.py` around lines 204 - 227, The current
verify_against_source flow uses gl.eq_principle.strict_eq on LLM output and
relies on gl.nondet.web.render, causing non-determinism; replace the single
strict_eq call with a deterministic aggregation: call verify_against_source (or
directly invoke the prompt) multiple times (e.g., 3 runs), catch and handle
failures from gl.nondet.web.render/exec_prompt, parse each JSON result, compute
a consensus (majority for "supported"/"source_quality", average or median for
"accuracy", merge "discrepancies"), and return that aggregated JSON instead of
using gl.eq_principle.strict_eq; ensure you add retries/timeouts and a cache or
snapshot for web_content to avoid differing renders between runs.
| import { createClient } from "genlayer-js"; | ||
| import { SimulatorTransport } from "genlayer-js/simulator"; | ||
| import { privateKeyToAccount } from "genlayer-js/accounts"; | ||
| import { abi } from "./SignalOracle.json"; |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if SignalOracle.json exists anywhere in the repository
fd -i "SignalOracle.json"
# Also check for any .json files in deploy/
fd -e json . deploy/Repository: genlayerlabs/genlayer-project-boilerplate
Length of output: 67
🏁 Script executed:
# Check if abi is used anywhere in deploy/deployScript.ts
rg "abi" deploy/deployScript.ts
# Also read the file to see context
wc -l deploy/deployScript.tsRepository: genlayerlabs/genlayer-project-boilerplate
Length of output: 153
🏁 Script executed:
cat -n deploy/deployScript.tsRepository: genlayerlabs/genlayer-project-boilerplate
Length of output: 1482
Remove unused import from non-existent file.
Line 4 imports abi from ./SignalOracle.json, which does not exist and the variable is never used in the script. This will cause a module resolution error when running npm run deploy.
Proposed fix
import { createClient } from "genlayer-js";
import { SimulatorTransport } from "genlayer-js/simulator";
import { privateKeyToAccount } from "genlayer-js/accounts";
-import { abi } from "./SignalOracle.json";📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| import { abi } from "./SignalOracle.json"; | |
| import { createClient } from "genlayer-js"; | |
| import { SimulatorTransport } from "genlayer-js/simulator"; | |
| import { privateKeyToAccount } from "genlayer-js/accounts"; |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@deploy/deployScript.ts` at line 4, Remove the unused import of abi from
"./SignalOracle.json" in deployScript.ts: delete the line "import { abi } from
\"./SignalOracle.json\""; ensure there are no other references to the symbol abi
or to SignalOracle.json elsewhere in this file and remove or replace any
remaining code that expected that JSON to avoid module resolution errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signal Quality Oracle
An AI-powered intelligent contract that autonomously evaluates news signals against editorial standards.
What it does
Why GenLayer?
This contract is impossible on any other blockchain — it uses:
gl.nondet.exec_prompt()for LLM inferencegl.nondet.web.render()for web content readinggl.eq_principle.strict_eq()for consensus validationTrack: Builders
Demonstrates GenLayer's core differentiator: AI-native smart contracts that can read, think, and judge.
See README.md for full details.
Summary by CodeRabbit
Release Notes
New Features
Documentation