Implemented semantic duplicate detection for doubts using embeddings

[Aditya8369]

User description

Completed the semantic duplicate detection upgrade end-to-end (embeddings + pgvector + vector search + fallback).

What’s now implemented

1. pgvector support + doubts.embedding column
   Added migration: drizzle/0009_add_doubt_embeddings_pgvector.sql
   enables vector extension
   adds doubts.embedding vector(1536)
   creates an HNSW cosine index
2. Drizzle schema updated
   Updated: src/configs/schema.ts
   added embedding: any() to doubtsTable
3. Embeddings generation + semantic search helper
   Added: src/lib/ai/embeddings.ts
   safeGenerateEmbedding(text) uses Groq embeddings (nomic-embed-text)
   findSemanticDuplicates(...) runs pgvector cosine similarity search
   returns results in the same shape as the existing API, including solvedAnswer by fetching repliesTable for solved doubts
4. Generate/store embeddings on doubt creation
   Updated: src/app/api/doubts/route.ts (POST)
   after inserting newDoubt, it generates an embedding from:
   subject + "\n" + content
   persists it into doubts.embedding
   fail-open: if embeddings fail, doubt creation still succeeds
5. Similarity endpoint now uses embeddings first, LLM fallback second

Updated: src/app/api/doubts/check-similarity/route.ts (POST)
Primary path: findSemanticDuplicates (vector search across full history for the same scope: community + classroomId or null)

closes #534

Summary by CodeRabbit

• New Features

  • Vector-based semantic duplicate detection with embedding generation and a similarity index (best-effort accelerated index).
  • Fast-path similarity check on question submission that can return matches immediately; embedding generation failures are non-blocking.

• Accessibility

  • File upload control updated with clearer labeling.

• UI/UX

  • Ask AI page layout refined with a smoother, scrollable content area.

• Bug Fixes

  • Safer count handling in members endpoint.

• Chores

  • New PR code-quality workflows to surface lint/build/security/quality feedback.

---

CodeAnt-AI Description

Detect duplicate doubts using meaning, not just exact wording

What Changed

• New doubt submissions now save an embedding so similar questions can be matched by meaning later
• Duplicate checks now return close matches from the full history first, which helps catch reworded versions of the same question
• If embedding-based matching does not return results, the system still falls back to the older similarity check
• Similar matches can now include the solved answer when the original doubt was already resolved
• The Ask AI page now keeps long content scrollable and gives the image upload control a clear label
• Member count handling no longer breaks when there are no results

Impact

✅ Fewer duplicate doubt posts
✅ Faster duplicate suggestions on submit
✅ Clearer image upload for Ask AI

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

[codeant-ai]

CodeAnt AI is reviewing your PR.

[vercel]

@Aditya8369 is attempting to deploy a commit to the Karan Mani Tripathi 's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds pgvector storage and index for 1536-dim embeddings, a Groq-backed embedding library with safe generation and pgvector nearest-neighbor search, persists embeddings on doubt creation, uses vector search as a fast-path in similarity checks with LLM fallback, and minor Ask AI UI/accessibility tweaks.

Changes

Semantic duplicate detection with embeddings

Layer / File(s)	Summary
Database vector column & index creation drizzle/0009_add_doubt_embeddings_pgvector.sql	Installs vector extension, adds doubts.embedding vector(1536) column, and creates an idempotent PL/pgSQL block that tries HNSW index then falls back to IVFFLAT.
Embedding generation and semantic search library src/lib/ai/embeddings.ts	Groq client and helpers: getEmbeddingInput, generateGroqEmbedding (1536-dim validation), safeGenerateEmbedding() (null-on-failure), and findSemanticDuplicates() which queries pgvector, computes 0–100 similarity, filters by type/classroom, and enriches solved items with reply content.
Persist embeddings on doubt creation src/app/api/doubts/route.ts	POST handler inserts doubt, then (fail-open) generates embedding from `${subject}\n${content
Similarity fast-path with LLM fallback src/app/api/doubts/check-similarity/route.ts	Adds an embedding-based fast-path calling findSemanticDuplicates({ type: "community", similarityThreshold: 80, topK: 5 }); returns early on results, otherwise logs errors and falls back to existing Groq/LLM comparison over recent doubts.
Ask AI layout and accessibility tweaks src/app/ask-ai/page.tsx	Wraps central content in a scrollable container and updates hidden image input to accept="image/*" with aria-label and title attributes.
CI: PR quality workflows and commenter .github/workflows/pr-code-quality.yml, .github/workflows/pr-code-quality-commenter.yml	Adds PR code-quality checks (typecheck, lint, build, security scan, quality gate, dependency audit) and a workflow to post artifact warnings as PR comments.
Tests and minor robustness fixes src/__tests__/*, src/app/api/rooms/members/route.ts, src/configs/db.tsx, src/app/api/replies/route.ts	Adds conditional test debug logging, hardens pagination total calculation, adds non-null assertions in a membership check, and minor formatting/trailing-line adjustments.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant CheckAPI as POST /api/doubts/check-similarity
  participant EmbeddingLib as findSemanticDuplicates
  participant GroqAPI as nomic-embed-text
  participant DB as Postgres (pgvector)
  Client->>CheckAPI: submit content to check
  CheckAPI->>EmbeddingLib: build input & request embedding
  EmbeddingLib->>GroqAPI: nomic-embed-text (float)
  GroqAPI-->>EmbeddingLib: embedding[1536]
  EmbeddingLib->>DB: pgvector cosine query (type/classroomId, LIMIT topK)
  DB-->>CheckAPI: candidate rows + similarity%
  alt candidates found
    CheckAPI-->>Client: return similarDoubts
  else none or error
    CheckAPI->>Client: fallback to existing Groq LLM comparison
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

• Add duplicate doubt suggestions before submission #513: Modifies the same similarity endpoint and is related to embedding-based fast-path improvements.

Possibly related PRs

• knoxiboy/DoubtDesk#453: Original Groq-based duplicate detection endpoint that this PR augments.
• knoxiboy/DoubtDesk#571: Modifies the same doubts endpoints and request flow, related at route logic level.
• knoxiboy/DoubtDesk#585: Overlaps on /api/doubts/check-similarity changes (AI provider gating).

Suggested labels

type:feature, size/xxl, mentor:knoxiboy, quality:clean

Suggested reviewers

• knoxiboy

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	While most changes directly support semantic duplicate detection, the PR includes unrelated improvements: GitHub workflow additions (pr-code-quality.yml, pr-code-quality-commenter.yml), UI fixes (ask-ai scrolling, image upload labeling), and minor bug fixes (room members count handling, replies route assertions) that fall outside issue #534 scope.	Consider separating unrelated improvements (workflows, UI fixes, bug fixes) into separate PRs to keep this PR focused on semantic duplicate detection and reduce review complexity.
Docstring Coverage	⚠️ Warning	Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main objective: implementing semantic duplicate detection using embeddings, which aligns with the primary focus of the changeset across embeddings, schema, migrations, and the API routes.
Linked Issues check	✅ Passed	The PR implements all major coding requirements from issue #534: embeddings generation via Groq (safeGenerateEmbedding), pgvector storage with HNSW index, semantic duplicate detection via vector search (findSemanticDuplicates), embedding persistence on doubt creation, and LLM fallback in check-similarity.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

@coderabbitai review

[codeant-ai]

CodeAnt AI finished reviewing your PR.

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/configs/schema.ts (1)

2-2: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Fix runtime crash in Drizzle schema: embedding: any()

src/configs/schema.ts defines doubtsTable.embedding as any() (around line 191). Since any is not a runtime value/symbol here, schema module initialization will throw (and it blocks the embeddings route).

Replace the embedding column with a real pgvector/vector(1536) Drizzle column type (the repo already sets the DB column via drizzle/0009_add_doubt_embeddings_pgvector.sql and uses pgvector operators in src/lib/ai/embeddings.ts). After that, update src/app/api/doubts/route.ts to stop writing embedding as any and pass the correct driver value type for that column.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/configs/schema.ts` at line 2, doubtsTable.embedding currently uses a
non-runtime symbol `any()` which crashes module initialization; replace that
column declaration with the pgvector Drizzle column type (a vector(1536) column)
by importing and using the appropriate pgvector/vector factory instead of
`any()` for doubtsTable.embedding, and then update the code in the doubts route
that writes embeddings (the writer that currently does `embedding as any`) to
pass the correct pgvector driver value type produced by your embedding code (no
`as any` cast). Ensure you update the imports where embedding is declared and
the insertion/update call in the route so the column type and the value type
match (vector(1536) for doubtsTable.embedding and the driver-specific vector
value when inserting).

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/app/api/doubts/route.ts`:
- Around line 263-271: The call to safeGenerateEmbedding is unbounded and can
stall the POST; wrap or replace the call to
safeGenerateEmbedding(embeddingInput) with a bounded/abortable request using a
timeout (AbortController or Promise.race with a timeout promise) so the request
path never waits indefinitely; if the embedding call times out or errors,
proceed with a safe fallback (e.g., null or empty array) and still perform
db.update(doubtsTable).set({ embedding: ... }) only when a valid embedding
arrives, and ensure you log the timeout/error context (include embeddingInput
and the error) for debugging.

In `@src/lib/ai/embeddings.ts`:
- Around line 94-96: The code currently treats an embedding generation failure
as "no duplicates" by doing if (!queryEmbedding) return [], which masks provider
outages; instead, when safeGenerateEmbedding(embeddingInput) yields no embedding
(queryEmbedding falsy) raise/propagate an error (or return a distinct sentinel
like null) so callers can detect the failure and trigger LLM fallback; replace
the early return in the block that calls safeGenerateEmbedding with an explicit
throw (or null return) and update any callers to handle that error/sentinel
rather than assuming an empty match.
- Around line 34-39: The current check in embeddings.ts lets non-empty
embeddings with the wrong length slip through; enforce exact dimensionality by
validating that vector is an array and vector.length === EMBEDDING_DIMENSIONS
before returning and throw a descriptive Error if it isn't. Update the existing
conditional around vector/EMBEDDING_DIMENSIONS so any length mismatch (including
zero) triggers an error like "Groq returned embedding with incorrect dimensions"
and include actual lengths in the message to aid debugging.

---

Outside diff comments:
In `@src/configs/schema.ts`:
- Line 2: doubtsTable.embedding currently uses a non-runtime symbol `any()`
which crashes module initialization; replace that column declaration with the
pgvector Drizzle column type (a vector(1536) column) by importing and using the
appropriate pgvector/vector factory instead of `any()` for
doubtsTable.embedding, and then update the code in the doubts route that writes
embeddings (the writer that currently does `embedding as any`) to pass the
correct pgvector driver value type produced by your embedding code (no `as any`
cast). Ensure you update the imports where embedding is declared and the
insertion/update call in the route so the column type and the value type match
(vector(1536) for doubtsTable.embedding and the driver-specific vector value
when inserting).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7d91b86b-fc2f-467e-98f8-a715f82faaee

📥 Commits

Reviewing files that changed from the base of the PR and between 291062f and 9fad971.

📒 Files selected for processing (6)

• drizzle/0009_add_doubt_embeddings_pgvector.sql
• src/app/api/doubts/check-similarity/route.ts
• src/app/api/doubts/route.ts
• src/app/ask-ai/page.tsx
• src/configs/schema.ts
• src/lib/ai/embeddings.ts

[knoxiboy]

Hi @Aditya8369! Thanks for your work on this. Before we can merge, please address the following:

• Implement the 8 unresolved suggestions from CodeRabbit/CodeAnt AI (make sure to reply or resolve the review threads).

Once these are fixed, the PR will be automatically evaluated again. Let us know if you need any help!

[codeant-ai]

CodeAnt AI Incremental review completed.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/app/api/replies/route.ts`:
- Around line 66-67: Remove the unsafe type assertions and add explicit runtime
guards before calling eq(...) with membershipsTable.userEmail and
membershipsTable.classroomId: validate that email is a non-empty string and that
doubt.classroomId is a non-null number (doubt.classroomId) and return a proper
error/abort if they are missing/invalid; then pass the validated values into
eq(...) instead of using as string/number. Update the code paths that branch on
teacher vs student checks so both validate email and classroomId where used
(references: email, doubt.classroomId, eq(), membershipsTable.userEmail,
membershipsTable.classroomId) and keep behavior consistent with the other
membership checks that do not use type assertions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4c375515-0838-4ece-a551-aeeda971c26f

📥 Commits

Reviewing files that changed from the base of the PR and between 3df4dae and e187f78.

📒 Files selected for processing (3)

• src/__tests__/api/doubts.test.ts
• src/app/api/doubts/check-similarity/route.ts
• src/app/api/replies/route.ts

💤 Files with no reviewable changes (1)

• src/tests/api/doubts.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/app/api/doubts/check-similarity/route.ts

[Aditya8369]

@knoxiboy check now

[knoxiboy]

Hi! This PR currently has merge conflicts with the main branch. Please rebase/merge main and resolve conflicts so we can review and merge it. Thank you!

[codeant-ai]

CodeAnt AI is running Incremental review

[Aditya8369]

@knoxiboy solved

[codeant-ai]

CodeAnt AI Incremental review completed.