AI Research Agent

A multi-step AI research pipeline for city intelligence — powered by Groq, Tavily, and Inngest.

---

What It Does

AI Research Agent takes a city name and a set of custom research criteria, then orchestrates a three-stage pipeline to produce a structured JSON report:

1. Prompt Enhancement — a Groq LLM refines your criteria into precise search queries
2. Web Research — Tavily performs deep web searches for each criterion
3. Data Extraction — a second Groq LLM reviews results and extracts structured JSON

All pipeline steps run as durable background jobs via Inngest, so long-running research survives network drops and server restarts.

---

Key Features

BYOK — Bring Your Own Keys

API keys (Groq, Tavily, Mixedbread, Pinecone) are entered at runtime in the dashboard UI and never stored server-side. You control your own usage and billing.

Research Pipeline

• Single-city or multi-criteria research
• Custom criteria builder with field descriptions
• Real-time activity logs and progress tracking
• Downloadable JSON reports
• Automatic fallback to a secondary model on rate-limit errors

Extended Research (Human-in-the-Loop)

• Searches for candidate sources, then pauses for manual link selection
• Resumes processing only the sources you choose
• Supports injecting a specific URL for targeted extraction

Vector Store

• Embeds custom text via Mixedbread's mxbai-embed-large-v1
• Upserts to Pinecone in batches of 10
• Chunked via LangChain's RecursiveCharacterTextSplitter

MongoDB Schema Detection

• Connect to any MongoDB instance with a connection string
• Lists real collections from your database
• Fetches sample documents and extracts unique field names
• One-click "Add to Search Criteria" to enrich existing schemas

Search History

• All completed jobs are stored per-user in MongoDB
• Browse, re-download, and delete past reports

Agent Planner

• Multi-step research plans generated by LLM
• Each plan step runs as an independent sub-job

---

Tech Stack

Layer	Technology
Framework	Next.js 16 (App Router)
Language	TypeScript
Styling	Tailwind CSS v4 + shadcn/ui
AI Models	Groq (LLaMA 3, Compound) via LangChain
Web Search	Tavily API
Embeddings	Mixedbread mxbai-embed-large-v1
Vector DB	Pinecone
Background Jobs	Inngest
Auth	NextAuth v5 (Google OAuth + credentials)
Database	MongoDB (Mongoose)
Error Tracking	Sentry
Analytics	PostHog

---

Getting Started

Prerequisites

• Node.js 18+
• MongoDB instance (local or Atlas)
• Inngest account (free tier works)

Installation

git clone https://github.com/vanshaj2023/AI-Research-Agent.git
cd AI-Research-Agent
npm install

Environment Variables

Copy .env.example to .env.local and fill in:

cp .env.example .env.local

Variable	Purpose
MONGODB_URI	MongoDB connection string
AUTH_SECRET	NextAuth secret — generate with npx auth secret
AUTH_GOOGLE_ID / AUTH_GOOGLE_SECRET	Google OAuth credentials
NEXT_PUBLIC_POSTHOG_KEY / NEXT_PUBLIC_POSTHOG_HOST	PostHog analytics
NEXT_PUBLIC_SENTRY_DSN	Sentry DSN

AI provider keys (Groq, Tavily, Mixedbread, Pinecone) are entered by users in the dashboard UI — no server-side config needed.

Running Locally

Two terminals are required:

Terminal 1 — Next.js:

npm run dev

Terminal 2 — Inngest dev server (required for background jobs):

npm run inngest

Open http://localhost:3000.

Background research will silently fail if Inngest is not running.

---

Docker

# Start app + MongoDB
npm run docker:up        # http://localhost:3002

# With hot reload
npm run docker:dev       # http://localhost:3001

# Stop
npm run docker:down

# Logs
npm run docker:logs

See docs/DOCKER.md for full Docker documentation.

---

Architecture

Research Request Lifecycle

POST /api/research
  → rate limit check
  → auth check (optional — unauthenticated users can research)
  → cache lookup
  → create job in jobs_db.json + MongoDB
  → emit Inngest event "research/start"
  → client polls GET /api/research/status?id=[jobId]

Inngest "research-flow":
  → createPromptEnhancer  — Groq LLM (query generation)
  → createResearcher      — Tavily web search
  → createReviewer        — Groq LLM (JSON extraction)
  → update job state in jobs_db.json + MongoDB

Dual Job Store

Jobs are written to two places simultaneously (lib/store.ts):

1. jobs_db.json — fast local read for status polling (ephemeral, not committed)
2. MongoDB AgentJob — durable, per-user history

Inngest Functions

Function	Event	Description
research-flow	research/start	Main research pipeline; cancellable via research/cancel
process-embeddings	vector/start-embedding	Chunk, embed, and upsert to Pinecone
extended-research-flow	research/extended	Human-in-the-loop with link selection
agent-runner	agent/execute-plan	Multi-step plan executor

---

Project Structure

app/
  api/             — API routes
  auth/            — Sign-in / sign-up pages
  dashboard/       — Main single-page application
  cache/           — Admin cache dashboard
components/        — React components (shadcn/ui based)
lib/
  agents/          — LLM agent factories (prompt-enhancer, researcher, reviewer)
  inngest/         — Background job functions
  models/          — Mongoose models (User, AgentJob, SearchHistory)
  store.ts         — Dual-write job store
  cache.ts         — In-memory TTL cache
  rate-limit.ts    — Per-IP rate limiting
  mongodb.ts       — Singleton Mongoose connection
docs/              — Extended documentation

---

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/my-feature
3. Commit your changes
4. Open a pull request

---

License

MIT — see LICENSE for details.