DialogLens-AI

Conversational Analytics Dashboard

A lightweight dashboard to analyze player/chat conversations in real-time using sentiment analysis, entity extraction, and topic clustering.

---

1) architecture diagram

+----------------------+        +----------------------+        +----------------------+
|      Frontend        | <----> |       Backend        | <----> |      NLP Worker      |
| (React, Chart.js)    |  HTTP  | (FastAPI/Express)    |  Queue | (Python: HF,spaCy,   |
| - Live dashboards    |        | - REST / SSE / WS    | <----> |  s-transformers)     |
+----------------------+        +---------+------------+        +---------+------------+
                                           |                               |
                                           | DB / Vector Store             | Object storage (logs)
                                           v                               v
                                 +----------------------+        +----------------------+
                                 |    Postgres / Redis  |        |       Faiss / Milvus |
                                 | (OLTP, sessions, CV) |        | (vector index for     |
                                 +----------------------+        |  topic clustering)    |
                                                                      +----------------------+

Optional: Alerting -> (Slack/Discord webhook) and Deployment -> Docker, GitHub Actions, Vercel (frontend) / Render or Cloud Run (backend)

---

2) Recommended folder structure

conversational-analytics/
├── frontend/                     # React app (dashboard)
│   ├── public/
│   └── src/
│       ├── components/
│       ├── pages/
│       ├── api/                  # client API wrappers
│       └── styles/
├── backend/                      # FastAPI (or Express) server
│   ├── app/
│   │   ├── api/                  # REST endpoints, SSE/ws
│   │   ├── core/                 # config, logging, auth
│   │   ├── db/                   # models, migrations
│   │   └── services/             # ingestion, alerting
│   └── tests/
├── nlp_worker/                   # NLP pipelines / batch & streaming jobs
│   ├── models/                   # model config and wrappers
│   ├── pipelines/                # sentiment.py, ner.py, embeddings.py
│   └── tasks/                    # worker entrypoints (celery / rq / simple loop)
├── data/                         # sample datasets, synthetic generators
├── infra/                        # docker-compose, terraform / k8s manifests
├── scripts/                      # helpers: seed_db.py, gen_synthetic_chat.py
├── docs/                         # architecture diagrams, notes
├── .env.example
├── docker-compose.yml
├── README.md                     # this file (starter template)
└── LICENSE

Notes:

• Keep nlp_worker separate so you can scale/replace it independently. Use Redis/RQ, Celery, or simple Pub/Sub depending on scale.
• backend/services should expose a streaming endpoint (SSE or WebSocket) for the dashboard's live updates.

---

Demo

• Live demo: (add link if deployed)
• Screenshot: docs/screenshot.png

Why this project

• Quickly surfaces player issues and trends from chat logs.
• Useful for product decisions, QA triage, and monitoring player sentiment around releases.

Tech stack

• Frontend: React, Chart.js / Recharts, Tailwind CSS
• Backend: FastAPI (or Express), SSE / WebSockets for streaming
• NLP: Hugging Face Transformers (DistilBERT), spaCy, sentence-transformers
• Vector store / search: Faiss (local) or Milvus
• Queue/Cache: Redis (pub/sub or RQ)
• Storage: Postgres (metadata), optionally S3 for raw logs
• Deployment: Docker, Vercel (frontend), Render/Cloud Run (backend)

Architecture

(Include the ASCII diagram or paste a rendered image from docs/architecture.png.)

Quick start (local, Docker Compose)

Prerequisites

• Docker & Docker Compose
• Node 18+ and npm/yarn (for frontend dev)

Run everything locally

# copy env example
cp .env.example .env
# build and run
docker-compose up --build

Frontend will be at http://localhost:3000 and backend at http://localhost:8000 by default.

Run only backend

cd backend
pip install -r requirements.txt
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Run only frontend

cd frontend
npm install
npm run dev

Sample environment variables (.env.example)

# Backend
BACKEND_PORT=8000
DATABASE_URL=postgresql://user:pass@db:5432/conv_analytics
REDIS_URL=redis://redis:6379/0
VECTOR_INDEX_DIR=/data/faiss

# NLP
SENTIMENT_MODEL=distilbert-base-...-finetuned-sentiment
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2

Features

• Message-level sentiment & conversation-level aggregation
• Named-entity frequency and co-occurrence
• Topic clustering (embeddings + k-means / HDBSCAN)
• Live dashboard with filters (region, VIP status, time range)
• Alerting via Slack/Discord webhook for sentiment spikes
• Full conversation pipeline with conversation-level metrics
• Entity Recognition & Topic Tagging using spaCy and Hugging Face models
• Vector Database for Search & Clustering using FAISS

API endpoints

• GET / — root endpoint
• GET /vector-stats — get statistics about the vector database
• POST /ingest — submit chat messages (bulk)
• GET /conversation/{conversation_id} — get a full conversation with all messages and metrics
• GET /search?text={text}&k={k} — search for messages similar to the provided text
• GET /api/sentiment?start=...&end=... — aggregated sentiment time series
• GET /api/topics?limit=10 — top N topics
• GET /api/stream — SSE stream of processed messages

Data

• data/sample_chat.jsonl — one message per line: {"user_id":..., "timestamp":..., "text":...}
• A synthetic generator is available at scripts/gen_synthetic_chat.py.

Evaluation & notes

• Use small holdout to sanity-check sentiment & topic coherence.
• Strip PII before storing logs; mention in docs how you anonymize.

Deploy

• Frontend: Vercel (connect repo)
• Backend: Render / Cloud Run / Heroku
• Use GitHub Actions for CI: run linters and tests, build Docker image and push to container registry

Roadmap / Stretch goals

• Fine-tune a sentiment model on game-chat style data
• Add real-time anomaly detection (seasonal-ESD)
• Slack integration for triage channels
• Support for Milvus as an alternative to FAISS
• Integration with NeonDatabase as requested

Contributing

• See CONTRIBUTING.md for code standards

License

MIT

Contact

• Author: Utkarsh Gual