🎙️ TEAM-5 — Speech Personality Analysis Pipeline

An advanced AI-powered speech analysis system that delivers real-time personality insights and communication feedback using NVIDIA NIM, RAG (Retrieval-Augmented Generation), and a multi-agent AI architecture.

---

📋 Table of Contents

• Overview
• Features
• Architecture
• System Requirements
• Quick Start
• Installation
• Usage
• Project Structure
• API Documentation
• Configuration
• Development
• Testing & Evaluation
• Troubleshooting
• Contributing
• License

---

🌟 Overview

TEAM-5 is a full-stack speech analysis pipeline that combines:

• 🎤 Faster-Whisper for high-accuracy speech-to-text
• 🔬 Acoustic feature extraction (pitch, energy, pauses, speech rate)
• 🤖 Multi-agent AI system powered by NVIDIA NIM (meta/llama-3.1-70b-instruct)
• 📚 RAG with ChromaDB for knowledge-augmented personality insights
• 🛡️ Guardrails AI for input/output safety validation
• 📊 LangChain Evaluators for quality assurance

Users can upload audio (any common format: WAV, WebM, MP3, OGG) via the React + TypeScript frontend, and receive structured analysis covering communication clarity, emotional confidence, and personality traits — along with actionable improvement tips.

---

✨ Features

Backend

Feature	Details
🎤 Audio Ingestion	Multi-format support (WebM, WAV, MP3, OGG); auto-converted to 16-bit PCM WAV via PyAV
📝 Speech-to-Text	High-accuracy transcription using Faster-Whisper
📊 Acoustic Analysis	Pitch variance, energy level, speech rate, pause ratio
🧠 Multi-Agent System	Communication, Confidence, and Personality agents run in parallel (ThreadPoolExecutor)
🔍 RAG System	ChromaDB + all-MiniLM-L6-v2 embeddings for expert knowledge retrieval
🛡️ Guardrails AI	Input/output validation and safety checks
📈 Evaluation Framework	LangChain-based quality assessment with optional refinement loop
🚀 REST API	FastAPI v2.0 with CORS, health check, and auto-generated docs

Frontend

Feature	Details
⚡ React 19 + TypeScript	Modern, type-safe UI with Vite 7 build tooling
🎨 UI Components	Radix UI primitives + Tailwind CSS v4
📤 Audio Upload	Drag-and-drop or file picker for any audio format
📊 Results Visualization	Tabbed display of transcript, acoustic features, and agent analyses
⏱️ Real-time Feedback	Live processing status updates
📱 Responsive Design	Works across desktop and mobile devices

---

🏗️ Architecture

┌──────────────────────────────────────────────────────────┐
│                     Frontend (React 19 + Vite)           │
│         Radix UI  ·  TypeScript  ·  Tailwind CSS v4      │
└─────────────────────────┬────────────────────────────────┘
                          │ HTTP / REST
                          ▼
┌──────────────────────────────────────────────────────────┐
│               Backend API  (FastAPI 2.0 + Uvicorn)       │
├──────────────────────────────────────────────────────────┤
│                   Speech Processing Pipeline              │
│                                                          │
│  [Audio Upload] → [PyAV Conversion] → [Preprocessing]   │
│        ↓                                                 │
│  [Faster-Whisper STT] → [Acoustic Feature Extraction]   │
│        ↓                                                 │
│  ┌─────────────────────────────────────┐                │
│  │       Multi-Agent System            │                │
│  │  ┌─────────────┐ ┌───────────────┐ │                │
│  │  │Communication│ │  Confidence   │ │  ← parallel    │
│  │  │   Agent     │ │    Agent      │ │                │
│  │  └─────────────┘ └───────────────┘ │                │
│  │           ↓ combined ↓             │                │
│  │  ┌──────────────────────────────┐  │                │
│  │  │      Personality Agent       │  │                │
│  │  └──────────────────────────────┘  │                │
│  └─────────────────────────────────────┘                │
│        ↓                                                 │
│  [RAG Retrieval (ChromaDB)] → [Report Generator]        │
│  [Guardrails Validation]    → [LangChain Evaluation]    │
└───────────────────────────────────────────────────────────┘
                          │
                          ▼
              ┌──────────────────────┐
              │    NVIDIA NIM API    │
              │  meta/llama-3.1-70b  │
              └──────────────────────┘

Agent Breakdown

Agent	Responsibility
Communication Agent	Analyzes clarity, fluency, vocabulary, and structural coherence
Confidence Agent	Evaluates vocal confidence, energy, and emotional tone
Personality Agent	Maps communication + confidence patterns to personality traits

Communication and Confidence agents run concurrently via ThreadPoolExecutor. Personality agent runs after both complete.

---

💻 System Requirements

Minimum

Requirement	Minimum
OS	Linux, macOS, or Windows 10+
Python	3.8+
Node.js	18.x+
RAM	8 GB
Storage	~5 GB (models + deps)
Microphone	Required for live recording

Recommended

• RAM: 16 GB+
• GPU: CUDA-compatible (optional, speeds up Whisper inference)
• Storage: SSD with 10 GB+ free

---

🚀 Quick Start

# 1. Clone the repository
git clone https://github.com/dinesh9997/TEAM-05.git
cd TEAM-05

# 2. Backend setup
cd backend
python -m venv venv
# Linux/macOS:
source venv/bin/activate
# Windows:
venv\Scripts\activate

pip install -r requirements.txt

# 3. Configure environment
cp .env.example .env
# Edit .env and add your NVIDIA_API_KEY

# 4. Start backend
uvicorn api:app --reload --port 8000

# 5. In a NEW terminal — Frontend
cd ../frontend
npm install
npm run dev

• Backend API: http://localhost:8000
• Frontend UI: http://localhost:5173
• API Docs: http://localhost:8000/docs

---

📦 Installation

1. Clone Repository

git clone https://github.com/dinesh9997/TEAM-05.git
cd TEAM-05

2. Backend Setup

cd backend
python -m venv venv

# Activate (Linux/macOS)
source venv/bin/activate

# Activate (Windows)
venv\Scripts\activate

pip install -r requirements.txt

3. Configure NVIDIA NIM API Key

Get a free API key at build.nvidia.com:

cp .env.example .env
# Open .env and set:
# NVIDIA_API_KEY=your_actual_api_key_here

Optionally override defaults in .env:

NVIDIA_API_KEY=your_key_here
NVIDIA_MODEL_NAME=meta/llama-3.1-70b-instruct   # default
LLM_TEMPERATURE=0.3                              # 0.0–1.0
LLM_MAX_TOKENS=1024

4. Frontend Setup

cd ../frontend
npm install

---

🎯 Usage

Full Stack

# Terminal 1 — Backend
cd backend && source venv/bin/activate
uvicorn api:app --reload --port 8000

# Terminal 2 — Frontend
cd frontend
npm run dev

Open http://localhost:5173, upload an audio file, and view your personality analysis report.

Backend Standalone (no frontend)

cd backend
python main.py

This will:

1. Record ~45 seconds of audio via microphone
2. Process and transcribe speech
3. Run the multi-agent analysis pipeline
4. Print a comprehensive report in the terminal

Using the API Directly

import requests

with open("audio.wav", "rb") as f:
    response = requests.post(
        "http://localhost:8000/analyze",
        files={"file": ("audio.wav", f, "audio/wav")}
    )

result = response.json()
print(result["final_report"])

---

📁 Project Structure

TEAM-05/
├── backend/                        # Python backend
│   ├── api.py                      # FastAPI app (v2.0) — REST endpoints
│   ├── main.py                     # Standalone pipeline (mic recording)
│   ├── link.py                     # Pipeline orchestrator (run_pipeline)
│   ├── agent.py                    # Agent orchestrator (run_agents)
│   ├── pipeline.py                 # Pipeline helper
│   ├── llm_helper.py               # LLM utility helpers
│   ├── guardrails_config.py        # Guardrails AI validation
│   ├── speech_to_text.py           # Faster-Whisper transcription
│   ├── speech_features.py          # Acoustic feature extraction
│   ├── record_audio.py             # Microphone recording
│   ├── preprocess_audio.py         # Audio normalization/denoising
│   ├── requirements.txt            # Python dependencies
│   ├── .env.example                # Environment variable template
│   │
│   ├── agents/                     # Multi-agent system
│   │   ├── communication_agent.py  # Clarity, fluency, structure
│   │   ├── confidence_agent.py     # Vocal confidence & emotion
│   │   └── personality_agent.py    # Personality trait mapping
│   │
│   ├── llm/                        # LLM wrapper (agents)
│   │   └── local_llm.py
│   │
│   ├── llm1/                       # LLM config & report generation
│   │   ├── llm_config.py
│   │   ├── local_llm.py
│   │   ├── prompt_templates.py
│   │   └── report_generator.py
│   │
│   ├── rag/                        # RAG system
│   │   ├── config.py               # ChromaDB settings
│   │   ├── retriever.py
│   │   ├── knowledge_base.py
│   │   ├── rag_pipeline.py
│   │   └── documents/              # Knowledge documents
│   │
│   ├── evals/                      # Evaluation framework
│   │   ├── eval_config.py
│   │   ├── eval_runner.py
│   │   ├── eval_refinement.py
│   │   └── test_evals.py
│   │
│   └── utils/                      # Shared utilities
│       ├── audio_loader.py
│       ├── parser.py
│       └── feature_scoring.py
│
├── frontend/                       # React frontend
│   ├── src/
│   │   ├── App.tsx                 # Root component
│   │   └── components/             # UI components (Radix UI)
│   ├── index.html
│   ├── package.json
│   ├── vite.config.ts
│   ├── tailwind.config.js
│   └── tsconfig.json
│
├── .gitignore
└── README.md                       # This file

---

📚 API Documentation

When the backend is running, interactive docs are available at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

GET /health

Check backend readiness and NVIDIA NIM configuration.

Response:

{
  "status": "healthy",
  "llm_provider": "nvidia-nim",
  "nvidia_configured": true,
  "model": "meta/llama-3.1-70b-instruct"
}

POST /analyze

Upload an audio file for full analysis.

Request:

POST /analyze HTTP/1.1
Content-Type: multipart/form-data

file: <audio_file>   (WAV, WebM, MP3, OGG, etc.)

Response:

{
  "transcript": "I am confident in my ability to...",
  "audio_features": {
    "speech_rate": 130,
    "pitch_variance": 22.5,
    "pause_ratio": 0.18,
    "energy_level": "medium-high"
  },
  "communication_analysis": { ... },
  "confidence_emotion_analysis": { ... },
  "personality_analysis": { ... },
  "final_report": "Detailed personalized feedback..."
}

Audio files are automatically converted to 16-bit PCM WAV internally, so any common format is accepted.

---

⚙️ Configuration

LLM — backend/llm1/llm_config.py

Configured via backend/.env:

Variable	Default	Description
NVIDIA_API_KEY	(required)	Your NVIDIA NIM API key
NVIDIA_MODEL_NAME	meta/llama-3.1-70b-instruct	LLM model to use
LLM_TEMPERATURE	0.3	Response creativity (0.0–1.0)
LLM_MAX_TOKENS	1024	Max response length

RAG — backend/rag/config.py

Setting	Default	Description
CHROMA_PERSIST_DIR	./rag/chroma_db	Vector DB storage path
COLLECTION_NAME	speech_analysis_knowledge	ChromaDB collection
TOP_K_RESULTS	3	Documents retrieved per query
CHUNK_SIZE	500	Text chunk size
CHUNK_OVERLAP	50	Chunk overlap

Recording — backend/main.py

Setting	Default
DURATION	45 seconds
SAMPLE_RATE	16000 Hz
CHANNELS	1 (mono)

---

🛠️ Development

Backend

cd backend
source venv/bin/activate   # or venv\Scripts\activate on Windows

# Dev server with hot reload
uvicorn api:app --reload --port 8000

# Code quality
black .
flake8 .

# Run tests
python -m pytest

Frontend

cd frontend

npm run dev       # Development server (http://localhost:5173)
npm run build     # Production build
npm run preview   # Preview production build
npm run lint      # ESLint check

---

🧪 Testing & Evaluation

Built-in Evaluation Suite

cd backend
python -m evals.test_evals

Criteria evaluated:

• Helpfulness, Relevance, Coherence
• Actionability, Specificity, Accuracy
• Completeness, Constructiveness

Manual Tests

# Test NVIDIA NIM LLM connection
python test_llm_step5.py

# Test RAG retrieval pipeline
python test_rag.py

# Run full standalone pipeline
python main.py

---

🔧 Troubleshooting

NVIDIA_API_KEY not set

1. Get a free key at build.nvidia.com
2. cp backend/.env.example backend/.env
3. Add your key to NVIDIA_API_KEY=...
4. Restart the backend

NVIDIA API not available

• Verify the key is valid and not expired
• Check your internet connection
• Review NVIDIA Build platform status

Import Errors / Missing Packages

# Ensure virtual environment is active
source venv/bin/activate  # venv\Scripts\activate on Windows

# Reinstall all dependencies
pip install -r requirements.txt

ChromaDB / Vector DB Issues

# Clear the vector database and reinitialize
rm -rf backend/rag/chroma_db/
# Restart the backend — it will rebuild on next run

Memory Issues

• Reduce LLM_MAX_TOKENS in .env
• Use a smaller NVIDIA NIM model
• Close memory-heavy applications
• Upgrade to 16 GB+ RAM for smooth operation

Audio Conversion Failures

The backend uses PyAV to convert all uploaded audio to 16-bit PCM WAV. If conversion fails:

• Ensure av is installed: pip install av
• Check that the uploaded file is a valid, non-empty audio file

---

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/my-feature
3. Commit your changes: git commit -m 'Add my feature'
4. Push to the branch: git push origin feature/my-feature
5. Open a Pull Request

Guidelines

• Follow PEP 8 for Python code
• Use TypeScript strictly for frontend code
• Add tests for new features
• Update documentation for any API or config changes

---

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

🙏 Acknowledgments

Library	Purpose
Faster-Whisper	Speech-to-text transcription
NVIDIA NIM	Cloud LLM API (LLaMA 3.1 70B)
LangChain	LLM evaluation framework
ChromaDB	Vector database for RAG
FastAPI	Python REST API framework
PyAV	Audio format conversion
React 19	Frontend UI library
Radix UI	Accessible UI primitives
Vite	Frontend build tooling

---

📞 Support

• Open an issue on GitHub
• Contact the project team through the repository

---

🗺️ Roadmap

• NVIDIA NIM cloud LLM integration
• Multi-agent parallel execution
• ChromaDB RAG system
• FastAPI REST backend
• React 19 + TypeScript frontend
• Multi-format audio upload (WebM, WAV, MP3, OGG)
• Real-time streaming analysis
• Multi-language speech support
• Advanced visualization dashboards
• Docker containerization
• Mobile app

---

Built with ❤️ by TEAM-5