🏥 Medical Prescription OCR System with Multi-Agent Architecture

An advanced OCR system for medical prescriptions using a multi-agent architecture with privacy-first PHI filtering and automated drug information extraction.

🎯 System Overview

This system processes medical prescription images through a sophisticated multi-agent pipeline that:

Extracts text from prescription images using Azure Vision OCR
Detects and redacts PHI (Protected Health Information) automatically
Identifies medications and queries multiple drug databases for alternatives
Provides drug information from FDA, RxNorm, and AI-powered sources

Key Features

🤖 6-Agent Architecture - Orchestrated multi-agent system for specialized tasks
🔒 HIPAA-Compliant PHI Filtering - Automatic detection and redaction of sensitive information
💊 Drug Information Extraction - Automated medication detection with database queries
🔄 Multi-Database Drug Lookup - RxNorm (NIH), FDA openFDA, and LLaMA AI fallback
🖼️ Image Segmentation - SAM2-powered region detection (with fallback)
📱 Modern React UI - Beautiful, responsive interface for image upload and results display

🏗️ Architecture

Multi-Agent System

The system uses 6 specialized agents coordinated by an orchestrator:

┌─────────────────────────────────────────────────────────────┐
│                    OrchestratorAgent                         │
│              (Routes tasks to specialized agents)            │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│                        OCRAgent                              │
│           (Coordinates full pipeline workflow)               │
└─────────────────────────────────────────────────────────────┘
           ↓              ↓              ↓              ↓
  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────────┐
  │Segmentation│  │    Text    │  │    PHI     │  │    Drug    │
  │   Agent    │  │Recognition │  │   Filter   │  │Information │
  │            │  │   Agent    │  │   Agent    │  │   Agent    │
  └────────────┘  └────────────┘  └────────────┘  └────────────┘

Processing Pipeline

Image Upload
    ↓
┌─────────────────────────────────────┐
│  STEP 1: Image Segmentation         │
│  Agent: SegmentationAgent           │
│  Tools: SAM2 / Fallback Contours    │
│  Output: Detected regions           │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  STEP 2: Text Recognition           │
│  Agent: TextRecognitionAgent        │
│  Tools: Azure Vision OCR / TrOCR    │
│  Output: Extracted text             │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  STEP 3: PHI Filtering              │
│  Agent: PHIFilterAgent              │
│  Tools: HuggingFace NER + Regex     │
│  Output: Redacted text, PHI list    │
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  STEP 4: Drug Information           │
│  Agent: DrugInformationAgent        │
│  Tools: RxNorm, FDA, LLaMA AI       │
│  Output: Medications + Alternatives │
└─────────────────────────────────────┘
    ↓
Final Response to UI

🤖 Agent Details

1. OrchestratorAgent

Purpose: Routes incoming tasks to the appropriate specialized agent
Routing Rules:
- "prescription" → OCRAgent
- "segment" → SegmentationAgent
- "phi" / "hipaa" → PHIFilterAgent
- "drug" / "medication" → DrugInformationAgent
Capabilities: Task delegation, agent discovery, priority-based routing

2. OCRAgent

Purpose: Coordinates the entire OCR pipeline
Workflow:
1. Delegates segmentation to SegmentationAgent
2. Delegates text extraction to TextRecognitionAgent
3. Delegates PHI filtering to PHIFilterAgent
4. Delegates drug extraction to DrugInformationAgent
Processing Modes:
- full - Complete pipeline (default)
- ocr_only - Text extraction only
- segment_only - Image segmentation only

3. SegmentationAgent

Purpose: Detects and extracts regions from prescription images
Tools:
- SAM2 (Segment Anything Model 2) - Advanced AI segmentation
- Fallback Contour Detection - OpenCV-based region detection
- Region Extraction - Crops and preprocesses detected regions
Output: List of regions with bounding boxes and cropped images

4. TextRecognitionAgent

Purpose: Extracts text from images using OCR
Tools:
- Azure Vision OCR - Microsoft's cloud OCR API (primary)
- TrOCR - Transformer-based handwriting recognition (optional)
Methods:
- auto - Tries all available methods
- azure - Azure Vision only
- trocr - TrOCR only
Output: Extracted text with confidence scores

5. PHIFilterAgent

Purpose: Detects and redacts Protected Health Information
Tools:
- HuggingFace NER - Named Entity Recognition for person names
- Regex Patterns - Pattern matching for structured PHI
Detects:
- 👤 Names - Patient and doctor names
- 📅 Dates - Birth dates, appointment dates
- 📞 Phone Numbers - All formats
- 📧 Email Addresses
- 🏠 Addresses - Street addresses
- 🔢 SSN - Social Security Numbers
- 🆔 Medical IDs - MRNs, Patient IDs
- 💳 Insurance Numbers
Output: Redacted text (PHI replaced with [TYPE_REDACTED]), PHI entity list

6. DrugInformationAgent

Purpose: Extracts medications and queries drug databases
Tools:
- FAISS Vector Database - Local database of 106 essential medications with semantic search
- Medication Extractor - Regex-based pattern matching
- RxNorm API - NIH drug database (free, international)
- FDA openFDA API - US FDA drug labels database
- LLaMA AI - AI-powered fallback for unknown drugs
Query Priority:
1. Vector Database (local, 10ms response time)
2. RxNorm API (NIH, international coverage)
3. FDA openFDA (US drug labels)
4. LLaMA AI (AI-generated fallback)
Extraction Patterns:
- Tab. Augmentin 625mg
- Cap. Amoxicillin 500mg
- Drug names with common suffixes (cillin, mycin, pril, etc.)
- 100+ common medication names
Output:
- Medication list (name + dosage)
- Drug alternatives from multiple sources
- Manufacturer information
- Indications and usage
- Category and usage classification
- Multi-source information combining all databases

🛠️ Technical Stack

Backend

Framework: FastAPI (Python)
Agent System: Custom multi-agent framework with async/await
OCR: Azure Vision API
Segmentation: SAM2 (Meta AI) + OpenCV fallback
NLP: HuggingFace Transformers (NER models)
Vector Database: FAISS + SentenceTransformers (all-MiniLM-L6-v2)
Drug APIs: RxNorm (NIH), openFDA, HuggingFace LLaMA
Image Processing: OpenCV, NumPy, PIL

Frontend

Framework: React 18
Build Tool: Vite
Styling: CSS3 with modern features
State Management: React Hooks

APIs & Services

Azure Vision API - Text extraction
RxNorm REST API - Drug information
FDA openFDA API - US drug labels
HuggingFace Inference API - NER and LLaMA
Meta SAM2 - Image segmentation
FAISS Vector Database - Local medication database (106 drugs)

📁 Project Structure

ocr/
├── backend/                          # Backend API server
│   ├── agent_system.py              # Agent system initialization
│   ├── main_agent_api.py            # FastAPI server with agent routes
│   ├── main.py                      # Legacy API (reference)
│   ├── requirements.txt             # Python dependencies
│   ├── .env                         # Configuration (keys, tokens)
│   ├── drugs.json                   # Essential medications dataset (106 drugs)
│   ├── load_drugs_json.py           # Load medications into vector DB
│   ├── medication_vector_db.py      # Vector database implementation
│   ├── agents/                      # Agent implementations
│   │   ├── __init__.py
│   │   ├── base_agent.py           # Base agent class
│   │   ├── orchestrator.py         # Task routing agent
│   │   ├── ocr_agent.py            # Pipeline coordinator
│   │   ├── segmentation_agent.py   # Image segmentation
│   │   ├── text_recognition_agent.py # Text extraction
│   │   ├── phi_filter_agent.py     # PHI detection/redaction
│   │   ├── drug_information_agent.py # Medication extraction
│   │   └── tools.py                # Tool implementations
│   ├── medication_db/               # Vector database storage
│   │   ├── faiss_index.bin         # FAISS vector index
│   │   └── metadata.json           # Medication metadata
│   ├── checkpoints/                 # Model checkpoints
│   │   └── sam2_hiera_large.pt     # SAM2 model (900MB)
│   └── segment-anything-2/          # SAM2 library
│
├── src/                             # React frontend
│   ├── components/
│   │   ├── LandingPage.jsx         # Main UI component
│   │   └── LandingPage.css         # Styling
│   ├── App.jsx
│   └── main.jsx
│
├── index.html
├── package.json
├── vite.config.js
└── README.md

🚀 Getting Started

Prerequisites

Python 3.8+
Node.js 16+
Azure Vision API credentials
HuggingFace API token (optional, for LLaMA)

Backend Setup

Install Python dependencies:

cd backend
pip install -r requirements.txt

Build Vector Database (essential medications):

# Load 106 essential medications from drugs.json into vector database
python load_drugs_json.py drugs.json

This creates a FAISS vector database in medication_db/ with semantic search capabilities.

Install SAM2 (optional, for advanced segmentation):

cd segment-anything-2
pip install -e .

Download SAM2 checkpoint (optional):

cd ..
python download_sam2_checkpoint.py

Configure environment variables (.env file):

# Azure Vision API (Required)
AZURE_VISION_ENDPOINT=https://your-resource.cognitiveservices.azure.com/
AZURE_VISION_KEY=your_azure_key

# HuggingFace API (Optional - for LLaMA AI fallback)
HF_TOKEN=your_huggingface_token

Start the agent API server:

python main_agent_api.py

Server will be available at http://localhost:8000

Frontend Setup

Install Node dependencies:

npm install

Start development server:

npm run dev

App will be available at http://localhost:5173

📡 API Endpoints

Main Endpoints

`POST /api/process-image`

Process an image through the full agent pipeline.

Request:

curl -X POST http://localhost:8000/api/process-image \
  -F "file=@prescription.jpg" \
  -F "mode=full" \
  -F "filter_phi=true"

Parameters:

file - Image file (JPEG, PNG)
mode - Processing mode: full, ocr_only, segment_only
filter_phi - Enable PHI filtering (default: true)
include_regions - Include region details (default: false)

Response:

{
  "success": true,
  "mode": "full",
  "agent_used": "OCRAgent",
  "tools_used": ["azure_vision_ocr", "filter_phi", "extract_medications"],
  "regions_detected": 3,
  "extracted_text": "Tab. Augmentin 625mg...",
  "redacted_text": "[NAME_REDACTED]...",
  "phi_summary": [
    {"type": "NAME", "original": "John Doe"},
    {"type": "DATE", "original": "12-09-2025"}
  ],
  "medications": [
    {"name": "augmentin", "dosage": "625mg"}
  ],
  "drug_alternatives": [
    {
      "original_drug": {"name": "augmentin", "dosage": "625mg"},
      "drug_info": {
        "found": true,
        "alternatives": [...],
        "source": "RxNorm (NIH)"
      }
    }
  ]
}

`POST /api/phi/filter`

Filter PHI from text directly (without image).

Request:

curl -X POST http://localhost:8000/api/phi/filter \
  -H "Content-Type: application/json" \
  -d '{"text": "Patient: John Doe, Age: 45, Phone: 555-1234"}'

`GET /health`

Check system health and status.

Response:

{
  "status": "healthy",
  "agent_system_initialized": true,
  "sam2_loaded": false,
  "trocr_loaded": false
}

`GET /api/agent-status`

Get detailed agent system status.

🔧 Configuration

Processing Modes

full - Complete pipeline (recommended)
- Segmentation → OCR → PHI Filtering → Drug Extraction
ocr_only - Text extraction only
- Skips segmentation, directly extracts text
segment_only - Image segmentation only
- Returns detected regions without text extraction

Agent Configuration

Edit agent_system.py to:

Enable/disable SAM2 segmentation
Enable/disable TrOCR handwriting recognition
Configure model checkpoints
Adjust agent routing rules

PHI Filtering Rules

Customize PHI detection in agents/phi_filter_agent.py:

Add custom regex patterns
Configure NER model
Adjust redaction format

Drug Extraction Patterns

Customize medication extraction in agents/drug_information_agent.py:

Add medication name patterns
Configure database priorities
Adjust extraction rules

🧪 Testing

Test with Sample Image

# Backend
cd backend
python main_agent_api.py

# In another terminal
curl -X POST http://localhost:8000/api/process-image \
  -F "file=@test_prescription.jpg" \
  -F "mode=full"

Test Individual Agents

from agent_system import get_agent_system
import asyncio

async def test():
    # Initialize system
    agent_system = await get_agent_system()
    
    # Test PHI filtering
    response = await agent_system.phi_filter_agent.process(
        "Filter PHI",
        {"text": "Patient: John Doe, DOB: 01/15/1980"}
    )
    print(response.data)

asyncio.run(test())

🔒 Privacy & Security

PHI Protection

All PHI is detected and redacted before drug information queries
No PHI is sent to external APIs (RxNorm, FDA, LLaMA)
Redacted text uses placeholder format: [TYPE_REDACTED]

HIPAA Compliance Features

✅ Automatic PHI detection and redaction
✅ No PHI logging in agent responses
✅ Privacy-first design (PHI filtered before external API calls)
⚠️ Note: This is a demonstration system. For production HIPAA compliance, additional measures required:
- Encrypted storage
- Access logging
- Audit trails
- BAA with cloud providers

🎨 UI Features

Image Upload

Drag & drop support
Format support: JPEG, PNG
Real-time processing feedback

Results Display

Agent Badge - Shows which agent processed the image
Tools Badge - Lists tools used in pipeline
Text Display - Original extracted text
Redacted Text - PHI-filtered version
PHI Summary - Detailed list of detected PHI entities
Medications - Extracted drugs with dosages
Drug Alternatives - Database results for each medication

Interactive Features

Toggle between original/segmented image view
Copy text to clipboard
Download full report with metadata
Show/hide detailed metadata
Region statistics display

🐛 Troubleshooting

"Agent system not initialized"

Solution: Wait a few seconds after server start for initialization.

"No text detected"

Solutions:

Verify Azure Vision API credentials in .env
Check image quality and resolution
Ensure prescription is clearly visible

"SAM2 not loaded"

Solution: This is optional. System uses fallback segmentation. To enable:

cd backend/segment-anything-2
pip install -e .
cd ..
python download_sam2_checkpoint.py

"TrOCR not loaded"

Solution: Optional for handwriting. To enable:

pip install "huggingface-hub>=0.24.0,<1.0"
pip install transformers --upgrade

📊 Performance

Processing Times (Approximate)

Segmentation: 1-3 seconds (with fallback)
SAM2 Segmentation: 5-10 seconds (if enabled)
Azure Vision OCR: 2-5 seconds
PHI Filtering: <1 second
Drug Information: 1-3 seconds per medication

Total Pipeline: 5-15 seconds per image

Optimization Tips

Use ocr_only mode if segmentation not needed
Disable SAM2 for faster processing (fallback is sufficient)
Cache drug information results
Process multiple images in batch mode

🔄 Future Enhancements

📝 License

This project is for educational and demonstration purposes.

🤝 Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Submit a pull request

📧 Support

For issues or questions, please open an issue on GitHub.

🙏 Acknowledgments

Azure Vision API - Microsoft Cognitive Services
SAM2 - Meta AI Research
RxNorm - National Library of Medicine (NLM)
openFDA - U.S. Food & Drug Administration
HuggingFace - Transformers and Inference API
FastAPI - Modern Python web framework
React - Facebook Open Source

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
backend		backend
src		src
.env.production		.env.production
.gitignore		.gitignore
README.md		README.md
SETUP.md		SETUP.md
drugs.json		drugs.json
index.html		index.html
package-lock.json		package-lock.json
package.json		package.json
vite.config.js		vite.config.js

Folders and files

Latest commit

History

Repository files navigation

🏥 Medical Prescription OCR System with Multi-Agent Architecture

🎯 System Overview

Key Features

🏗️ Architecture

Multi-Agent System

Processing Pipeline

🤖 Agent Details

1. OrchestratorAgent

2. OCRAgent

3. SegmentationAgent

4. TextRecognitionAgent

5. PHIFilterAgent

6. DrugInformationAgent

🛠️ Technical Stack

Backend

Frontend

APIs & Services

📁 Project Structure

🚀 Getting Started

Prerequisites

Backend Setup

Frontend Setup

📡 API Endpoints

Main Endpoints

POST /api/process-image

POST /api/phi/filter

GET /health

GET /api/agent-status

🔧 Configuration

Processing Modes

Agent Configuration

PHI Filtering Rules

Drug Extraction Patterns

🧪 Testing

Test with Sample Image

Test Individual Agents

🔒 Privacy & Security

PHI Protection

HIPAA Compliance Features

🎨 UI Features

Image Upload

Results Display

Interactive Features

🐛 Troubleshooting

"Agent system not initialized"

"No text detected"

"SAM2 not loaded"

"TrOCR not loaded"

📊 Performance

Processing Times (Approximate)

Optimization Tips

🔄 Future Enhancements

📝 License

🤝 Contributing

📧 Support

🙏 Acknowledgments

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

`POST /api/process-image`

`POST /api/phi/filter`

`GET /health`

`GET /api/agent-status`

Packages