+ Built with โค๏ธ by the AutoFlow Team +
diff --git a/SYSTEM_ARCHITECTURE.md b/SYSTEM_ARCHITECTURE.md new file mode 100644 index 0000000..b364045 --- /dev/null +++ b/SYSTEM_ARCHITECTURE.md @@ -0,0 +1,676 @@ +# ๐๏ธ AutoFlow OSS โ System Architecture + +> This document provides a comprehensive technical deep-dive into AutoFlow's system design, data flows, component interactions, and design decisions. + +--- + +## Table of Contents + +1. [High-Level Architecture](#1-high-level-architecture) +2. [Service Topology](#2-service-topology) +3. [Data Flow Diagrams](#3-data-flow-diagrams) +4. [Frontend Architecture](#4-frontend-architecture) +5. [Backend Architecture](#5-backend-architecture) +6. [WhatsApp Bridge Architecture](#6-whatsapp-bridge-architecture) +7. [LLM Integration Layer](#7-llm-integration-layer) +8. [Workflow Executor Engine](#8-workflow-executor-engine) +9. [Database Design](#9-database-design) +10. [Inter-Service Communication](#10-inter-service-communication) +11. [Security Considerations](#11-security-considerations) +12. [Scalability & Future Architecture](#12-scalability--future-architecture) + +--- + +## 1. High-Level Architecture + +AutoFlow is a **4-service microservice architecture** orchestrated via Docker Compose, designed for simplicity and self-hosted deployment on commodity hardware. + +``` +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ HOST MACHINE โ +โ โ +โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ +โ โ Docker Compose Network โ โ +โ โ โ โ +โ โ โโโโโโโโโโโโโโโ REST โโโโโโโโโโโโโโโ REST โโโโโโโ โ โ +โ โ โ FRONTEND โโโโโโโโโโโโโ BACKEND โโโโโโโโโโโโโOLLAMโ โ โ +โ โ โ React+Vite โ โ FastAPI โ โ A โ โ โ +โ โ โ :3000 โ โ :8000 โ โ:1143โ โ โ +โ โ โโโโโโโโโโโโโโโ โโโโโโโโฌโโโโโโโโ โ 4 โ โ โ +โ โ โ โโโโโโโ โ โ +โ โ โ REST โ โ +โ โ โผ โ โ +โ โ โโโโโโโโโโโโโโโ โ โ +โ โ โ WHATSAPP โ โ โ +โ โ โ BRIDGE โ โ โ +โ โ โ Node.js โ โ โ +โ โ โ :3001 โ โ โ +โ โ โโโโโโโโฌโโโโโโโ โ โ +โ โ โ โ โ +โ โ โผ โ โ +โ โ WhatsApp Web โ โ +โ โ Protocol โ โ +โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ +โ โ +โ โโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โ +โ โ ./data/ โ โ Docker โ โ +โ โ SQLite โ โ Volumes โ โ +โ โ CSVs โ โ (ollama, โ โ +โ โ โ โ whatsapp) โ โ +โ โโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +``` + +### Design Principles + +| Principle | Implementation | +|-----------|---------------| +| **Self-Hosted First** | No external APIs. LLM runs locally via Ollama. | +| **Zero Configuration** | Single `docker-compose up` starts everything. | +| **Offline Capable** | Works without internet after initial setup. | +| **SMB-Friendly** | SQLite (no Postgres setup), CSV imports (no fancy integrations). | +| **Developer Experience** | Hot-reload for both frontend and backend via volume mounts. | + +--- + +## 2. Service Topology + +### 2.1 Frontend Service + +```yaml +Image: node:18-alpine +Port: 3000 +Framework: React 18 + Vite 5 +Styling: TailwindCSS 3 +State: Zustand +Flow: React Flow v11 +Charts: Recharts +``` + +**Responsibilities:** +- Render visual workflow builder with drag-and-drop nodes +- Accept natural language input and call backend `/api/workflow/generate` +- Display AI-generated explanations +- Show analytics dashboard with performance metrics +- Handle WhatsApp deployment flow with QR code display + +### 2.2 Backend Service + +```yaml +Image: python:3.11-slim +Port: 8000 +Framework: FastAPI (async) +ORM: SQLModel + SQLAlchemy +Database: SQLite +LLM: Ollama (via httpx) +``` + +**Responsibilities:** +- REST API for all CRUD operations +- Natural language โ workflow JSON conversion via Ollama +- Workflow execution engine (graph traversal) +- WhatsApp webhook listener (`/api/whatsapp/incoming`) +- Audit logging for every execution +- Database seeding with templates and demo workflows + +### 2.3 WhatsApp Bridge Service + +```yaml +Image: node:18-alpine + Chromium +Port: 3001 +Framework: Express.js +Library: whatsapp-web.js (Puppeteer-based) +``` + +**Responsibilities:** +- Manage WhatsApp Web session via headless Chromium +- Generate QR codes for phone linking +- Forward incoming messages to backend webhook +- Send individual and broadcast messages +- Persist session data across container restarts + +### 2.4 Ollama Service + +```yaml +Image: ollama/ollama (official) +Port: 11434 +Model: qwen3:8b +``` + +**Responsibilities:** +- Serve the Qwen3:8B model for inference +- Auto-pull the model on first startup +- Provide `/api/chat` endpoint for structured JSON generation + +--- + +## 3. Data Flow Diagrams + +### 3.1 Workflow Generation Flow + +``` +User types: "When customer asks about stock, check inventory and reply" + โ + โผ +โโโโโโโโโโโโโโโโ POST /api/workflow/generate +โ Frontend โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ NLInputPanelโ โ Backend โ +โโโโโโโโโโโโโโโโ โ main.py โ + โโโโโโโโฌโโโโโโโโ + โ + POST /api/chat โ + (system + user โ + prompts) โผ + โโโโโโโโโโโโโโโโ + โ Ollama โ + โ qwen3:8b โ + โโโโโโโโฌโโโโโโโโ + โ + JSON response โ + { nodes, edges } โ + โผ + โโโโโโโโโโโโโโโโ + โ Backend โ + โ validates โ + โ against โ + โ WorkflowSchemaโ + โโโโโโโโฌโโโโโโโโ + โ + explanation text โ + + workflow JSON โ + โผ +โโโโโโโโโโโโโโโโ { workflow, explanation } โโโโโโโโโโโโโโโโ +โ Frontend โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Backend โ +โ WorkflowGraphโ โโโโโโโโโโโโโโโโ +โ renders nodesโ +โโโโโโโโโโโโโโโโ +``` + +### 3.2 Message Processing Flow (Runtime) + +``` +Customer sends WhatsApp message: "Do you have sugar?" + โ + โผ +โโโโโโโโโโโโโโโโ POST WEBHOOK +โ WhatsApp โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ Bridge โ โ Backend โ +โ server.js โ โ /api/whatsappโ +โโโโโโโโโโโโโโโโ โ /incoming โ + โโโโโโโโฌโโโโโโโโ + โ + For each active + workflow, check + keyword match + โ + Match found! + "sugar" โ keywords + โ + โผ + โโโโโโโโโโโโโโโโ + โ Executor โ + โ engine โ + โ traverses โ + โ node graph โ + โโโโโโโโฌโโโโโโโโ + โ + โโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโ + โ โ โ + โผ โผ โผ + Inventory Condition WhatsApp + Lookup (qty > 0?) Send Reply + (CSV search) โโ True โโโ "Sugar available" + โโ False โโโ "Out of stock" + โ + โผ + โโโโโโโโโโโโโโโโ POST /send + โ Backend โโโโโโโโโโโโโโโโโโโโโโโโโโ + โ โ โ Bridge โ + โโโโโโโโโโโโโโโโ โโโโโโโโฌโโโโ + โ โ + Audit Log WhatsApp + saved to reply sent + SQLite to customer +``` + +### 3.3 Deploy Agent Flow + +``` +User clicks "Deploy Agent" on frontend + โ + โผ +โโโโโโโโโโโโโโโโ GET /status (poll every 2s) +โ DeployPage โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ frontend โ โ WhatsApp โ +โโโโโโโโโโโโโโโโ โ Bridge โ + โ โโโโโโโโฌโโโโโโโโ + โ โ + โโโโโโ { qr: "data:..." } โโโโโโโโโโโโโโโโโโโโโโโโ + โ + โผ + Display QR Code + (QRCodeSVG component) + โ + User scans with phone + โ + โผ +โโโโโโโโโโโโโโโโ GET /status โ { connected: true } +โ DeployPage โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ shows โ โ โ +โ "Agent Live"โ โโโโโโโโโโโโโโ +โโโโโโโโโโโโโโโโ โ Bridge โ + โ emits โ + โ "ready" โ + โโโโโโโโโโโโโโ +``` + +--- + +## 4. Frontend Architecture + +### 4.1 Component Hierarchy + +``` +App.jsx (Router) +โโโ LandingPage.jsx +โโโ Dashboard.jsx +โ โโโ ROIDashboard.jsx (Recharts metrics) +โโโ Builder.jsx +โ โโโ TopBar.jsx (Save / Activate) +โ โโโ WorkflowGraph.jsx (React Flow canvas) +โ โ โโโ CustomNode.jsx (base renderer) +โ โ โโโ TriggerNode.jsx +โ โ โโโ ActionNode.jsx +โ โ โโโ ConditionNode.jsx +โ โ โโโ WhatsAppNode.jsx +โ โ โโโ InventoryNode.jsx +โ โ โโโ DelayNode.jsx +โ โโโ NodeConfigPanel.jsx (side panel) +โ โโโ NLInputPanel.jsx (bottom bar) +โ โโโ AIExplainerPanel.jsx +โโโ Templates.jsx +โโโ DeployPage.jsx (QR + Chat simulator) +โโโ NotFound.jsx +``` + +### 4.2 State Management (Zustand) + +```javascript +workflowStore = { + nodes: [], // React Flow node objects + edges: [], // React Flow edge objects + currentWorkflow: {}, // Metadata (id, name, description) + isActive: false, // Workflow activation status + explanation: null, // AI-generated explanation text + selectedNode: null, // Currently selected node for config panel + + // Actions + setNodes, setEdges, + onNodesChange, onEdgesChange, // React Flow change handlers + setCurrentWorkflow, setIsActive, + setExplanation, setSelectedNode, + updateNodeData(id, newData) // Granular node data update +} +``` + +### 4.3 Custom Node System + +Each node type is a React component that wraps React Flow's `Handle` components: + +| Node | Color | Icon | Handles | +|------|-------|------|---------| +| TriggerNode | Dark green | โก Lightning | Output only (right) | +| ActionNode | Dark blue | โ๏ธ Gear | Input (left) + Output (right) | +| ConditionNode | Amber | ๐ท Diamond | Input (left) + True (green right) + False (red right) | +| WhatsAppNode | Emerald | ๐ฌ WhatsApp | Input (left) + Output (right) | +| InventoryNode | Orange | ๐ฆ Box | Input (left) + Output (right) | +| DelayNode | Purple | โฐ Clock | Input (left) + Output (right) | + +### 4.4 Auto-Layout Algorithm + +The `getLayoutedElements()` function in `WorkflowGraph.jsx` implements a **horizontal directed graph layout**: + +1. **Depth Calculation** โ Uses edge relaxation (similar to Bellman-Ford) to compute the longest path depth for each node +2. **Column Grouping** โ Nodes at the same depth are placed in the same vertical column +3. **Position Assignment** โ X position = `depth ร 350 + 100`, Y position = centered within column + +This ensures AI-generated workflows are automatically arranged left-to-right. + +--- + +## 5. Backend Architecture + +### 5.1 Module Dependency Graph + +``` +main.py (FastAPI app) +โโโ database.py (SQLite engine + sessions) +โโโ models.py (SQLModel ORM: Workflow, Template, AuditLog, InventorySource) +โโโ schemas.py (Pydantic: WorkflowSchema, WorkflowGenerateRequest) +โโโ ollama_client.py (LLM: generate_workflow, explain_workflow) +โโโ executor.py (WorkflowExecutor class) +โโโ seeder.py (Database seeding) +โโโ routers/ + โโโ inventory.py (CSV upload + search) + โโโ templates.py (CRUD + fork) + โโโ audit.py (Query logs) +``` + +### 5.2 Request Processing Pipeline + +``` +HTTP Request โ CORSMiddleware โ FastAPI Router โ Handler โ Response + โ + โโโโโโโดโโโโโโ + โ SQLModel โ + โ Session โ + โ (dep inj) โ + โโโโโโโโโโโโโโ +``` + +### 5.3 Middleware Stack + +| Order | Middleware | Purpose | +|-------|-----------|---------| +| 1 | CORSMiddleware | Allow cross-origin requests from frontend (:3000) | +| 2 | FastAPI default | Request parsing, validation, response serialization | + +### 5.4 API Router Structure + +``` +/api/ +โโโ workflow/ +โ โโโ POST /generate โ ollama_client.generate_workflow() +โ โโโ POST /save โ Workflow.create() +โ โโโ GET /list โ Workflow.select_all() +โ โโโ GET /{id} โ Workflow.get() +โ โโโ POST /{id}/activate โ Workflow.update(is_active=True) +โ โโโ POST /{id}/deactivate โ Workflow.update(is_active=False) +โ โโโ POST /{id}/execute โ WorkflowExecutor.execute() +โโโ templates/ +โ โโโ GET / โ Template.select_all() +โ โโโ GET /{id} โ Template.get() +โ โโโ POST /{id}/fork โ Template โ Workflow copy +โโโ inventory/ +โ โโโ POST /upload โ Save CSV to data/ +โ โโโ GET /list โ InventorySource.select_all() +โ โโโ GET /{id}/data โ Parse CSV โ JSON +โโโ audit/ +โ โโโ GET / โ AuditLog.select_all() +โ โโโ GET /{workflow_id} โ AuditLog.filter(workflow_id) +โโโ whatsapp/ + โโโ POST /incoming โ Webhook from Bridge โ Executor +``` + +--- + +## 6. WhatsApp Bridge Architecture + +### 6.1 Session Lifecycle + +``` +Container Start + โ + โโโ Clean stale SingletonLock files + โโโ Initialize whatsapp-web.js Client + โ โโโ Puppeteer launches headless Chromium + โ + โโโ Event: 'qr' โ Store QR string, log to terminal + โโโ Event: 'ready' โ Set connected=true, clear QR + โโโ Event: 'message' โ Forward to backend webhook + โโโ Event: 'disconnected' โ Reset state +``` + +### 6.2 Message Flow + +``` +Incoming WhatsApp Message + โ + โผ +whatsapp-web.js client.on('message') + โ + โโโ Filter: Ignore status@broadcast + โ + โผ +POST http://backend:8000/api/whatsapp/incoming + { + from: "919876543210@c.us", + message: "do you have sugar?", + timestamp: 1710518400 + } +``` + +### 6.3 Session Persistence + +WhatsApp session data is stored in a Docker named volume `whatsapp_session` mapped to `/app/.wwebjs_auth`. This ensures the session survives container restarts without needing to re-scan the QR code. + +--- + +## 7. LLM Integration Layer + +### 7.1 Prompt Architecture + +``` +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ SYSTEM PROMPT โ +โ - Available node types โ +โ - Response format rules โ +โ - {{variable}} syntax requirement โ +โ - "Return ONLY valid JSON" โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ + + +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ USER PROMPT โ +โ Business automation request: "{nl_input}" โ +โ Generate complete workflow JSON โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ + โ + โผ + Ollama /api/chat + (qwen3:8b, temp=0.2) + โ + โผ +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ JSON CLEANING โ +โ - Strip ```json``` markers โ +โ - Trim whitespace โ +โ - json.loads() parse โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ + โ + On failure + โผ +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +โ STRICT RETRY PROMPT โ +โ "You failed to generate valid JSON" โ +โ "Start with { and end with }" โ +โ Temperature lowered to 0.1 โ +โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ +``` + +### 7.2 Error Handling Strategy + +| Error | Handling | Max Retries | +|-------|----------|-------------| +| Ollama connection timeout | Wait 2s, retry | 3 | +| Invalid JSON from LLM | Retry with stricter prompt | 1 (strict mode) | +| Schema validation failure | Return 500 with error | 0 | + +--- + +## 8. Workflow Executor Engine + +### 8.1 Execution Algorithm + +```python +def execute(workflow_json, trigger_data): + # 1. Build node lookup map: { node_id โ node } + # 2. Build adjacency map: { source_id โ [{ target, label }] } + # 3. Initialize context: { trigger: trigger_data, results: {} } + # 4. Find trigger node + # 5. DFS traversal starting from trigger: + # - For each node: + # a. Record in actions_taken + # b. Execute node-type-specific logic + # c. Store result in context.results[node_id] + # d. Follow outgoing edges: + # - Unconditional โ always follow + # - Conditional โ only follow matching "true"/"false" branch + # 6. Log audit entry (success or failure) +``` + +### 8.2 Supported Action Types + +| Action Type | Input | Output | Effect | +|-------------|-------|--------|--------| +| `inventory_lookup` | CSV file, search query from trigger message | `{ found, product, quantity, price }` | Searches CSV for matching product | +| `send_whatsapp` | `{ to, message }` with template vars | `{ status: "sent" }` | Sends WhatsApp via bridge | +| `send_broadcast` | Customer list from CSV + message | `{ status: "broadcast_sent" }` | Mass message via bridge | +| `load_udhaar_list` | Udhaar CSV file | `{ status: "reminders_sent" }` | Sends reminders to overdue customers | +| `load_customers` | Customers CSV file | Phone number list | Loads broadcast recipient list | +| `log_payment` | Payment data | `{ status: "logged" }` | Records payment | + +### 8.3 Template Variable Resolution + +The `_resolve_template()` method uses regex to replace `{{path.to.value}}` placeholders: + +```python +# Input: "Hello {{trigger.from}}, {{results.inventory_1.product}} has {{results.inventory_1.quantity}} units" +# Context: { trigger: { from: "John" }, results: { inventory_1: { product: "Sugar", quantity: 50 } } } +# Output: "Hello John, Sugar has 50 units" +``` + +--- + +## 9. Database Design + +### 9.1 Entity-Relationship Diagram + +``` +โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ +โ Workflow โ โ Template โ +โโโโโโโโโโโโโโโโค โโโโโโโโโโโโโโโโค +โ id (PK) โ โ id (PK) โ +โ name โ โ name โ +โ description โ โ category โ +โ nl_input โ โ nodes (JSON) โ +โ nodes (JSON) โ โ edges (JSON) โ +โ edges (JSON) โ โ use_count โ +โ is_active โ โ created_at โ +โ created_at โ โโโโโโโโโโโโโโโโ +โ updated_at โ +โโโโโโโโฌโโโโโโโโ + โ 1:N + โผ +โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโ +โ AuditLog โ โ InventorySource โ +โโโโโโโโโโโโโโโโค โโโโโโโโโโโโโโโโโโโโค +โ id (PK) โ โ id (PK) โ +โ workflow_id โโFK โ name โ +โ workflow_nameโ โ type โ +โ trigger_type โ โ file_path โ +โ trigger_data โ โ sheet_url โ +โ actions_takenโ โ last_synced โ +โ result โ โ created_at โ +โ error_messageโ โโโโโโโโโโโโโโโโโโโโ +โ executed_at โ +โโโโโโโโโโโโโโโโ +``` + +### 9.2 Design Decisions + +| Decision | Rationale | +|----------|-----------| +| **SQLite over Postgres** | Zero-config, single file, perfect for single-machine deployment | +| **JSON columns for nodes/edges** | Avoids complex relational modeling of graph structures | +| **Separate Template table** | Templates are read-only references, workflows are user-editable forks | +| **Audit table per-execution** | Full traceability; every trigger โ result is recorded | + +--- + +## 10. Inter-Service Communication + +### 10.1 Communication Matrix + +| From | To | Protocol | Endpoint | Purpose | +|------|-----|----------|----------|---------| +| Frontend | Backend | HTTP REST | `:8000/api/*` | All CRUD + AI generation | +| Backend | Ollama | HTTP REST | `:11434/api/chat` | LLM inference | +| Backend | Bridge | HTTP REST | `:3001/send`, `/broadcast` | Send WhatsApp messages | +| Bridge | Backend | HTTP POST | `:8000/api/whatsapp/incoming` | Forward incoming messages | +| Frontend | Bridge | HTTP GET | `:3001/status`, `/qr` | Check connection, get QR | + +### 10.2 Docker Networking + +All services communicate via Docker Compose's default bridge network. Service names (`backend`, `frontend`, `whatsapp-bridge`, `ollama`) resolve to container IPs automatically. + +--- + +## 11. Security Considerations + +| Area | Current State | Recommendation | +|------|--------------|----------------| +| **Authentication** | None (dev mode) | Add JWT-based auth for production | +| **CORS** | `allow_origins=["*"]` | Restrict to specific frontend origin | +| **WhatsApp Session** | Persisted in Docker volume | Encrypt session data at rest | +| **Database** | SQLite file on host | Use file permissions, consider encryption | +| **LLM Prompt Injection** | Basic input cleaning | Add input sanitization layer | +| **API Rate Limiting** | None | Add FastAPI rate limiter middleware | + +--- + +## 12. Scalability & Future Architecture + +### Current Limitations +- Single-instance SQLite (no concurrent writes) +- Single WhatsApp session per bridge +- CPU-only LLM inference (slow on large models) + +### Proposed Future Architecture + +``` + โโโโโโโโโโโโโโโโ + โ Nginx โ + โ Reverse โ + โ Proxy โ + โโโโโโโโฌโโโโโโโโ + โ + โโโโโโโโโโโโโโผโโโโโโโโโโโโโ + โผ โผ โผ + โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ + โ Frontend โ โ Backend โ โ Backend โ + โ (CDN) โ โ Worker 1 โ โ Worker 2 โ + โโโโโโโโโโโโ โโโโโโฌโโโโโโ โโโโโโฌโโโโโโ + โ โ + โโโโโโดโโโโโโโโโโโโโดโโโโโ + โ PostgreSQL โ + โ + Redis Cache โ + โโโโโโโโโโโโโโโโโโโโโโโโโ + โ + โโโโโโโโโโโโดโโโโโโโโโโโ + โ Message Queue โ + โ (RabbitMQ/Redis) โ + โโโโโโโโโโโโฌโโโโโโโโโโโ + โ + โโโโโโโโโโโโดโโโโโโโโโโโ + โ Executor Workers โ + โ (async processing) โ + โโโโโโโโโโโโโโโโโโโโโโโ +``` + +### Roadmap Items +- [ ] PostgreSQL migration for multi-user support +- [ ] Redis caching for LLM responses +- [ ] Message queue for async workflow execution +- [ ] GPU acceleration for Ollama (CUDA/ROCm) +- [ ] Multi-phone WhatsApp session management +- [ ] Scheduled trigger support (cron-based) +- [ ] Google Sheets inventory connector +- [ ] Multi-language UI (Hindi, Tamil, Telugu) +- [ ] Mobile-responsive workflow builder +- [ ] Workflow versioning and rollback + +--- + ++ Architecture designed for simplicity today, scalability tomorrow. +
diff --git a/TEST_CHECKLIST.md b/TEST_CHECKLIST.md new file mode 100644 index 0000000..2a9f9a5 --- /dev/null +++ b/TEST_CHECKLIST.md @@ -0,0 +1,29 @@ +# AutoFlow OSS Testing Checklist + +## ๐๏ธ Start up +- [ ] `docker-compose up --build` runs without throwing terminal errors. +- [ ] Container `ollama` successfully pulls the `qwen3:8b` model implicitly. (Wait ~1min) +- [ ] Container `whatsapp-bridge` logs `WhatsApp Bridge running on port 3001` and generates a QR code. + +## ๐ข Endpoints & Health +- [ ] Navigate to `http://localhost:8000/docs` to see the FastAPI Swagger UI. +- [ ] Navigate to `http://localhost:3000` to see the React Frontend wrapper. +- [ ] Run `curl http://localhost:3001/status` and receive JSON `{ "connected": false }` (or true if registered). + +## ๐ ๏ธ Feature Testing +- [ ] **AI Generation**: On Frontend Builder, type "When user asks about order, check tracking" and see valid ReactFlow nodes render in UI. +- [ ] **SQLite Seeding**: `backend/data/autoflow.db` exists. +- [ ] **Data Read**: Ensure hitting `GET /api/templates` on Swagger returns the 5 default demo templates. + +## ๐ฑ Demo 1: Stock Query E2E +- [ ] Go to WhatsApp and scan the QR printed by docker-compose logs for `whatsapp-bridge`. +- [ ] Turn ON the `Stock Query Auto-Reply` inside `Builder` top bar (Set to Active). +- [ ] Send message from another phone: *"Do you have any sugar left?"* +- [ ] Backends should log standard trigger matching keywords: `sugar`. +- [ ] `executor.py` runs `inventory_lookup`. It loads `inventory.csv` mapped locally. +- [ ] Automated reply is sent back: *"Sorry, sugar is out of stock right now."* (Qty is 0 in CSV). +- [ ] Message *"What about rice?"* -> Returns *"Yes! rice is available. Stock: 50 units."* +- [ ] Verify execution is logged in SQLite `audit_logs` table (Viewable via `GET /api/audit`). + +## โ Conclusion +If all above pass, AutoFlow OSS offline core architecture is completely functional. diff --git a/WORKFLOW.md b/WORKFLOW.md new file mode 100644 index 0000000..0f51dc1 --- /dev/null +++ b/WORKFLOW.md @@ -0,0 +1,97 @@ +# ๐ AutoFlow: Message Processing Workflow + +This document visualizes the complete lifecycle of a message within the AutoFlow system, from the user's phone to the final response. + +## ๐งฉ High-Level Flow + +1. **User** sends a message via WhatsApp. +2. **Baileys** (via WebSocket) receives and decrypts the payload. +3. **Logic Engine** sanitizes the input and detects intent. +4. **Routing**: + * **Simple Intents** (Greetings, Menu) are handled locally. + * **Data Intents** (Price, Stock) query **Google Sheets**. + * **Complex Intents** are forwarded to **Gemini AI**. +5. **Response** is formatted and sent back to the user. + +--- + +## ๐ Detailed Workflow Diagram + +```mermaid +graph TD + %% STYLES + classDef user fill:#E5F4FD,stroke:#3B82F6,stroke-width:2px,color:#1e3a8a; + classDef whatsapp fill:#DCFCE7,stroke:#22C55E,stroke-width:2px,color:#14532d; + classDef engine fill:#F3E8FF,stroke:#A855F7,stroke-width:2px,color:#581c87; + classDef sheets fill:#FEF3C7,stroke:#F59E0B,stroke-width:2px,color:#78350f; + classDef ai fill:#FCE7F3,stroke:#EC4899,stroke-width:2px,color:#831843; + + %% EXTERNAL + User((๐ค User Phone)):::user + WAServer[โ๏ธ WhatsApp Cloud]:::whatsapp + + %% BACKEND CONTEXT + subgraph Processing_System ["โก AutoFlow Processing System"] + direction TB + + Baileys[๐ Baileys Socket Service]:::whatsapp + + subgraph Engine_Core ["โ๏ธ Logic Engine Core"] + Decrypt[๐ Decrypt & Parse]:::engine + PreProcess[๐งน Sanitize & Normalization]:::engine + Router{๐ฆ Intent Router}:::engine + end + + subgraph Handlers ["๐ ๏ธ Action Handlers"] + H_Static[๐ Static Response]:::engine + H_Sheet[๐ Sheet Service]:::sheets + H_AI[๐ง AI Service]:::ai + end + + ResponseBuilder[โ๏ธ Response Formatter]:::engine + end + + %% EXTERNAL DATA + GSheets[(Google Sheets DB)]:::sheets + GeminiAPI[โจ Gemini Pro API]:::ai + + %% CONNECTIONS & LOGIC FLOW + User -->|1. Sends Msg| WAServer + WAServer -->|2. WebSocket Push| Baileys + Baileys -->|3. Event: messages.upsert| Decrypt + Decrypt -->|4. Clean Text| PreProcess + PreProcess -->|5. Analyzed Text| Router + + %% ROUTING LOGIC + Router -->|'Hi', 'Menu'| H_Static + Router -->|'Price', 'Stock'| H_Sheet + Router -->|'Order', 'Buy'| H_Sheet + Router -->|Unmatched/Complex| H_AI + + %% HANDLER EXECUTION + H_Static -->|Get Template| ResponseBuilder + + H_Sheet -->|6a. Read/Write| GSheets + GSheets -->|6b. Data Return| H_Sheet + H_Sheet -->|Format Data| ResponseBuilder + + H_AI -->|7a. Prompt + Context| GeminiAPI + GeminiAPI -->|7b. Generated Text| H_AI + H_AI -->|Natural Response| ResponseBuilder + + %% FINAL REPLY + ResponseBuilder -->|8. Final Payload| Baileys + Baileys -->|9. Send Message| WAServer + WAServer -->|10. Delivers| User +``` + +--- + +## ๐ Key Components + +| Component | Responsibility | Tech Stack | +| :--- | :--- | :--- | +| **Baileys Socket** | Manages the persistent WebSocket connection, auth, and encryption. | `@whiskeysockets/baileys` | +| **Intent Router** | Regex-based classification for high-speed routing (0ms latency). | JavaScript `RegExp` | +| **Sheet Service** | Acts as a pseudo-database for inventory and logging. | `google-spreadsheet` | +| **AI Service** | Handles fallback and complex conversational queries. | `@google/generative-ai` | diff --git a/autoflow-frontend/.eslintrc.cjs b/autoflow-frontend/.eslintrc.cjs deleted file mode 100644 index 3e212e1..0000000 --- a/autoflow-frontend/.eslintrc.cjs +++ /dev/null @@ -1,21 +0,0 @@ -module.exports = { - root: true, - env: { browser: true, es2020: true }, - extends: [ - 'eslint:recommended', - 'plugin:react/recommended', - 'plugin:react/jsx-runtime', - 'plugin:react-hooks/recommended', - ], - ignorePatterns: ['dist', '.eslintrc.cjs'], - parserOptions: { ecmaVersion: 'latest', sourceType: 'module' }, - settings: { react: { version: '18.2' } }, - plugins: ['react-refresh'], - rules: { - 'react/jsx-no-target-blank': 'off', - 'react-refresh/only-export-components': [ - 'warn', - { allowConstantExport: true }, - ], - }, -} diff --git a/autoflow-frontend/Dockerfile b/autoflow-frontend/Dockerfile new file mode 100644 index 0000000..fad0419 --- /dev/null +++ b/autoflow-frontend/Dockerfile @@ -0,0 +1,20 @@ +# Use node for frontend build +FROM node:18-alpine + +WORKDIR /app + +# Copy package files +COPY package*.json ./ + +# Install dependencies +RUN npm install + +# Copy source code +COPY . . + +# Expose port (Vite default is 5173, but compose uses 3000) +EXPOSE 3000 + +# Start dev server (Vite) +# We use --host to allow external access +CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0", "--port", "3000"] diff --git a/autoflow-frontend/README.md b/autoflow-frontend/README.md index f768e33..74872fd 100644 --- a/autoflow-frontend/README.md +++ b/autoflow-frontend/README.md @@ -1,4 +1,4 @@ -# React + Vite +# React + TypeScript + Vite This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules. @@ -6,3 +6,45 @@ Currently, two official plugins are available: - [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react/README.md) uses [Babel](https://babeljs.io/) for Fast Refresh - [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react-swc) uses [SWC](https://swc.rs/) for Fast Refresh + +## Expanding the ESLint configuration + +If you are developing a production application, we recommend updating the configuration to enable type aware lint rules: + +- Configure the top-level `parserOptions` property like this: + +```js +export default tseslint.config({ + languageOptions: { + // other options... + parserOptions: { + project: ['./tsconfig.node.json', './tsconfig.app.json'], + tsconfigRootDir: import.meta.dirname, + }, + }, +}) +``` + +- Replace `tseslint.configs.recommended` to `tseslint.configs.recommendedTypeChecked` or `tseslint.configs.strictTypeChecked` +- Optionally add `...tseslint.configs.stylisticTypeChecked` +- Install [eslint-plugin-react](https://github.com/jsx-eslint/eslint-plugin-react) and update the config: + +```js +// eslint.config.js +import react from 'eslint-plugin-react' + +export default tseslint.config({ + // Set the react version + settings: { react: { version: '18.3' } }, + plugins: { + // Add the react plugin + react, + }, + rules: { + // other rules... + // Enable its recommended rules + ...react.configs.recommended.rules, + ...react.configs['jsx-runtime'].rules, + }, +}) +``` diff --git a/autoflow-frontend/build.txt b/autoflow-frontend/build.txt new file mode 100644 index 0000000..617f23c Binary files /dev/null and b/autoflow-frontend/build.txt differ diff --git a/autoflow-frontend/eslint.config.js b/autoflow-frontend/eslint.config.js new file mode 100644 index 0000000..092408a --- /dev/null +++ b/autoflow-frontend/eslint.config.js @@ -0,0 +1,28 @@ +import js from '@eslint/js' +import globals from 'globals' +import reactHooks from 'eslint-plugin-react-hooks' +import reactRefresh from 'eslint-plugin-react-refresh' +import tseslint from 'typescript-eslint' + +export default tseslint.config( + { ignores: ['dist'] }, + { + extends: [js.configs.recommended, ...tseslint.configs.recommended], + files: ['**/*.{ts,tsx}'], + languageOptions: { + ecmaVersion: 2020, + globals: globals.browser, + }, + plugins: { + 'react-hooks': reactHooks, + 'react-refresh': reactRefresh, + }, + rules: { + ...reactHooks.configs.recommended.rules, + 'react-refresh/only-export-components': [ + 'warn', + { allowConstantExport: true }, + ], + }, + }, +) diff --git a/autoflow-frontend/index.html b/autoflow-frontend/index.html index 0c589ec..f2883c8 100644 --- a/autoflow-frontend/index.html +++ b/autoflow-frontend/index.html @@ -2,12 +2,12 @@ - + -