Automated, Continuous Web Monitoring Platform
Monitor any webpage. Track changes. Detect price movements. Store versions. Get alerts. No code needed.
Define runtime intervals (every X minutes/hours) and Webloom keeps scraping automatically, providing ongoing surveillance of your target webpages.
For each monitored page, Webloom stores structured snapshots and creates diffs over time:
- Text changes
- Price movements
- Image updates
- Link removals
- Metadata shifts
Automatically extracts, normalizes & tracks price values with visualization:
- Price drops
- Price increases
- Stable periods
Every version is compared to previous ones using:
- Similarity scoring
- Difference weighting
- Element-wise changes
You don't manually diff pages — Webloom does.
Webloom follows a microservices architecture with independent agents communicating through RabbitMQ:
| Agent | Responsibility |
|---|---|
| selector-agent | Infer CSS selectors using heuristic algorithms |
| discovery-agent | Find additional links to scrape |
| scraper-agent | Fetch page content securely |
| parser-agent | Extract structured data |
| classifier-agent | Classify page types |
| change-detector-agent | Detect content differences |
| price-tracker-agent | Extract and track price history |
| notifier-agent | Send alerts via email/Telegram |
| scheduler-agent | Automate periodic runs |
| storage-agent | Persist data to MongoDB |
| control-agent | Handle pause/resume/stop |
| health-monitor-agent | Pipeline metrics & heartbeats |
A responsive, real-time dashboard built with Next.js 14 featuring:
- Job analytics and management
- Run history and status
- Version timeline visualization
- Live event streaming
- Price trend graphs
- Aggregated insights
- Node.js + Docker
- RabbitMQ for message orchestration
- MongoDB Atlas free tier for data storage
- Distributed microservice model
- Next.js 14 (App Router)
- TailwindCSS for styling
- Chart.js/Recharts for data visualization
- Railway (backend services)
- Vercel (frontend)
- MongoDB Atlas (database)
- GitHub Actions for CI/CD
Works fully in free environments with zero hosting costs.
┌───────────────────┐
│ Frontend │
│ (Next.js) │
└─────────┬─────────┘
│
▼
┌──────────────────┐
│ API Gateway │
│ (Node.js/Express)│
└─────────┬────────┘
│
▼
┌────────────────────────┐
│ RabbitMQ Broker │
└────────────────────────┘
┌──────────────┬──────────────┬───────────────┬──────────────┬──────────────┐
▼ ▼ ▼ ▼ ▼
Selector Discovery Scraper Parser Classifier
Agent Agent Agent Agent Agent
│ │ │ │ │
└───────────┬──┴─────────────┴───────────────┴──────────────┘
│
▼
Change Detector Agent
│
▼
Price Tracker Agent
│
▼
Storage Agent
│
▼
MongoDB Database
The project includes comprehensive documentation organized into the following categories:
- features.md - Key features and capabilities
- glossary.md - Technical terms and definitions
- intro.md - Introduction to Webloom
- roadmap.md - Development roadmap
- vision.md - Project vision and goals
- architecture-overview.md - High-level architecture
- data-model.md - Database schema
- dlq-backpressure.md - Dead letter queue handling
- job-lifecycle.md - Job lifecycle management
- queues.md - Message queue system
- retention.md - Data retention policies
- state-machine.md - Finite state machines
- system-flow.md - System workflow
- change-detector-agent.md
- classifier-agent.md
- control-agent.md
- discovery-agent.md
- health-monitor-agent.md
- notifier-agent.md
- parser-agent.md
- price-tracker-agent.md
- scheduler-agent.md
- scraper-agent.md
- selector-agent.md
- storage-agent.md
- api-spec.md - Detailed API specification
- api.md - API reference
- cicd.md - CI/CD pipeline
- deployment.md - Deployment guides
- self-hosting.md - Self-hosting instructions
- templates.md - Job templates
- testing.md - Testing strategies
- dashboard.md - Dashboard documentation
- error-handling.md - Error handling strategies
- free-tier-limits.md - Free-tier limitations
- observability.md - Observability and monitoring
- security.md - Security guide
- financial-model.md - Financial model & monetization strategy
- design-language.md - Design language & interaction guidelines
- ui-copy.md - UI copy for all user-facing elements
- web-client.md - Web client (Next.js dashboard)
- Docker Desktop installed
- Node.js 18+
git clone https://github.com/your-org/webloom.git
cd webloom
docker-compose up --build- Dashboard: http://localhost:3001
- API Gateway: http://localhost:3000/api
- RabbitMQ UI: http://localhost:15672 (user: guest, pass: guest)
Webloom is designed to run completely free on cloud platforms:
| Infrastructure | Provider | Notes |
|---|---|---|
| Frontend | Vercel | Free tier |
| Backend | Railway | Free tier |
| Database | MongoDB Atlas | Free tier |
| Message Broker | Railway Add-on | Free tier |
No credit card required for any service.
See docs/PLATFORM-ENGINEERING/deployment.md for detailed deployment instructions.
Contributions are welcome!
- Fork the repository
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a pull request
See LICENSE for details.
"Watch the web evolve — automatically."