Binance Collector

Real-time trading data collector for Binance Futures/Spot with Polymarket prediction market integration.

Features

Data Collection

• Binance WebSocket Streams: aggTrade, bookTicker, markPrice, forceOrder, depth
• Multi-market Support: Futures and Spot markets simultaneously
• OHLCV Bar Aggregation: 5s, 15s, 1m timeframes with delta volume
• Order Book Depth: Top N levels with configurable update speed (100ms/500ms)

Trading Features

• CVD (Cumulative Volume Delta): Real-time buy/sell pressure tracking
• RVOL (Relative Volume): Volume compared to rolling average
• Impulse Detection: Price momentum spikes
• Microprice: Volume-weighted mid price
• Basis: Futures vs Spot premium/discount
• Anchored VWAP: Session VWAP with deviation tracking
• Liquidation Tracking: Large forced liquidations

Polymarket Integration

• Real-time Order Book: UP/DOWN token prices via WebSocket
• Fair Value Model: Probability estimation with fast/smooth modes
• Spike Detection: Microstructure spike/dip detection
• Edge Calculation: Trading edge in basis points

Advanced Analytics

• TAAPI Integration: External technical indicators (RSI, MACD, etc.)
• Bias Model: Multi-timeframe directional bias
• Volatility Estimation: Fast/slow/blend sigma calculation
• Quality Mode: Data quality monitoring (OK/DEGRADED/BAD)

API & Monitoring

• HTTP REST API: Health checks, features, control endpoints
• TUI Dashboard: Terminal-based real-time monitoring
• Prometheus Metrics: Lag, throughput, error tracking
• Structured JSON Logging: Machine-parseable logs

Requirements

• Python 3.10+
• Network access to Binance and Polymarket APIs
• (Optional) TAAPI.io API key for technical indicators

Installation

# Clone the project
git clone https://github.com/txbabaxyz/mlmodelpoly.git
cd mlmodelpoly

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # Linux/Mac
# or: venv\Scripts\activate  # Windows

# Install dependencies
pip install -r requirements.txt

# Copy and configure environment
cp .env.example .env
# Edit .env with your settings

Configuration

All settings are configured via environment variables or .env file.

Core Settings

Variable	Default	Description
SYMBOL	BTCUSDT	Trading pair symbol
LOG_LEVEL	INFO	Logging level (DEBUG/INFO/WARNING/ERROR)
HTTP_HOST	0.0.0.0	API server host
HTTP_PORT	8000	API server port

WebSocket Settings

Variable	Default	Description
FUTURES_WS	wss://fstream.binance.com	Futures WebSocket URL
SPOT_WS	wss://stream.binance.com:9443	Spot WebSocket URL
DEPTH_ENABLED	true	Enable order book depth stream
DEPTH_SPEED	100ms	Depth update speed (100ms/500ms/none)
TOPN	10	Order book levels to track

Context Bootstrap

Variable	Default	Description
CONTEXT_ENABLED	true	Enable HTF klines bootstrap
CONTEXT_TFS	1m,5m,15m,1h	Timeframes to bootstrap
CONTEXT_BOOTSTRAP_LIMIT	500	Klines per timeframe

TAAPI Integration

Variable	Default	Description
TAAPI_ENABLED	true	Enable TAAPI indicators
TAAPI_SECRET	-	TAAPI.io API secret key
TAAPI_EXCHANGE	binancefutures	Exchange for TAAPI
TAAPI_TFS	1m,5m,15m,1h	Timeframes to fetch

Polymarket Integration

Variable	Default	Description
POLYMARKET_ENABLED	true	Enable Polymarket integration
POLYMARKET_WS_URL	wss://ws-subscriptions-clob.polymarket.com/ws/market	WebSocket URL
POLYMARKET_STALE_THRESHOLD_SEC	5.0	Stale data threshold
POLYMARKET_MIN_DEPTH	200.0	Min depth for execution
POLYMARKET_MAX_SPREAD_BPS	500.0	Max spread before veto
POLY_UP_IS_YES	true	UP = YES token mapping

Trading Parameters

Variable	Default	Description
SLICE_USD	20.0	Standard slice size
MAX_SLICES_PER_WINDOW	30	Max slices per 15-min window
MAX_USD_PER_WINDOW	300.0	Max USD per window
COOLDOWN_SEC	2.0	Cooldown between executions
EDGE_BUFFER_BPS	25.0	Required edge buffer

Event Recording

Variable	Default	Description
RECORD_ENABLED	false	Enable event recording
RECORD_DIR	data/recordings	Recording output directory
RECORD_BUFFER_SIZE	100	Events before flush

Usage

Start the Collector

cd mlmodelpoly
source venv/bin/activate

# Run with default settings
python -m collector

# Or with custom symbol
SYMBOL=ETHUSDT python -m collector

Start TUI Dashboard

In a separate terminal:

source venv/bin/activate
python scripts/tui_dashboard.py --host localhost --port 8000

API Endpoints

Endpoint	Method	Description
/health	GET	Health check
/state	GET	Current system state
/latest/features	GET	Latest computed features
/latest/bars	GET	Latest OHLCV bars
/latest/edge	GET	Latest edge decision
/control/anchor/reset	POST	Reset VWAP anchor

Example API Usage

# Health check
curl http://localhost:8000/health

# Get latest features
curl http://localhost:8000/latest/features | jq .

# Get edge decision
curl http://localhost:8000/latest/edge | jq .

Project Structure

mlmodelpoly/
├── src/
│   ├── collector/              # Main collector package
│   │   ├── __init__.py
│   │   ├── main.py            # Entry point
│   │   ├── config.py          # Configuration (pydantic-settings)
│   │   ├── ws_client.py       # Binance WebSocket client
│   │   ├── pipeline.py        # Event processing pipeline
│   │   ├── features.py        # Feature computation engine
│   │   ├── bars.py            # OHLCV bar aggregation
│   │   ├── edge_engine.py     # Trading edge calculation
│   │   ├── fair_model.py      # Fair value estimation
│   │   ├── bias_model.py      # Directional bias model
│   │   ├── volatility.py      # Volatility estimation
│   │   ├── accumulate_engine.py # Trade accumulation logic
│   │   ├── decision_logger.py # Structured decision logging
│   │   ├── http_api.py        # FastAPI REST endpoints
│   │   ├── metrics.py         # Prometheus metrics
│   │   ├── polymarket/        # Polymarket integration
│   │   │   ├── ws_client.py   # PM WebSocket client
│   │   │   ├── book_store.py  # Order book storage
│   │   │   ├── market_resolver.py # Token ID resolution
│   │   │   └── normalize_updown.py # UP/DOWN normalization
│   │   ├── taapi/             # TAAPI.io integration
│   │   │   ├── client.py      # Async HTTP client
│   │   │   ├── store.py       # Indicator storage
│   │   │   └── context_engine.py # Context aggregation
│   │   └── utils/             # Utility functions
│   └── strategies/            # Trading strategies
│       └── z_contra_fav_dip_hedge.py
├── scripts/
│   ├── tui_dashboard.py       # Terminal UI dashboard
│   ├── backtest_*.py          # Backtesting scripts
│   └── analyze_*.py           # Analysis tools
├── data/                      # Data storage (gitignored)
│   ├── decisions/             # Decision logs
│   └── recordings/            # Event recordings
├── logs/                      # Log files (gitignored)
├── requirements.txt           # Python dependencies
├── .env.example              # Example configuration
├── .gitignore                # Git ignore rules
└── README.md                 # This file

Troubleshooting

Connection Issues

Problem: WebSocket disconnects frequently

Solution:

1. Check network connectivity to Binance
2. Reduce DEPTH_SPEED to 500ms for lower bandwidth
3. Check firewall/proxy settings

High CPU Usage

Problem: Collector using too much CPU

Solution:

1. Set DEPTH_SPEED=500ms instead of 100ms
2. Reduce TOPN from 10 to 5
3. Disable unused features (TAAPI_ENABLED=false)

Missing Features

Problem: Features showing null or 0

Solution:

1. Wait for warmup period (1-5 minutes)
2. Check CONTEXT_ENABLED=true for HTF data
3. Verify Binance API is accessible

TAAPI Errors

Problem: TAAPI indicators not updating

Solution:

1. Verify TAAPI_SECRET is valid
2. Check TAAPI.io rate limits
3. Set TAAPI_ENABLED=false if not needed

Polymarket Issues

Problem: Polymarket prices not updating

Solution:

1. Verify WebSocket URL is accessible
2. Check if market is currently active
3. Verify token IDs are resolving correctly

Development

Running Tests

source venv/bin/activate
# Tests coming soon

Code Style

# Format code
black src/ scripts/

# Check types
mypy src/collector/

Adding New Features

1. Add feature computation to src/collector/features.py
2. Expose via HTTP API in src/collector/http_api.py
3. Add to TUI dashboard if needed

License

MIT License

Contributing

1. Fork the repository
2. Create feature branch
3. Submit pull request

Support

For issues and questions, please open a GitHub issue.