Skip to content

caomengxuan666/AstraDB

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

390 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

AstraDB

A high-performance, Redis-compatible database written in modern C++23.

🎯 Vision

AstraDB is designed to surpass DragonflyDB in every aspect while maintaining full Redis compatibility.

Our goal: 2x DragonflyDB performance, 50% less memory usage, and superior scalability.

✨ Features

Core Features βœ…

  • Redis Protocol Compatible: Full support for RESP2 and RESP3 protocols
  • High Performance: C++23 implementation with Asio coroutines and multi-threading
  • NO SHARING Architecture: Each Worker is completely independent with private resources
  • MPSC Communication: Lock-free cross-worker communication via concurrent queues
  • SIMD Optimizations: AVX2, SSE4.2, and NEON support for vectorized operations
  • Zero-Copy Serialization: FlatBuffers-based efficient serialization
  • Flexible Threading: Support for both single-threaded and multi-threaded modes
  • Dual Backend: epoll (stable) and io_uring (high performance) support
  • Vector Search: In-memory ANN search with hnswlib (cosine/L2/IP)
  • JSON Documents: Native JSON.SET/GET/DEL/TYPE with nlohmann_json
  • Rich Commands: Support for 270+ Redis commands across all data types
  • Persistence: AOF (Append Only File), RDB snapshots, and ROCKSDB integration
  • Cluster Support: Gossip-based cluster management with libgossip
  • Security: Access Control List (ACL) support
  • Monitoring: Prometheus metrics integration
  • Logging: High-performance structured logging with spdlog

Data Structures βœ…

  • String: Basic key-value operations
  • Hash: Field-value pairs
  • List: Linked list with push/pop operations
  • Set: Unique string collections
  • Sorted Set: Ordered sets with scores (B+ tree implementation)
  • Stream: Redis Streams for message queues
  • Bitmap: Bit manipulation operations
  • HyperLogLog: Probabilistic cardinality estimation
  • Geospatial: Geospatial indexing and queries
  • JSON: Document storage with JSON Pointer path support

Advanced Features βœ…

  • Transactions: MULTI/EXEC with optimistic locking
  • Pub/Sub: Publish/Subscribe messaging
  • Lua Scripting: Server-side scripting with Lua 5.4
  • ACL: User-based access control
  • TTL: Time-to-live for automatic key expiration
  • Replication: Master-slave replication (partial)
  • Cluster: Distributed cluster with gossip protocol

Planned Features 🚧

  • Real Blocking: Full blocking command implementation (BLPOP, BRPOP, etc.)
  • Raft Consensus: Distributed consensus for cluster management
  • TLS Encryption: Secure connections with OpenSSL
  • Scatter-Gather VSEARCH: Parallel multi-worker vector search
  • Redis Modules compatibility: Support for Redis Modules API

πŸ“Š Performance

Current Baseline (2026-05-04)

Test setup:

  • AstraDB binary: build-linux-release-debuginfo-noasan/bin/astradb
  • AstraDB config: /tmp/astradb-mgetmset-test.toml (thread_count=2, num_shards=2, persistence disabled)
  • Redis baseline: redis-server 8.0.2 (--save '' --appendonly no)
  • Client: redis-benchmark
  • Run shape: -n 300000 -c 80 -P 1 (non-pipeline)
Command AstraDB QPS Redis QPS AstraDB vs Redis
SET 220,102.70 219,780.22 +0.15%
GET 223,380.48 221,402.20 +0.89%
MSET k1..k4 187,617.27 211,118.94 -11.13%
MGET k1..k4 182,815.36 205,761.33 -11.15%

Conclusion: non-pipeline SET/GET is roughly at Redis level on this setup; MSET/MGET still needs major optimization.

Peak values from the same run shape (-n 300000 -c 80 -P 1):

  • SET: 223,498
  • GET: 228,116
  • MSET k1..k4: 190,056
  • MGET k1..k4: 184,532

Benchmark note: run commands sequentially (not in parallel) when collecting baseline numbers.

Historical Mixed Pipeline Baseline (2026-04-23)

All numbers below are from one mixed benchmark mode (-t set,get) to avoid split-test bias.

Pipeline (-P) SET QPS GET QPS
1 209,424.08 209,117.52
16 1,381,215.50 1,805,054.12
64 1,612,903.25 2,145,922.75

For historical investigation notes and older experiments, see PERFORMANCE.md.

πŸ—οΈ Architecture

Current Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                        AstraDB Core                            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Server Core (Asio + Thread Pool)                              β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚  Server | Shard Manager | Command Handler                 β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                              ↓                                   β”‚
β”‚  Network Layer (Asio)                                          β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚  Connection Management | RESP2 Parser | Command Registry  β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                              ↓                                   β”‚
β”‚  Command Layer (100+ Redis Commands)                          β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚  String | Hash | List | Set | ZSet | Stream | Transaction  β”‚  β”‚
β”‚  β”‚  PubSub | Script | Admin | ACL | Bitmap | HyperLogLog       β”‚  β”‚
β”‚  β”‚  Geospatial | Client | Cluster | Replication | TTL           β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                              ↓                                   β”‚
β”‚  Data Structures Layer                                         β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚  DashMap | B+ Tree ZSet | String Pool | Linked List       β”‚  β”‚
β”‚  β”‚  Stream Data | HNSW Vector Index | FlatBuffers Serialize   β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                              ↓                                   β”‚
β”‚  Storage Layer                                                 β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚  Key Metadata | AOF Writer | RDB Writer | ROCKSDB Adapter  β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Cluster & Security:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Gossip Manager (libgossip) | ACL Manager | Replication Manager β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Planned Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    Blocking Manager                            β”‚
β”‚  Wait Queue | Timeout Management | Async Notification           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    Raft Consensus                              β”‚
β”‚  Leader Election | Log Replication | Consensus Protocol         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ› οΈ Technology Stack

Core Dependencies

Component Library Version Purpose
Networking Asio 1.30.2 Async networking and coroutines
Serialization FlatBuffers 24.3.25 Zero-copy serialization
Logging spdlog 1.17.0 High-performance logging
Memory mimalloc 2.1.7 Fast memory allocator
Container Abseil 20240116.1 High-performance containers
Thread Pool Intel TBB 2021.12.0 Work-stealing scheduler
Concurrent Queue concurrentqueue 1.0.4 Lock-free queue
Cluster libgossip 1.2.0 Gossip protocol
Vector Search hnswlib 0.8.0 HNSW approximate nearest neighbor
Storage ROCKSDB Latest Key-value store
Metrics Prometheus Client 1.2.4 Metrics collection
Compression zstd 1.5.6 Fast compression
JSON nlohmann_json 3.11.2 JSON parsing
Lua Lua 5.4.7 Scripting support
Config tomlplusplus 3.4.0 TOML configuration
CLI cxxopts 3.2.1 Command-line parsing
Testing GoogleTest 1.14.0 Unit testing
Benchmarking Google Benchmark 1.8.5 Performance benchmarking

Build System

  • CMake: 3.20+
  • C++ Standard: C++23
  • Compiler: GCC 13+ / Clang 16+
  • Build Tool: Ninja (recommended) or Make
  • Package Manager: CPM (C++ Package Manager)

πŸ“¦ Project Structure

AstraDB/
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ astra/
β”‚   β”‚   β”œβ”€β”€ base/              # Core utilities and logging
β”‚   β”‚   β”œβ”€β”€ commands/          # Redis command implementations
β”‚   β”‚   β”œβ”€β”€ container/         # Data structures (DashMap, ZSet, List, Stream)
β”‚   β”‚   β”œβ”€β”€ core/              # Core functionality (memory, metrics)
β”‚   β”‚   β”œβ”€β”€ network/           # Networking layer (RESP protocol)
β”‚   β”‚   β”œβ”€β”€ server/            # Server core (Server, Shard)
β”‚   β”‚   β”œβ”€β”€ persistence/       # Persistence layer (AOF, RDB, ROCKSDB)
β”‚   β”‚   β”œβ”€β”€ cluster/           # Cluster management (Gossip)
β”‚   β”‚   β”œβ”€β”€ security/          # Security layer (ACL)
β”‚   β”‚   β”œβ”€β”€ replication/       # Replication manager
β”‚   β”‚   └── storage/           # Storage utilities
β”‚   └── main.cpp               # Application entry point
β”œβ”€β”€ tests/
β”‚   β”œβ”€β”€ unit/                  # Unit tests
β”‚   β”œβ”€β”€ benchmark/             # Performance benchmarks
β”‚   └── integration/           # Integration tests
β”œβ”€β”€ cmake/                     # CMake configuration
β”œβ”€β”€ third_party/               # Third-party dependencies
β”œβ”€β”€ AstraDB_DESIGN.md          # Design documentation
β”œβ”€β”€ README.md                  # This file
β”œβ”€β”€ LICENSE                    # Apache 2.0 License
└── CMakeLists.txt             # Root CMake file

πŸš€ Quick Start

Prerequisites

  • CMake 3.20+
  • C++23 compatible compiler (GCC 13+, Clang 16+)
  • Ninja build system (recommended)

Build

# Clone the repository
git clone https://github.com/caomengxuan666/AstraDB.git
cd AstraDB

# Configure and build (Release mode)
cmake -B build-release -G "Ninja" -DCMAKE_BUILD_TYPE=Release
ninja -C build-release

# Run the server
./build-release/bin/astradb --port 6379

Build Options

# Debug mode
cmake -B build-debug -G "Ninja" -DCMAKE_BUILD_TYPE=Debug
ninja -C build-debug

# Release mode with debug information (recommended for development)
cmake --preset linux-release-debuginfo-clang
ninja -C build-linux-release-debuginfo-clang

# Enable LTO (Link-Time Optimization)
cmake -B build-release -G "Ninja" -DCMAKE_BUILD_TYPE=Release -DASTRADB_ENABLE_LTO=ON
ninja -C build-release

# Enable SIMD optimizations (default: ON)
cmake -B build-release -G "Ninja" -DCMAKE_BUILD_TYPE=Release -DASTRADB_ENABLE_SIMD=ON
ninja -C build-release

# Build without examples (for package builds)
cmake -B build-release -G "Ninja" -DCMAKE_BUILD_TYPE=Release -DASTRADB_BUILD_EXAMPLES=OFF
ninja -C build-release

# Use io_uring backend (Linux 5.1+, high performance)
cmake --preset linux-package-clang-iouring

# Use epoll backend (stable, compatible with all Linux versions)
cmake --preset linux-package-clang

Configuration

You can configure AstraDB using command-line options or a configuration file:

# Command-line options
./build-release/bin/astradb \
  --port 6379 \
  --threads 20 \
  --shards 16 \
  --databases 16 \
  --max-connections 10000

# Single-threaded mode (for testing or low-load scenarios)
./build-release/bin/astradb \
  --port 6379 \
  --threads 1 \
  --shards 1

# Using configuration file
./build-release/bin/astradb --config astradb.toml

Example astradb.toml:

[server]
host = "0.0.0.0"
port = 6379
max_connections = 10000
thread_count = 20
shard_count = 16
database_count = 16

[logging]
level = "info"
async = true
queue_size = 8192

# Storage mode configuration
[storage]
mode = "redis"  # Options: "redis" (default) or "rocksdb"

# Redis mode persistence (when mode = "redis")
[persistence]
aof_enabled = true
rdb_enabled = true
snapshot_interval = 300  # seconds

# RocksDB configuration (when mode = "rocksdb")
[rocksdb]
data_dir = "./data/rocksdb_allin"
# All data is persisted to RocksDB in all-in mode
# This is different from Redis mode where RocksDB is only used for cold data

Storage Modes

AstraDB supports two storage modes for different use cases:

Redis Mode (Default)

  • Memory-first: Data primarily stored in memory
  • Persistence: AOF and RDB for durability
  • Cold Data Storage: RocksDB for evicted data (optional)
  • Use Case: High-performance scenarios with low latency requirements
[storage]
mode = "redis"

[persistence]
aof_enabled = true
rdb_enabled = true
rocksdb_cold_data = true  # Optional: enable RocksDB for cold data

RocksDB All-in Mode

  • Disk-first: All data persisted to RocksDB
  • Memory Cache: Hot data cached in memory with automatic eviction
  • Persistence: FlatBuffer serialization for type-safe storage
  • Use Case: Large datasets, memory-constrained environments
[storage]
mode = "rocksdb"

[rocksdb]
data_dir = "./data/rocksdb_allin"

Key Differences:

  • Redis Mode: Best for performance, lower latency, requires more memory
  • RocksDB All-in Mode: Best for memory efficiency, handles large datasets, slightly higher latency

Both modes support all Redis data types and commands with automatic persistence.

πŸ§ͺ Testing

Run Tests

# Run unit tests
./build-release/bin/astradb_tests

# Run specific test suite
./build-release/bin/astradb_tests --gtest_filter=StringCommandsTest.*

# Run benchmarks
./build-release/bin/astradb_benchmarks

# Run specific benchmark
./build-release/bin/astradb_benchmarks --benchmark_filter=SetPerformance

Redis Compatibility Testing

# Using redis-cli (from Redis)
redis-cli -p 6379 PING

# Test basic operations
redis-cli -p 6379 SET mykey "Hello, AstraDB!"
redis-cli -p 6379 GET mykey

# Test list operations
redis-cli -p 6379 LPUSH mylist item1 item2 item3
redis-cli -p 6379 LRANGE mylist 0 -1

# Test sorted set operations
redis-cli -p 6379 ZADD myzset 1 "one" 2 "two" 3 "three"
redis-cli -p 6379 ZRANGE myzset 0 -1 WITHSCORES

Vector Search Quick Start πŸ†•

# Start AstraDB
./build-release/bin/astradb --port 6379

# Use the bundled Python benchmark script
python3 scripts/vector_bench.py --dim 512 --count 5000 --search-qps-duration 10

# Or test manually with raw RESP commands:
# Create a 512-dim cosine index
printf '*4\r\n$7\r\nVCREATE\r\n$2\r\ndb\r\n$3\r\n512\r\n$6\r\ncosine\r\n' | nc 127.0.0.1 6379

# Search with a query vector (requires raw float32 bytes)
# Python example using raw sockets
import socket, struct, random

def resp_cmd(sock, *args):
    parts = [f'*{len(args)}\r\n'.encode()]
    for a in args:
        b = a if isinstance(a, bytes) else a.encode()
        parts.append(f'${len(b)}\r\n'.encode() + b + b'\r\n')
    sock.sendall(b''.join(parts))
    return sock.recv(1024)

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('127.0.0.1', 6379))

# Create index
resp_cmd(sock, 'VCREATE', 'docs', '768', 'cosine')

# Insert a vector (768-dimensional random)
vec = struct.pack('768f', *[random.gauss(0, 1) for _ in range(768)])
resp_cmd(sock, 'VSET', 'docs', 'doc:1', vec)

# Search top-10
query = struct.pack('768f', *[random.gauss(0, 1) for _ in range(768)])
result = resp_cmd(sock, 'VSEARCH', 'docs', query, '10')
print(result)

Benchmarking

# Canonical mixed read/write benchmark (recommended)
redis-benchmark -h 127.0.0.1 -p 6379 -t set,get -n 1000000 -c 256 -P 1 -q
redis-benchmark -h 127.0.0.1 -p 6379 -t set,get -n 1000000 -c 256 -P 16 -q
redis-benchmark -h 127.0.0.1 -p 6379 -t set,get -n 1000000 -c 256 -P 64 -q

# Optional: command-specific stress
redis-benchmark -h 127.0.0.1 -p 6379 -t lpush,lpop,rpush,rpop -n 1000000 -c 256 -P 16 -q

πŸ“š Documentation

🀝 Contributing

Contributions are welcome! Please follow these guidelines:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Coding Standards

  • Follow the existing code style
  • Use C++23 features where appropriate
  • Add unit tests for new features
  • Update documentation as needed
  • Run clang-format and clang-tidy before submitting

Commit Guidelines

  • Use clear, descriptive commit messages
  • Follow conventional commit format:
    • feat: add new feature
    • fix: fix bug
    • docs: update documentation
    • test: add tests
    • refactor: refactor code
    • perf: performance improvement

πŸ—ΊοΈ Roadmap

v1.0.0 (Current Target)

  • Core server infrastructure
  • 250+ Redis commands (96%+ Redis 7.4.1 compatibility)
  • All data types (String, Hash, List, Set, ZSet, Stream)
  • Persistence (AOF, RDB, ROCKSDB)
  • Cluster support (Gossip)
  • ACL support
  • Lua scripting
  • Transactions
  • Pub/Sub
  • RESP2 protocol support
  • RESP3 protocol support (HELLO command)
  • Bitmap commands
  • HyperLogLog commands
  • Geospatial commands
  • TTL commands
  • Real blocking commands (simplified implementation)
  • Raft consensus
  • Full Redis compatibility (100%)
  • Comprehensive documentation
  • Vector search (in-memory, hnswlib, cosine/L2/IP)

v1.2.0 (Future)

  • TLS encryption
  • Full RESP3 protocol
  • Scatter-gather multi-worker VSEARCH
  • Redis Modules compatibility
  • Web UI for monitoring

Implemented Commands (250+)

AstraDB has implemented 270+ Redis commands, covering all major data types and features. We aim to achieve 100% Redis 7.4.1 compatibility., covering all major data types and features. We aim to achieve 100% Redis 7.4.1 compatibility.

String Commands (30+)

  • GET, SET, DEL, EXISTS, MGET, MSET, MSETNX
  • INCR, DECR, INCRBY, DECRBY, INCRBYFLOAT
  • APPEND, STRLEN, GETRANGE, SETRANGE, SUBSTR
  • SETEX, PSETEX, SETNX, GETSET, GETDEL, GETEX
  • STRALGO, LCS, COPY, DUMP, RESTORE, UNLINK
  • TYPE, ECHO, RANDOMKEY, RENAME, RENAMENX, MOVE, TOUCH

Hash Commands (20+)

  • HSET, HGET, HGETALL, HKEYS, HVALS, HLEN
  • HMGET, HMSET, HEXISTS, HDEL
  • HINCRBY, HINCRBYFLOAT, HSTRLEN, HRANDFIELD
  • HSCAN, HSETNX, HTTL, HEXPIRE, HEXPIREAT
  • HEXPIRETIME, HPERSIST, HPEXPIRE, HPEXPIREAT, HPEXPIRETIME, HPTTL

List Commands (20+)

  • LPUSH, RPUSH, LPOP, RPOP, LPUSHX, RPUSHX
  • LLEN, LRANGE, LINDEX, LINSERT, LSET, LTRIM, LREM, LPOS
  • RPOPLPUSH, LMOVE, LMPOP
  • BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, BLMPOP

Set Commands (20+)

  • SADD, SREM, SISMEMBER, SCARD, SMEMBERS
  • SRANDMEMBER, SPOP, SMOVE, SMISMEMBER
  • SDIFF, SINTER, SUNION, SINTERCARD
  • SDIFFSTORE, SINTERSTORE, SUNIONSTORE
  • SSCAN, SORT, SORT_RO
  • SPUBLISH, SSUBSCRIBE, SUNSUBSCRIBE

Sorted Set Commands (30+)

  • ZADD, ZREM, ZCARD, ZSCORE, ZINCRBY, ZCOUNT, ZMSCORE, ZRANDMEMBER
  • ZRANGE, ZREVRANGE, ZRANGEBYSCORE, ZREVRANGEBYSCORE, ZRANGESTORE
  • ZRANK, ZREVRANK, ZPOPMIN, ZPOPMAX
  • ZRANGEBYLEX, ZREVRANGEBYLEX, ZLEXCOUNT, ZREMRANGEBYLEX
  • ZREMRANGEBYRANK, ZREMRANGEBYSCORE
  • ZUNION, ZINTER, ZUNIONSTORE, ZINTERSTORE, ZDIFF, ZDIFFSTORE, ZINTERCARD
  • ZMPOP, BZPOPMIN, BZPOPMAX, BZMPOP
  • ZSCAN

Stream Commands (15+)

  • XADD, XREAD, XRANGE, XREVRANGE, XREADGROUP
  • XLEN, XDEL, XTRIM, XSETID
  • XGROUP, XACK, XPENDING, XINFO, XCLAIM, XAUTOCLAIM

Transaction Commands (5+)

  • MULTI, EXEC, DISCARD, WATCH, UNWATCH

Pub/Sub Commands (5+)

  • SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, PUBLISH

Script Commands (5+)

  • EVAL, EVALSHA, SCRIPT, SCRIPT EXISTS, SCRIPT FLUSH

Admin Commands (40+)

  • INFO, CONFIG, DBSIZE, KEYS, FLUSHDB, FLUSHALL, SELECT
  • PING, ECHO, QUIT, SAVE, BGSAVE, LASTSAVE, BGREWRITEAOF
  • COMMAND, DEBUG, CLUSTER, MIGRATE, MODULE, SCAN, MEMORY, ASKING
  • TYPE, RANDOMKEY, RENAME, RENAMENX, MOVE, OBJECT, TOUCH
  • TIME, SHUTDOWN, SWAPDB, WAIT, WAITAOF, READONLY, READWRITE, RESET
  • FAILOVER, LATENCY, MONITOR, SLOWLOG, LOLWUT, AUTH, ACL, HELLO, SLAVEOF, REPLICAOF

ACL Commands (5+)

  • ACL SETUSER, ACL GETUSER, ACL DELUSER, ACL LIST, ACL USERS

Bitmap Commands (5+)

  • SETBIT, GETBIT, BITCOUNT, BITPOS, BITOP

HyperLogLog Commands (5+)

  • PFADD, PFCOUNT, PFMERGE, PFDEBUG, PFSELFTEST

Geospatial Commands (10+)

  • GEOADD, GEODIST, GEOHASH, GEOPOS, GEORADIUS, GEORADIUSBYMEMBER

Client Commands (5+)

  • CLIENT, CLIENT LIST, CLIENT KILL, CLIENT SETNAME, CLIENT GETNAME

Cluster Commands (5+)

  • CLUSTER, CLUSTER INFO, CLUSTER NODES, CLUSTER MEET, CLUSTER SLOTS

Replication Commands (5+)

  • SYNC, PSYNC, REPLCONF, SLAVEOF, REPLICAOF, ROLE

Vector Search Commands (10) πŸ†•

  • VCREATE β€” Create a vector index with dimension and distance metric
  • VDROP β€” Drop a vector index
  • VLIST β€” List all vector indexes
  • VSET β€” Insert/update a vector with optional metadata
  • MVSET β€” Batch insert multiple vectors
  • VGET β€” Retrieve a vector entry
  • VDEL β€” Delete a vector entry
  • VSEARCH β€” KNN search by vector similarity
  • VINFO β€” Get vector index statistics
  • VCOMPACT β€” Compact vector index (remove deleted entries)

TTL Commands (9+)

  • TTL, PTTL, EXPIRE, PEXPIRE, EXPIREAT, PEXPIREAT
  • EXPIRETIME, PEXPIRETIME, PERSIST

JSON Commands (10) πŸ†•

  • JSON.SET β€” Set JSON document at key
  • JSON.GET β€” Get JSON document or sub-path
  • JSON.DEL β€” Delete key or sub-path
  • JSON.TYPE β€” Get type of value at path
  • JSON.ARRAPPEND β€” Append values to array
  • JSON.ARRLEN β€” Get array length
  • JSON.OBJLEN β€” Get object key count
  • JSON.NUMINCRBY β€” Increment number at path
  • JSON.STRAPPEND β€” Append to string at path
  • JSON.ARRINDEX β€” Find value index in array

Command Completion Status

Category Commands Status
String 30+ βœ… Complete
Hash 20+ βœ… Complete
List 20+ βœ… Complete
Set 20+ βœ… Complete
Sorted Set 30+ βœ… Complete
Stream 15+ βœ… Complete
Transaction 5 βœ… Complete
Pub/Sub 6 βœ… Complete
Script 9 βœ… Complete
Admin 40+ βœ… Complete
ACL 5+ βœ… Complete
Bitmap 6 βœ… Complete
HyperLogLog 5 βœ… Complete
Geospatial 10+ βœ… Complete
Client 4 βœ… Complete
Cluster 5+ βœ… Complete
Replication 6 βœ… Complete
TTL 9 βœ… Complete
JSON 10 βœ… New
Vector Search 10 βœ… New
Total 270+ 96%+

Command Implementation Notes

  • Full Implementation: Commands with complete Redis 7.4.1 compatibility
  • Simplified Implementation: Commands with basic functionality, marked for future enhancement
  • Experimental: Commands with partial support or in development

All 250+ commands are registered and functional. We aim for 100% Redis 7.4.1 compatibility by v1.0.0.

πŸ“ˆ Performance Metrics

Metric Current Target Status
MGET(4) QPS (Non-Pipeline) 182,815.36 (Peak 184,532) >= Redis ⚠️ 89% of Redis (205,761.33)
MSET(4) QPS (Non-Pipeline) 187,617.27 (Peak 190,056) >= Redis ⚠️ 89% of Redis (211,118.94)
Pipeline QPS ~several million 10M+ ⏳ Testing planned
Command Coverage 250+ (96%+) 250+ (100%) 96%
Memory Overhead TBD 0 bytes TBD
Scalability (1β†’8 threads) TBD 8x TBD
Startup Time TBD 0.5s TBD

Note: on this machine, non-pipeline single-key throughput is NIC-limited at around 220k QPS (similar to Redis/Dragonfly), so optimization focus is on multi-key and pipeline paths.

πŸ† Comparison

AstraDB vs Redis vs DragonflyDB

Feature Redis DragonflyDB AstraDB
Single-threaded βœ… ❌ ❌
Multi-threaded ❌ βœ… βœ…
Sharding Manual Automatic Automatic
Persistence RDB/AOF Snapshot AOF/RDB/ROCKSDB
Clustering Redis Cluster Built-in Built-in
ACL βœ… βœ… βœ…
Lua Scripting βœ… βœ… βœ…
Transactions βœ… βœ… βœ…
Pub/Sub βœ… βœ… βœ…
Streams βœ… βœ… βœ…
HyperLogLog βœ… βœ… βœ…
Geospatial βœ… βœ… βœ…
Bitmaps βœ… βœ… βœ…
SIMD ❌ βœ… βœ…
MPSC Queues ❌ βœ… βœ…
Zero-Copy I/O ❌ βœ… βœ…
C++23 ❌ ❌ βœ…
Vector Search βœ… (RediSearch) ❌ βœ… (hnswlib)

πŸ” Security

Access Control List (ACL)

AstraDB supports Redis-compatible ACL for user-based access control:

# Create a new user
ACL SETUSER myuser on >mypassword ~* +@all

# Get user information
ACL GETUSER myuser

# List all users
ACL LIST

# Delete a user
ACL DELUSER myuser

TLS Support (Planned)

TLS encryption support is planned for v1.1.0:

# Enable TLS in configuration
[server]
tls_enabled = true
tls_cert_file = "/path/to/cert.pem"
tls_key_file = "/path/to/key.pem"

πŸ“ž Support

πŸ“„ License

Licensed under the Apache License, Version 2.0. See the LICENSE file for details.

πŸ™ Acknowledgments

  • Redis: The inspiration and protocol reference
  • DragonflyDB: Performance optimization ideas
  • Asio: Excellent async networking library
  • FlatBuffers: Zero-copy serialization
  • ROCKSDB: Lightweight key-value store
  • libgossip: Gossip protocol implementation
  • spdlog: High-performance logging
  • mimalloc: Fast memory allocator
  • All contributors: Thank you for your contributions!

Let's build the fastest Redis-compatible database! πŸš€

About

A high-performance, Redis-compatible database written in modern C++23.

Resources

License

Stars

20 stars

Watchers

0 watching

Forks

Packages

 
 
 

Contributors