Meta DB Backup Guide

Meta DB Backup & Recovery Guide

Language: English | 한국어

1. Overview

• CB-Spider stores all management metadata (CSP connection configs, resource IIDs, etc.) in a single SQLite3 database file (meta_db/cb-spider.db).
• If this file is corrupted or accidentally deleted, it becomes very difficult to identify and manage resources that Spider was tracking across multiple CSPs and Regions.
• To mitigate this risk, CB-Spider provides an automatic background backup feature that periodically creates consistent copies of the Meta DB using SQLite3's VACUUM INTO.

┌───────────────────────────────────────────────────────────────┐
│                   CB-Spider Meta DB Backup                    │
│                                                               │
│  meta_db/                                                     │
│  ├── cb-spider.db                     ← Active Meta DB        │
│  └── backups/                         ← Backup Directory      │
│      ├── cb-spider_backup_20260305_060000.db                  │
│      ├── cb-spider_backup_20260305_120000.db                  │
│      ├── cb-spider_backup_20260305_180000.db                  │
│      └── ...  (up to MaxCount files, oldest rotated out)      │
└───────────────────────────────────────────────────────────────┘

2. Backup Policy

2.1 Online Backup (VACUUM INTO)

• Backups are performed using SQLite3's VACUUM INTO command.
• This is a fully online operation: reads and writes to the Meta DB continue normally during backup.
• The backup file is an optimized, consistent snapshot of the database at the time of execution.

2.2 Backup Schedule

Item	Default	Description
Startup Backup	Always	An immediate backup is performed when Spider server starts
Periodic Backup	Every 6 hours	Background scheduler runs at the configured interval
Minimum Interval	1 minute	Values below 1 minute are automatically adjusted to 1 minute

2.3 Retention (Rotation)

• Only the maximum backup count (MaxCount) is enforced.
• When the number of backup files exceeds MaxCount, the oldest files are deleted first.
• There is no time-based expiration; retention is purely count-based.

Item	Default	Description
Max Backup Count	10	Number of backup files to keep

2.4 Backup File Naming

cb-spider_backup_YYYYMMDD_HHMMSS.db

• Example: cb-spider_backup_20260305_180000.db
• Files are sorted by name (timestamp order) for rotation.

3. Configuration

All backup settings are configured via environment variables in setup.env.
All settings are optional. If not set, defaults are used without errors.

Environment Variable	Default	Description
SPIDER_BACKUP_ENABLED	true	Enable/disable backup (true, false, on, off, 1, 0)
SPIDER_BACKUP_INTERVAL	6h	Backup interval in Go duration format (1h, 30m, 24h, etc.)
SPIDER_BACKUP_DIR	$CBSPIDER_ROOT/meta_db/backups	Directory to store backup files
SPIDER_BACKUP_MAX_COUNT	10	Maximum number of backup files to retain

3.1 Configuration Examples

Default (no configuration needed)

# All defaults apply: enabled, 6h interval, 10 backups
# No environment variables need to be set

High-frequency backup

export SPIDER_BACKUP_INTERVAL=1h
export SPIDER_BACKUP_MAX_COUNT=24

Daily backup

export SPIDER_BACKUP_INTERVAL=24h
export SPIDER_BACKUP_MAX_COUNT=7

Disable backup

export SPIDER_BACKUP_ENABLED=false

4. Recovery Procedure

When the Meta DB (cb-spider.db) is corrupted or deleted, restore it from the latest backup:

4.1 Recovery Steps

Step 1.  Stop the CB-Spider server.
Step 2.  (Optional) Move or rename the corrupted DB file:
           $ mv meta_db/cb-spider.db meta_db/cb-spider.db.corrupted
Step 3.  Copy the latest backup file as the active DB:
           $ cp meta_db/backups/cb-spider_backup_20260305_180000.db meta_db/cb-spider.db
Step 4.  Restart the CB-Spider server.

Important: The server must be stopped before replacing the DB file, because SQLite uses file-level locking.

4.2 Identifying the Latest Backup

# List backups sorted by time (newest last)
$ ls -lt meta_db/backups/

The file with the most recent timestamp in its name is the latest backup.

5. Logging

All backup operations are logged via cb-log with the [MSB] (Meta Store Backup) prefix:

Log Level	Message	Description
INFO	[MSB] Meta DB Backup Scheduler started.	Scheduler initialization
INFO	[MSB] Starting meta DB backup...	Backup cycle begins
INFO	[MSB] Meta DB backup completed: <path> (took <duration>)	Backup success
INFO	[MSB] Deleted old backup: <filename>	Rotation cleanup
WARN	[MSB] Meta DB file not found: <path>. Skipping backup.	Source DB missing
ERROR	[MSB] Meta DB backup failed: <error>	Backup failure
ERROR	[MSB] Backup rotation failed: <error>	Rotation failure

6. Architecture

apiserver.go (main)
    │
    ├── LoadBackupConfig()         ← Read env vars (backup_config.go)
    │
    └── StartBackupScheduler()     ← Launch background goroutine (backup_scheduler.go)
            │
            ├── Immediate backup on startup
            │
            └── Periodic loop (time.Ticker)
                    │
                    ├── BackupMetaDB()      ← VACUUM INTO (backup.go)
                    │
                    └── RotateBackups()     ← Delete oldest if > MaxCount (backup.go)

Source File	Description
info-store/backup_config.go	Configuration parsing with safe defaults
info-store/backup.go	Core backup logic (VACUUM INTO) and rotation
info-store/backup_scheduler.go	Background scheduler with graceful shutdown