Refactor of the library core #98

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Mirrowel merged 37 commits into dev from refactor/core

Jan 24, 2026

.gitignore

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -126,9 +126,10 @@ staged_changes.txt
  
    launcher_config.json

    quota_viewer_config.json

    cache/antigravity/thought_signatures.json

    logs/

    cache/

    /logs/

    /cache/

    *.env

    oauth_creds/

    /oauth_creds/

    /usage/

DOCUMENTATION.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -22,9 +22,9 @@ This architecture cleanly separates the API interface from the resilience logic,
  
    This library is the heart of the project, containing all the logic for managing a pool of API keys, tracking their usage, and handling provider interactions to ensure application resilience.

    ### 2.1. `client.py` - The `RotatingClient`

    ### 2.1. `client/rotating_client.py` - The `RotatingClient`

    The `RotatingClient` is the central class that orchestrates all operations. It is designed as a long-lived, async-native object.

    The `RotatingClient` is the central class that orchestrates all operations. It is now a slim facade that delegates to modular components (executor, filters, transforms) while remaining a long-lived, async-native object.

    #### Initialization

    @@ -35,7 +35,7 @@ client = RotatingClient(
  
        api_keys=api_keys,

        oauth_credentials=oauth_credentials,

        max_retries=2,

        usage_file_path="key_usage.json",

        usage_file_path="usage.json",

        configure_logging=True,

        global_timeout=30,

        abort_on_callback_error=True,

    @@ -50,7 +50,7 @@ client = RotatingClient(
  
    -   `api_keys` (`Optional[Dict[str, List[str]]]`, default: `None`): A dictionary mapping provider names to a list of API keys.

    -   `oauth_credentials` (`Optional[Dict[str, List[str]]]`, default: `None`): A dictionary mapping provider names to a list of file paths to OAuth credential JSON files.

    -   `max_retries` (`int`, default: `2`): The number of times to retry a request with the *same key* if a transient server error occurs.

    -   `usage_file_path` (`str`, default: `"key_usage.json"`): The path to the JSON file where usage statistics are persisted.

    -   `usage_file_path` (`str`, optional): Base path for usage persistence (defaults to `usage/` in the data directory). The client stores per-provider files under `usage/usage_<provider>.json`.

    -   `configure_logging` (`bool`, default: `True`): If `True`, configures the library's logger to propagate logs to the root logger.

    -   `global_timeout` (`int`, default: `30`): A hard time limit (in seconds) for the entire request lifecycle.

    -   `abort_on_callback_error` (`bool`, default: `True`): If `True`, any exception raised by `pre_request_callback` will abort the request.

    @@ -96,9 +96,9 @@ The `_safe_streaming_wrapper` is a critical component for stability. It:
  
    *   **Error Interception**: Detects if a chunk contains an API error (like a quota limit) instead of content, and raises a specific `StreamedAPIError`.

    *   **Quota Handling**: If a specific "quota exceeded" error is detected mid-stream multiple times, it can terminate the stream gracefully to prevent infinite retry loops on oversized inputs.

    ### 2.2. `usage_manager.py` - Stateful Concurrency & Usage Management

    ### 2.2. `usage/manager.py` - Stateful Concurrency & Usage Management

    This class is the stateful core of the library, managing concurrency, usage tracking, cooldowns, and quota resets.

    This class is the stateful core of the library, managing concurrency, usage tracking, cooldowns, and quota resets. Usage tracking now lives in the `rotator_library/usage/` package with per-provider managers and `usage/usage_<provider>.json` storage.

    #### Key Concepts

    @@ -419,7 +419,7 @@ The `CooldownManager` handles IP or account-level rate limiting that affects all
  
    - All subsequent `acquire_key()` calls for that provider will wait until the cooldown expires

    ### 2.10. Credential Prioritization System (`client.py` & `usage_manager.py`)

    ### 2.10. Credential Prioritization System (`client/rotating_client.py` & `usage/manager.py`)

    The library now includes an intelligent credential prioritization system that automatically detects credential tiers and ensures optimal credential selection for each request.

    @@ -762,7 +762,7 @@ Acquiring key for model antigravity/claude-opus-4.5. Tried keys: 0/12(17,cd:3,fc
  
    ```

    **Persistence:**

    Cycle state is persisted in `key_usage.json` under the `__fair_cycle__` key.

    Cycle state is persisted alongside usage data in `usage/usage_<provider>.json`.

    ### 2.20. Custom Caps

    @@ -1773,7 +1773,7 @@ The system follows a strict hierarchy of survival:
  
    2. **Credential Management (Level 2)**: OAuth tokens are cached in memory first. If credential files are deleted, the proxy continues using cached tokens. If a token refresh succeeds but the file cannot be written, the new token is buffered for retry and saved on shutdown.

    3. **Usage Tracking (Level 3)**: Usage statistics (`key_usage.json`) are maintained in memory via `ResilientStateWriter`. If the file is deleted, the system tracks usage internally and attempts to recreate the file on the next save interval. Pending writes are flushed on shutdown.

    3. **Usage Tracking (Level 3)**: Usage statistics (`usage/usage_<provider>.json`) are maintained in memory via `ResilientStateWriter`. If the file is deleted, the system tracks usage internally and attempts to recreate the file on the next save interval. Pending writes are flushed on shutdown.

    4. **Provider Cache (Level 4)**: The provider cache tracks disk health and continues operating in memory-only mode if disk writes fail. Has its own shutdown mechanism.

    @@ -1813,7 +1813,7 @@ INFO:rotator_library.resilient_io:Shutdown flush: all 2 write(s) succeeded
  
    This architecture supports a robust development workflow:

    - **Log Cleanup**: You can safely run `rm -rf logs/` while the proxy is serving traffic. The system will recreate the directory structure on the next request.

    - **Config Reset**: Deleting `key_usage.json` resets the persistence layer, but the running instance preserves its current in-memory counts for load balancing consistency.

    - **Config Reset**: Deleting `usage/usage_<provider>.json` resets the persistence layer, but the running instance preserves its current in-memory counts for load balancing consistency.

    - **File Recovery**: If you delete a critical file, the system attempts directory auto-recreation before every write operation.

    - **Safe Exit**: Ctrl+C triggers graceful shutdown with final data flush attempt.

README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -53,20 +53,21 @@ docker run -d \
  
      -v $(pwd)/.env:/app/.env:ro \

      -v $(pwd)/oauth_creds:/app/oauth_creds \

      -v $(pwd)/logs:/app/logs \

      -v $(pwd)/usage:/app/usage \

      -e SKIP_OAUTH_INIT_CHECK=true \

      ghcr.io/mirrowel/llm-api-key-proxy:latest

    ```

    **Using Docker Compose:**

    ```bash

    # Create your .env file and key_usage.json first, then:

    # Create your .env file and usage directory first, then:

    cp .env.example .env

    touch key_usage.json

    mkdir usage

    docker compose up -d

    ```

    > **Important:** You must create both `.env` and `key_usage.json` files before running Docker Compose. If `key_usage.json` doesn't exist, Docker will create it as a directory instead of a file, causing errors.

    > **Important:** Create the `usage/` directory before running Docker Compose so usage stats persist on the host.

    > **Note:** For OAuth providers, complete authentication locally first using the credential tool, then mount the `oauth_creds/` directory or export credentials to environment variables.

    @@ -335,17 +336,20 @@ The proxy includes a powerful text-based UI for configuration and management.
  
    ### TUI Features

    - **🚀 Run Proxy** — Start the server with saved settings

    - **⚙️ Configure Settings** — Host, port, API key, request logging

    - **⚙️ Configure Settings** — Host, port, API key, request logging, raw I/O logging

    - **🔑 Manage Credentials** — Add/edit API keys and OAuth credentials

    - **📊 View Status** — See configured providers and credential counts

    - **🔧 Advanced Settings** — Custom providers, model definitions, concurrency

    - **📊 View Provider & Advanced Settings** — Inspect providers and launch the settings tool

    - **📈 View Quota & Usage Stats (Alpha)** — Usage, quota windows, fair-cycle status

    - **🔄 Reload Configuration** — Refresh settings without restarting

    ### Configuration Files

    | File | Contents |

    |------|----------|

    | `.env` | All credentials and advanced settings |

    | `launcher_config.json` | TUI-specific settings (host, port, logging) |

    | `quota_viewer_config.json` | Quota viewer remotes + per-provider display toggles |

    | `usage/usage_<provider>.json` | Usage persistence per provider |

    ---

    @@ -446,6 +450,7 @@ The proxy includes a powerful text-based UI for configuration and management.
  
    <summary><b>📝 Logging & Debugging</b></summary>

    - **Per-request file logging** with `--enable-request-logging`

    - **Raw I/O logging** with `--enable-raw-logging` (proxy boundary payloads)

    - **Unique request directories** with full transaction details

    - **Streaming chunk capture** for debugging

    - **Performance metadata** (duration, tokens, model used)

    @@ -801,6 +806,7 @@ Options:
  
      --host TEXT                Host to bind (default: 0.0.0.0)

      --port INTEGER             Port to run on (default: 8000)

      --enable-request-logging   Enable detailed per-request logging

      --enable-raw-logging       Capture raw proxy I/O payloads

      --add-credential           Launch interactive credential setup tool

    ```

    @@ -813,6 +819,9 @@ python src/proxy_app/main.py --host 127.0.0.1 --port 9000
  
    # Run with logging

    python src/proxy_app/main.py --enable-request-logging

    # Run with raw I/O logging

    python src/proxy_app/main.py --enable-raw-logging

    # Add credentials without starting proxy

    python src/proxy_app/main.py --add-credential

    ```

    @@ -850,8 +859,8 @@ The proxy is available as a multi-architecture Docker image (amd64/arm64) from G
  
    cp .env.example .env

    nano .env

    # 2. Create key_usage.json file (required before first run)

    touch key_usage.json

    # 2. Create usage directory (usage_*.json files are created automatically)

    mkdir usage

    # 3. Start the proxy

    docker compose up -d

    @@ -860,13 +869,13 @@ docker compose up -d
  
    docker compose logs -f

    ```

    > **Important:** You must create `key_usage.json` before running Docker Compose. If this file doesn't exist on the host, Docker will create it as a directory instead of a file, causing the container to fail.

    > **Important:** Create the `usage/` directory before running Docker Compose so usage stats persist on the host.

    **Manual Docker Run:**

    ```bash

    # Create key_usage.json if it doesn't exist

    touch key_usage.json

    # Create usage directory if it doesn't exist

    mkdir usage

    docker run -d \

      --name llm-api-proxy \

    @@ -875,7 +884,7 @@ docker run -d \
  
      -v $(pwd)/.env:/app/.env:ro \

      -v $(pwd)/oauth_creds:/app/oauth_creds \

      -v $(pwd)/logs:/app/logs \

      -v $(pwd)/key_usage.json:/app/key_usage.json \

      -v $(pwd)/usage:/app/usage \

      -e SKIP_OAUTH_INIT_CHECK=true \

      -e PYTHONUNBUFFERED=1 \

      ghcr.io/mirrowel/llm-api-key-proxy:latest

    @@ -895,7 +904,7 @@ docker compose -f docker-compose.dev.yml up -d --build
  
    | `.env`           | Configuration and API keys (read-only) |

    | `oauth_creds/`   | OAuth credential files (persistent)    |

    | `logs/`          | Request logs and detailed logging      |

    | `key_usage.json` | Usage statistics persistence           |

    | `usage/`       | Usage statistics persistence (`usage_*.json`) |

    **Image Tags:**

docker-compose.dev.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -19,8 +19,8 @@ services:
  
          - ./oauth_creds:/app/oauth_creds

          # Mount logs directory for persistent logging

          - ./logs:/app/logs

          # Mount key_usage.json for usage statistics persistence

          - ./key_usage.json:/app/key_usage.json

          # Mount usage directory for usage statistics persistence

          - ./usage:/app/usage

          # Optionally mount additional .env files (e.g., combined credential files)

          # - ./antigravity_all_combined.env:/app/antigravity_all_combined.env:ro

        environment:

docker-compose.tls.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -36,8 +36,8 @@ services:
  
          - ./oauth_creds:/app/oauth_creds

          # Mount logs directory for persistent logging

          - ./logs:/app/logs

          # Mount key_usage.json for usage statistics persistence

          - ./key_usage.json:/app/key_usage.json

          # Mount usage directory for usage statistics persistence

          - ./usage:/app/usage

          # Optionally mount additional .env files (e.g., combined credential files)

          # - ./antigravity_all_combined.env:/app/antigravity_all_combined.env:ro

        environment:

docker-compose.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -17,8 +17,8 @@ services:
  
          - ./oauth_creds:/app/oauth_creds

          # Mount logs directory for persistent logging

          - ./logs:/app/logs

          # Mount key_usage.json for usage statistics persistence

          - ./key_usage.json:/app/key_usage.json

          # Mount usage directory for usage statistics persistence

          - ./usage:/app/usage

          # Optionally mount additional .env files (e.g., combined credential files)

          # - ./antigravity_all_combined.env:/app/antigravity_all_combined.env:ro

        environment:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor of the library core #98

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!