This guide covers all configuration options for the BigQuery MCP Server.
-This guide covers all configuration options for the BigQuery MCP Server v1.1.1 with CLI-first architecture.
+The preferred way to configure the server is using command-line arguments:
-# Basic usage
-python src/server.py sandbox-dev:dev_* sandbox-main:main_*
-
-# Multiple patterns for same project
-python src/server.py cayzer-xyz:demo_* cayzer-xyz:analytics_*
+# Basic usage with new CLI-first format
+python src/server.py --project "your-project-1:dev_*" --project "your-project-2:main_*" --billing-project "your-billing-project"
-# All datasets in a project
-python src/server.py your-project:*
+# Single project with all datasets
+python src/server.py --project "your-project:*" --billing-project "your-project"
-# With additional options
-python src/server.py sandbox-dev:dev_* \
- --billing-project cayzer-xyz \
- --location EU
+# Multiple patterns for same project
+python src/server.py --project "your-project:demo_*,analytics_*" --billing-project "your-project"
+
+# Enterprise usage with multiple projects and settings
+python src/server.py \
+ --project "analytics-prod:user_*,session_*" \
+ --project "logs-prod:application_*,system_*" \
+ --project "ml-dev:training_*,models_*" \
+ --billing-project "my-billing-project" \
+ --log-level "INFO" \
+ --compact-format "true" \
+ --timeout 300 \
+ --max-limit 50000
-Command-Line Options¶
+Complete CLI Arguments Reference¶
+Core Arguments¶
-- Project patterns:
project_id:dataset_pattern format
---billing-project: Project ID for billing (overrides environment variable)--project: Project access patterns (can be repeated). Format: project_id:dataset_pattern[:table_pattern]--billing-project: BigQuery billing project (overrides environment variable)--config: Path to config file (fallback when no projects specified)--location: BigQuery location (default: EU)--config: Path to config file (fallback only)--version: Show version information
-Configuration File (Deprecated)¶
-For backward compatibility, the server still supports YAML configuration files:
-cp config/config.yaml.example config/config.yaml
+Logging Options¶
+
+--log-level: Logging level (DEBUG, INFO, WARNING, ERROR). Default: INFO--log-queries: Log queries for audit purposes (true/false). Default: true--log-results: Log query results - be careful with sensitive data (true/false). Default: false
+Performance & Limits¶
+
+--timeout: Query timeout in seconds. Default: 60--max-limit: Maximum rows that can be requested. Default: 10000--max-bytes-processed: Maximum bytes processed for cost control. Default: 1073741824 (1GB)
+Security Options¶
+
+--select-only: Allow only SELECT statements (true/false). Default: true--require-explicit-limits: Require explicit LIMIT clause in SELECT queries (true/false). Default: false--banned-keywords: Comma-separated list of banned SQL keywords. Default: CREATE,DELETE,DROP,TRUNCATE,ALTER,INSERT,UPDATE
+Formatting¶
+
+--compact-format: Use compact response format (true/false). Default: false
+Configuration Precedence¶
+CLI Arguments > Config File > Environment Variables > Hardcoded Defaults
+This means CLI arguments always take precedence over config file settings, which take precedence over environment variables.
+Project Pattern Examples¶
+Simple Patterns¶
+# All datasets in a project
+--project "my-project:*"
+
+# Specific dataset patterns
+--project "my-project:analytics_*,logs_*"
+
+# Multiple projects
+--project "project1:*" --project "project2:staging_*"
+
+Enterprise Patterns¶
+# Complex multi-project setup
+--project "analytics-prod:user_*,session_*,conversion_*" \
+--project "logs-prod:application_*,system_*,error_*" \
+--project "ml-dev:training_*,features_*,models_*" \
+--project "warehouse:daily_*,weekly_*,monthly_*"
+
+Table Patterns (Future Enhancement)¶
+# Table patterns will be supported in future versions
+--project "project:dataset:table_pattern"
+
+Configuration File (Fallback)¶
+For backward compatibility and complex setups, the server still supports YAML configuration files:
+# Use config file when no --project arguments provided
+python src/server.py --config config/config.yaml
Complete Configuration Reference¶
Server Section¶
server:
name: "BigQuery MCP Server"
- version: "1.1.0"
+ version: "1.1.1"
-
-name: Display name for the server (used in logs and responses)version: Server version (should match the installed version)
BigQuery Section¶
bigquery:
+ # Required: Project for billing
billing_project: "your-billing-project"
- location: "US"
- service_account_path: ""
+
+ # Optional: BigQuery location/region
+ location: "EU"
+
+ # Optional: Service account path
+ service_account_path: "/path/to/service-account.json"
-
-billing_project: Project ID used for billing BigQuery queries (required)location: BigQuery location/region (default: "US")- Common values: "US", "EU", "asia-northeast1"
-service_account_path: Path to service account JSON file (optional)- If not provided, uses Application Default Credentials
-
Projects Section¶
-Define which BigQuery projects and datasets the server can access:
projects:
- project_id: "analytics-prod"
project_name: "Analytics Production"
@@ -1215,35 +1231,25 @@ Projects Section - project_id: "raw-data-lake"
project_name: "Raw Data Lake"
description: "Raw data ingestion layer"
- datasets: ["*"] # Allow all datasets
-
- - project_id: "ml-features"
- project_name: "ML Feature Store"
- description: "Machine learning features and training data"
- datasets: ["features_*", "training_*", "models_*"]
+ datasets: ["*"] # All datasets
-Dataset Patterns:
-- "*" - Allow all datasets in the project
-- "dataset_name" - Allow specific dataset
-- "prefix_*" - Allow datasets starting with prefix
-- ["dataset1", "dataset2"] - Allow multiple specific datasets
Limits Section¶
-Control query execution and resource usage:
limits:
+ # Default rows returned if not specified
default_limit: 20
- max_limit: 10000
+
+ # Maximum query execution time in seconds
max_query_timeout: 60
+
+ # Maximum rows that can be requested
+ max_limit: 10000
+
+ # Maximum bytes processed (cost control)
max_bytes_processed: 1073741824 # 1GB
-
-default_limit: Default number of rows returned (if not specified in query)max_limit: Maximum rows that can be requested in a single querymax_query_timeout: Maximum query execution time in secondsmax_bytes_processed: Maximum bytes processed per query (for cost control)
Security Section¶
-SQL safety and validation settings:
security:
+ # Banned SQL keywords
banned_sql_keywords:
- "CREATE"
- "DELETE"
@@ -1255,253 +1261,86 @@ Security Section - "GRANT"
- "REVOKE"
- "MERGE"
- - "CALL"
- - "EXECUTE"
- - "SCRIPT"
- require_explicit_limits: false
+
+ # Allow only SELECT statements
select_only: true
+
+ # Require explicit LIMIT clause
+ require_explicit_limits: false
-
-banned_sql_keywords: SQL keywords that will cause query rejectionrequire_explicit_limits: If true, all SELECT queries must include LIMIT clauseselect_only: If true, only SELECT statements and CTEs (WITH clauses) are allowed (recommended)
Formatting Section¶
-Response formatting options:
formatting:
- compact_mode: false
+ # Use compact format by default
+ compact_format: false
-
-compact_mode: Use compact response format (reduces token usage)
-Note: Field descriptions are always included in schema responses.
Logging Section¶
-Logging and audit configuration:
logging:
+ # Log queries for audit purposes
log_queries: true
+
+ # Log query results (be careful with sensitive data)
log_results: false
- max_query_log_length: 1000
+Environment Variables¶
+The following environment variables are supported:
-log_queries: Log SQL queries for audit purposeslog_results: Log query results (be careful with sensitive data)max_query_log_length: Maximum length of logged SQL queriesBIGQUERY_BILLING_PROJECT: Default billing projectGOOGLE_APPLICATION_CREDENTIALS: Path to service account JSONBIGQUERY_LOCATION: BigQuery locationLOG_LEVEL: Logging levelCOMPACT_FORMAT: Use compact format (true/false)LOG_QUERIES: Log queries (true/false)LOG_RESULTS: Log results (true/false)
-Environment Variables¶
-Environment variables override YAML configuration values:
-Core Settings¶
-# BigQuery configuration
-export BIGQUERY_BILLING_PROJECT=your-billing-project
-export BIGQUERY_LOCATION=US
-export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
-
-# Server behavior
-export LOG_LEVEL=INFO # DEBUG, INFO, WARNING, ERROR
-export COMPACT_FORMAT=true # Override formatting.compact_mode
-
-Available Environment Variables¶
-
-
-
-Variable
-Description
-Default
-
-
-
-
-BIGQUERY_BILLING_PROJECTOverride billing project
-From config
-
-
-BIGQUERY_LOCATIONOverride BigQuery location
-From config
-
-
-GOOGLE_APPLICATION_CREDENTIALSPath to service account JSON
-Auto-detect
-
-
-LOG_LEVELLogging verbosity
-INFO
-
-
-COMPACT_FORMATEnable compact mode
-From config
-
-
-
-Configuration Examples¶
-Development Setup¶
-For local development with personal projects:
-server:
- name: "BigQuery MCP Development"
- version: "1.1.0"
-
-bigquery:
- billing_project: "my-dev-project"
- location: "US"
-
-projects:
- - project_id: "my-dev-project"
- project_name: "Development Project"
- description: "Personal development data"
- datasets: ["*"]
-
-limits:
- default_limit: 10
- max_limit: 1000
- max_query_timeout: 30
- max_bytes_processed: 104857600 # 100MB
-
-formatting:
- compact_mode: true
-
-logging:
- log_queries: true
- log_results: true # OK for development
-
-Production Setup¶
-For production deployments with multiple projects:
-server:
- name: "BigQuery MCP Production"
- version: "1.1.0"
-
-bigquery:
- billing_project: "analytics-billing"
- location: "US"
- service_account_path: "/app/credentials/service-account.json"
-
-projects:
- - project_id: "data-warehouse"
- project_name: "Data Warehouse"
- description: "Production data warehouse"
- datasets: ["prod_*", "reporting_*"]
-
- - project_id: "analytics-sandbox"
- project_name: "Analytics Sandbox"
- description: "Analytics team sandbox"
- datasets: ["sandbox_*", "experiments_*"]
-
-limits:
- default_limit: 100
- max_limit: 10000
- max_query_timeout: 300 # 5 minutes
- max_bytes_processed: 10737418240 # 10GB
-
-security:
- banned_sql_keywords:
- - "CREATE"
- - "DELETE"
- - "DROP"
- - "INSERT"
- - "UPDATE"
- - "TRUNCATE"
- require_explicit_limits: true
- select_only: true
-
-formatting:
- compact_mode: true
-
-logging:
- log_queries: true # Query length limited to 500 chars
-
-Multi-Region Setup¶
-For organizations with data in multiple regions:
-bigquery:
- billing_project: "global-analytics"
- location: "US" # Default location
-
-projects:
- - project_id: "us-data-warehouse"
- project_name: "US Data Warehouse"
- description: "US region data"
- datasets: ["us_*"]
-
- - project_id: "eu-data-warehouse"
- project_name: "EU Data Warehouse"
- description: "EU region data"
- datasets: ["eu_*"]
-
- - project_id: "asia-data-warehouse"
- project_name: "Asia Data Warehouse"
- description: "Asia region data"
- datasets: ["asia_*"]
-
Docker Configuration¶
-Environment File¶
-Create a .env file for Docker deployments:
-# .env file
-BIGQUERY_BILLING_PROJECT=your-billing-project
-BIGQUERY_LOCATION=US
-LOG_LEVEL=INFO
-COMPACT_FORMAT=true
+CLI-First Docker (Recommended)¶
+docker run --rm -i \
+ --volume ~/.config/gcloud:/home/mcpuser/.config/gcloud:ro \
+ --volume ./logs:/app/logs \
+ bigquery-mcp:latest \
+ python src/server.py \
+ --project "your-project:*" \
+ --billing-project "your-project" \
+ --log-level "INFO"
Docker Compose¶
-Use environment variables in docker-compose.yml:
services:
bigquery-mcp:
build: .
- environment:
- - BIGQUERY_BILLING_PROJECT=${BIGQUERY_BILLING_PROJECT}
- - BIGQUERY_LOCATION=${BIGQUERY_LOCATION:-US}
- - LOG_LEVEL=${LOG_LEVEL:-INFO}
- - COMPACT_FORMAT=${COMPACT_FORMAT:-true}
+ image: bigquery-mcp:latest
+ container_name: bigquery-mcp
+ command: [
+ "python", "src/server.py",
+ "--project", "your-project:*",
+ "--billing-project", "your-project",
+ "--log-level", "INFO"
+ ]
volumes:
- - ./config:/app/config:ro
+ - ./logs:/app/logs
- ~/.config/gcloud:/home/mcpuser/.config/gcloud:ro
+ stdin_open: true
+ tty: false
-Validation¶
-Configuration Validation¶
-The server validates configuration on startup:
-
-- Required fields must be present
-- Project IDs must be valid
-- Numeric limits must be positive
-- Dataset patterns must be valid
-
-Testing Configuration¶
-Test your configuration:
-# Test configuration loading
-python -c "from src.config import load_config; print('Config OK')"
-
-# Test BigQuery access
-python -c "from src.client import BigQueryClient; client = BigQueryClient('config/config.yaml'); print('BigQuery OK')"
-
+Migration from v1.1.0¶
+If you're upgrading from v1.1.0:
+
+- Update CLI usage: Change from
python src/server.py project:pattern to python src/server.py --project "project:pattern"
+- Add new arguments: Take advantage of new CLI options like
--log-level, --timeout, etc.
+- Update Docker configs: Use new CLI-first Docker approach
+- Check config files: Ensure
log_results attribute is present in config files
+
Troubleshooting¶
-Common Configuration Issues¶
-Invalid Project Access¶
-Error: Project 'project-id' not found or access denied
-
-Solution: Check project ID spelling and IAM permissions
-Dataset Pattern Errors¶
-Error: No datasets match pattern 'invalid_*'
-
-Solution: Verify dataset names and patterns in your BigQuery project
-Resource Limit Errors¶
-Error: Query exceeded maximum bytes processed
-
-Solution: Increase max_bytes_processed or optimize your query
-Configuration Best Practices¶
+Common Issues¶
-- Use environment variables for sensitive values (project IDs, paths)
-- Set appropriate limits based on your use case and costs
-- Use specific dataset patterns rather than
"*" where possible
-- Enable query logging for audit purposes
-- Disable result logging in production for security
-- Test configuration changes in development first
+- Missing log_results attribute: Update config files to include
log_results: false in the logging section
+- CLI argument parsing: Ensure you're using
--project flag instead of positional arguments
+- Docker issues: Use absolute paths for volume mounts and ensure gcloud auth is set up
-Next Steps¶
-
-- Tools Reference - Learn about available MCP tools
-- Development Guide - Set up development environment
-- Installation Guide - Deployment options
-
+Debug Mode¶
+# Enable debug logging
+python src/server.py --project "your-project:*" --billing-project "your-project" --log-level DEBUG --log-queries true --log-results true
+
+This will provide detailed information about configuration loading, query execution, and error handling.
diff --git a/site/installation/index.html b/site/installation/index.html
index 4d08377..529099a 100644
--- a/site/installation/index.html
+++ b/site/installation/index.html
@@ -244,6 +244,25 @@
+
+
+
+
+
+
+
+ AI Setup Assistant
+
+
+
+
+
+
+
+
+
+
+
@@ -705,6 +724,28 @@
+
+
+
+
+
+
+ AI Setup Assistant
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@@ -1287,13 +1328,13 @@ Getting HelpCheck the logs/ directory in your project root for detailed error messages
Verify your configuration matches the examples
Test BigQuery access independently with gcloud or bq commands
-Review the development guide for debugging tips
+Review the setup guide for debugging tips
Next Steps¶
- Configuration Guide - Detailed configuration options
- Tools Reference - Available MCP tools and examples
-- Integration Examples - Claude Desktop and Cursor setup
+- Client Setup Guide - Claude Desktop and Cursor setup
diff --git a/site/sitemap.xml b/site/sitemap.xml
index ead519a..bf31f6f 100644
--- a/site/sitemap.xml
+++ b/site/sitemap.xml
@@ -1,19 +1,23 @@
+
+ https://github.com/aicayzer/bigquery-mcp/ai-setup/
+ 2025-07-17
+
https://github.com/aicayzer/bigquery-mcp/configuration/
- 2025-07-16
+ 2025-07-17
https://github.com/aicayzer/bigquery-mcp/installation/
- 2025-07-16
+ 2025-07-17
https://github.com/aicayzer/bigquery-mcp/setup/
- 2025-07-16
+ 2025-07-17
https://github.com/aicayzer/bigquery-mcp/tools/
- 2025-07-16
+ 2025-07-17
\ No newline at end of file
diff --git a/site/sitemap.xml.gz b/site/sitemap.xml.gz
index 8e01285..012cbf8 100644
Binary files a/site/sitemap.xml.gz and b/site/sitemap.xml.gz differ
diff --git a/site/tools/index.html b/site/tools/index.html
index de811b1..d4f50a5 100644
--- a/site/tools/index.html
+++ b/site/tools/index.html
@@ -13,7 +13,7 @@
-
+
@@ -243,6 +243,25 @@
+
+
+
+
+
+
+
+
+ AI Setup Assistant
+
+
+
+
+
+
+
+
+
+
@@ -372,6 +391,28 @@
+
+
+
+
+
+
+
+
+ AI Setup Assistant
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/src/config.py b/src/config.py
index 0e97372..b84f22f 100644
--- a/src/config.py
+++ b/src/config.py
@@ -44,7 +44,7 @@ class LimitsConfig:
"""Query and response limits configuration."""
default_limit: int = 20
- max_query_timeout: int = 60
+ max_query_timeout: int = 20
max_limit: int = 10000
max_bytes_processed: int = 1073741824 # 1GB
@@ -65,6 +65,7 @@ def __init__(self, config_path: Optional[str] = None):
config_path = self._find_config_file()
self.config_path = config_path
+ self._from_cli_args = False # Indicate this instance was created from config file
self._raw_config = self._load_yaml(config_path)
self._parse_config()
self._apply_env_overrides()
@@ -75,15 +76,26 @@ def from_cli_args(
project_patterns: Dict[str, List[str]],
billing_project: Optional[str] = None,
location: str = "EU",
+ log_level: str = "INFO",
+ log_queries: bool = True,
+ log_results: bool = False,
+ timeout: int = 20,
+ max_limit: int = 10000,
+ max_bytes_processed: int = 1073741824,
+ compact_format: bool = False,
+ select_only: bool = True,
+ require_explicit_limits: bool = False,
+ banned_keywords: str = "CREATE,DELETE,DROP,TRUNCATE,ALTER,INSERT,UPDATE",
):
"""Create configuration from command-line arguments."""
instance = cls.__new__(cls)
instance.config_path = None
instance._raw_config = {}
+ instance._from_cli_args = True # Indicate that this instance was created from CLI args
# Server info
instance.server_name = "BigQuery MCP Server"
- instance.server_version = "1.1.0"
+ instance.server_version = "1.1.1"
# BigQuery settings
instance.billing_project = billing_project or os.getenv("BIGQUERY_BILLING_PROJECT", "")
@@ -102,39 +114,30 @@ def from_cli_args(
)
)
- # Default security settings
+ # Security settings from CLI args
+ banned_keywords_list = [kw.strip().upper() for kw in banned_keywords.split(",")]
instance.security = SecurityConfig(
- banned_sql_keywords=[
- "CREATE",
- "DELETE",
- "DROP",
- "TRUNCATE",
- "INSERT",
- "UPDATE",
- "ALTER",
- "GRANT",
- "REVOKE",
- "MERGE",
- ],
- select_only=True,
- require_explicit_limits=False,
+ banned_sql_keywords=banned_keywords_list,
+ select_only=select_only,
+ require_explicit_limits=require_explicit_limits,
)
- # Default limits
+ # Limits from CLI args
instance.limits = LimitsConfig(
- default_limit=20,
- max_query_timeout=60,
- max_limit=10000,
- max_bytes_processed=1073741824, # 1GB
+ default_limit=20, # Keep default for backward compatibility
+ max_query_timeout=timeout,
+ max_limit=max_limit,
+ max_bytes_processed=max_bytes_processed,
)
- # Default formatting
+ # Formatting from CLI args
instance.formatting = FormattingConfig(
- compact_format=os.getenv("COMPACT_FORMAT", "").lower() == "true",
+ compact_format=compact_format,
)
- # Default logging
- instance.log_queries = True
+ # Logging from CLI args
+ instance.log_queries = log_queries
+ instance.log_results = log_results
# Apply environment overrides
instance._apply_env_overrides()
@@ -252,30 +255,69 @@ def _parse_config(self) -> None:
# Logging
logging_dict = self._raw_config.get("logging", {})
self.log_queries = logging_dict.get("log_queries", True)
+ self.log_results = logging_dict.get("log_results", False)
def _apply_env_overrides(self) -> None:
- """Apply environment variable overrides."""
+ """Apply environment variable overrides.
+
+ This method applies environment variable overrides to the configuration.
+ Note: CLI arguments (when using from_cli_args) take precedence over both
+ config file and environment variables.
+ """
# Billing project
if env_billing := os.getenv("BIGQUERY_BILLING_PROJECT"):
- self.billing_project = env_billing
- logger.info(f"Overriding billing project from env: {env_billing}")
+ if not self.billing_project: # Only override if not already set
+ self.billing_project = env_billing
+ logger.info(f"Using billing project from env: {env_billing}")
# Service account
if env_creds := os.getenv("GOOGLE_APPLICATION_CREDENTIALS"):
- self.service_account_path = env_creds
- logger.info("Using service account from GOOGLE_APPLICATION_CREDENTIALS")
+ if not self.service_account_path: # Only override if not already set
+ self.service_account_path = env_creds
+ logger.info("Using service account from GOOGLE_APPLICATION_CREDENTIALS")
# Location
if env_location := os.getenv("BIGQUERY_LOCATION"):
- self.location = env_location
- logger.info(f"Overriding location from env: {env_location}")
+ if self.location == "EU": # Only override if using default
+ self.location = env_location
+ logger.info(f"Using location from env: {env_location}")
- # Compact format
+ # Compact format (only if not explicitly set)
if env_compact := os.getenv("COMPACT_FORMAT"):
- self.formatting.compact_format = env_compact.lower() == "true"
- logger.info(f"Overriding compact format from env: {self.formatting.compact_format}")
+ # For CLI args, this would have been set explicitly, so only override for config file usage
+ if hasattr(self, "_from_cli_args") and not self._from_cli_args:
+ self.formatting.compact_format = env_compact.lower() == "true"
+ logger.info(f"Using compact format from env: {self.formatting.compact_format}")
+
+ # Log level (handled by logging setup in server.py)
+ # Additional environment variables for comprehensive support
+ if env_log_queries := os.getenv("LOG_QUERIES"):
+ if not hasattr(self, "_from_cli_args") or not self._from_cli_args:
+ self.log_queries = env_log_queries.lower() == "true"
+ logger.info(f"Using log_queries from env: {self.log_queries}")
+
+ if env_log_results := os.getenv("LOG_RESULTS"):
+ if not hasattr(self, "_from_cli_args") or not self._from_cli_args:
+ self.log_results = env_log_results.lower() == "true"
+ logger.info(f"Using log_results from env: {self.log_results}")
+
+ def log_configuration_source(self) -> None:
+ """Log the source of configuration values for debugging."""
+ logger.info("Configuration precedence: CLI > Config File > Environment > Defaults")
+
+ if hasattr(self, "_from_cli_args") and self._from_cli_args:
+ logger.info("Configuration source: CLI arguments (highest precedence)")
+ elif self.config_path:
+ logger.info(
+ f"Configuration source: Config file ({self.config_path}) + Environment overrides"
+ )
+ else:
+ logger.info("Configuration source: Environment variables + Defaults")
- # Log level is handled by logging setup, not here
+ logger.info(
+ f"Final configuration: billing_project={self.billing_project}, "
+ f"location={self.location}, compact_format={self.formatting.compact_format}"
+ )
def get_project(self, project_id: str) -> Optional[ProjectConfig]:
"""Get project configuration by ID."""
diff --git a/src/server.py b/src/server.py
index 5b6b612..3f2405a 100644
--- a/src/server.py
+++ b/src/server.py
@@ -64,23 +64,31 @@ def parse_arguments():
epilog="""
Examples:
# Single project with all datasets
- python src/server.py sandbox-dev:*
+ python src/server.py --project "sandbox-dev:*"
# Multiple projects with specific patterns
- python src/server.py sandbox-dev:dev_* sandbox-main:main_*
+ python src/server.py --project "sandbox-dev:dev_*" --project "sandbox-main:main_*"
- # Multiple patterns for same project
- python src/server.py cayzer-xyz:demo_* cayzer-xyz:analytics_*
+ # Multiple dataset patterns for same project
+ python src/server.py --project "cayzer-xyz:demo_*,analytics_*"
- # Fallback to config file if no arguments
- python src/server.py
+ # Complex enterprise usage with multiple projects and patterns
+ python src/server.py \\
+ --project "analytics-prod:user_*,session_*" \\
+ --project "logs-prod:application_*,system_*" \\
+ --billing-project "my-project" \\
+ --log-level INFO
+
+ # Fallback to config file if no projects specified
+ python src/server.py --config config.yaml
""",
)
parser.add_argument(
- "projects",
- nargs="*",
- help="Project access patterns in format 'project_id:dataset_pattern' (e.g., 'sandbox-dev:dev_*')",
+ "--project",
+ action="append",
+ dest="projects",
+ help="Project access patterns in format 'project_id:dataset_pattern[:table_pattern]' (e.g., 'sandbox-dev:dev_*'). Can be used multiple times.",
)
parser.add_argument(
@@ -99,37 +107,149 @@ def parse_arguments():
help="BigQuery location (default: EU)",
)
+ # Logging configuration
+ parser.add_argument(
+ "--log-level",
+ choices=["DEBUG", "INFO", "WARNING", "ERROR"],
+ default="INFO",
+ help="Logging level (default: INFO)",
+ )
+
+ parser.add_argument(
+ "--log-queries",
+ type=lambda x: x.lower() == "true",
+ default=True,
+ help="Log queries for audit purposes (default: true)",
+ )
+
+ parser.add_argument(
+ "--log-results",
+ type=lambda x: x.lower() == "true",
+ default=False,
+ help="Log query results - be careful with sensitive data (default: false)",
+ )
+
+ # Query execution limits
+ parser.add_argument(
+ "--timeout",
+ type=int,
+ default=20,
+ help="Query timeout in seconds (default: 20)",
+ )
+
+ parser.add_argument(
+ "--max-limit",
+ type=int,
+ default=10000,
+ help="Maximum rows that can be requested (default: 10000)",
+ )
+
+ parser.add_argument(
+ "--max-bytes-processed",
+ type=int,
+ default=1073741824,
+ help="Maximum bytes that will be processed for cost control (default: 1073741824 = 1GB)",
+ )
+
+ # Response formatting
+ parser.add_argument(
+ "--compact-format",
+ type=lambda x: x.lower() == "true",
+ default=False,
+ help="Use compact response format (default: false)",
+ )
+
+ # Security settings
+ parser.add_argument(
+ "--select-only",
+ type=lambda x: x.lower() == "true",
+ default=True,
+ help="Allow only SELECT statements (default: true)",
+ )
+
+ parser.add_argument(
+ "--require-explicit-limits",
+ type=lambda x: x.lower() == "true",
+ default=False,
+ help="Require explicit LIMIT clause in SELECT queries (default: false)",
+ )
+
+ parser.add_argument(
+ "--banned-keywords",
+ default="CREATE,DELETE,DROP,TRUNCATE,ALTER,INSERT,UPDATE",
+ help="Comma-separated list of banned SQL keywords (default: CREATE,DELETE,DROP,TRUNCATE,ALTER,INSERT,UPDATE)",
+ )
+
parser.add_argument(
"--version",
action="version",
- version="BigQuery MCP Server 1.1.0",
+ version="BigQuery MCP Server 1.1.1",
)
return parser.parse_args()
def parse_project_patterns(project_args):
- """Parse project:dataset patterns from command line arguments."""
+ """Parse project:dataset:table patterns from command line arguments.
+
+ Supports formats:
+ - project_id:* (all datasets)
+ - project_id:dataset_pattern (specific dataset patterns)
+ - project_id:dataset_pattern:table_pattern (dataset and table patterns)
+ - project_id:dataset1,dataset2:table1,table2 (multiple patterns)
+
+ Args:
+ project_args: List of project pattern strings
+
+ Returns:
+ Dict mapping project_id to list of dataset patterns
+ """
projects = {}
for pattern in project_args:
if ":" not in pattern:
logger.error(
- f"Invalid project pattern '{pattern}'. Expected format: 'project_id:dataset_pattern'"
+ f"Invalid project pattern '{pattern}'. Expected format: 'project_id:dataset_pattern[:table_pattern]'"
+ )
+ sys.exit(1)
+
+ # Split into components
+ parts = pattern.split(":", 2) # Max 3 parts: project:dataset:table
+
+ if len(parts) < 2:
+ logger.error(
+ f"Invalid project pattern '{pattern}'. Expected format: 'project_id:dataset_pattern[:table_pattern]'"
)
sys.exit(1)
- project_id, dataset_pattern = pattern.split(":", 1)
+ project_id = parts[0].strip()
+ dataset_part = parts[1].strip()
+ table_part = parts[2].strip() if len(parts) > 2 else None
- if not project_id or not dataset_pattern:
+ if not project_id or not dataset_part:
logger.error(
f"Invalid project pattern '{pattern}'. Both project_id and dataset_pattern are required"
)
sys.exit(1)
+ # Parse dataset patterns (comma-separated)
+ dataset_patterns = [p.strip() for p in dataset_part.split(",") if p.strip()]
+
+ if not dataset_patterns:
+ logger.error(f"Invalid project pattern '{pattern}'. Dataset pattern cannot be empty")
+ sys.exit(1)
+
+ # For now, we'll store dataset patterns as-is
+ # Table patterns will be handled in future enhancement
+ if table_part:
+ logger.warning(
+ f"Table patterns not yet fully implemented. Ignoring table pattern '{table_part}' for project '{project_id}'"
+ )
+
+ # Add to projects dict
if project_id not in projects:
projects[project_id] = []
- projects[project_id].append(dataset_pattern)
+ projects[project_id].extend(dataset_patterns)
return projects
@@ -153,15 +273,24 @@ def initialize_server():
project_patterns=project_patterns,
billing_project=args.billing_project,
location=args.location,
+ log_level=args.log_level,
+ log_queries=args.log_queries,
+ log_results=args.log_results,
+ timeout=args.timeout,
+ max_limit=args.max_limit,
+ max_bytes_processed=args.max_bytes_processed,
+ compact_format=args.compact_format,
+ select_only=args.select_only,
+ require_explicit_limits=args.require_explicit_limits,
+ banned_keywords=args.banned_keywords,
)
else:
- # Fall back to config file with deprecation warning
- logger.warning(
- "DEPRECATED: Using config file. Please migrate to command-line arguments."
- )
- logger.warning("Example: python src/server.py sandbox-dev:dev_* sandbox-main:main_*")
+ # Use config file
config = get_config(args.config)
+ # Log configuration source for debugging
+ config.log_configuration_source()
+
# Initialize BigQuery client
logger.info("Initializing BigQuery client...")
bq_client = BigQueryClient(config)
diff --git a/src/tools/analysis.py b/src/tools/analysis.py
index 34aa2dc..498bf53 100644
--- a/src/tools/analysis.py
+++ b/src/tools/analysis.py
@@ -534,10 +534,10 @@ def analyze_columns(
analysis_query,
job_config=QueryJobConfig(
use_query_cache=True,
- job_timeout_ms=60000, # 60 second timeout
+ job_timeout_ms=20000, # 20 second timeout
),
)
- query_results = list(query_job.result(timeout=60))
+ query_results = list(query_job.result(timeout=20))
if not query_results:
logger.warning(f"No results returned for column {col_name}")
diff --git a/src/tools/execution.py b/src/tools/execution.py
index b11cdd1..5e168bd 100644
--- a/src/tools/execution.py
+++ b/src/tools/execution.py
@@ -10,7 +10,11 @@
from google.cloud.bigquery import QueryJobConfig, ScalarQueryParameter
-from utils.errors import QueryExecutionError, SecurityError, SQLValidationError
+from utils.errors import (
+ SecurityError,
+ SQLValidationError,
+ create_ai_friendly_error,
+)
from utils.validation import SQLValidator
logger = logging.getLogger(__name__)
@@ -349,33 +353,19 @@ def execute_query(
error_str = str(e)
logger.error(f"Query execution error: {error_str}")
- # Handle specific error types
- if "403" in error_str or "Permission denied" in error_str:
- raise QueryExecutionError(
- "Permission denied. Ensure you have bigquery.jobs.create permission "
- "and access to the referenced tables."
- )
- elif "404" in error_str or "Not found" in error_str:
- raise QueryExecutionError("Table not found. Check the table references in your query.")
- elif "Syntax error" in error_str or "syntax" in error_str.lower():
- raise QueryExecutionError(f"SQL syntax error: {error_str}")
- elif "Array cannot have a null element" in error_str:
- raise QueryExecutionError(
- "BigQuery array contains NULL values. Use COALESCE() or filter NULL values: "
- f"{error_str}"
- )
- elif isinstance(e, TimeoutError) or "timeout" in error_str.lower():
- raise QueryExecutionError(
- f"Query timeout after {timeout} seconds. "
- f"Try adding LIMIT clause, filtering data, or increase timeout parameter."
- )
- elif "quota" in error_str.lower() or "rate limit" in error_str.lower():
- raise QueryExecutionError(
- f"BigQuery quota exceeded. Try again later or reduce query complexity: {error_str}"
- )
- else:
- # For any other error, preserve the original message
- raise QueryExecutionError(f"Query execution failed: {error_str}")
+ # Create AI-friendly error with context
+ context = {
+ "query_length": len(query),
+ "query_complexity": _estimate_query_complexity(query),
+ "timeout_used": timeout,
+ "limit_used": limit,
+ "dry_run": dry_run,
+ }
+
+ # Use the new AI-friendly error creation system
+ ai_friendly_error = create_ai_friendly_error(e, context)
+ logger.error(f"AI-friendly error: {ai_friendly_error.to_dict()}")
+ raise ai_friendly_error
def _estimate_query_complexity(query: str) -> str:
diff --git a/src/utils/errors.py b/src/utils/errors.py
index 12003b6..b587b78 100644
--- a/src/utils/errors.py
+++ b/src/utils/errors.py
@@ -4,70 +4,194 @@
class BigQueryMCPError(Exception):
"""Base exception for all BigQuery MCP errors."""
- pass
+ def __init__(
+ self,
+ message: str,
+ error_source: str = "MCP_SERVER",
+ error_code: str = None,
+ suggested_action: str = None,
+ context: dict = None,
+ ):
+ """Initialize BigQuery MCP error with enhanced information.
+
+ Args:
+ message: Error message
+ error_source: Source of error (BIGQUERY_API, MCP_SERVER, USER_QUERY, CONFIGURATION)
+ error_code: Specific error code for categorization
+ suggested_action: Actionable suggestion for resolution
+ context: Additional context information
+ """
+ super().__init__(message)
+ self.error_source = error_source
+ self.error_code = error_code
+ self.suggested_action = suggested_action
+ self.context = context or {}
+
+ def to_dict(self) -> dict:
+ """Convert error to dictionary for structured logging."""
+ return {
+ "error_type": self.__class__.__name__,
+ "message": str(self),
+ "error_source": self.error_source,
+ "error_code": self.error_code,
+ "suggested_action": self.suggested_action,
+ "context": self.context,
+ }
class ConfigurationError(BigQueryMCPError):
"""Raised when configuration is invalid or missing."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(message, error_source="CONFIGURATION", **kwargs)
class AuthenticationError(BigQueryMCPError):
"""Raised when authentication fails."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(message, error_source="BIGQUERY_API", error_code="AUTH_FAILED", **kwargs)
class ProjectAccessError(BigQueryMCPError):
"""Raised when attempting to access a disallowed project."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(
+ message, error_source="MCP_SERVER", error_code="PROJECT_ACCESS_DENIED", **kwargs
+ )
class DatasetAccessError(BigQueryMCPError):
"""Raised when attempting to access a disallowed dataset."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(
+ message, error_source="MCP_SERVER", error_code="DATASET_ACCESS_DENIED", **kwargs
+ )
class TableNotFoundError(BigQueryMCPError):
"""Raised when a requested table doesn't exist."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(
+ message, error_source="BIGQUERY_API", error_code="TABLE_NOT_FOUND", **kwargs
+ )
class InvalidTablePathError(BigQueryMCPError):
"""Raised when table path format is invalid."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(
+ message, error_source="USER_QUERY", error_code="INVALID_TABLE_PATH", **kwargs
+ )
class SQLValidationError(BigQueryMCPError):
"""Raised when SQL query contains forbidden operations."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(
+ message, error_source="MCP_SERVER", error_code="SQL_VALIDATION_FAILED", **kwargs
+ )
class SecurityError(BigQueryMCPError):
"""Raised when a security policy is violated."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(
+ message, error_source="MCP_SERVER", error_code="SECURITY_VIOLATION", **kwargs
+ )
class QueryExecutionError(BigQueryMCPError):
"""Raised when query execution fails."""
- pass
+ def __init__(self, message: str, **kwargs):
+ # Extract error_code from kwargs if provided, otherwise use default
+ error_code = kwargs.pop("error_code", "QUERY_EXECUTION_FAILED")
+ super().__init__(message, error_source="BIGQUERY_API", error_code=error_code, **kwargs)
class QueryTimeoutError(QueryExecutionError):
"""Raised when query exceeds timeout limit."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(message, error_source="BIGQUERY_API", error_code="QUERY_TIMEOUT", **kwargs)
class ResourceLimitError(BigQueryMCPError):
"""Raised when a resource limit is exceeded."""
- pass
+ def __init__(self, message: str, **kwargs):
+ super().__init__(
+ message, error_source="BIGQUERY_API", error_code="RESOURCE_LIMIT_EXCEEDED", **kwargs
+ )
+
+
+def create_ai_friendly_error(original_error: Exception, context: dict = None) -> BigQueryMCPError:
+ """Create an AI-friendly error from any exception.
+
+ Args:
+ original_error: The original exception
+ context: Additional context information
+
+ Returns:
+ Enhanced BigQueryMCPError with AI-friendly information
+ """
+ error_str = str(original_error)
+ context = context or {}
+
+ # Classify error source and provide suggestions
+ if "403" in error_str or "Permission denied" in error_str:
+ return QueryExecutionError(
+ f"Permission denied: {error_str}",
+ error_code="PERMISSION_DENIED",
+ suggested_action="Ensure you have bigquery.jobs.create permission and access to the referenced tables. Check your service account permissions.",
+ context=context,
+ )
+ elif "404" in error_str or "Not found" in error_str:
+ return QueryExecutionError(
+ f"Resource not found: {error_str}",
+ error_code="RESOURCE_NOT_FOUND",
+ suggested_action="Verify the project ID, dataset name, and table name are correct. Check if the resource exists in BigQuery.",
+ context=context,
+ )
+ elif "Syntax error" in error_str or "syntax" in error_str.lower():
+ return BigQueryMCPError(
+ f"SQL syntax error: {error_str}",
+ error_source="USER_QUERY",
+ error_code="SYNTAX_ERROR",
+ suggested_action="Review your SQL syntax. Common issues: missing quotes, incorrect keywords, or malformed expressions.",
+ context=context,
+ )
+ elif "Array cannot have a null element" in error_str:
+ return BigQueryMCPError(
+ f"BigQuery array contains NULL values: {error_str}",
+ error_source="USER_QUERY",
+ error_code="ARRAY_NULL_ELEMENT",
+ suggested_action="Use COALESCE() to handle NULL values or filter them out before creating arrays.",
+ context=context,
+ )
+ elif "timeout" in error_str.lower():
+ return QueryTimeoutError(
+ f"Query timeout: {error_str}",
+ suggested_action="Try adding LIMIT clause, filtering data, or increase timeout parameter. Consider breaking complex queries into smaller parts.",
+ context=context,
+ )
+ elif "quota" in error_str.lower() or "rate limit" in error_str.lower():
+ return ResourceLimitError(
+ f"BigQuery quota exceeded: {error_str}",
+ suggested_action="Wait and retry later, or reduce query complexity. Check your BigQuery quotas and limits.",
+ context=context,
+ )
+ else:
+ return QueryExecutionError(
+ f"Query execution failed: {error_str}",
+ error_code="UNKNOWN_ERROR",
+ suggested_action="Check the error details and BigQuery documentation. If the issue persists, verify your query and data.",
+ context=context,
+ )
diff --git a/src/utils/formatting.py b/src/utils/formatting.py
index 5b7c03e..ac30b7c 100644
--- a/src/utils/formatting.py
+++ b/src/utils/formatting.py
@@ -94,22 +94,53 @@ def format_query_results(
def format_error(self, error: Exception) -> Dict[str, Any]:
"""
- Format error response.
+ Format error response with AI-friendly information.
Args:
error: Exception to format
Returns:
- Error response dictionary
+ Error response dictionary with enhanced information
"""
error_type = type(error).__name__
error_message = str(error)
- # Make error messages more user-friendly
- if "Not found" in error_message:
- error_message = self._improve_not_found_message(error_message)
+ # Check if this is an enhanced BigQueryMCPError
+ if hasattr(error, "error_source") and hasattr(error, "to_dict"):
+ # Use the enhanced error information
+ error_dict = error.to_dict()
- return {"status": "error", "error_type": error_type, "error": error_message}
+ response = {
+ "status": "error",
+ "error_type": error_dict["error_type"],
+ "error": error_dict["message"],
+ "error_source": error_dict["error_source"],
+ }
+
+ # Add optional fields if available
+ if error_dict.get("error_code"):
+ response["error_code"] = error_dict["error_code"]
+
+ if error_dict.get("suggested_action"):
+ response["suggested_action"] = error_dict["suggested_action"]
+
+ # Add context in non-compact mode
+ if not self.compact_mode and error_dict.get("context"):
+ response["context"] = error_dict["context"]
+
+ return response
+ else:
+ # Fallback to original format for non-enhanced errors
+ # Make error messages more user-friendly
+ if "Not found" in error_message:
+ error_message = self._improve_not_found_message(error_message)
+
+ return {
+ "status": "error",
+ "error_type": error_type,
+ "error": error_message,
+ "error_source": "UNKNOWN",
+ }
def format_table_info(self, table_info: Dict[str, Any]) -> Dict[str, Any]:
"""
diff --git a/tests/conftest.py b/tests/conftest.py
index 8acb8f8..142fb69 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -30,7 +30,7 @@ def sample_config():
"datasets": ["test_*", "sample_*"],
}
],
- "limits": {"default_limit": 20, "max_query_timeout": 60},
+ "limits": {"default_limit": 20, "max_query_timeout": 20},
"security": {"banned_sql_keywords": ["CREATE", "DROP", "DELETE"]},
}
diff --git a/tests/integration/test_bigquery_integration.py b/tests/integration/test_bigquery_integration.py
index 1a01428..ebc989a 100644
--- a/tests/integration/test_bigquery_integration.py
+++ b/tests/integration/test_bigquery_integration.py
@@ -37,7 +37,7 @@ def mock_config(self):
"datasets": ["*"],
},
],
- "limits": {"default_limit": 20, "max_query_timeout": 60},
+ "limits": {"default_limit": 20, "max_query_timeout": 20},
"security": {
"banned_sql_keywords": ["CREATE", "DROP"],
"select_only": True,
diff --git a/tests/unit/test_execution.py b/tests/unit/test_execution.py
index c1afaff..a02aacf 100644
--- a/tests/unit/test_execution.py
+++ b/tests/unit/test_execution.py
@@ -41,7 +41,7 @@ def mock_dependencies(self):
execution.config.security.require_explicit_limits = False
execution.config.limits.default_limit = 100
execution.config.limits.max_limit = 10000
- execution.config.limits.max_query_timeout = 60
+ execution.config.limits.max_query_timeout = 20
execution.config.limits.max_bytes_processed = 1073741824
yield execution
@@ -141,7 +141,7 @@ def mock_dependencies(self):
execution.config.security.require_explicit_limits = False
execution.config.limits.default_limit = 100
execution.config.limits.max_limit = 10000
- execution.config.limits.max_query_timeout = 60
+ execution.config.limits.max_query_timeout = 20
execution.config.limits.max_bytes_processed = 1073741824
execution.config.log_queries = True
execution.config.log_results = False
@@ -372,7 +372,7 @@ def test_execute_query_table_not_found(self, mock_dependencies):
with pytest.raises(QueryExecutionError) as exc_info:
execute_query("SELECT * FROM dataset.table")
- assert "Table not found" in str(exc_info.value)
+ assert "Resource not found" in str(exc_info.value)
class TestToolRegistration: