Refactor/docs

[thealphacubicle]

This pull request introduces several improvements and refactors to the project's developer experience, CI/CD workflows, and documentation. The most significant changes are the addition of pre-release validation to the release workflow, enhanced Terraform validation and linting in CI, improvements to the interactive configuration CLI (including automatic creation of the Terraform S3 state bucket), and updates to documentation and CLI usage to reflect these changes.

CI/CD Workflow Improvements:

• Added a dedicated validate job to the release workflow (.github/workflows/release.yml) that runs linting, security checks, and both Python and Go tests before building or deploying artifacts. The build and build-lambda-zip jobs now depend on this validation step. [1] [2]
• Split CI jobs for linting, security auditing, and testing, and improved dependency management and job dependencies in .github/workflows/ci.yml. Security audits now run in a dedicated job after linting. [1] [2] [3]
• Refactored the Terraform CI workflow to lint, format, and validate all Terraform directories, removing redundant caching and simplifying validation logic. TFLint and tfsec now run across all directories. (.github/workflows/infra.ymlL17-R69, Fa101fd6L45R48)

CLI and Configuration Enhancements:

• The opencontext configure CLI command now automatically checks for and creates the required Terraform S3 state bucket if it does not exist, enabling versioning and encryption by default. The state bucket name is configurable via a CLI option. [1] [2] [3]
• Improved the plugin configuration prompts in the CLI to be more robust and user-friendly. [1] [2] [3]

Developer Experience and Documentation:

• Updated documentation and example commands to use the new opencontext serve and opencontext test CLI commands instead of direct script calls, and expanded the CLI reference in the docs. [1] [2] [3] [4]
• Updated project layout and key files in documentation to reflect the migration of the dev server from scripts/local_server.py to cli/commands/serve.py. [1] [2]
• Updated the VSCode tasks to use the new CLI command for starting the local server.
• Clarified Lambda deployment size validation and other usage notes in documentation.

These changes streamline the development and deployment process, improve security and quality checks, and make the project easier to use and maintain.

[Copilot]

Pull request overview

This PR refactors the project’s developer experience by moving common workflows (serve/test/deploy/configure) into the opencontext Typer CLI, strengthening CI/CD validation (including pre-release checks), and updating Terraform + documentation to match the new workflow (including auto-creating the Terraform state S3 bucket).

Changes:

• Added a new opencontext serve command (aiohttp dev server) plus tests, and registered it in the CLI app.
• Enhanced opencontext configure to ensure the Terraform state S3 bucket exists (and added --state-bucket), with extensive unit tests.
• Refactored GitHub Actions workflows (CI, release, infra) and updated docs to replace removed scripts/* helpers with CLI commands.

Reviewed changes

Copilot reviewed 38 out of 41 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
uv.lock	Locks new/updated Python dependencies (notably boto3/botocore/jmespath/etc.).
tests/test_cli_serve.py	Adds unit tests for opencontext serve behavior and wiring.
tests/test_cli_configure.py	Adds unit tests for _ensure_state_bucket + --state-bucket behavior.
tests/conftest.py	Adds boto3/botocore import stubs for tests when deps aren’t installed.
terraform/README.md	Updates Terraform docs to use opencontext deploy/configure instead of scripts.
terraform/bootstrap/variables.tf	Removes DynamoDB lock table variable from bootstrap module.
terraform/bootstrap/README.md	Updates bootstrap docs to reflect S3-only backend bootstrap and CLI usage.
terraform/bootstrap/outputs.tf	Removes DynamoDB lock table output.
terraform/bootstrap/main.tf	Removes DynamoDB lock table resource; keeps S3 bucket setup.
terraform/aws/main.tf	Updates comments to reflect opencontext deploy packaging behavior.
scripts/test_streamable_http.sh	Removes legacy curl/jq test script (replaced by opencontext test).
scripts/setup-backend.sh	Removes legacy backend setup script (replaced by opencontext configure --state-bucket).
scripts/README.md	Removes scripts directory documentation (scripts removed).
scripts/local_server.py	Removes legacy local server script (replaced by opencontext serve).
scripts/deploy.sh	Removes legacy deploy script (replaced by opencontext deploy).
requirements.txt	Adds pre-commit (now installed into Lambda package during packaging workflows).
requirements-dev.txt	Expands dev deps for tests/lint/CLI and AWS SDK packages.
README.md	Updates quick usage to use opencontext serve and links new CLI docs.
pyproject.toml	Adds boto3/botocore/click to project deps (and updates dependency set).
local_server.py	Removes legacy root-level local server entrypoint.
docs/TESTING.md	Updates local testing instructions to use opencontext serve/test.
docs/QUICKSTART.md	Updates quickstart to use CLI serve flow and plugin ordering tweaks.
docs/GETTING_STARTED.md	Updates setup/deploy/testing guidance and expands CLI command list.
docs/FAQ.md	Updates operational guidance to use opencontext deploy/serve.
docs/DEPLOYMENT.md	Updates endpoint guidance and AWS services list to match current infra model.
docs/CUSTOM_PLUGINS.md	Updates deploy instructions + expands DataPlugin interface documentation.
docs/CLI.md	Adds a full CLI reference for lifecycle commands and flags.
docs/BUILT_IN_PLUGINS.md	Expands plugin docs (CKAN + adds ArcGIS Hub, updates Socrata section).
docs/ARCHITECTURE.md	Updates architecture and layout docs for new CLI + plugin set + infra notes.
custom_plugins/template/plugin_template.py	Fixes tool naming example to match plugin__tool convention.
core/validators.py	Updates validation error guidance to reference opencontext deploy.
config-example.yaml	Adds Socrata example configuration block.
cli/main.py	Registers the new serve command group.
cli/commands/serve.py	Implements opencontext serve local dev server command.
cli/commands/configure.py	Adds _ensure_state_bucket + --state-bucket; reorders init/workspace flow.
CLAUDE.md	Updates contributor docs to use the CLI instead of removed scripts.
.vscode/tasks.json	Updates VSCode task to start server via opencontext serve.
.gitignore	Ignores .claude/ directory.
.github/workflows/release.yml	Adds a pre-release validate job and gates build jobs on it.
.github/workflows/infra.yml	Refactors Terraform CI to validate/lint all terraform directories.
.github/workflows/ci.yml	Splits CI into lint, security audit, and tests with explicit dependencies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 39 out of 42 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The boto3/botocore stubs are installed based on sys.modules, which will be empty before imports even when the real packages are installed. This will unintentionally shadow real boto3/botocore in normal test environments. Consider detecting availability via importlib.util.find_spec(...) or a try: import boto3, botocore / except ImportError gate before installing stubs.

[Copilot]

yaml.safe_load(f) can return None for an empty config file; returning that as config will later break _run_server() / get_logging_config() which expect a dict. Consider defaulting to {} (and/or validating the loaded type) before returning.

Suggested change

	return yaml.safe_load(f), resolved
	loaded_config = yaml.safe_load(f)
	if loaded_config is None:
	loaded_config = {}
	elif not isinstance(loaded_config, dict):
	console.print(
	f"[red]Invalid config file:[/red] {resolved} "
	"(top-level YAML value must be a mapping)"
	)
	raise typer.Exit(1)
	return loaded_config, resolved

[Copilot]

The devcontainer postCreateCommand downloads and executes a remote install script (curl ... | bash). This is a supply-chain/security risk and can also break in offline or restricted environments. Consider removing it, pinning to a specific version/checksum, or making it an explicit, manual opt-in step documented in the README.

Suggested change

	"claude": "curl -fsSL https://claude.ai/install.sh | bash"
	"claude": "echo 'Optional manual step: install Claude by reviewing and running the vendor-provided installer yourself if needed.'"