diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..49ce9a7 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,20 @@ +--- +name: Bug Report +about: Something isn't working +title: "" +labels: bug +assignees: '' +--- + +**What happened?** + +**What did you expect?** + +**Steps to reproduce** +1. +2. + +**Logs or errors** (if any) + +``` +``` diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..59309ce --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature Request +about: Suggest an idea +title: "" +labels: enhancement +assignees: '' +--- + +## Summary + +A brief, one-line summary of the feature. + +## Motivation + +Why is this feature needed? What problem does it solve or what workflow does it +improve? Link any related issues if applicable. + +## Proposed Solution + +Describe how you envision this working. Be as specific as possible. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..8c6bfec --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,14 @@ +**What does this PR do?** + + +**Related issues** + + +**How to test** + + +**Checklist** +- [ ] `ruff check .` and `ruff format --check .` pass +- [ ] `bandit -ll -ii -c pyproject.toml -r .` passes +- [ ] `pytest` passes +- [ ] Tests added/updated if needed diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..741d34f --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,104 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in the +DeepZero community a harassment-free experience for everyone, regardless of age, +body size, visible or invisible disability, ethnicity, sex characteristics, +gender identity and expression, level of experience, education, socio-economic +status, nationality, personal appearance, race, caste, color, religion, or +sexual identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment: + +- Using welcoming and inclusive language +- Being respectful of differing viewpoints and experiences +- Giving and gracefully accepting constructive feedback +- Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +- Focusing on what is best not just for us as individuals, but for the overall + community + +Examples of unacceptable behavior: + +- The use of sexualized language or imagery, and sexual attention or advances of + any kind +- Trolling, insulting or derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information, such as a physical or email address, + without their explicit permission +- Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Responsible Security Research + +DeepZero is a vulnerability research pipeline engine. As a community built +around security tooling, we hold ourselves to an additional standard: + +- **Use DeepZero for defensive research only.** Contributions and discussions + must focus on improving security posture, not enabling attacks. +- **Follow coordinated disclosure.** If your work with DeepZero uncovers a + real-world vulnerability, report it to the affected vendor before any public + disclosure. See our [Security Policy](SECURITY.md) for guidance. +- **Never share live exploits** or weaponized payloads in issues, discussions, + or pull requests. +- **Respect data boundaries.** Do not include proprietary binaries, copyrighted + corpora, or sensitive data in contributions. + +## Enforcement Responsibilities + +Project maintainers are responsible for clarifying and enforcing our standards +of acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Project maintainers have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned with this Code of Conduct, and will communicate reasons for +moderation decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, including the GitHub +repository, issue tracker, pull requests, discussions, and any other channels +associated with DeepZero. It also applies when an individual is officially +representing the community in public spaces. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the project maintainers via: + +- **GitHub**: Open a private report through the repository's security advisories +- **Email**: Contact a maintainer directly through the email listed on their + GitHub profile + +All complaints will be reviewed and investigated promptly and fairly. All +maintainers are obligated to respect the privacy and security of the reporter. + +## Enforcement Guidelines + +### 1. Warning + +For first-time or minor violations, maintainers will reach out privately to +explain what was inappropriate and why. The expectation is that the behavior +stops. + +### 2. Block + +For repeated or severe violations, the individual will be blocked from the +repository, preventing further issues, comments, and pull requests. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.1, available at +[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. + +[homepage]: https://www.contributor-covenant.org +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..cba040e --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,281 @@ +# Contributing to DeepZero + +Thank you for your interest in contributing to DeepZero! This document explains +how to get started, what we expect from contributions, and how the review +process works. + +Please also read our [Code of Conduct](CODE_OF_CONDUCT.md) before +participating. + +--- + +## Table of Contents + +- [Getting Started](#getting-started) +- [Development Setup](#development-setup) +- [Project Structure](#project-structure) +- [Making Changes](#making-changes) +- [Code Style & Linting](#code-style--linting) +- [Testing](#testing) +- [Commit Messages](#commit-messages) +- [Pull Request Process](#pull-request-process) +- [What to Contribute](#what-to-contribute) +- [Security](#security) + +--- + +## Getting Started + +1. **Fork** the repository on GitHub. +2. **Clone** your fork locally: + ```bash + git clone https://github.com//DeepZero.git + cd DeepZero + ``` +3. **Add the upstream remote**: + ```bash + git remote add upstream https://github.com/416rehman/DeepZero.git + ``` +4. **Create a feature branch** from `main`: + ```bash + git checkout -b my-feature main + ``` + +--- + +## Development Setup + +DeepZero requires **Python 3.11+**. + +```bash +# Create and activate a virtual environment +python -m venv .venv +source .venv/bin/activate # Linux/macOS +# .venv\Scripts\activate # Windows + +# Install in editable mode with dev dependencies +pip install -e ".[dev]" + +# Install linting and security tools +pip install ruff bandit +``` + +The `[dev]` extra pulls in all optional dependency groups (`llm`, `serve`, `pe`) +plus `pytest` and `pytest-asyncio`. + +If you plan to work on the Ghidra-related processors, you will also need: +- **Java JDK 17+** +- **Ghidra 11.x** (set `GHIDRA_INSTALL_DIR` accordingly) + +--- + +## Project Structure + +``` +src/deepzero/ +├── cli.py # CLI (click + rich) +├── api/ # REST API server (starlette) +├── engine/ # Core engine: runner, pipeline loader, state, registry +└── stages/ # Built-in processors + +processors/ # Community / external processors (shipped as examples) + +pipelines/ # Example pipeline definitions + +tests/ # pytest test suite (28 test files) +``` + +- **Built-in processors** live in `src/deepzero/stages/` and are registered in + `src/deepzero/stages/__init__.py`. +- **Community processors** live under `processors//` and are referenced + by path in pipeline YAML. +- **Pipelines** live under `pipelines//` with a `pipeline.yaml` and any + supporting files (prompt templates, semgrep rules, etc.). + +--- + +## Making Changes + +### Branching Strategy + +- All work should be done on a feature branch off of `main`. +- Keep branches focused on a single change. Avoid combining unrelated fixes. + +### Types of Changes + +| Change Type | Where | +|---|---| +| New built-in processor | `src/deepzero/stages/` + register in `__init__.py` | +| New community processor | `processors//` directory | +| New pipeline | `pipelines//` directory | +| Engine changes | `src/deepzero/engine/` | +| CLI changes | `src/deepzero/cli.py` | +| API changes | `src/deepzero/api/` | +| Tests | `tests/` | + +--- + +## Code Style & Linting + +We use [Ruff](https://docs.astral.sh/ruff/) for linting and formatting, and +[Bandit](https://bandit.readthedocs.io/) for security scanning. **CI will reject +PRs that fail these checks.** + +Run all checks locally before pushing: + +```bash +# Linting +ruff check . + +# Format check (does not modify files) +ruff format --check . + +# Auto-format (modifies files in place) +ruff format . + +# Security scan +bandit -ll -ii -c pyproject.toml -r . +``` + +### Style Summary + +- **Line length**: 100 characters (E501 is ignored, but keep it reasonable) +- **Quote style**: double quotes +- **Indent style**: spaces +- **Import sorting**: handled by Ruff (`I` rules) +- **Target version**: Python 3.11 + +--- + +## Testing + +Tests live in the `tests/` directory and are run with `pytest`: + +```bash +# Run the full suite +pytest + +# Run a specific test file +pytest tests/test_runner.py + +# Run with verbose output +pytest -v + +# Run only tests matching a keyword +pytest -k "test_pipeline" +``` + +### Writing Tests + +- Place test files in `tests/` and name them `test_.py`. +- Use `tmp_path` fixtures for filesystem operations. Never write to the + project directory. +- Mock external dependencies (LLM APIs, Ghidra, semgrep) rather than requiring + them to be installed. +- If your change adds a new processor, add corresponding tests covering at + least: valid config, `ok` / `filter` / `fail` result paths, and edge cases. + +--- + +## Commit Messages + +We prefer [Conventional Commits](https://www.conventionalcommits.org/) style: + +``` +(): + + +``` + +**Types**: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `ci`, `chore` + +**Examples**: +``` +feat(stages): add regex_filter built-in processor +fix(runner): prevent deadlock on Ctrl+C during BulkMap stage +docs: update CLI reference with new --timeout flag +test(state): add atomicity tests for concurrent writes +``` + +--- + +## Pull Request Process + +1. **Ensure CI passes.** The GitHub Actions workflow runs linting, security + scanning, and the full test suite on Python 3.11 and 3.12. + +2. **Fill out the PR template.** Describe what changed, why, and how to test + it. Link any related issues. + +3. **Keep PRs focused.** One logical change per PR. If you find an unrelated + bug while working, open a separate issue or PR for it. + +4. **Respond to review feedback.** Maintainers may request changes. Please + address them or explain your reasoning. + +5. **Clean history.** PRs may be squash-merged into `main` at the + maintainer's discretion. + +### PR Checklist + +Before requesting review, verify: + +- [ ] `ruff check .` passes +- [ ] `ruff format --check .` passes +- [ ] `bandit -ll -ii -c pyproject.toml -r .` passes +- [ ] `pytest` passes +- [ ] New code has corresponding tests +- [ ] Documentation is updated if behavior changed + +--- + +## What to Contribute + +We welcome contributions of all kinds. Here are some ideas: + +### Good First Issues + +Look for issues labeled +[`good first issue`](https://github.com/416rehman/DeepZero/labels/good%20first%20issue) +on GitHub. + +### New Processors + +Community processors are the easiest way to contribute. Create a directory +under `processors/`, subclass one of the four base classes +(`IngestProcessor`, `MapProcessor`, `BulkMapProcessor`, `ReduceProcessor`), +and include a README explaining what it does. See the +[Building Processors](README.md#-building-processors) section of the README. + +### New Pipelines + +Have a creative vulnerability research workflow? Add it under +`pipelines//` with a `pipeline.yaml`, any prompt templates or rules, and +a README describing the use case and required tooling. + +### Bug Fixes & Improvements + +Check the [issue tracker](https://github.com/416rehman/DeepZero/issues) for +reported bugs. Engine improvements (better error messages, performance +optimizations, new CLI features) are also highly valued. + +### Documentation + +README clarifications, docstring improvements, and usage examples are always +appreciated. + +--- + +## Security + +If you discover a security vulnerability, **do not open a public issue.** +Follow the process in our [Security Policy](SECURITY.md) to report it +privately. + +When writing processors, follow the security best practices outlined in +SECURITY.md. Avoid `eval`/`exec`, don't hardcode credentials, and ensure +`setup()`/`teardown()` are symmetric. + +--- + +Thank you for helping make DeepZero better! 🚀