Skip to content

Feature: Add a security patch skill, and companion tool for grabbing skill specific context#148

Open
QuinnDACollins wants to merge 7 commits intomainfrom
Security_Patch_Skill
Open

Feature: Add a security patch skill, and companion tool for grabbing skill specific context#148
QuinnDACollins wants to merge 7 commits intomainfrom
Security_Patch_Skill

Conversation

@QuinnDACollins
Copy link
Contributor

No description provided.

@github-actions
Copy link

🤖 Hi @QuinnDACollins, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a robust set of features aimed at streamlining the security analysis and remediation process. It provides new tools that guide users through setting up isolated environments for Proof-of-Concept (PoC) generation, detecting the project's programming language, and accessing a dedicated knowledge base for vulnerability remediation strategies. These enhancements enable a more structured and secure approach to identifying, verifying, and fixing security flaws within projects.

Highlights

  • New Security Tools: Introduced security:setup_poc for preparing Proof-of-Concept (PoC) environments and security:fix for gathering context for vulnerability remediation.
  • Vulnerability Knowledge Base: Added a new system to load and utilize knowledge base articles for specific vulnerability types, starting with Path Traversal.
  • Enhanced PoC Execution: The runPoc tool was significantly updated to include project language detection, dynamic dependency management for Node.js PoCs using npx, and automatic cleanup of temporary files.
  • Isolated Dependency Management: A new install_dependencies tool was added to safely resolve and install dependencies in isolated environments for PoC execution.
  • Security Workflow Integration: The main security analysis workflow (commands/security/analyze.toml) was updated to prompt users for actions on identified vulnerabilities, such as generating PoCs or patching.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

## 📋 Review Summary

This pull request introduces a significant new feature for generating security patches and proofs-of-concept. The implementation is well-structured, with a clear separation of concerns between knowledge loading, PoC execution, and tool definitions. The use of skills and dedicated tools for security fixing is a great addition.

🔍 General Feedback

  • The changes to PoC execution to use npx for isolated dependency installation is a clever approach to avoid modifying the user's project.
  • The introduction of a knowledge base for vulnerabilities is a solid foundation for providing accurate and context-aware security fixes.
  • I've highlighted a few areas for improvement, mainly around strengthening security and removing redundant code. Overall, this is a great feature.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new security patch skill and companion tools that significantly enhance the security analysis and remediation capabilities. The changes include adding an ID field to vulnerability reports, enhancing the analyze.toml workflow for user interaction, and introducing a security:setup_poc tool to prepare the environment for PoC generation. The run_poc tool has been refactored for improved robustness, Node.js dependency handling, and temporary file cleanup. Additionally, new security:fix and install_dependencies tools are added to facilitate vulnerability patching and isolated dependency installation. The knowledge.ts module securely loads knowledge base articles with input sanitization, and comprehensive documentation is provided for the new skills. The new features are well-tested with unit and integration tests, demonstrating a strong focus on security and a clear workflow.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

## 📋 Security Analysis Summary

This pull request introduces a new skill and a companion tool. The new skill and tool are well-structured and follow the existing coding style. However, I found a critical command injection vulnerability in the new install_dependencies tool.

🔍 General Feedback

  • The new install_dependencies tool uses execAsync to execute a script from user input, which is a critical security vulnerability.

@QuinnDACollins
Copy link
Contributor Author

Benchmarked on PatchEval for 68.89 accuracy%

* **Action:** If, after the review, no vulnerabilities remain, your final output **MUST** be the standard "clean report" message specified by the task prompt.
* **Action:** ONLY IF the user requested JSON output (e.g., via `--json` in context or natural language), call the `convert_report_to_json` tool. Inform the user that the JSON version of the report is available at .gemini_security/security_report.json.
* **Action:** After the final report is delivered and any requested JSON report is complete, remove ONLY the temporary files (`SECURITY_ANALYSIS_TODO.md` and `DRAFT_SECURITY_REPORT.md`, you must keep `security_report.json` if generated) from the `.gemini_security/` directory. Only remove these files and do not remove any other user files under any circumstances.
* **Action:** Use the `ask_user` tool for the following TWO questions:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to have a "no questions asked mode" where we select the answers without prompting?

This would be useful for devs who want the remediation done without being prompted so many times. This would also directly apply to our CR use case.

server.registerPrompt(
'security:poc',
server.registerTool(
'security:setup_poc',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we change the description here given that we have an install dependencies tool?


server.tool(
'install_dependencies',
'Executes a script file inside workspace.',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this script user provided or agent generated?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants