Feature: Add a security patch skill, and companion tool for grabbing skill specific context#148
Feature: Add a security patch skill, and companion tool for grabbing skill specific context#148QuinnDACollins wants to merge 7 commits intomainfrom
Conversation
|
🤖 Hi @QuinnDACollins, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a robust set of features aimed at streamlining the security analysis and remediation process. It provides new tools that guide users through setting up isolated environments for Proof-of-Concept (PoC) generation, detecting the project's programming language, and accessing a dedicated knowledge base for vulnerability remediation strategies. These enhancements enable a more structured and secure approach to identifying, verifying, and fixing security flaws within projects. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
This pull request introduces a significant new feature for generating security patches and proofs-of-concept. The implementation is well-structured, with a clear separation of concerns between knowledge loading, PoC execution, and tool definitions. The use of skills and dedicated tools for security fixing is a great addition.
🔍 General Feedback
- The changes to PoC execution to use
npxfor isolated dependency installation is a clever approach to avoid modifying the user's project. - The introduction of a knowledge base for vulnerabilities is a solid foundation for providing accurate and context-aware security fixes.
- I've highlighted a few areas for improvement, mainly around strengthening security and removing redundant code. Overall, this is a great feature.
There was a problem hiding this comment.
Code Review
This pull request introduces a new security patch skill and companion tools that significantly enhance the security analysis and remediation capabilities. The changes include adding an ID field to vulnerability reports, enhancing the analyze.toml workflow for user interaction, and introducing a security:setup_poc tool to prepare the environment for PoC generation. The run_poc tool has been refactored for improved robustness, Node.js dependency handling, and temporary file cleanup. Additionally, new security:fix and install_dependencies tools are added to facilitate vulnerability patching and isolated dependency installation. The knowledge.ts module securely loads knowledge base articles with input sanitization, and comprehensive documentation is provided for the new skills. The new features are well-tested with unit and integration tests, demonstrating a strong focus on security and a clear workflow.
There was a problem hiding this comment.
This pull request introduces a new skill and a companion tool. The new skill and tool are well-structured and follow the existing coding style. However, I found a critical command injection vulnerability in the new install_dependencies tool.
🔍 General Feedback
- The new
install_dependenciestool usesexecAsyncto execute a script from user input, which is a critical security vulnerability.
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
Benchmarked on PatchEval for 68.89 accuracy% |
| * **Action:** If, after the review, no vulnerabilities remain, your final output **MUST** be the standard "clean report" message specified by the task prompt. | ||
| * **Action:** ONLY IF the user requested JSON output (e.g., via `--json` in context or natural language), call the `convert_report_to_json` tool. Inform the user that the JSON version of the report is available at .gemini_security/security_report.json. | ||
| * **Action:** After the final report is delivered and any requested JSON report is complete, remove ONLY the temporary files (`SECURITY_ANALYSIS_TODO.md` and `DRAFT_SECURITY_REPORT.md`, you must keep `security_report.json` if generated) from the `.gemini_security/` directory. Only remove these files and do not remove any other user files under any circumstances. | ||
| * **Action:** Use the `ask_user` tool for the following TWO questions: |
There was a problem hiding this comment.
Is there a way to have a "no questions asked mode" where we select the answers without prompting?
This would be useful for devs who want the remediation done without being prompted so many times. This would also directly apply to our CR use case.
| server.registerPrompt( | ||
| 'security:poc', | ||
| server.registerTool( | ||
| 'security:setup_poc', |
There was a problem hiding this comment.
Should we change the description here given that we have an install dependencies tool?
|
|
||
| server.tool( | ||
| 'install_dependencies', | ||
| 'Executes a script file inside workspace.', |
There was a problem hiding this comment.
Is this script user provided or agent generated?
No description provided.