From 49db407567e01ae1a0e8bdbb4dd4bd60ecbe91bb Mon Sep 17 00:00:00 2001 From: ambuj Date: Tue, 3 Sep 2024 21:31:12 +0530 Subject: [PATCH 1/3] Add gsoc'24 Add more data sources to vulnerablecode report Signed-off-by: ambuj --- docs/source/archive/gsoc-toc.rst | 1 + .../reports/2024/vulnerablecode_ambuj.rst | 150 ++++++++++++++++++ 2 files changed, 151 insertions(+) create mode 100644 docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst diff --git a/docs/source/archive/gsoc-toc.rst b/docs/source/archive/gsoc-toc.rst index a0695b5..f79bcb6 100755 --- a/docs/source/archive/gsoc-toc.rst +++ b/docs/source/archive/gsoc-toc.rst @@ -17,6 +17,7 @@ GSoC 2024 gsoc/reports/2024/scancode_toolkit_swastkk gsoc/reports/2024/scancodeio_scorecode_pranay gsoc/reports/2024/vulntotal_extension_michael + gsoc/reports/2024/vulnerablecode_ambuj GSoC 2022 --------- diff --git a/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst b/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst new file mode 100644 index 0000000..7e29626 --- /dev/null +++ b/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst @@ -0,0 +1,150 @@ +====================================================================================== +Add more data sources and mine the graph to find correlations between vulnerabilities +====================================================================================== + + +| **Organization:** `AboutCode `_ +| **Project:** `Vulnerablecode `_ +| **Mentee:** `Ambuj Kulshreshtha (ambuj-1211) `_ +| **Mentors:** +- `Keshav Priyadarshi `_ +- `Ziad Hany `_ +- `Tushar Goel `_ + +Overview +-------- + +There is a large number of pending tickets for datasources. This project focuses on adding more vulnerability data sources and consume them. I have considered following issues to solve `Collect advisories for AlmaLinux #1201`, `Collect vulnerabilities from Amazon Linux #72` , `Collect Oracle Linux #75` , `Add data in CSAF format from https://github.com/cisagov/CSAF #1315`, `VCIO does not collect some Severity (cvssv3.1) scores for a CVE #1238`, `Add CWE support in all importers #1093`. Consuming these datasources will help to create a large database for vulnerabilities. + + +Implementation +-------------- + +- **Created Importers to add more advisory data from different data sources:** + + - I have added a few new importer modules to the VulnerableCode project to incorporate advisory data from different data sources. Some of the importers I created include the `Curl Importer`_, `RockyLinux Importer`_, `AlmaLinux Importer`_, and `Amazon Linux Importer`_. I also worked on creating an importer to retrieve data in CSAF format from the `cisagov repo `_. + +- **Added CWE support in multiple importers:** + + - Many importers did not include CWE information, this was mentioned here: `https://github.com/aboutcode-org/vulnerablecode/issues/1093`_, so I solved this issue to add cwe data in multiple importers. There are still many importers that do not have CWE data available in their root data sources. I will add CWE data for them in the future if their data sources are updated. + +- **Found bugs in some Vulnerablities** + + - There is an issue `https://github.com/aboutcode-org/vulnerablecode/issues/1238`_, that I need to resolve regarding specific CVE data, specifically addressing the inconsistency in the severity information. +- **Testing:** + + - I have built proper doctests for each importer, describing each function in the module in terms of its parameters and return values. + - Proper unit tests have been created for each module I built to ensure the proper functioning of these modules. + +Linked Pull Requests +-------------------- + +.. list-table:: + :widths: 10 60 30 + :header-rows: 1 + + * - Sr. no + - Name + - Link + - Status + * - 1 + - Added Curl Advisories + - `aboutcode.org/vulnerablecode#1439 `_ + - Open + * - 2 + - Added AlmaLinux Advisories + - `aboutcode.org/vulnerablecode#1491 `_ + - Open + * - 3 + - Added CWE support in multiple importers + - `aboutcode.org/vulnerablecode#1526 `_ + - Open + * - 4 + - Added RockyLinux advisories + - `aboutcode.org/vulnerablecode#1535 `_ + - Open + * - 5 + - Added Amazon Linux advisories + - `aboutcode.org/vulnerablecode#1569 `_ + - Open + + +Related Issues +-------------- + +.. list-table:: + :widths: 10 60 30 + :header-rows: 1 + + * - Sr. no + - Name + - Link + * - 1 + - Add CURL advisories data source + - `#1166 `_ + * - 2 + - Collect advisories for AlmaLinux + - `#1201 `_ + * - 3 + - Add CWE support in all importers + - `#1093 `_ + * - 4 + - Collect rockylinux advisories + - `#753 `_ + * - 5 + - Collect vulnerabilities from Amazon Linux + - `#72 `_ + * - 6 + - Add data in CSAF format from https://github.com/cisagov/CSAF + - `#1315 `_ + * - 7 + - Collect Oracle Linux + - `#75 `_ + + +Pre GSoC Work +--------- + +I started my contributions to AboutCode by the `Add Curl Advisories issue `_, I added the curl advisories datasources to vulnerablecode database. This issue helped me to: +- Understand the importers. +- Understand the database models of VulnerableCode. +- Understand the structure of `AdvisoryData`_. +- I also explored many components, such as `PackageURL`_, `AffectedPackage`_, `Severities`_, etc. + +Post GSoC +--------- + +I am committed to working on the pull request to ensure it is merged successfully, addressing any reviews and feedback from the mentors. I will prioritize completing any remaining tasks related to my GSoC work. This includes fixing issues such as bugs for specific CVEs that lack severity CVSSv3 scores and references from NVD (as there are a few of these CVEs). Once these tasks are completed, I plan to explore and contribute to more projects within AboutCode. + +Links +----- + +* `Project Idea `_ + +* `Official GSoC project page `_ + +* `GSoC Proposal `_ + +* `Project Board `_ + +Acknowledgements +---------------- + +I would like to thank my mentors: + +- `Ziad Hany`_ +- `Tushar Goel`_ +- `Philippe Ombredanne`_ +- `Ayan Sinha Mahapatra`_ +- `Keshav Priyadarshi`_ + +This summer was full of new challenges and learning. I got to learn a lot from everyone on the team. The weekly status calls were incredibly helpful in solving all my doubts. It was fun building for AboutCode, and I will continue to contribute to the codebase of VulnerableCode and other projects as well. I plan to explore more projects in AboutCode and contribute to them because I would love to be a part of this wonderful project. + +Thank you, everyone, for your continuous support and belief in me. Your guidance and encouragement have been invaluable, and I am truly grateful for all the help and trust you've shown me throughout this journey. + + +.. _Ziad Hany: https://github.com/ziadhany +.. _Tushar Goel: https://github.com/TG1999 +.. _Philippe Ombredanne: https://github.com/pombredanne +.. _Ayan Sinha Mahapatra: https://github.com/AyanSinhaMahapatra +.. _Keshav Priyadarshi: https://github.com/keshav-space From 7ac394054ad1683d1a3e64c71e9c967ca3a43245 Mon Sep 17 00:00:00 2001 From: ambuj Date: Fri, 6 Sep 2024 11:31:20 +0530 Subject: [PATCH 2/3] resolve integration test failures Signed-off-by: ambuj --- .../reports/2024/vulnerablecode_ambuj.rst | 69 +++++++++++++------ 1 file changed, 47 insertions(+), 22 deletions(-) diff --git a/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst b/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst index 7e29626..4c78eb3 100644 --- a/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst +++ b/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst @@ -3,18 +3,23 @@ Add more data sources and mine the graph to find correlations between vulnerabil ====================================================================================== -| **Organization:** `AboutCode `_ -| **Project:** `Vulnerablecode `_ -| **Mentee:** `Ambuj Kulshreshtha (ambuj-1211) `_ -| **Mentors:** +**Organization:** `AboutCode `_ + +**Project:** `Vulnerablecode `_ + +**Mentee:** `Ambuj Kulshreshtha (ambuj-1211) `_ + +**Mentors:** + - `Keshav Priyadarshi `_ - `Ziad Hany `_ - `Tushar Goel `_ + Overview -------- -There is a large number of pending tickets for datasources. This project focuses on adding more vulnerability data sources and consume them. I have considered following issues to solve `Collect advisories for AlmaLinux #1201`, `Collect vulnerabilities from Amazon Linux #72` , `Collect Oracle Linux #75` , `Add data in CSAF format from https://github.com/cisagov/CSAF #1315`, `VCIO does not collect some Severity (cvssv3.1) scores for a CVE #1238`, `Add CWE support in all importers #1093`. Consuming these datasources will help to create a large database for vulnerabilities. +There is a large number of pending tickets for datasources. This project focuses on adding more vulnerability data sources and consume them. I have considered following issues to solve `Collect advisories for AlmaLinux #1201 `_, `Collect vulnerabilities from Amazon Linux #72 `_ , `Collect Oracle Linux #75 `_ , `Add data in CSAF format #1315 `_, `VCIO does not collect some Severity (cvssv3.1) scores for a CVE #1238 `_, `Add CWE support in all importers #1093 `_ and `Collect rockylinux advisories #753 `_. Consuming these datasources will help to create a large database for vulnerabilities. Implementation @@ -22,25 +27,28 @@ Implementation - **Created Importers to add more advisory data from different data sources:** - - I have added a few new importer modules to the VulnerableCode project to incorporate advisory data from different data sources. Some of the importers I created include the `Curl Importer`_, `RockyLinux Importer`_, `AlmaLinux Importer`_, and `Amazon Linux Importer`_. I also worked on creating an importer to retrieve data in CSAF format from the `cisagov repo `_. + - I have added a few new importer modules to the VulnerableCode project to incorporate advisory data from different data sources. Some of the importers I created include the `Curl Importer`, `RockyLinux Importer`, `AlmaLinux Importer`, and `Amazon Linux Importer`. I also worked on creating an importer to retrieve data in CSAF format from the `cisagov repo `_. - **Added CWE support in multiple importers:** - - Many importers did not include CWE information, this was mentioned here: `https://github.com/aboutcode-org/vulnerablecode/issues/1093`_, so I solved this issue to add cwe data in multiple importers. There are still many importers that do not have CWE data available in their root data sources. I will add CWE data for them in the future if their data sources are updated. + - Many importers did not include CWE information, this was mentioned here: `Add CWE support in all importers #1093 `_, so I solved this issue to add cwe data in multiple importers. There are still many importers that do not have CWE data available in their root data sources. I will add CWE data for them in the future if their data sources are updated. - **Found bugs in some Vulnerablities** - - There is an issue `https://github.com/aboutcode-org/vulnerablecode/issues/1238`_, that I need to resolve regarding specific CVE data, specifically addressing the inconsistency in the severity information. + - There is an issue `VCIO does not collect some Severity (cvssv3.1) scores for a CVE #1238 `_, that I need to resolve regarding specific CVE data, specifically addressing the inconsistency in the severity information. - **Testing:** - - I have built proper doctests for each importer, describing each function in the module in terms of its parameters and return values. - - Proper unit tests have been created for each module I built to ensure the proper functioning of these modules. + - I have built proper doctests for each importer, describing each + function in the module in terms of its parameters and return values. + + - Proper unit tests have been created for each module I built + to ensure the proper functioning of these modules. Linked Pull Requests -------------------- .. list-table:: - :widths: 10 60 30 + :widths: 10 60 30 10 :header-rows: 1 * - Sr. no @@ -67,7 +75,6 @@ Linked Pull Requests - Added Amazon Linux advisories - `aboutcode.org/vulnerablecode#1569 `_ - Open - Related Issues -------------- @@ -93,31 +100,43 @@ Related Issues - `#753 `_ * - 5 - Collect vulnerabilities from Amazon Linux - - `#72 `_ + - `#72 `_ * - 6 - - Add data in CSAF format from https://github.com/cisagov/CSAF + - Add data in CSAF format - `#1315 `_ * - 7 - Collect Oracle Linux - `#75 `_ + * - 8 + - VCIO does not collect some Severity (cvssv3.1) scores for a CVE + - `#1238 `_ Pre GSoC Work ---------- +--------------- I started my contributions to AboutCode by the `Add Curl Advisories issue `_, I added the curl advisories datasources to vulnerablecode database. This issue helped me to: + - Understand the importers. + - Understand the database models of VulnerableCode. -- Understand the structure of `AdvisoryData`_. -- I also explored many components, such as `PackageURL`_, `AffectedPackage`_, `Severities`_, etc. + +- Understand the structure of `AdvisoryData`. + +- I also explored many components, such as `PackageURL`, `AffectedPackage`, `Severities`, etc. Post GSoC ---------- +---------- -I am committed to working on the pull request to ensure it is merged successfully, addressing any reviews and feedback from the mentors. I will prioritize completing any remaining tasks related to my GSoC work. This includes fixing issues such as bugs for specific CVEs that lack severity CVSSv3 scores and references from NVD (as there are a few of these CVEs). Once these tasks are completed, I plan to explore and contribute to more projects within AboutCode. +I am committed to working on the pull request to ensure it is merged +successfully, addressing any reviews and feedback from the mentors. I will prioritize +completing any remaining tasks related to my GSoC work. This includes fixing issues +such as bugs for specific CVEs that lack severity CVSSv3 scores and references +from NVD (as there are a few of these CVEs). Once these tasks are completed, +I plan to explore and contribute to more projects within AboutCode. Links ------ +------ * `Project Idea `_ @@ -138,9 +157,15 @@ I would like to thank my mentors: - `Ayan Sinha Mahapatra`_ - `Keshav Priyadarshi`_ -This summer was full of new challenges and learning. I got to learn a lot from everyone on the team. The weekly status calls were incredibly helpful in solving all my doubts. It was fun building for AboutCode, and I will continue to contribute to the codebase of VulnerableCode and other projects as well. I plan to explore more projects in AboutCode and contribute to them because I would love to be a part of this wonderful project. +This summer was full of new challenges and learning. I got to learn a lot from everyone on the team. +The weekly status calls were incredibly helpful in solving all my doubts. It was fun building for +AboutCode, and I will continue to contribute to the codebase of VulnerableCode and +other projects as well. I plan to explore more projects in AboutCode and contribute to +them because I would love to be a part of this wonderful project. -Thank you, everyone, for your continuous support and belief in me. Your guidance and encouragement have been invaluable, and I am truly grateful for all the help and trust you've shown me throughout this journey. +Thank you, everyone, for your continuous support and belief in me. +Your guidance and encouragement have been invaluable, and I am truly grateful +for all the help and trust you've shown me throughout this journey. .. _Ziad Hany: https://github.com/ziadhany From 1c9e03f0d510415c098937748dc77dc0b58f0aee Mon Sep 17 00:00:00 2001 From: ambuj Date: Fri, 6 Sep 2024 18:19:44 +0530 Subject: [PATCH 3/3] Update documentation Signed-off-by: ambuj --- .../archive/gsoc/reports/2024/vulnerablecode_ambuj.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst b/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst index 4c78eb3..5f3e4f2 100644 --- a/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst +++ b/docs/source/archive/gsoc/reports/2024/vulnerablecode_ambuj.rst @@ -1,6 +1,6 @@ -====================================================================================== -Add more data sources and mine the graph to find correlations between vulnerabilities -====================================================================================== +=============================================== +Add more data sources and improve data quality +=============================================== **Organization:** `AboutCode `_ @@ -11,9 +11,11 @@ Add more data sources and mine the graph to find correlations between vulnerabil **Mentors:** -- `Keshav Priyadarshi `_ +- `Philippe Ombredanne `_ +- `Ayan Sinha Mahapatra `_ - `Ziad Hany `_ - `Tushar Goel `_ +- `Keshav Priyadarshi `_ Overview