APPENG-3181: Feature Add C support to transitive tool (#104)

* feat: Add C/C++ support to transitive code search tool

- Add C language support and enhance function parsing capabilities
- Implement singleton pattern for RPMDependencyManager
- Add container source download for C/C++ dependencies
- Add C extended segmenter class with comprehensive testing
- Add new Function Name Locator tool with fuzzy matching
- Enhance transitive code search with assistant tool
- Add C/C++ function parsers and language support
- Add comprehensive test suite for C code analysis
- Improve function name extraction and parsing
- Add source RPM downloader for C/C++ dependencies
- Update configuration files for NIM and OpenAI endpoints
- Fix various bugs and improve code quality
- Add proper documentation and type hints
- Clean up imports and remove debug prints

This commit consolidates 39 individual commits that add comprehensive
C/C++ language support to the vulnerability analysis toolchain.

* update docs, kustomize service config, tests and rpm downloader support UT

* review fix move C_DEP_LIBS_NAME locaiton

* fixed review comments

* Add cache for std lib names for lang python,go,java,javascript

* Update src/vuln_analysis/utils/standard_library_cache.py

Co-authored-by: Zvi Grinberg <[email protected]>

* fix indentation

* add persistance to cache

* code review comments

* add comments and update docker for bsdtar app to be install in image

* update docfile

* small improvments

* handle invalid utf-8 files

---------

Co-authored-by: Zvi Grinberg <[email protected]>

Author: RedTanny

Dockerfile

@@ -33,9 +33,13 @@ RUN apt-get update && apt-get install -y \
       git \
       git-lfs \
       wget \
+      skopeo \
+      libarchive-tools \
     && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* \
     && update-ca-certificates
 
+
 RUN curl -L -X GET https://go.dev/dl/go1.24.1.linux-amd64.tar.gz -o /tmp/go1.24.1.linux-amd64.tar.gz \
     && tar -C /usr/local -xzf /tmp/go1.24.1.linux-amd64.tar.gz \
     && rm /tmp/go1.24.1.linux-amd64.tar.gz

README.md

@@ -216,6 +216,15 @@ To run the workflow you need to obtain API keys for the following APIs. These wi
         - Click on your account in the top right, select "Setup" from the dropdown.
         - Click the "Generate Personal Key" option and then the "+ Generate Personal Key" button to create your API key.
     - This will be used in the `NVIDIA_API_KEY` environment variable.
+  - REDHAT container registry (Recommended but not compulsory)
+    - To get source images from a Red Hat container registry using registry service account tokens. You will need to create a [registry service account](https://access.redhat.com/terms-based-registry/)
+    - Steps:
+      - Sign in to the registry service accout
+      - Press on New service Account Button
+      - Fill the Name (ex. 'test-case') and Description fields and Click the Create botton
+      - Token user name for example '11008101|test-case' for REGISTRY_REDHAT_USERNAME environment variable
+      - Token password a long string for REGISTRY_REDHAT_PASSWORD environment variable
+    
 
 The workflow can be configured to use other LLM services as well, see the [Customizing the LLM models](#customizing-the-llm-models) section for more info.
 

kustomize/README.md

@@ -38,24 +38,47 @@ export NVIDIA_API_BASE=http://YOUR_SELF_HOSTED_OPENAI_LLM_ADDRESSS/v1
 export PYTHONUNBUFFERED=1
 export SERPAPI_API_KEY=YOUR_SERPAPI_KEY
 export SUMMARIZE_MODEL_NAME=hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4
+export REGISTRY_REDHAT_USERNAME="your_username"
+export REGISTRY_REDHAT_PASSWORD="your_password_or_token"
 ```
 
 6. **Run the Application**: After a successful installation, run the application.
-
 ```shell
 nat --log-level debug  serve --config_file=src/vuln_analysis/configs/config-http-openai.yml --host 0.0.0.0 --port 26466
 ```
 
+### Container Source Download Configuration
+
+For C/C++ projects, you can enable container source download to extract RPM dependencies from Red Hat container registries. This feature uses skopeo to download container source layers and automatically extracts RPM packages, excluding the main application RPMs to focus on dependencies only.
+**Required Environment Variables:**
+# Enable container source download mode (default behavior )
+export USE_CONTAINER_SOURCES=true
+
+**How it works:**
+1. Downloads container source layers using skopeo
+2. Extracts RPM packages from the downloaded layers
+3. Filters out main application RPMs (e.g., `postgresql-*` for PostgreSQL containers)
+4. Copies dependency RPMs to the standard RPM cache directory
+
+
+**Prerequisites:**
+- `skopeo` must be installed on the system
+- Valid Red Hat registry credentials
+- Network access to `registry.redhat.io`
+
+
 ## Deploy And Run On OCP
 
-1. Create a `base/secrets.env` file containing the API keys for external services `ExploitIQ` might use. Not all keys are mandatory. Refer to the main [README](../README.md) for details.
+1. Create a `base/secrets.env` file containing the API keys for external services `ExploitIQ` might use. Not all keys are mandatory. Refer to the main [README](../README.md#obtain-api-keys) for details on how to create the Red Hat credentials and other API keys.
 
 ```shell
 cat > base/secrets.env << EOF
 nvd_api_key=you_api_key
 serpapi_api_key=your_api_key
 nvidia_api_key=your_api_key
 ghsa_api_key=your_api_key
+registry_redhat_username=your_registry_username
+registry_redhat_password=your_registry_pass_token
 EOF
 ```
 

kustomize/base/excludes.json

@@ -17,6 +17,15 @@
     "pom.xml",
     "build.gradle"
   ],
+  "C": [
+    "**/tests/**/*",
+    "**/test/**/*",
+    "**/demos/**/*",
+    "**/examples/**/*",
+    "**/samples/**/*",
+    "**/doc/**/*",              
+    "**/docs/**/*"
+  ],
   "JavaScript": [
     "node_modules/**/*",
     "dist/**/*",

kustomize/base/exploit-iq-config.yml

@@ -32,7 +32,8 @@ functions:
     base_git_dir: ${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}git
     base_vdb_dir: ${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}vdb
     base_code_index_dir: ${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}code_index
-    base_pickle_dir: ${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}pickle
+    base_pickle_dir: ${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}pickle                
+    base_rpm_dir: ${EXPLOIT_IQ_DATA_DIR:-/exploit-iq-data/}rpms
     ignore_code_embedding: true
   cve_fetch_intel:
     _type: cve_fetch_intel
@@ -59,6 +60,8 @@ functions:
   Calling Function Name Extractor:
     _type: calling_function_name_extractor
     enable_functions_usage_search: true
+  Package and Function Locator:
+    _type: package_and_function_locator
   Container Image Code QA System:
     _type: local_vdb_retriever
     embedder_name: nim_embedder
@@ -89,6 +92,7 @@ functions:
       - Internet Search
       - Transitive code search tool
       - Calling Function Name Extractor
+      - Package and Function Locator
     max_concurrency: null
     max_iterations: 10
     prompt_examples: false

kustomize/base/exploit_iq_service.yaml

@@ -79,6 +79,16 @@ spec:
               value: EXPLOIT_IQ
             - name: NGC_API_KEY
               value: EXPLOIT_IQ
+            - name: REGISTRY_REDHAT_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  key: registry_redhat_password
+                  name: exploit-iq-secret
+            - name: REGISTRY_REDHAT_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  key: registry_redhat_username
+                  name: exploit-iq-secret
             - name: CVE_DETAILS_BASE_URL
               value: http://nginx-cache:8080/cve-details
             - name: CWE_DETAILS_BASE_URL

kustomize/base/includes.json

@@ -48,6 +48,10 @@
     "public/**/*",
     "assets/**/*"
   ],
+  "C":[
+    "**/*.c",        
+    "**/*.h"
+  ],
   "Dockerfile": [
     "Dockerfile*",
     "docker-compose.yml",

kustomize/config-http-openai-local.yml

@@ -37,6 +37,7 @@ functions:
     base_vdb_dir: .cache/am_cache/vdb
     base_code_index_dir: .cache/am_cache/code_index
     base_pickle_dir: .cache/am_cache/pickle
+    base_rpm_dir: .cache/am_cache/rpms
     ignore_code_embedding: true
   cve_fetch_intel:
     _type: cve_fetch_intel
@@ -60,6 +61,8 @@ functions:
   Calling Function Name Extractor:
     _type: calling_function_name_extractor
     enable_functions_usage_search: true
+  Package and Function Locator:
+    _type: package_and_function_locator
   Container Image Code QA System:
     _type: local_vdb_retriever
     embedder_name: nim_embedder
@@ -90,6 +93,7 @@ functions:
       - Internet Search
       - Transitive code search tool
       - Calling Function Name Extractor
+      - Package and Function Locator
     max_concurrency: null
     max_iterations: 10
     prompt_examples: false