Add pagination, filters, and collapse to active jobs page; implement custom workflow builder with Kali Linux Docker support

[Copilot]

Redesigned active jobs page with pagination, filtering, and collapsible job details. Added custom workflow system for creating multi-phase reconnaissance pipelines with template variables, including automatic default workflow creation and workflow selection when launching scans. Updated Docker infrastructure to use Kali Linux base image with comprehensive test coverage.

Active Jobs Page

Pagination & Filters

• Configurable items per page (5/10/20/50) with prev/next navigation
• Domain search and status filtering with localStorage persistence
• Jobs collapsed by default showing only summary (domain, status, progress)
• Expand All / Collapse All bulk controls

UI Changes

// Jobs now render with collapsible state tracking
const isCollapsed = jobsCollapsedState[jobId] !== false; // Default collapsed
<div class="job-summary" onclick="toggleJobDetails('${jobId}')">
  <span class="job-toggle-icon">${isCollapsed ? '▶' : '▼'}</span>
  ...
</div>
<div class="job-details" style="display: ${isCollapsed ? 'none' : 'block'};">
  <!-- Steps, logs, metadata -->
</div>

Custom Workflows

Data Model

• SQLite workflows table: id, name, description, phases (JSON), is_default
• Each phase: tool selection (16 built-ins + custom), flags, input/output templates

Template Variables

• $DOMAIN$ - Target domain
• $INPUT$ / $OUTPUT$ - Phase I/O chaining
• $WORDLIST$ - Configured wordlist path

API Endpoints

• GET /api/workflows - List with default indicator
• POST /api/workflows/{create,update,delete,set-default} - CRUD operations

UI

• Phase builder with tool dropdown or custom command textarea
• Per-phase flags and I/O template configuration
• Default workflow selection for automatic scan launches
• Workflow selector in Launch Scan form - dropdown shows all workflows with default pre-selected

Example custom phase:

echo "$DOMAIN$" | custom-subdomain-tool --output $OUTPUT$ --threads 20

Default Workflow

Auto-Creation

• On first run, automatically creates "Default Recon Pipeline" workflow
• Contains all standard tools: amass, subfinder, assetfinder, findomain, sublist3r, crtsh, github-subdomains, dnsx, ffuf, httpx, waybackurls, gau, nmap, nuclei, nikto
• Marked as default workflow (is_default = 1)

Protection

• Default workflow cannot be deleted
• Users must set another workflow as default before deletion is allowed
• Ensures standard pipeline is always available

Usage

• Default workflow automatically selected when launching scans
• Users can choose different workflow from dropdown
• Jobs track which workflow was used via workflow_id field

Docker Infrastructure

Kali Linux Base Image

• Migrated from python:3.11-slim to kalilinux/kali-rolling
• Provides comprehensive security-focused environment with pre-installed tools
• Better tool compatibility and reduced build time
• Platform support: linux/amd64, linux/arm64

Key Changes:

FROM kalilinux/kali-rolling AS base
RUN apt-get update && apt-get install -y python3 python3-pip ...
RUN pip3 install --no-cache-dir --break-system-packages psutil sublist3r nikto-parser

Test Coverage

Comprehensive Workflow Tests

• Added TestWorkflows class with 9 test cases covering:
  • Default workflow auto-creation and protection
  • Workflow CRUD operations (Create, Read, Update, Delete)
  • Workflow-job integration
  • API endpoint validation
  • Database persistence

Test Results:

test_main.py::TestWorkflows::test_ensure_default_workflow PASSED
test_main.py::TestWorkflows::test_create_workflow PASSED
test_main.py::TestWorkflows::test_update_workflow PASSED
test_main.py::TestWorkflows::test_delete_non_default_workflow PASSED
test_main.py::TestWorkflows::test_cannot_delete_default_workflow PASSED
test_main.py::TestWorkflows::test_set_default_workflow PASSED
test_main.py::TestWorkflows::test_list_workflows PASSED
test_main.py::TestWorkflows::test_start_pipeline_job_with_workflow PASSED
test_main.py::TestWorkflows::test_workflow_api_endpoints PASSED

========================= 9 passed in 0.46s =========================

Technical Notes

• All state (filters, pagination, collapse) persists in localStorage
• Workflow phases stored as JSON array in SQLite for flexibility
• Default workflow created via ensure_default_workflow() during database initialization
• start_pipeline_job() accepts optional workflow_id, falls back to default if not provided
• Backward compatible - existing jobs display unchanged, just organized better
• Single-file architecture maintained per project convention
• Documentation updated: DOCKER_BUILD.md, TESTING.md with Kali Linux information and workflow test instructions

Original prompt

Please re-design the active jobs page to be paged and have filters and start collapsed

Please give users the options to make workflows which can use the building blocks with flags , input and output template string where they can make their own phases. The user should be able to add their own tools in it by using a custom command.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[The-XSS-Rat]

@copilot The app is not using the workflow i have set as the workflow when launching a new scan

make sure there is 1 workflow as "Default" that can never be deleted and contains the current workflow that the app normally runs

Allow the user to select the workflow when launching a new scan from a dropdown of existing workflows

[Copilot]

@copilot The app is not using the workflow i have set as the workflow when launching a new scan

make sure there is 1 workflow as "Default" that can never be deleted and contains the current workflow...

Implemented in commit 30433e2.

Changes made:

1. Default Workflow Created: On first run, a "Default Recon Pipeline" workflow is automatically created containing all standard tools from PIPELINE_STEPS (amass, subfinder, httpx, nuclei, etc.)

2. Protected Default: The default workflow cannot be deleted. Users must set another workflow as default before deletion is allowed (enforced in delete_workflow())

3. Workflow Selector in Launch Form: Added a dropdown to select workflow when launching scans. The default workflow is pre-selected automatically.

4. Backend Integration:

   • Updated start_pipeline_job() to accept and store workflow_id
   • Jobs now track which workflow was used
   • /api/run endpoint accepts workflow_id parameter
   • Falls back to default workflow if none selected

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• esm.ubuntu.com
  • Triggering command: /usr/lib/apt/methods/https /usr/lib/apt/methods/https (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[The-XSS-Rat]

@copilot Can you please also update all the tests and the dockerfile + also make sure that the dockerfile uses the base image of kali

[Copilot]

@copilot Can you please also update all the tests and the dockerfile + also make sure that the dockerfile uses the base image of kali

Completed in commit e0ad97d.

Dockerfile Updates:

• Changed base image from python:3.11-slim to kalilinux/kali-rolling
• Added explicit python3 and python3-pip installation
• Added psutil for system resource monitoring
• Updated platform support to linux/amd64 and linux/arm64
• Added --break-system-packages flag for pip3 (Kali requirement)

Test Suite:

• Added comprehensive TestWorkflows class with 9 test cases
• All workflow tests passing ✅
• Tests cover: workflow creation, updating, deletion, default workflow protection, job integration, and API endpoints

Documentation:

• Updated DOCKER_BUILD.md with Kali Linux information
• Updated TESTING.md with workflow test section
• Added commands for running workflow-specific tests