[JDBC] Connectors Forge using Bob by Thomasgloria · Pull Request #38 · IBM/cp4d-connector-sdk

Thomasgloria · 2026-05-04T12:28:01Z

Overview

This PR introduces automated JDBC connector generation capabilities and streamlined AWS deployment for custom JDBC connectors in the CP4D Connector SDK.

What's New

Automated JDBC Connector Generation

Customers can now automatically generate custom JDBC connector code using the Custom JDBC Connector Mode. This eliminates manual configuration and reduces setup time significantly.

Connectors Forge JDBC Mode - Complete Documentation

Overview

The Connectors Forge JDBC mode is an AI-powered assistant that guides users through creating custom JDBC connectors for IBM Cloud Pak for Data (CP4D). It automates the entire process from database information gathering to deployment on multiple platforms (AWS ECS Fargate, OpenShift, or self-hosted servers).

Key Features

Two Deployment Modes: Codeless (pre-built image) or Developer (custom code generation)
Multi-Platform Deployment: AWS, OpenShift, or self-hosted
Automated Driver Management: Automatic download or manual provision of JDBC drivers
DSL-Based Configuration: JSON-based Domain Specific Language for connector definition
Complete Automation: From DSL creation to production deployment
AWS EFS Integration: Persistent driver storage with automatic upload

Mode Purpose and Capabilities

What It Does

The Connectors Forge JDBC mode is a specialized AI assistant that:

Gathers Database Information: Collects JDBC driver details, connection properties, and database-specific features
Creates DSL Configuration: Generates database-specific DSL files from templates
Manages JDBC Drivers: Downloads or copies driver JARs to appropriate locations
Generates Code (Developer mode): Creates complete connector skeleton with customizations
Deploys to Cloud: Automates deployment to AWS ECS Fargate, OpenShift, or self-hosted servers
Handles Infrastructure: Creates EFS, security groups, task definitions, and services

When to Use

Use this mode when you need to:

Create a new JDBC connector for any database (MySQL, PostgreSQL, Oracle, SQL Server, etc.)
Deploy connectors to AWS ECS Fargate with persistent driver storage
Deploy connectors to OpenShift or self-hosted servers
Customize existing JDBC connector implementations
Set up complete connector infrastructure from scratch

Architecture

High-Level Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Connectors Forge JDBC Mode                    │
│                                                                  │
│  ┌────────────────┐              ┌──────────────────┐          │
│  │  Information   │              │   DSL Creator    │          │
│  │   Gathering    │─────────────▶│   (JSON Config)  │          │
│  └────────────────┘              └──────────────────┘          │
│         │                                 │                     │
│         │                                 │                     │
│         ▼                                 ▼                     │
│  ┌────────────────┐              ┌──────────────────┐          │
│  │ Driver Manager │              │  Code Generator  │          │
│  │ (Download/Copy)│              │  (Developer Mode)│          │
│  └────────────────┘              └──────────────────┘          │
│         │                                 │                     │
│         │                                 │                     │
│         └─────────────┬───────────────────┘                     │
│                       │                                         │
│                       ▼                                         │
│              ┌──────────────────┐                              │
│              │   Deployment     │                              │
│              │   Orchestrator   │                              │
│              └──────────────────┘                              │
│                       │                                         │
│         ┌─────────────┼─────────────┐                          │
│         │             │             │                          │
│         ▼             ▼             ▼                          │
│    ┌────────┐   ┌──────────┐  ┌──────────┐                   │
│    │  AWS   │   │OpenShift │  │Self-Host │                   │
│    │  ECS   │   │          │  │          │                   │
│    └────────┘   └──────────┘  └──────────┘                   │
└─────────────────────────────────────────────────────────────────┘

Component Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Mode Components                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  1. Workflow Engine (.bob/rules-create-custom-connector-jdbc/)  │
│     ├── 1_workflow.xml - Complete workflow phases               │
│     ├── 2_best_practices.xml - Best practices and guidelines    │
│     ├── 2_database_information.xml - Database-specific info     │
│     ├── 3_tool_usage_and_best_practices.xml - Tool guidelines   │
│     └── 4_user_interaction.xml - User interaction patterns      │
│                                                                  │
│  2. Configuration Templates (connectors-forge/codeless/jdbc/)   │
│     ├── DSL-jdbc.json - DSL template                            │
│     ├── connector-config.env.template - Connector config        │
│     ├── deploy-container.env.template - Container config        │
│     ├── deploy.env.template - OpenShift deployment config       │
│     └── aws-deployment.properties.template - AWS config         │
│                                                                  │
│  3. Deployment Scripts (connectors-forge/codeless/jdbc/)        │
│     ├── deploy-to-aws.sh - AWS ECS deployment (1074 lines)     │
│     ├── upload-driver-to-efs.sh - EFS driver upload            │
│     ├── deploy-openshift.sh - OpenShift deployment             │
│     ├── deploy-container.sh - Self-hosted deployment           │
│                                                                  │
│  4. Documentation (connectors-forge/codeless/jdbc/)             │
│     ├── AUTOMATIC-DRIVER-UPLOAD.md - Driver upload guide       │
│     ├── EFS-DRIVER-UPLOAD-GUIDE.md - EFS upload details        │
│     ├── UPLOAD-TO-EFS-UI-GUIDE.md - UI-based upload            │
│     └── FIX-EFS-ATTACHMENT.md - Troubleshooting guide          │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Deployment Approaches

1. Codeless Mode (Recommended for Quick Setup)

Overview: Uses pre-built Docker image from GitHub Container Registry

Docker Image: ghcr.io/thomasgloria/wdp-connect-sdk-gen-jdbc-connectors-forge:latest

Benefits:

✅ No code generation required
✅ No build process needed
✅ Faster deployment (minutes vs hours)
✅ Best for standard JDBC use cases
✅ Automatic driver management

Workflow:

1. Information Gathering → 2. DSL Creation → 3. Driver Management → 4. Deployment

Use Cases:

Standard database connectors (MySQL, PostgreSQL, etc.)
Quick prototyping and testing
Production deployments with standard requirements
When you don't need custom connector logic

2. Developer Mode (Full Customization)

Overview: Generates complete customizable connector code

Benefits:

✅ Full control over connector implementation
✅ Custom business logic and transformations
✅ Advanced data type mappings
✅ Custom authentication methods
✅ Comprehensive test suite generation

Workflow:

1. Information Gathering → 2. DSL Creation → 3. Code Generation → 
4. Customization → 5. Test Generation → 6. Build → 7. Deployment

Use Cases:

Custom JDBC implementations
Advanced data transformations
Proprietary database connectors
When you need to modify connector behavior
Enterprise requirements with specific customizations

Complete Workflow

Phase 0: Deployment Approach Selection

Purpose: Choose between Codeless or Developer mode

Steps:

Mode presents deployment options
User selects Codeless or Developer
Mode configures workflow path accordingly

Decision Factors:

Codeless: Standard use cases, quick deployment
Developer: Custom requirements, full control

Phase 1: Information Gathering

Purpose: Collect database-specific information

Information Collected:

Connector Metadata:
- Connector name (e.g., mysql, postgresql)
- Connector label (e.g., "MySQL", "PostgreSQL")
- Description
JDBC Driver Information:
- Driver class name (e.g., com.mysql.cj.jdbc.Driver)
- Maven coordinates (e.g., com.mysql:mysql-connector-j:8.0.33)
- JDBC URL template (e.g., jdbc:mysql://{host}:{port}/{database})
Database Features:
- Default port number
- Schema/catalog support
- System schemas to exclude
- Identifier quote character
Driver JAR (Codeless mode only):
- Automatic download from Maven Central (for open-source drivers)
- Manual provision (for proprietary drivers like Oracle, DB2)

Supported Databases (with pre-configured information):

IBM Db2
MySQL
PostgreSQL
Oracle
Microsoft SQL Server
MariaDB
Snowflake
Amazon Redshift
Teradata
Apache Hive

Phase 2: DSL Creation

Purpose: Generate database-specific DSL configuration file

Process:

Read DSL-jdbc.json template
Populate with database-specific values
Create DSL-jdbc-{connector_name}.json
Remove example/comment fields
Validate JSON structure

DSL File Structure:

{
  "connector_metadata": {
    "connector_name": "mysql",
    "connector_label": "MySQL",
    "connector_description": "JDBC connector for MySQL databases"
  },
  "jdbc_driver": {
    "jdbc_url": "jdbc:mysql://{host}:{port}/{database}",
    "driver_class_name": "com.mysql.cj.jdbc.Driver"
  },
  "connection_properties": {
    "additional_properties": [...]
  },
  "database_specific_features": {
    "supports_schemas": false,
    "supports_catalogs": true,
    "system_schemas": ["information_schema", "mysql", "performance_schema", "sys"]
  },
  "implementation_notes": {
    "dependencies": [{
      "driver_dependencies": ["com.mysql:mysql-connector-j:8.0.33"]
    }]
  }
}

Output: connectors-forge/codeless/jdbc/DSL-jdbc-{connector_name}.json

Phase 2.5: Deployment Preparation (Codeless Mode Only)

Purpose: Prepare configuration files for deployment

Steps:

Select Deployment Type:
- AWS ECS Fargate
- OpenShift
- Self-hosted server
Create connector-config.env:
- Populated from DSL values
- Contains connector metadata
- Includes JDBC driver configuration
Confirm Configuration:
- Display connector-config.env contents
- Verify connector label (UI display name)
- Allow modifications if needed
Create Deployment-Specific Config (if OpenShift):
- Generate deploy.env from template
- Configure OpenShift-specific settings
- Review and update values

connector-config.env Example:

# Connector Configuration
CONNECTOR_DATASOURCE_TYPE=mysql
CONNECTOR_LABEL=MySQL
CONNECTOR_DESCRIPTION=JDBC connector for MySQL databases
JDBC_DRIVER_CLASS=com.mysql.cj.jdbc.Driver
JDBC_DRIVER_PATH=/mnt/efs/driver.jar  # AWS
# JDBC_DRIVER_PATH=/drivers/driver.jar  # OpenShift

Phase 3-9: Developer Mode Only

Phase 3: Label Confirmation
Phase 4: Code Generation (./gradlew generateJavaJdbc)
Phase 5: DSL Relocation
Phase 6: Connector Customization (./gradlew customizeJdbcConnector)
Phase 7: Test Generation (./gradlew generateJdbcTests)
Phase 8: Code Formatting (./gradlew spotlessApply)
Phase 9: Build (./gradlew :jdbc_{name}:build)

Phase 10/11: Deployment

Purpose: Deploy connector to selected platform

AWS ECS Fargate Deployment

Infrastructure Created:

EFS File System: Persistent driver storage
EFS Security Group: NFS access control
EFS Mount Targets: Multi-AZ availability
EFS Access Point: Path isolation
ECS Cluster: Container orchestration
ECS Task Definition: Container configuration with EFS mount
ECS Service: Auto-scaling and load balancing
CloudWatch Logs: Centralized logging

Deployment Process:

cd connectors-forge/codeless/jdbc
./deploy-to-aws.sh

What Happens:

Load configuration from aws-deployment.properties
Validate AWS credentials and parameters
Create VPC resources (if needed)
Create EFS infrastructure
Automatically upload JDBC driver to EFS
Create ECS cluster
Register task definition with EFS mount
Create and start ECS service
Configure CloudWatch logging
Display deployment summary with endpoints

Automatic Driver Upload:

Launches temporary EC2 instance
Mounts EFS file system
Uploads driver.jar via SCP
Verifies upload completion
Cleans up EC2 instance
Total time: ~3-5 minutes

OpenShift Deployment

Process:

cd connectors-forge/codeless/jdbc
./deploy-openshift.sh

Resources Created:

Project/Namespace
PersistentVolumeClaim (for driver storage)
Deployment
Service
Route (external access)

Self-Hosted Deployment

Process:

cd connectors-forge/codeless/jdbc
./deploy-container.sh

Requirements:

Docker installed on target server
SSH access configured
Driver JAR available locally

Implementation Details

DSL Configuration System

Purpose: Define connector behavior without code

Key Sections:

connector_metadata: Basic connector information
jdbc_driver: Driver class and URL template
connection_properties: Additional connection parameters
authentication: Supported auth methods (basic, Kerberos, OAuth)
database_specific_features: Schema/catalog support, system schemas
data_type_mappings: Custom type conversions
url_template: JDBC URL construction
validation_rules: Property validation logic
custom_actions: Database-specific operations
implementation_notes: Driver dependencies and customization notes

Driver Management

Automatic Download (Codeless Mode)

Supported Databases:

MySQL, PostgreSQL, MariaDB (open-source drivers)
H2, Derby, SQLite (embedded databases)

Process:

Extract Maven coordinates from DSL
Construct Maven Central URL
Download driver JAR using curl
Save to connectors-forge/codeless/jdbc/driver/driver.jar
Verify file integrity

Maven Central URL Pattern:

https://repo1.maven.org/maven2/{group_path}/{artifact}/{version}/{artifact}-{version}.jar

Manual Provision (Codeless Mode)

Required for:

Oracle (proprietary license)
IBM Db2 (proprietary license)
Microsoft SQL Server (with license)
Other proprietary databases

Process:

User provides local path to driver JAR
Mode copies JAR to connectors-forge/codeless/jdbc/driver/driver.jar
Verifies file exists and is readable

Developer Mode

Process:

Add Maven dependency to build.gradle
Gradle downloads driver during build
Driver included in final JAR

AWS Deployment Architecture

Complete Infrastructure Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                            AWS Cloud                                 │
│                                                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │                         VPC                                     │ │
│  │                                                                 │ │
│  │  ┌──────────────────────────────────────────────────────────┐ │ │
│  │  │                    Public Subnet                          │ │ │
│  │  │                                                           │ │ │
│  │  │  ┌─────────────────┐                                     │ │ │
│  │  │  │  NAT Gateway    │                                     │ │ │
│  │  │  └─────────────────┘                                     │ │ │
│  │  └──────────────────────────────────────────────────────────┘ │ │
│  │                           │                                    │ │
│  │  ┌────────────────────────┼─────────────────────────────────┐ │ │
│  │  │                 Private Subnet                            │ │ │
│  │  │                        │                                  │ │ │
│  │  │  ┌─────────────────────▼──────────────────────────────┐  │ │ │
│  │  │  │           ECS Cluster                              │  │ │ │
│  │  │  │                                                    │  │ │ │
│  │  │  │  ┌──────────────────────────────────────────────┐ │  │ │ │
│  │  │  │  │  ECS Service (Fargate)                       │ │  │ │ │
│  │  │  │  │                                              │ │  │ │ │
│  │  │  │  │  ┌────────────────┐  ┌────────────────┐    │ │  │ │ │
│  │  │  │  │  │  Task 1        │  │  Task 2        │    │ │  │ │ │
│  │  │  │  │  │  ┌──────────┐  │  │  ┌──────────┐  │    │ │  │ │ │
│  │  │  │  │  │  │Container │  │  │  │Container │  │    │ │  │ │ │
│  │  │  │  │  │  │          │  │  │  │          │  │    │ │  │ │ │
│  │  │  │  │  │  │/mnt/efs/ │◄─┼──┼──┤/mnt/efs/ │  │    │ │  │ │ │
│  │  │  │  │  │  └──────────┘  │  │  └──────────┘  │    │ │  │ │ │
│  │  │  │  │  └────────────────┘  └────────────────┘    │ │  │ │ │
│  │  │  │  └──────────────────────────────────────────────┘ │  │ │ │
│  │  │  └─────────────────────────────────────────────────────┘  │ │ │
│  │  │                        │                                   │ │ │
│  │  │                        │ EFS Mount (NFS)                   │ │ │
│  │  │                        │                                   │ │ │
│  │  │  ┌─────────────────────▼──────────────────────────────┐   │ │ │
│  │  │  │              AWS EFS File System                    │   │ │ │
│  │  │  │                                                     │   │ │ │
│  │  │  │  /jdbc-drivers/                                    │   │ │ │
│  │  │  │    └── driver.jar  ← Persistent Storage            │   │ │ │
│  │  │  │                                                     │   │ │ │
│  │  │  │  Features:                                          │   │ │ │
│  │  │  │  - Multi-AZ replication                            │   │ │ │
│  │  │  │  - Encryption in transit (TLS)                     │   │ │ │
│  │  │  │  - Access point isolation                          │   │ │ │
│  │  │  └─────────────────────────────────────────────────────┘   │ │ │
│  │  └──────────────────────────────────────────────────────────┘ │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                                                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │                    CloudWatch Logs                              │ │
│  │  /ecs/jdbc-connector                                           │ │
│  │    - Container logs                                            │ │
│  │    - Application logs                                          │ │
│  │    - Error tracking                                            │ │
│  └────────────────────────────────────────────────────────────────┘ │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

EFS Driver Upload Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Driver Upload Process                         │
│                                                                  │
│  Step 1: Launch Temporary EC2                                   │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  - Amazon Linux 2023                                        │ │
│  │  - t2.micro instance                                        │ │
│  │  - Same VPC/Subnet as ECS                                   │ │
│  │  - Temporary SSH key pair                                   │ │
│  └────────────────────────────────────────────────────────────┘ │
│                           │                                      │
│                           ▼                                      │
│  Step 2: Mount EFS                                              │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  sudo yum install -y amazon-efs-utils                      │ │
│  │  sudo mkdir -p /mnt/efs                                    │ │
│  │  sudo mount -t efs -o tls fs-xxxxx:/ /mnt/efs             │ │
│  └────────────────────────────────────────────────────────────┘ │
│                           │                                      │
│                           ▼                                      │
│  Step 3: Upload Driver via SCP                                  │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  scp -i key.pem driver.jar ec2-user@instance:/tmp/        │ │
│  │  sudo cp /tmp/driver.jar /mnt/efs/jdbc-drivers/           │ │
│  │  sudo chmod 644 /mnt/efs/jdbc-drivers/driver.jar          │ │
│  └────────────────────────────────────────────────────────────┘ │
│                           │                                      │
│                           ▼                                      │
│  Step 4: Verify and Cleanup                                     │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  - Verify file exists and is readable                      │ │
│  │  - Terminate EC2 instance                                  │ │
│  │  - Delete SSH key pair                                     │ │
│  │  - Remove temporary files                                  │ │
│  └────────────────────────────────────────────────────────────┘ │
│                                                                  │
│  Total Time: ~3-5 minutes                                       │
└─────────────────────────────────────────────────────────────────┘