SNOW-2235287 : Update RunInstance info like nodetype, requested min/max count, provvide count etc to snowhouse

EC2_MONITORING.md

@@ -0,0 +1,89 @@
+# EC2 Provisioning Monitoring
+
+This feature provides real-time monitoring for AWS EC2 node provisioning issues by integrating with Snowhouse database using JDBC.
+
+## Overview
+
+The monitoring system captures detailed information about EC2 provisioning attempts and sends them to a Snowhouse database for real-time analysis and alerting. This helps identify provisioning issues faster than relying on application logs and CloudTrail.
+
+## Data Captured
+
+For each provisioning attempt, the following information is recorded:
+
+- **region**: AWS region where provisioning was attempted
+- **availability_zone**: Specific AZ within the region
+- **request_id**: Unique identifier for the provisioning request
+- **requested_instance_type**: EC2 instance type requested (e.g., m5.large)
+- **requested_max_count**: Maximum number of instances requested
+- **requested_min_count**: Minimum number of instances requested
+- **provisioned_instances_count**: Actual number of instances provisioned
+- **controller_name**: Name of the Jenkins controller
+- **timestamp**: When the event occurred
+- **phase**: Event phase (REQUEST, SUCCESS, FAILURE, REQUEST_FALLBACK, SUCCESS_FALLBACK)
+- **error_message**: Error details if provisioning failed
+- **jenkins_url**: URL of the Jenkins instance
+
+## Database Schema
+
+The monitoring system creates the following table in Snowhouse:
+
+```sql
+CREATE TABLE IF NOT EXISTS EC2_PROVISIONING_EVENTS (
+    ID NUMBER AUTOINCREMENT,
+    CREATE_TIME TIMESTAMP_NTZ,
+    REGION VARCHAR(50),
+    AVAILABILITY_ZONE VARCHAR(50),
+    REQUEST_ID VARCHAR(100),
+    REQUESTED_INSTANCE_TYPE VARCHAR(50),
+    REQUESTED_MAX_COUNT NUMBER,
+    REQUESTED_MIN_COUNT NUMBER,
+    PROVISIONED_INSTANCES_COUNT NUMBER,
+    CONTROLLER_NAME VARCHAR(200),
+    PHASE VARCHAR(50),
+    ERROR_MESSAGE VARCHAR(2000),
+    JENKINS_URL VARCHAR(500),
+    EVENT_DATA VARIANT,
+    PRIMARY KEY (ID)
+);
+```
+
+## Configuration
+
+1. **Install Database Plugin**: The monitoring requires the Jenkins Database plugin to be installed.
+
+2. **Configure Snowhouse Database**: In Jenkins system configuration, add a new Snowflake database connection with:
+   - Account Name: Your Snowflake account
+   - Database: Target database name
+   - Warehouse: Snowflake warehouse to use
+   - Credentials: Username/password credentials for Snowflake
+   - Timeouts: Network, query, and login timeouts
+
+3. **Set as Global Database**: Configure the Snowflake connection as the global database in Jenkins.
+
+## Event Flow
+
+1. **Provisioning Request**: When Jenkins attempts to provision EC2 instances, a "REQUEST" event is recorded
+2. **Success/Failure**: Based on the AWS API response, either "SUCCESS" or "FAILURE" events are recorded
+3. **Fallback Scenarios**: For spot instances that fall back to on-demand, additional "REQUEST_FALLBACK" and "SUCCESS_FALLBACK" events are recorded
+4. **Batch Processing**: Events are queued and sent to Snowhouse in batches every 30 seconds
+
+## Monitoring Points
+
+The system monitors both:
+
+- **On-demand instances**: Direct EC2 instance provisioning
+- **Spot instances**: Spot instance requests and their fallback scenarios
+
+## Benefits
+
+- **Real-time alerting**: Immediate visibility into provisioning issues
+- **Trend analysis**: Historical data for capacity planning
+- **Failure investigation**: Detailed error information for troubleshooting
+- **Performance monitoring**: Track provisioning success rates and response times
+
+## Implementation Details
+
+- **Non-blocking**: Event recording doesn't impact provisioning performance
+- **Fault-tolerant**: Monitoring failures don't affect EC2 provisioning
+- **Scalable**: Batch processing handles high-volume environments
+- **Configurable**: Database connection is configurable through Jenkins UI 

README.md

@@ -364,7 +364,7 @@ def slaveTemplateUsEast1Parameters = [
 ]
 
 def EC2CloudParameters = [
-  name:      'MyCompany',
+  cloudName:      'MyCompany',
   credentialsId:  'jenkins-aws-key',
   instanceCapStr: '2',
   privateKey:     '''-----BEGIN RSA PRIVATE KEY-----
@@ -465,7 +465,7 @@ SlaveTemplate slaveTemplateUsEast1 = new SlaveTemplate(
 
 // https://javadoc.jenkins.io/plugin/ec2/hudson/plugins/ec2/EC2Cloud.html
 EC2Cloud ec2Cloud = new EC2Cloud(
-  EC2CloudParameters.name,
+  EC2CloudParameters.cloudName,
   EC2CloudParameters.useInstanceProfileForCredentials,
   EC2CloudParameters.credentialsId,
   EC2CloudParameters.region,

jenkins_build.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+# auth setup
+eval $(sf artifact maven auth)
+
+ENV="sandbox"
+
+if [ -n "${JENKINS_ENVIRONMENT}" ]; then
+    ENV="${JENKINS_ENVIRONMENT}"
+fi
+
+# Generate version depending on the env
+generate_version() {
+    local git_sha=$(git rev-parse --short=7 HEAD)
+
+    local timestamp=$(date +"%Y%m%d-%H%M%S")
+
+    local version="${timestamp}.${git_sha}"
+
+    # build num not set during sandbox dev generally
+    if [ -n "${BUILD_NUMBER}" ]; then
+        version="${version}.${BUILD_NUMBER}"
+    fi
+
+    echo "${version}"
+}
+
+DYNAMIC_VERSION=$(generate_version)
+echo "Generated version: ${DYNAMIC_VERSION}"
+
+# build & deploy
+mvn clean deploy -P "${ENV}" -Dchangelist="${DYNAMIC_VERSION}"

pom.xml

@@ -72,13 +72,26 @@ THE SOFTWARE.
     <url>https://github.com/${gitHubRepo}</url>
   </scm>
 
+  <distributionManagement>
+    <repository>
+      <id>snowflake-jenkins-plugins</id>
+      <name>Snowflake Jenkins Plugins Repository</name>
+      <url>${deploy.repo.url}</url>
+    </repository>
+    <snapshotRepository>
+      <id>snowflake-jenkins-plugins</id>
+      <name>Snowflake Jenkins Plugins Repository</name>
+      <url>${deploy.repo.url}</url>
+    </snapshotRepository>
+  </distributionManagement>
+
   <properties>
-    <changelist>999999-SNAPSHOT</changelist>
+    <!-- Default changelist for local development - will be overridden by build script -->
+    <changelist>20250819-nthirumoorthy-aws-monitoring-info</changelist>
     <!-- https://www.jenkins.io/doc/developer/plugin-development/choosing-jenkins-baseline/ -->
     <jenkins.baseline>2.479</jenkins.baseline>
     <jenkins.version>${jenkins.baseline}.3</jenkins.version>
     <gitHubRepo>jenkinsci/${project.artifactId}-plugin</gitHubRepo>
-    <hpi.compatibleSinceVersion>1885</hpi.compatibleSinceVersion>
     <spotless.check.skip>false</spotless.check.skip>
   </properties>
 
@@ -158,6 +171,20 @@ THE SOFTWARE.
       <groupId>org.jenkins-ci.plugins.workflow</groupId>
       <artifactId>workflow-step-api</artifactId>
     </dependency>
+    <dependency>
+      <groupId>net.snowflake</groupId>
+      <artifactId>snowflake-jdbc</artifactId>
+      <version>3.10.1</version>
+    </dependency>
+    <dependency>
+      <groupId>org.jenkins-ci.plugins</groupId>
+      <artifactId>database</artifactId>
+      <version>274.vea_2e859b_2661</version>
+    </dependency>
+    <dependency>
+      <groupId>io.jenkins.plugins</groupId>
+      <artifactId>json-api</artifactId>
+    </dependency>
     <dependency>
       <groupId>io.jenkins</groupId>
       <artifactId>configuration-as-code</artifactId>
@@ -208,16 +235,56 @@ THE SOFTWARE.
   </dependencies>
 
   <repositories>
+    <!-- Central repository redirected to artifactory -->
+    <repository>
+      <releases>
+        <enabled>true</enabled>
+      </releases>
+      <snapshots>
+        <enabled>false</enabled>
+      </snapshots>
+      <id>central</id>
+      <name>Maven Central</name>
+      <url>https://artifactory.ci1.us-west-2.aws-dev.app.snowflake.com/artifactory/development-maven-virtual</url>
+    </repository>
+    <!-- Jenkins CI repository -->
     <repository>
+      <releases>
+        <enabled>true</enabled>
+      </releases>
+      <snapshots>
+        <enabled>true</enabled>
+      </snapshots>
       <id>repo.jenkins-ci.org</id>
-      <url>https://repo.jenkins-ci.org/public/</url>
+      <name>Jenkins CI</name>
+      <url>https://artifactory.ci1.us-west-2.aws-dev.app.snowflake.com/artifactory/development-jenkins-ci-virtual</url>
     </repository>
   </repositories>
 
   <pluginRepositories>
+    <!-- Central plugin repository -->
     <pluginRepository>
+      <releases>
+        <enabled>true</enabled>
+      </releases>
+      <snapshots>
+        <enabled>false</enabled>
+      </snapshots>
+      <id>central</id>
+      <name>Maven Central Plugins</name>
+      <url>https://artifactory.ci1.us-west-2.aws-dev.app.snowflake.com/artifactory/development-maven-virtual</url>
+    </pluginRepository>
+    <!-- Jenkins CI plugin repository -->
+    <pluginRepository>
+      <releases>
+        <enabled>true</enabled>
+      </releases>
+      <snapshots>
+        <enabled>true</enabled>
+      </snapshots>
       <id>repo.jenkins-ci.org</id>
-      <url>https://repo.jenkins-ci.org/public/</url>
+      <name>Jenkins CI Plugins</name>
+      <url>https://artifactory.ci1.us-west-2.aws-dev.app.snowflake.com/artifactory/development-jenkins-ci-virtual</url>
     </pluginRepository>
   </pluginRepositories>
 
@@ -233,4 +300,25 @@ THE SOFTWARE.
       </plugin>
     </plugins>
   </build>
+
+  <profiles>
+    <profile>
+      <id>prod</id>
+      <properties>
+        <deploy.repo.url>https://artifactory.ci1.us-west-2.aws-dev.app.snowflake.com/artifactory/internal-production-maven-jenkins_plugins-local/</deploy.repo.url>
+      </properties>
+    </profile>
+    <profile>
+      <id>dev</id>
+      <properties>
+        <deploy.repo.url>https://artifactory.ci1.us-west-2.aws-dev.app.snowflake.com/artifactory/internal-development-maven-jenkins_plugins-local/</deploy.repo.url>
+      </properties>
+    </profile>
+    <profile>
+      <id>sandbox</id>
+      <properties>
+        <deploy.repo.url>https://artifactory.ci1.us-west-2.aws-dev.app.snowflake.com/artifactory/internal-sandbox-maven-jenkins_plugins-local/</deploy.repo.url>
+      </properties>
+    </profile>
+  </profiles>
 </project>