feat: optimize S3 lifecycle management and modernize infrastructure configuration (#170)

* feat: add S3 lifecycle management for cost optimization and automated cleanup

This commit implements comprehensive S3 lifecycle rules for the nf-core-awsmegatests bucket to optimize storage costs and automatically clean up temporary workflow files.

Changes:
- Add create_s3_lifecycle_configuration() function with 4 lifecycle rules
- Rule 1: Preserve metadata files with cost optimization (IA after 30 days, Glacier after 90 days)
- Rule 2: Clean up temporary files after 30 days (based on nextflow.io/temporary tag)
- Rule 3: Clean up work directory after 90 days (prefix-based cleanup)
- Rule 4: Clean up incomplete multipart uploads after 7 days

The implementation includes proper error handling to gracefully fall back to manual management if AWS permissions are insufficient.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* refactor: remove all try/except statements to let Pulumi handle errors natively

This commit removes all manual try/except error handling blocks throughout the codebase, allowing Pulumi to handle errors using its built-in error management system.

Changes:
- Remove try/except from S3 lifecycle configuration creation
- Remove try/except from Seqera compute environment deployment
- Remove try/except from GitHub integration resource creation
- Remove try/except from Seqera provider initialization
- Remove try/except from TowerForge credential upload
- Remove try/except from configuration file loading
- Remove try/except from workspace ID validation
- Remove unused validate_environment() function
- Simplify numeric validation logic in settings

Pulumi's native error handling provides better diagnostics and stack traces than custom exception wrapping. This simplifies the codebase while maintaining robust error reporting.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* feat: optimize S3 bucket lifecycle and enable CORS for Seqera Data Explorer

- Update work directory cleanup from 90 days to 14 days for aggressive cost optimization
- Add lifecycle rules for scratch/ (7 days) and cache/ directories (30 days)
- Enable CORS configuration for Seqera Data Explorer compatibility
- Add one-time batch job script for tagging existing log files
- Preserve tagged log files while aggressively cleaning untagged work files

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* fix: adjust S3 lifecycle cleanup to be less aggressive

- Update work directory cleanup from 14 days to 30 days
- Update scratch directory cleanup from 7 days to 30 days
- Maintain 30-day cleanup for cache directories
- Keep tagged log files preserved for 90 days with storage transitions

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* refactor: externalize nextflow configuration to separate files

- Move embedded nextflowConfig strings from JSON to external .config files
- Create modular nextflow configs with base + environment-specific settings
- Add load_nextflow_config() function to read external config files
- Update compute environment creation to use external nextflow configs
- Clean JSON files by removing embedded nextflowConfig fields
- Improve maintainability and readability of nextflow configurations

Config structure:
- nextflow-base.config: Common settings (AWS Batch, error handling, fusion tags)
- nextflow-cpu.config: CPU-specific settings (x86_64, CPU tags)
- nextflow-gpu.config: GPU-specific settings (x86_64, GPU tags)
- nextflow-arm.config: ARM-specific settings (arm64, ARM tags)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* chore: update Seqera Terraform provider version to 0.25.2

- Update provider version from 0.13.0 to 0.25.2
- Attempt to resolve pulumi_seqera module import issues
- Provider configuration ready but SDK generation still needs resolution

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* feat: generate Seqera Terraform provider SDK and verify infrastructure changes

- Generate pulumi-seqera SDK from terraform-provider registry.terraform.io/seqeralabs/seqera
- Add pulumi-seqera dependency with local SDK path configuration
- Update package dependencies and lock file with generated SDK
- Verify infrastructure changes work correctly with pulumi preview:
  * S3 CORS configuration for Seqera Data Explorer ✅
  * S3 lifecycle optimization with 30-day cleanup ✅
  * External nextflow config files integration ✅
  * Compute environment replacements with updated configs ✅

All infrastructure optimizations tested and ready for deployment.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* fix: correct Seqera provider version to 0.25.2

- Fix provider version that was incorrectly reverted to 0.13.0
- Ensure we're using the latest Seqera provider version 0.25.2 as intended
- SDK generation accidentally used older version, now corrected

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* fix: correct S3 CORS configuration to comply with AWS and Seqera requirements

- Remove unsupported x-amz-meta-* wildcard from ExposeHeaders
- Simplify ExposeHeaders to only include ETag per Seqera documentation
- Update documentation link to correct Seqera Data Explorer CORS guide

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* fix: add missing EC2 permissions to TowerForge IAM policy

Add ec2:DescribeAccountAttributes, ec2:DescribeLaunchTemplateVersions,
and ec2:DescribeInstanceTypeOfferings permissions to align with the
official Seqera forge policy requirements. This resolves 403 forbidden
errors when Seqera Platform attempts to describe AWS account attributes.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* test: add IAM policy compliance validation against Seqera reference

Add comprehensive unit tests to validate TowerForge IAM policy includes
all required permissions from the official Seqera forge policy. The test:

- Fetches reference policy from seqeralabs/nf-tower-aws repository
- Compares our policy permissions against the reference
- Validates critical EC2 permissions are present
- Ensures proper policy structure

Also adds missing EFS permissions (elasticfilesystem:*) and iam:GetRole
permission that were in the reference policy but missing from ours.

Includes TODO comment for implementing Pulumi CrossGuard policy validation
for automated compliance checking at deployment time.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* feat: add explicit compute environment dependencies and IAM policy change detection

Implement proper dependency management and prevent authorization errors by:

- Add explicit dependencies between IAM resources → Seqera credentials → compute environments
- Generate IAM policy hash to force compute environment recreation on policy changes
- Embed policy version hash in CE descriptions to trigger replacement when policies update
- Pass Seqera credential resource for explicit dependency tracking

This ensures compute environments are always created with fully propagated IAM permissions,
preventing "not authorized" errors during resource creation when policies are updated.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* feat: implement Python file injection for Nextflow config merging

Replace includeConfig statements with programmatic config merging to resolve
Seqera Platform compatibility issues. Add comprehensive test suite with 10 test cases.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

---------

Co-authored-by: Claude <[email protected]>

Author: edmundmiller

pulumi/AWSMegatests/Pulumi.yaml

@@ -9,11 +9,11 @@ config:
   # GitHub Provider Configuration
   github:owner:
     value: nf-core
-  # Note: AWS and other tokens are provided via ESC environment
-  # AWS credentials should come from ESC OIDC integration
+    # Note: AWS and other tokens are provided via ESC environment
+    # AWS credentials should come from ESC OIDC integration
 packages:
   seqera:
     source: terraform-provider
-    version: 0.13.0
+    version: 0.25.2
     parameters:
       - registry.terraform.io/seqeralabs/seqera

pulumi/AWSMegatests/__main__.py

@@ -38,78 +38,41 @@ def main():
     # Note: lifecycle_configuration is managed manually, not used in exports
 
     # Step 5: Create TowerForge IAM credentials and upload to Seqera Platform
-    towerforge_access_key_id, towerforge_access_key_secret, seqera_credentials_id = (
-        create_towerforge_credentials(
-            aws_provider,
-            nf_core_awsmegatests_bucket,
-            seqera_provider,
-            float(config["tower_workspace_id"]),
-        )
+    (
+        towerforge_access_key_id,
+        towerforge_access_key_secret,
+        seqera_credentials_id,
+        seqera_credential_resource,
+        iam_policy_hash,
+    ) = create_towerforge_credentials(
+        aws_provider,
+        nf_core_awsmegatests_bucket,
+        seqera_provider,
+        float(config["tower_workspace_id"]),
     )
 
     # Step 6: Deploy Seqera Platform compute environments using Terraform provider
-    try:
-        pulumi.log.info(
-            "Deploying Seqera compute environments using Terraform provider"
-        )
-
-        # Deploy using Seqera Terraform provider with dynamic credentials ID
-        terraform_resources = deploy_seqera_environments_terraform(
-            config,
-            seqera_credentials_id,  # Dynamic TowerForge credentials ID from Seqera Platform
-            seqera_provider,  # Reuse existing Seqera provider
-        )
-
-        # Get compute environment IDs from Terraform provider
-        compute_env_ids = get_compute_environment_ids_terraform(terraform_resources)
-        deployment_method = "terraform-provider"
-
-        pulumi.log.info(
-            "Successfully deployed compute environments using Seqera Terraform provider"
-        )
-    except Exception as e:
-        error_msg = (
-            f"Seqera deployment failed: {e}. "
-            "Common solutions: "
-            "1. Verify TOWER_ACCESS_TOKEN has WORKSPACE_ADMIN permissions "
-            "2. Check workspace ID is correct in ESC environment "
-            "3. Ensure TowerForge credentials were successfully uploaded to Seqera Platform "
-            "4. Verify network connectivity to api.cloud.seqera.io"
-        )
-        pulumi.log.error(error_msg)
-        raise RuntimeError(error_msg)
+    # Deploy using Seqera Terraform provider with dynamic credentials ID
+    terraform_resources = deploy_seqera_environments_terraform(
+        config,
+        seqera_credentials_id,  # Dynamic TowerForge credentials ID from Seqera Platform
+        seqera_provider,  # Reuse existing Seqera provider
+        seqera_credential_resource,  # Seqera credential resource for dependency
+        iam_policy_hash,  # IAM policy hash to force CE recreation on policy changes
+    )
+
+    # Get compute environment IDs from Terraform provider
+    compute_env_ids = get_compute_environment_ids_terraform(terraform_resources)
+    deployment_method = "terraform-provider"
 
     # Step 8: Create GitHub resources
     # Full GitHub integration enabled - creates both variables and secrets
-    try:
-        pulumi.log.info("Creating GitHub organization variables and secrets")
-
-        github_resources = create_github_resources(
-            github_provider,
-            compute_env_ids,
-            config["tower_workspace_id"],
-            tower_access_token=config["tower_access_token"],
-        )
-
-        pulumi.log.info(
-            "Successfully created GitHub variables. Manual secret commands available in outputs."
-        )
-    except Exception as e:
-        error_msg = (
-            f"GitHub integration failed: {e}. "
-            "This is often harmless if variables already exist (409 errors). "
-            "Common issues: "
-            "1. GitHub token lacks org-level permissions "
-            "2. Variables already exist (409 Already Exists - harmless) "
-            "3. Network connectivity to api.github.com"
-        )
-        pulumi.log.warn(error_msg)
-        github_resources = {
-            "variables": {},
-            "secrets": {},
-            "gh_cli_commands": [],
-            "note": f"GitHub integration failed: {e}",
-        }
+    github_resources = create_github_resources(
+        github_provider,
+        compute_env_ids,
+        config["tower_workspace_id"],
+        tower_access_token=config["tower_access_token"],
+    )
 
     # Exports - All within proper Pulumi program context
     pulumi.export(

pulumi/AWSMegatests/pyproject.toml

@@ -12,11 +12,12 @@ dependencies = [
     "pulumi-seqera",
 ] 
 
-[tool.uv.sources]
-pulumi-seqera = { path = "sdks/seqera" }
-
-
 [dependency-groups]
 dev = [
     "mypy>=1.17.1",
+    "pytest>=7.0.0",
+    "requests>=2.28.0",
 ]
+
+[tool.uv.sources]
+pulumi-seqera = { path = "sdks/seqera" }

pulumi/AWSMegatests/scripts/README.md

@@ -0,0 +1,59 @@
+# AWS Megatests Utility Scripts
+
+This directory contains utility scripts for AWS Megatests infrastructure management.
+
+## Log File Tagging Script
+
+**Purpose**: One-time batch job to tag existing log files in S3 work directories for preservation during lifecycle cleanup.
+
+### Usage
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Dry run to see what would be tagged
+python tag_existing_log_files.py --bucket nf-core-awsmegatests --dry-run
+
+# Actually tag the files
+python tag_existing_log_files.py --bucket nf-core-awsmegatests
+
+# With custom threading (default is 10 workers)
+python tag_existing_log_files.py --bucket nf-core-awsmegatests --max-workers 20
+```
+
+### Prerequisites
+
+- AWS credentials configured (AWS CLI, IAM role, or environment variables)
+- S3 permissions: `s3:ListBucket`, `s3:GetObjectTagging`, `s3:PutObjectTagging`
+
+### What it does
+
+1. Scans the `work/` directory for log files matching Nextflow patterns
+2. Tags log files with `nextflow.io/metadata=true`
+3. Preserves existing tags while adding the metadata tag
+4. Uses multi-threading for performance with large buckets
+
+### Log File Patterns
+
+The script identifies these Nextflow log files:
+
+- `.command.log` - Main command log
+- `.command.err` - Error log
+- `.command.out` - Standard output
+- `.exitcode` - Exit code file
+- `.command.sh` - Command script
+- `.command.run` - Run script
+- `.command.begin` - Begin timestamp
+- `trace.txt` - Trace file
+- `timeline.html` - Timeline report
+- `report.html` - Execution report
+- `dag.html` - DAG visualization
+
+### Integration with Lifecycle Rules
+
+Tagged files are preserved by S3 lifecycle rules:
+
+- Tagged log files: Kept for 90 days, then moved to cheaper storage classes
+- Untagged work files: Deleted after 14 days for aggressive cleanup
+- Future log files will be tagged automatically by Nextflow (no need to run this script again)

pulumi/AWSMegatests/scripts/requirements.txt

@@ -0,0 +1,3 @@
+# Requirements for log file tagging script
+boto3>=1.26.0
+botocore>=1.29.0