Era5 pipeline

[ludolara]

Summary

This PR adds a config-driven ERA5/MODIS acquisition pipeline for generating EcoPerceiver-compatible SQLite datasets and ERA5-style NetCDF outputs.

The new pipeline consolidates ERA5 download, ERA5 processing, MODIS download, IGBP assignment, MODIS processing, indexing, and shard post-processing behind a unified era5_pipeline workflow.

What Changed

• Added a YAML-configured era5_pipeline workflow with CLI orchestration.
• Added a SLURM entrypoint for running the pipeline on cluster infrastructure.
• Added ERA5 CDS acquisition and NetCDF-to-SQLite processing.
• Migrated MODIS download, IGBP assignment, MODIS processing, and ERA5 indexing scripts into the unified pipeline.
• Updated ERA5 shard merging to emit ERA5-like NetCDF cubes with:
  • valid_time
  • latitude/longitude coordinates
  • metadata
  • compression
  • chunking
• Removed older standalone MODIS pipeline scripts now covered by era5_pipeline.
• Added plotting/verifier utilities for checking generated outputs.

Validation

• Ran ERA5 processing using the 2016-2017 configuration.
• Verified the full end-to-end flow across:
  • ERA5 acquisition
  • ERA5 processing
  • ERA5 indexing
  • MODIS acquisition
  • MODIS processing
  • post-processing into yearly ERA5-style NetCDF shards

Notes

The main review areas are the new pipeline orchestration, ERA5 NetCDF/SQLite conversion logic, shard post-processing behavior, and the migration of legacy MODIS scripts into the unified era5_pipeline layout.