Fast DS impl

.pre-commit-config.yaml

@@ -8,7 +8,7 @@ ci:
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v5.0.0
     hooks:
       # list of supported hooks: https://pre-commit.com/hooks.html
       - id: trailing-whitespace
@@ -26,7 +26,7 @@ repos:
           )$
 
   - repo: https://github.com/python-poetry/poetry
-    rev: "1.5.0"
+    rev: "1.7.1"
     hooks:
       - id: poetry-check
       # too slow
@@ -35,29 +35,29 @@ repos:
         args: ["-f", "requirements.txt", "-o", "requirements.txt"]
 
   - repo: https://github.com/asottile/pyupgrade
-    rev: v3.15.0
+    rev: v3.20.0
     hooks:
       - id: pyupgrade
         args: [--py312-plus]
         name: Upgrade code
 
   # python formatting
   - repo: https://github.com/psf/black
-    rev: 23.3.0
+    rev: 25.1.0
     hooks:
       - id: black
         name: Format code
         args: ["--line-length=100"]
 
   - repo: https://github.com/hadialqattan/pycln
-    rev: v2.1.5 # Possible releases: https://github.com/hadialqattan/pycln/releases
+    rev: v2.5.0 # Possible releases: https://github.com/hadialqattan/pycln/releases
     hooks:
       - id: pycln
         args: [--all]
 
   # ref: https://github.com/microsoft/vscode-isort]
   - repo: https://github.com/pycqa/isort
-    rev: 5.12.0
+    rev: 6.0.1
     hooks:
       - id: isort
         name: isort (python)
@@ -72,14 +72,14 @@ repos:
 
   # yaml formatting
   - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v3.0.0-alpha.9-for-vscode
+    rev: v4.0.0-alpha.8
     hooks:
       - id: prettier
         types: [yaml]
 
   # markdown formatting
   - repo: https://github.com/executablebooks/mdformat
-    rev: 0.7.16
+    rev: 0.7.22
     hooks:
       - id: mdformat
         additional_dependencies:

README.md

@@ -21,7 +21,9 @@ Make sure to register for an API key and validate licences at:
 
 Before proceeding further, it is required to accept all the licenses in the section "Your profile" in the website of [Copernicus](https://cds.climate.copernicus.eu/profile?tab=licences).
 
-### Passing the key as environment variable
+### CDS/ADS credentials
+
+#### Passing the key as environment variable
 
 This is the easiest method. Use:
 
@@ -30,7 +32,7 @@ import os
 os.environ["CDSADS_API_KEY"] = "YOUR_KEY"
 ```
 
-### Installing the API key in your filesytem
+#### Installing the API key in your filesytem
 
 Then create the file `~/.cdsapirc` with the following content:
 
@@ -42,6 +44,33 @@ key: <your_api_key>
 Note: the URL will be dynamically managed by the script depending on the data source.
 The API key doesn't vary, it's the same for both ERA5 and CAMS data.
 
+### Earth Data Hub credentials (optional)
+
+If you want to use the [Earth Data Hub](https://earthdatahub.destine.eu/) (EDH) as a data source instead of CDS, you need to set up your EDH personal access token.
+
+> **Note:** Earth Data Hub has a monthly quota of 500,000 requests per user. Data is available until the last closed month.
+
+#### Passing the token as environment variable
+
+```python
+import os
+os.environ["EDH_TOKEN"] = "edh_pat_..."
+```
+
+Or in the shell:
+
+```bash
+export EDH_TOKEN="edh_pat_..."
+```
+
+#### Installing the token in your filesystem
+
+Create the file `~/.edh_token` with your personal access token (plain text, first line):
+
+```
+edh_pat_...
+```
+
 ## Install the package
 
 ### From PyPI
@@ -61,7 +90,7 @@ poetry install
 
 # Usage
 
-> \[!NOTE\]
+> [!NOTE]
 > When running in a Jupyter notebook, to make progress bars and interactive widgets work, make sure to install `ipywidgets` and to enable the widgets extension.
 
 ```bash
@@ -82,6 +111,9 @@ poetry run era5epw_download --year 2024 --latitude 49.4 --longitude 0.1 --city-n
 
 # using installed binary, after pypi package installation
 era5epw_download --year 2024 --latitude 49.4 --longitude 0.1 --city-name "Le Havre" --elevation 0 --time-zone 1
+
+# using Earth Data Hub as data source (faster downloads, requires EDH token)
+era5epw_download --year 2024 --latitude 49.4 --longitude 0.1 --city-name "Le Havre" --elevation 0 --time-zone 1 --era5-data-source edh
 ```
 
 By default, the `time-zone` argument is used only to populate the `LOCATION` header and data time is UTC. Use `--apply-time-zone-to-data` to apply it to the date and time fields (this will shift the UTC time by the provided time zone offset).
@@ -105,6 +137,42 @@ download_and_make_epw(
     output_file="/tmp/era5epw_paris_2025.epw",
     apply_time_zone_to_data=True,
 )
+
+# Or using Earth Data Hub as data source (faster downloads):
+download_and_make_epw(
+    year=2025,
+    latitude=48.8,
+    longitude=2.4,
+    city_name="Paris",
+    time_zone=1,
+    elevation=0,
+    output_file="/tmp/era5epw_paris_2025.epw",
+    apply_time_zone_to_data=True,
+    era5_data_source="edh",
+)
+```
+
+## Reading EPW Files into a DataFrame
+
+The package provides a reader to load EPW files into Pandas DataFrames for inspection, data analysis, and further processing.
+
+### Python API
+
+```python
+from era5epw.reader import read_epw_file
+
+# Load EPW file into a DataFrame
+df = read_epw_file("path/to/file.epw")
+
+# Inspect the data
+print(df.head())
+print(df.describe())
+
+# Access specific weather series
+print(df["Dry Bulb Temperature"].mean())
+
+# Filter by date
+january_data = df[df.index.month == 1]
 ```
 
 ## Visualizing EPW Files
@@ -180,6 +248,7 @@ Datasets home pages:
 - [ERA5 Land hourly time-series data from 1950 to present](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-land-timeseries) (Experimental)
 - [ERA5 hourly data on single levels from 1940 to present](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels)
 - [CAMS solar radiation time-series](https://ads.atmosphere.copernicus.eu/datasets/cams-solar-radiation-timeseries)
+- [Earth Data Hub - ERA5 reanalysis single levels](https://earthdatahub.destine.eu/collections/era5/datasets/reanalysis-era5-single-levels) (License: [Copernicus License](https://earthdatahub.destine.eu/collections/era5/datasets/reanalysis-era5-single-levels))
 
 View your API requests and download responses at:
 

era5epw/ads.py

@@ -1,4 +1,5 @@
 import tempfile
+from typing import Any
 
 import pandas as pd
 import xarray as xr
@@ -19,7 +20,7 @@ def make_cams_solar_radiation_request(
     time_step: str = "1hour",
     time_reference: str = "universal_time",
     time_zone: int | None = None,
-) -> dict[str, any] | None:
+) -> dict[str, Any] | None:
     assert sky_type in [
         "clear",
         "observed_cloud",

era5epw/cds.py

@@ -1,6 +1,7 @@
 import os
 from multiprocessing import Pool
 from tempfile import TemporaryDirectory
+from typing import Any
 
 import pandas as pd
 from tqdm.auto import tqdm
@@ -30,23 +31,25 @@
         "10m_u_component_of_wind",
         "10m_v_component_of_wind",
         "surface_pressure",
+        "total_cloud_cover",
     ],
     "reanalysis-era5-land-timeseries": [
         "soil_temperature_level_1",
+        "snow_depth",
     ],
     "reanalysis-era5-single-levels": ["*"],
 }
 
 
 def make_cds_request(
     ds: str | None,
-    variables: [str],
+    variables: list[str],
     year: int,
     month: int | None,
     latitude: float,
     longitude: float,
     time_zone: int | None = None,
-) -> list[dict[str, any]] | None:
+) -> list[dict[str, Any]] | None:
     """Create a CDS request for the specified parameters.
 
     :param ds: The dataset to use, e.g., 'reanalysis-era5-single-levels-timeseries'.
@@ -179,7 +182,7 @@ def make_cds_request(
         )
 
 
-def make_intermediate_file_names(tmpdir: str, cds_requests: list[dict[str, any]]) -> list[str]:
+def make_intermediate_file_names(tmpdir: str, cds_requests: list[dict[str, Any]]) -> list[str]:
     """Generate a list of temporary file names for storing intermediate results of CDS requests.
 
     :param tmpdir: Temporary directory where intermediate files will be stored.
@@ -227,7 +230,7 @@ def make_intermediate_file_names(tmpdir: str, cds_requests: list[dict[str, any]]
 
 
 def download_era5_data(
-    variables: [str],
+    variables: list[str],
     year: int,
     latitude: float,
     longitude: float,