Failure to read HLS data when dask client is explicitly run simultaneously

[pritamd47]

Environment Information

• easysnowdata version: 0.0.4
• Python version: 3.12.3
• Operating System: Linux

Description

I tried to download Harmonized Landsat-Sentinel (HLS) data which was failing while reading the data. I get an error something like this for all the bands I am trying to access. -

Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.B8A.tif:1

Notice the :1 at the end, which is getting added after the link making it an invalid link. Without it the link points to the correct dataset. I get this error consistently when I have explicitly started a dask cluster.

What I Did

Code to reproduce:

from easysnowdata.remote_sensing import HLS
from dask.distributed import Client, LocalCluster
import pandas as pd


cluster = LocalCluster(n_workers=4)
client = Client()

start_date = pd.to_datetime('2019-01-10').tz_localize('UTC')
end_date = pd.to_datetime('2019-01-17').tz_localize('UTC')

ROI = (91.81, 23.44, 91.90, 23.55)

ds = HLS(
    bbox_input=ROI, start_date=start_date, end_date=end_date,
    bands=[
        'blue', 'green', 'red', 'nir narrow', 'swir 1', 'swir 2', 'Fmask'
    ]
)
data = ds.data
data = data.drop_vars(['AssociatedBrowseImageUrls', 'geometry'])

data.to_netcdf(f'./reservoir.nc')

Leads to the following errors:

Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCM.2019016T041804.v2.0/HLS.L30.T46QCM.2019016T041804.v2.0.Fmask.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019012T043141.v2.0/HLS.S30.T46QCM.2019012T043141.v2.0.B8A.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCM.2019016T041804.v2.0/HLS.L30.T46QCM.2019016T041804.v2.0.B02.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCM.2019016T041804.v2.0/HLS.L30.T46QCM.2019016T041804.v2.0.B04.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCM.2019016T041804.v2.0/HLS.L30.T46QCM.2019016T041804.v2.0.B07.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019012T043141.v2.0/HLS.S30.T46QCM.2019012T043141.v2.0.B02.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCM.2019016T041804.v2.0/HLS.L30.T46QCM.2019016T041804.v2.0.B03.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019012T043141.v2.0/HLS.S30.T46QCM.2019012T043141.v2.0.Fmask.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCL.2019016T041828.v2.0/HLS.L30.T46QCL.2019016T041828.v2.0.Fmask.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019012T043141.v2.0/HLS.S30.T46QCL.2019012T043141.v2.0.B8A.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCL.2019016T041828.v2.0/HLS.L30.T46QCL.2019016T041828.v2.0.B04.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCL.2019016T041828.v2.0/HLS.L30.T46QCL.2019016T041828.v2.0.B02.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCL.2019016T041828.v2.0/HLS.L30.T46QCL.2019016T041828.v2.0.B07.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019012T043141.v2.0/HLS.S30.T46QCL.2019012T043141.v2.0.Fmask.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019012T043141.v2.0/HLS.S30.T46QCL.2019012T043141.v2.0.B02.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCL.2019016T041828.v2.0/HLS.L30.T46QCL.2019016T041828.v2.0.B03.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019014T042129.v2.0/HLS.S30.T46QCM.2019014T042129.v2.0.Fmask.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019014T042129.v2.0/HLS.S30.T46QCM.2019014T042129.v2.0.B02.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019014T042129.v2.0/HLS.S30.T46QCM.2019014T042129.v2.0.B12.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCM.2019016T041804.v2.0/HLS.L30.T46QCM.2019016T041804.v2.0.B05.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019014T042129.v2.0/HLS.S30.T46QCM.2019014T042129.v2.0.B04.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019014T042129.v2.0/HLS.S30.T46QCM.2019014T042129.v2.0.B03.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019012T043141.v2.0/HLS.S30.T46QCM.2019012T043141.v2.0.B03.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019012T043141.v2.0/HLS.S30.T46QCM.2019012T043141.v2.0.B11.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.B02.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.B04.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.B03.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.B12.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019012T043141.v2.0/HLS.S30.T46QCL.2019012T043141.v2.0.B03.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.Fmask.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019012T043141.v2.0/HLS.S30.T46QCL.2019012T043141.v2.0.B11.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCL.2019016T041828.v2.0/HLS.L30.T46QCL.2019016T041828.v2.0.B05.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019012T043141.v2.0/HLS.S30.T46QCM.2019012T043141.v2.0.B12.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019012T043141.v2.0/HLS.S30.T46QCM.2019012T043141.v2.0.B04.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCM.2019016T041804.v2.0/HLS.L30.T46QCM.2019016T041804.v2.0.B06.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019014T042129.v2.0/HLS.S30.T46QCM.2019014T042129.v2.0.B8A.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCM.2019014T042129.v2.0/HLS.S30.T46QCM.2019014T042129.v2.0.B11.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019012T043141.v2.0/HLS.S30.T46QCL.2019012T043141.v2.0.B12.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.B8A.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T46QCL.2019016T041828.v2.0/HLS.L30.T46QCL.2019016T041828.v2.0.B06.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019014T042129.v2.0/HLS.S30.T46QCL.2019014T042129.v2.0.B11.tif:1
Ignoring read failure while reading: https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T46QCL.2019012T043141.v2.0/HLS.S30.T46QCL.2019012T043141.v2.0.B04.tif:1

[egagli]

Thanks for the issue @pritamd47!! Just getting to this now, could you try adding os.environ['GDAL_HTTP_NETRC'] = 'YES' somewhere before you call HLS?