You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Reading it using earthkit-data's from_source() method takes 30 seconds, while extracting all the numpy arrays using the netCDF4 package takes 0.02 seconds!
Having inspected the file we can see this structure (not all variables are shown):
earthkit-data is splitting this data by variable, level and time and generates 403006 (!) fields out of it, which obviously takes a lot of time. However, we can argue that this data should not be represented as a fieldlist since each "field" only contains a single point.
So the question is how to decide automatically whether a NetCDF dataset should be treated as a fieldlist.
The text was updated successfully, but these errors were encountered:
Sometimes reading a small NetCDF file (< 2 Mb) as a fieldlist takes a very long time.
There is one particular example reported to be very slow to read: https://get.ecmwf.int/repository/test-data/earhtkit-data/test-data/htessel_points.nc
Reading it using earthkit-data's
from_source()
method takes 30 seconds, while extracting all the numpy arrays using the netCDF4 package takes 0.02 seconds!Having inspected the file we can see this structure (not all variables are shown):

earthkit-data is splitting this data by variable, level and time and generates 403006 (!) fields out of it, which obviously takes a lot of time. However, we can argue that this data should not be represented as a fieldlist since each "field" only contains a single point.
So the question is how to decide automatically whether a NetCDF dataset should be treated as a fieldlist.
The text was updated successfully, but these errors were encountered: