Downloading of the 2020 data

[gherbox]

I can't download the 2020 dataset here https://app.globus.org/file-manager?origin_id=0b226e2c-4de0-11ea-971a-021304b0cca7&origin_path=%2F .

The download option is disabled. (See image)

How can we download these data?

[sparticlesteve]

It doesn't allow direct downloads, but you can do a data transfer to a local endpoint.
If you have globus deployed at your institution, you can transfer to whatever endpoint may be already setup on your end.
If not, you can setup a global personal endpoint. It's just a little tedious to do but it's pretty much our only option for reliably hosting such a big dataset at NERSC.

[jaidayal]

Is there anyway to get a smaller data set to use for testing various things? We want to do some tests on various parameters before we run a giant data set. The Cosmoflow MLPerf benchmark created a smaller (<200mb) dataset exactly for this purpose and one doesn't need Globus for downloading such a small thing.

From looking at the MLPerf-HPC submissions, it seems that most people run cosmoflow and from others I've talked to, most cite the above reason as a limiter for running DeepCam.

[sparticlesteve]

yeah I believe we have a smaller dataset around somewhere already. I will take a look.

[jaidayal]

Thank you SO much. That would be amazing. This is a really interesting benchmark and we have a test cluster with some new hardware and system software we want to experiment with, so having a smaller dataset is definitely helpful.

[sparticlesteve]

I've now made a deepcam-data-mini subfolder in the globus endpoint which has a 24GB tarball.
It's also available here: https://portal.nersc.gov/project/dasrepo/deepcam/climate-data/deepcam-data-mini/

It has 512 files and the stats file. There is a script there you can use to make N copies and copy to train/ val/ subfolders, or adapt as needed.

If this is still too big, let me know.

[jaidayal]

This is really great. Thanks! Is reducing the data further as simple as just deleting some of the .h5 files?

[sparticlesteve]

Yeah, I could just make a tarball with fewer files, and you could make more copies as desired.

[gherbox]

Hello @sparticlesteve,

I tried to run the code with the small data. I first generated the stats.h5 file.
But it looks like there is an incompatibility between the code and the attribute of the h5 files in the file /data/cam_hdf5_dataset.py.
For the h5 images I changed this code from
self.label_shape = fin['climate']['labels_0'].shape
to
self.label_shape = fin['climate']['labels'].shape

But there is also an incompatibility with the stats.h5 file.
The file cam_hdf5_dataset.py suggests that there is an attribute called "minval" for the normalisation of the data.

        with h5.File(self.statsfile, "r") as f:
            data_shift = f["climate"]["minval"][self.channels]

But in the stats.h5 file that I generated, I don't have this attribute.
So I get this error

The stats file that I generated has 2 keys for for "climate" that are either 'label_stats' or 'stats'

[sparticlesteve]

There should've been a stats.h5 file included in the tarball. I'm not sure about the state of the script which generates the stats file. Maybe Thorsten had an uncommitted version..?

Can you try with the stats file that was included?

[gherbox]

Hey @sparticlesteve
I used the stats.h5 file included in the tarball and I got the same error.
When I look in the stats.h5 here is what I find.

Maybe it is the /data/cam_hdf5_dataset.py file that is not updated

[jaidayal]

Thanks a lot for your help. In regards to this, I just want to clarify:

Yeah, I could just make a tarball with fewer files, and you could make more copies as desired.

Instead of having you do it each time, I could just delete .h5 files and it will all work? Or would I need to regenerate a stats file as well? I mostly ask because some people might want to test something for a few minutes (others may want to go longer). Having that flexibility would help and we'd hate to ask you to do that for us each time :).

[sparticlesteve]

@gherbox ,

I used the stats.h5 file included in the tarball and I got the same error.

given that you have climseg-data-small in your data path, and the contents of that h5 file, I have to ask if you're using the old small dataset or if you've grabbed the new one from here: https://portal.nersc.gov/project/dasrepo/deepcam/climate-data/deepcam-data-mini/
also available in the globus endpoint directory: https://app.globus.org/file-manager?origin_id=0b226e2c-4de0-11ea-971a-021304b0cca7&origin_path=%2Fdeepcam-data-mini%2F

I thought I've removed all public links to the old small dataset, but that folder was named climseg-data-small so maybe you grabbed it.

@jaidayal ,

Yeah you should just be able to delete or copy/link subsets of what you have. For this kind of testing, the actual stats stored in the stats.h5 really aren't important. Just keep using that same file.

[sparticlesteve]

The correct stats file has these keys:

In [7]: f['climate'].keys()
Out[7]: <KeysViewHDF5 ['count', 'maxval', 'mean', 'minval', 'sqmean']>

[gherbox]

Hey @sparticlesteve
Thanks a lot for your help. Indeed, my rootdir path was pointing to my previous data.
It works fine now.