Package downscaling data for download

[dlebauer]

• Identify all input
• Compress and zip in a tar.gz
• Put on S3 bucket
• Document access

[divine7022]

I don't see this all_crops_2018-2023_same_uids_try5.shp file, only all_crops_2016-2023_join.* exists in /projectnb/dietzelab/ccmmf/data_raw/cadwr_land_use/2015-2023_crops_same_uid.
should we include only what exists, or is there a missing file?

[dlebauer]

@divine7022 sorry about that! I shouldn't have included that specific list. The only data that needs to be available for download is what is required by the pipeline.

It should be organized "nicely" under data_raw or data. the distinction "raw data" is what we don't have a pipeline to reproduce (like anything directly downloaded, created for the purpose, or otherwise modified in a way that would not be easily reproduced. "data" is made from raw data, any data that can be reliably downloaded.

I will update the ticket

[divine7022]

Data package created and staged for download.
package location -- /projectnb/dietzelab/ccmmf/packaging/ccmmf_landiq_data.tar.gz

Package structure:

ccmmf_landiq_data
├── README.md
├── data
│   └── cadwr_land_use
│       ├── crops_all_years.csv
│       └── crops_all_years_metadata.csv
└── data_raw
    └── cadwr_land_use
        ├── CARB_Metadata_ref.csv
        ├── CARB_PFTs_table.csv
        ├── landiq_crop_mapping_codes.tsv
        └── landiq_shapefiles
            ├── i15_Crop_Mapping_2014_SHP
            │   ├── Standard_LandUse_Legend.lyr
            │   ├── i15_Crop_Mapping_2014.cpg
            │   ├── i15_Crop_Mapping_2014.dbf
            │   ├── i15_Crop_Mapping_2014.prj
            │   ├── i15_Crop_Mapping_2014.sbn
            │   ├── i15_Crop_Mapping_2014.sbx
            │   ├── i15_Crop_Mapping_2014.shp
            │   ├── i15_Crop_Mapping_2014.shp.xml
            │   ├── i15_Crop_Mapping_2014.shx
            │   └── i15_Crop_Mapping_2014_Metadata.pdf
            ├── i15_Crop_Mapping_2016_SHP
            │   ├── Standard_LandUse_Legend.lyr
            │   ├── i15_Crop_Mapping_2016.cpg
            │   ├── i15_Crop_Mapping_2016.dbf
            │   ├── i15_Crop_Mapping_2016.prj
            │   ├── i15_Crop_Mapping_2016.sbn
            │   ├── i15_Crop_Mapping_2016.sbx
            │   ├── i15_Crop_Mapping_2016.shp
            │   ├── i15_Crop_Mapping_2016.shp.xml
            │   ├── i15_Crop_Mapping_2016.shx
            │   └── i15_Crop_Mapping_2016_Metadata.pdf
            ├── i15_Crop_Mapping_2018_SHP
            │   ├── Standard_LandUse_Legend.lyr
            │   ├── i15_Crop_Mapping_2018.cpg
            │   ├── i15_Crop_Mapping_2018.dbf
            │   ├── i15_Crop_Mapping_2018.prj
            │   ├── i15_Crop_Mapping_2018.sbn
            │   ├── i15_Crop_Mapping_2018.sbx
            │   ├── i15_Crop_Mapping_2018.shp
            │   ├── i15_Crop_Mapping_2018.shp.xml
            │   ├── i15_Crop_Mapping_2018.shx
            │   └── i15_Crop_Mapping_2018_Metadata.pdf
            ├── i15_Crop_Mapping_2019_SHP
            │   ├── Standard_LandUse_Legend.lyr
            │   ├── i15_Crop_Mapping_2019.cpg
            │   ├── i15_Crop_Mapping_2019.dbf
            │   ├── i15_Crop_Mapping_2019.pdf
            │   ├── i15_Crop_Mapping_2019.prj
            │   ├── i15_Crop_Mapping_2019.sbn
            │   ├── i15_Crop_Mapping_2019.sbx
            │   ├── i15_Crop_Mapping_2019.shp
            │   ├── i15_Crop_Mapping_2019.shp.xml
            │   └── i15_Crop_Mapping_2019.shx
            ├── i15_Crop_Mapping_2020_SHP
            │   ├── 15_Crop_Mapping_2020.pdf
            │   ├── CropMapping2020_Legend.lyrx
            │   ├── i15_Crop_Mapping_2020.cpg
            │   ├── i15_Crop_Mapping_2020.dbf
            │   ├── i15_Crop_Mapping_2020.prj
            │   ├── i15_Crop_Mapping_2020.sbn
            │   ├── i15_Crop_Mapping_2020.sbx
            │   ├── i15_Crop_Mapping_2020.shp
            │   ├── i15_Crop_Mapping_2020.shp.xml
            │   └── i15_Crop_Mapping_2020.shx
            ├── i15_Crop_Mapping_2021_SHP
            │   ├── CropMapping2021_Legend.lyrx
            │   ├── i15_Crop_Mapping_2021.cpg
            │   ├── i15_Crop_Mapping_2021.dbf
            │   ├── i15_Crop_Mapping_2021.prj
            │   ├── i15_Crop_Mapping_2021.sbn
            │   ├── i15_Crop_Mapping_2021.sbx
            │   ├── i15_Crop_Mapping_2021.shp
            │   ├── i15_Crop_Mapping_2021.shp.xml
            │   └── i15_Crop_Mapping_2021.shx
            ├── i15_Crop_Mapping_2022_Provisional_SHP
            │   ├── CropMapping2022_Provisional_Legend.lyrx
            │   ├── i15_Crop_Mapping_2022_Provisional.cpg
            │   ├── i15_Crop_Mapping_2022_Provisional.dbf
            │   ├── i15_Crop_Mapping_2022_Provisional.prj
            │   ├── i15_Crop_Mapping_2022_Provisional.sbn
            │   ├── i15_Crop_Mapping_2022_Provisional.sbx
            │   ├── i15_Crop_Mapping_2022_Provisional.shp
            │   ├── i15_Crop_Mapping_2022_Provisional.shp.xml
            │   └── i15_Crop_Mapping_2022_Provisional.shx
            ├── i15_Crop_Mapping_2023_Provisional_SHP
            │   ├── i15_Crop_Mapping_2023_Provisional.cpg
            │   ├── i15_Crop_Mapping_2023_Provisional.dbf
            │   ├── i15_Crop_Mapping_2023_Provisional.prj
            │   ├── i15_Crop_Mapping_2023_Provisional.shp
            │   ├── i15_Crop_Mapping_2023_Provisional.shp.xml
            │   └── i15_Crop_Mapping_2023_Provisional.shx
            └── spatial_joins
                ├── all_crops_2016-2023_join.cpg
                ├── all_crops_2016-2023_join.dbf
                ├── all_crops_2016-2023_join.prj
                ├── all_crops_2016-2023_join.shp
                └── all_crops_2016-2023_join.shx

S3 access:

aws s3 cp \
    s3://carb/data/ccmmf_landiq_data.tar.gz \
    ccmmf_landiq_data.tar.gz \
    --endpoint-url https://s3.garage.ccmmf.ncsa.cloud

tar -xzvf ccmmf_landiq_data.tar.gz