Skip to content

Conversation

@getty708
Copy link
Contributor

@getty708 getty708 commented Aug 11, 2024

Overview

The official data downloader is litten in a shell script and it only supports downloading data from zenodo, Google Drives are not supported. A stronger and more flexible data downloader is required.

Major Changes

  • Python Based Downloader
    • Generate metadata (and save it as a YAML file on GitHub) and register it to Artifactory (Zenodo)
    • Generate metadata (and save it as a YAML file on GitHub) and register it to Artifactory (Google Drive)
    • Add python-based data downloader for zenodo. (Download them to the .cache directory)
    • Add a downloader for data on the Google Drive. (Download them to the .cache directory)
    • Generate metadata (and save it as a YAML file on GitHub) and register it to Artifactory (Zenodo File Contents)
    • Generate metadata (and save it as a YAML file on GitHub) and register it to Artifactory (Google Drive File Contents)
    • Support selective download.
  • Publish the code for the preprocessed dataset generation.
  • WandB-based data management support.

Minor Changes

  • Visualization on WandB

@getty708 getty708 force-pushed the feat/getty708/add-python-based-download-tool branch 3 times, most recently from 601f556 to d18c0b1 Compare August 11, 2024 06:19
@getty708 getty708 force-pushed the feat/getty708/add-python-based-download-tool branch from d18c0b1 to 8489a5f Compare August 11, 2024 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants