Skip to content

chore(ci): log basic details of downloaded ncbi datasets #5513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
corneliusroemer merged 7 commits into from
Nov 24, 2025
Merged

chore(ci): log basic details of downloaded ncbi datasets #5513

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6c7aeb2
chore(ci): log basic details of downloaded ncbi datasets
corneliusroemer Nov 23, 2025
3cc853c
Run prio 2 with 1hr offset from 2-hourly prio 1 to not coincide
corneliusroemer Nov 23, 2025
c014843
Run prio 2 with 1hr offset from 2-hourly prio 1 to not coincide add b…
corneliusroemer Nov 23, 2025
37ecba5
Nicer log
corneliusroemer Nov 23, 2025
6c53e2e
try again
corneliusroemer Nov 23, 2025
64ff17c
again
corneliusroemer Nov 23, 2025
477eb02
never mind
corneliusroemer Nov 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/datasets-mirror-priority-2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Mirror NCBI Datasets (Priority 2 - Daily)
on:
workflow_dispatch: # Allows manual triggering to re-run all priority-2 taxa
schedule:
- cron: '12 2 * * *' # Runs daily at 02:12 UTC
- cron: '12 3 * * *' # Runs daily at 03:12 UTC
jobs:
mirror:
strategy:
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/datasets-mirror.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,16 @@ jobs:
- uses: mamba-org/setup-micromamba@v2
with:
environment-name: datasets
create-args: ncbi-datasets-cli s3cmd
- name: Download NCBI Dataset
create-args: -c conda-forge -c bioconda ncbi-datasets-cli s3cmd seqkit
- name: Download NCBI Dataset and create tzst archive
shell: bash -l {0}
run: |
datasets download virus genome taxon ${{ inputs.taxon_id }} --no-progressbar --filename ${{ inputs.taxon_id }}.zip
unzip -o ${{ inputs.taxon_id }}.zip
tar -I 'zstd -T0 -18' -cvf ${{ inputs.taxon_id }}.tar.zst ncbi_dataset
cat md5sum.txt
ls -lh ${{ inputs.taxon_id }}.* ncbi_dataset/data/*
seqkit stats ncbi_dataset/data/genomic.fna
- name: Create S3cmd config
run: |
cat <<EOF > ~/.s3cfg
Expand Down
Loading