Skip to content

Bulk download speed-up #19

Open
Open
@RRemelgado

Description

@RRemelgado

I was trying to download thousands of range maps using BIEN_ranges_species(), and it took over a day to reach 10,000 (without parallelization). Therefore, I had a look at your code.

When dealing with multiple species, you are currently iterating through each and returning a single ESRI shapefile. Instead, I would suggest using the output of .BIEN_sql() with st_as_sf() from the sf package. This require a column with WKT data, which is returned by .BIEN_sql(). Doing so translates the full query into a single sf object, greatly limiting the overhead time.

Using this approach, it took me half a day to download the full set of range maps (> 98,000).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions