GPU example

[jbednar]

Since Numba has good support for using GPUs, we should evaluate how well GPU performance maps onto Datashader's algorithms, and see if there are cases where we recommend using a GPU. Presumably it would be for datasets that fit entirely into memory on the GPU, and on which one needs to do multiple reaggregations; otherwise it seems like speedups will be hard to find due to the I/O time.

[ariejdl]

These folks know about this! https://github.com/mapd/mapd-core Is the architecture comparable?

[jbednar]

We've talked to them and compared notes, and their approach does share some similarities. But we'll need to move forward with our own GPU implementation before being able to compare in any detail. I'd expect to make use of https://github.com/gpuopenanalytics/pygdf , but don't currently have any funding to work on getting it working...

[ariejdl]

Thanks, like this project, was using it today at work in a notebook. All the best.

[mrocklin]

Checking in here, is GPU acceleration still a topic of active interest within the Datashader community?

I was talking with @exactlyallan @AjayThorve about this and they seemed interested.

[jbednar]

We're not only interested, we recently obtained funding for it, and have promised to deliver it sometime in the next 12 months. The agreed task is:

Support GPU DataFrames in Datashader

The new RAPIDS project developed by Nvidia and Anaconda provides a GPU-based DataFrame analogous to the Pandas and Dask DataFrames already supported by Datashader for rendering large datasets for tasks like regridding and resampling (including interpolation and aggregation). Once this support has been added to Datashader, users will be able to choose between Pandas dataframes (typically on a single node, possibly covering multiple CPU cores), Dask dataframes (which can support arbitrarily distributed computations across separate CPU nodes), or GPU dataframes (using data structures held entirely within the GPU memory. Each of these alternatives has different use cases and is optimized for different applications, and users will be able to choose whichever approach is most suitable for their applications.

[mrocklin]

My guess is that @exactlyallan and @AjayThorve might be good resources while pursuing this work. I'd encourage you all to chat at some point.

[kkraus14]

I'll also throw my name in the hat here as a maintainer of https://github.com/rapidsai/cudf that I'd be happy to chat if you have any questions or issues 😄

[jbednar]

Sounds great! I'll send a meeting invite to all of you once we are ready to start.

[jbednar]

We've looked over our various staff and project and schedules, and we currently plan to start this work in September of this year. It would be nice to start it sooner, but all the people involved at our end are fully booked until then. Could change, but if not, I'll be in touch in August!

[jbednar]

Took a bit longer than expected, but GPU support is now in Datashader 0.9.0. See #793 .