Support for Arrow

[rafapereirabr]

Opening this issue with the suggestion that we include support for Arrow in r5r.

As documented on their website, Arrow specifies a standardized language-independent columnar memory format for flat and hierarchical data. This would mean two most obvious advantages: (1) passing data from Java to R (from R5 to r5r) would become seamless, (2) saving outputs in .parquet format. Both of these advantages would probably make r5r substantially faster, with more efficiency gains for large scale analyses.

There are robust implementations of Arrow in Java, R and also in Python (in case we want to implement this in r5py).

I'm not sure this could be done entirely within the Java side of r5r or whether it would require some change to R5 upstream. In any case, this might be something that the @conveyal would be interested in, since this would speed improve up the process of passing R5 results to interactive visualization in Conveyal Analysis.

[botanize]

I've been working with the csv output of travel_time_matrix for a large region and many of the csv files contain only a few lines, performance reading 30k files is poor. It would be a bit more work, but writing the matrices to parquet files, aggregating multiple from_ids would be hugely helpful.

[rafapereirabr]

yes, this could be a great improvment to te package. @botanize , if you're familiar with Java and would like to have a look at this, we would appreciate PR from collaborators

[rafapereirabr]

this would require major change upstream in R5. Closing the issue here

[rafapereirabr]

The idea is that instead of deserializing the output table from Java to R with java_to_dt(travel_times), the small Jar would convert tha output to Arrow, which could be seamlessly passed to R. Cc @BardyBard

[alexmgns]

@rafapereirabr I have successfully transferred a travel times matrix from the little jar to R using arrow.

PR soon...

[rafapereirabr]

yahoo ! amazing. now the key question, how did it affect performance (computation time and memory) ?

[alexmgns]

@rafapereirabr I tried a square matrix, 1500 origins to 1500 destinations on my laptop.

10-15% improvement in total CPU time, and 20% improvement in real time passed. I assume the discrepancy between the real life time and CPU time is because arrow does a bit more multi threaded work compared to Marcus's code.

This is a good result I guess considering we're just changing the language translation code. But a bit disappointing nonetheless

[alexmgns]

I also tried a 15 origins to 150,000 observations test, 10% improvement in both CPU and real time.

However it does it use less memory. With only setting 2gb of Ram I was able to run the query only 3 times using java_to_dt implementation, but 4 times with arrow implementation before running out of memory.

Although it is weird that java ends up running out of memory. Any unused variables should be automatically cleaned up by the JVM from my understanding, so if the query ran once it should never run out of memory since the JVM should clean and reuse that memory during the next call. This is either a bug in how we're calling rJava, or just a bug in rJava.

@rafapereirabr

[rafapereirabr]

Ok, so using Arrow to pass data from Java to R does not bring the big improvment I was expecting, but it's still something. I guess @mvpsaraiva code inside java_to_dt() was already pretty efficient then! @BardyBard , have you had the chance of testing it in larger travel time matrices?

[alexmgns]

Ok, so using Arrow to pass data from Java to R does not bring the big improvment I was expecting, but it's still something. I guess @mvpsaraiva code inside java_to_dt() was already pretty efficient then! @BardyBard , have you had the chance of testing it in larger travel time matrices?

How large do you suggest? You can let me know or we can discuss tomorrow what you want me to test.

I can also put up a PR with a branch you can test yourself.