Large number of files to 'put' blows JVM heap

[markjfisher]

Environment info

groovy-ssh-2.8.0 (java-1.8.0_111, groovy-2.4.7, jsch-0.1.54)

Steps to reproduce

1. 'put' a directory with large number of files and directories

I'm using put to copy a directory of exploded EARs to target destinations.

The directory I'm trying to push has 58k files (largest directory depth 19). When trying to send, the code that creates the instructions to execute through the connection eventually kills the JVM process with heap errors.

My work-around is to compress the directory into a single file, push that, then decompress on the target, but this is adding overhead (even with 'store' of zip it's adding ~60 seconds to the process).

[amake]

In theory, the "real" problem is that the files to be PUT are gathered eagerly first, and then processed; a true fix would process the files without gathering them all into memory first.

However I wonder if the effective problem isn't that child directories are double-traversed (see #230). With a large number of heavily nested directories, the total number of files gathered would explode. When 2.9.0 is released I suspect this issue may be resolved.