Windows Support: Hadoop fails to load Loaded Libraries

[kadaan]

Windows tries to load the native hadoop binaries which do not exist. It tries to load them from a wide range of folder because Windows returns Windows-<Version> from os.name. After changing this to use 'Windows' as the os.name for all Windows versions and the addition of the native libraries to /nativelib/Windows_amd64, HadoopNative correctly loads the libraries from the temp files that are created. Unfortunately, when loadAllCodecs runs it subsequently fails with the following error:

java.lang.RuntimeException: failed to load Hadoop native library
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:60)
        at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:51)
Caused by: java.lang.RuntimeException: native snappy library not available: SnappyCompressor has not been loaded.
        at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:68)
        at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:189)
        at com.facebook.presto.hadoop.HadoopNative.loadAllCodecs(HadoopNative.java:73)
        at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:55)

Changing HadoopNative.requireHadoopNative to create a temporary directory and making loadLibrary copy the embedded files with the original names into this directory allows the libraries to correctly loaded.

[electrum]

Are Hadoop native libraries for Windows available anywhere? I did a quick search and couldn't find them.

[kadaan]

For sure. Hadoop runs on Windows Azure (HDInight), is available to be installed on Windows machines (Hortonworks), and there are instructions for building Hadoop with native libraries for Windows (http://www.srccodes.com/m/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os).

[kadaan]

Did this answer your question? I'm interested in getting Presto to work on Windows, but could use feedback so that the correct changes are made.

[electrum]

Do you know if the pre-built native libraries are available from Hortonworks or another official source? None of us have a Windows machine.

[kadaan]

Yes, there are prebuilt native libraries available. I'll send the link a little later.

[kadaan]

It is available from Hortonworks at: http://hortonworks.com/products/releases/hdp-2-windows/

[kadaan]

The pre-built binaries are missing bzip2 support which cause the NativeLibraryChecker to fail. They will need to be re-built to support bzip2 or the code that calls NativeLibraryChecker will need to take over the check and special case Windows.

[dain]

Either is fine. bzip2 isn't widely used anyway.

[kokosing]

Closing this as parent task prestodb/presto#2351 was also closed.