PyTorch integration

[AlexandreBrown]

Hello,

With PyTorch gaining more and more traction these days, does the KotlinDL team plan on supporting PyTorch as well in the future or is this an uncertainty for now?

Thanks.

[zaleslaw]

Well, it's a good question, and I prepared two answers: short and long.

The short answer: Yes, currently, it's on the roadmap, and the earliest support is planned for a 0.5 or 0.6 release.

The long answer: both TensorFlow and PyTorch have Java APIs, but only TensorFlow Java API gives us the ability to train models. The PyTorch Java API is built for inference on mobile devices and has no additional support for transfer learning and training. Also, I could not find any plans from Facebook to publish better Java API in the nearest future.

There are a few ways: to contribute in PyTorch Java API or wrap PyTorch C++ API with JNI or JavaCPP project. There are a few experiments with writing JNI bindings for this C++ API in Java/Kotlin community, we hope they become more mature during this year (like Java API for TensorFlow) to switch on them or start our own if these experiments will be frozen.

Also, for high-level API is not so important what GPU kernels are used under the hood to perform tensor multiplications and additions. To start to implement PyTorch-like API we don't need the Torch runtime at all :)

[fabianishere]

Thank you for the answer @zaleslaw. I am also interested in using KotlinDL with a different machine learning runtimes. When I looked into this last time, the main issue from KotlinDL's side is that the model description is currently tied to TensorFlow. Are you planning to separate KotlinDL's runtime implementation (using TensorFlow) from the API for creating model descriptions (which can be independent of runtime implementation)?

[zaleslaw]

@fabianishere You are absolutely right, it has strong ties under the hood, and in custom implementations, we need to make a step forward and make runtime-independent API, but before it, we need to build a successful and popular library for DL in the Kotlin ecosystem and attract new contributors and users who could say us what is really important for them.

Now we have a demand for Transfer Learning/ONNX integration/fine-tuning on mobile devices/having easy and Kotlin-idiomatic API for preprocessing and model usage. So, then it will be stabilized, we "can go deeper" to the next things.

Experimental ONNX integration will be available in 0.3, I hope.

So, could you please write more about your interest in the different runtimes? Happy to read more about that.

[fabianishere]

I agree with your assessment and am eager to see where KotlinDL is heading! :)

As for my use-case, it is rather niche and non-standard, so I don't expect you to take it into account during development. My use-case is about simulating Keras models. In essence, this works by providing the same API as Keras, but replacing the part that performs actual computation by some simulation code. If at some point KotlinDL supports multiple runtimes, any model using KotlinDL could be simulated by just swapping the runtime implementation. The current approach is that we provide a very similar API to KotlinDL, but with the TensorFlow-specific code removed.

[zaleslaw]

@fabianishere Wow, sounds very interesting, is this somehow related to the Neural
Architecture Search problem or to the alternative methods for model
training (for example evolution algorithms to update weights)?

[fabianishere]

We are currently using ML simulations to estimate their environmental impact and test how we can better optimize infrastructure for ML workloads (e.g., different scheduling policies or different hardware)

[saudet]

BTW, there are at least a few people currently happily using the JavaCPP Presets for PyTorch:
https://github.com/bytedeco/javacpp-presets/tree/master/pytorch
This only maps the C++ API, it does not provide an idiomatic Java API, but it does save us from writing, building, and loading JNI code manually. AFAIU, @zaleslaw would accept contributions to KotinDL using those presets, but please correct me if I'm wrong!