Suggestion: Move to SBT for faster development cycle?

[EmilyFlarionIO]

What is the problem the feature request solves?

Running a simple "make" after a change in *.scala takes a few whole minutes to recompile,
not to mention the complexity of the current xml files, and inconvenience of the repeated formatting and import order changes required.

I believe moving to SBT can greatly hasten this, and provide an improved developer experience and velocity.
As well as simplification for plugin usage.
Is this something that this project can consider? should I try and make a POC?

Describe the potential solution

1. Moving to build.sbt and plugins.sbt files.
2. Optional: Use sbt-assembly for the single Comet .jar file.
3. Optional: Use scalafmt in tandem with sbt-scalafmt for formatting and checking.
4. Optional: Use sbt-jni to reduce boilerplate in the Native class(as sbt-jni takes care of loading and linking).

[andygrove]

Thanks for raising this, @EmilyFlarionIO. The build times can be frustrating. If we were to start by adopting step 1, I assume we would not need to change any of the Maven infrastructure and could still rely on that for the current release process.

My only concern is that I have no experience working with SBT, but I have many years of experience working with Maven.

@comphead @kazuyukitanimura @viirya @parthchandra What are your thoughts on this?

[comphead]

Thanks @EmilyFlarionIO indeed the SBT is more natural to Scala, even Apache Spark uses SBT inside their build. We probably can still live with the Makefile calling the SBT instead of Maven. My concern though: the important that Maven does now is the profiles, so we can build an artifact for specific Apache Spark version and Scala version, I'm not sure about SBT but it has been problems to support same in Gradle.

[kazuyukitanimura]

It would be great if we can use sbt, but I am not sure if we have all the functionalities that we use today.
Spark still is using maven

For short term, I guess we can turn off the style checks for local builds that makes the development faster?

[viirya]

Note that Spark uses sbt for daily development but it still uses Maven for release binary. I remember sometimes sbt build produces issues. But for daily development it is okay.

In short, I don't think "moving" to sbt from Maven is okay but it is okay to have both sbt and Maven build like Spark.

[andygrove]

In short, I don't think "moving" to sbt from Maven is okay but it is okay to have both sbt and Maven build like Spark.

Yes, exactly. I would be okay with SBT being an option for daily dev use. One advantage is that SBT supports incremental compilation, leading to faster dev cycles.

[parthchandra]

I have little experience with sbt myself so as long as maven is available, I am happy if other developers also have the option of using sbt.
My understanding is that the maven pom file is the source of truth for Spark; in my (few) contributions to Spark, I have had to update the pom file but never any sbt related file(s). I've never understood how that works, but perhaps we can have something similar?
FWIW, I do find it easier to run scala tests via sbt.

[andygrove]

@EmilyFlarionIO It looks like there is concensus that it would be good to have SBT as an option but keep Maven as the official build and release system. Do you want to create a PR to add SBT support?

[EmilyFlarionIO]

@andygrove I will take a look at this after the weekend(Sunday) and investigate the best way to implement this :)