Building Apache Storm

Apache Storm

The instructions provided below specify the steps to build Apache Storm version 1.2.2 on Linux on IBM Z for the following distributions:

• RHEL (7.3, 7.4, 7.5)
• SLES (12 SP3)
• Ubuntu (16.04, 18.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1: Building and installing Apache Storm

1.1) Install Dependencies

• RHEL (7.3, 7.4, 7.5)

  sudo yum install git tar java-1.8.0-openjdk-devel.s390x wget which unzip

• SLES (12 SP3)

  sudo zypper install git java-1_8_0-openjdk-devel tar wget curl which unzip

• Ubuntu (16.04)

  sudo apt-get install git tar ant wget openjdk-8-jdk curl gpgv nodejs gnupg2 python nodejs-legacy

• Ubuntu (18.04)

  sudo apt-get install git tar ant wget openjdk-8-jdk curl gpgv nodejs gnupg2 python

1.2) Install Maven 3.2.5

wget https://archive.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
tar -xzf apache-maven-3.2.5-bin.tar.gz
export PATH=$PWD/apache-maven-3.2.5/bin:$PATH

Note: Maven v3.2.5 or below is required. There is a bug in later versions of maven as linked to from https://issues.apache.org/jira/browse/MSHADE-206 that cause shaded dependencies to not be packaged correctly.

1.3) Install NodeJS SDK (For RHEL and SLES only)

Install nodejs sdk from https://nodejs.org/en/download/ for Linux on System z

Add the nodejs executable to the path:

export PATH=$PATH:<Nodejs-Install-Directory>/bin  

1.4) Install rvm and nvm

command curl -sSL https://rvm.io/mpapis.asc | gpg2 --import -
curl -L https://get.rvm.io | bash -s stable --autolibs=enabled && source $HOME/.profile
wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.26.1/install.sh | bash && source $HOME/.bashrc

1.5) Set environment variables

Setup up Maven and JAVA:

export JAVA_HOME=<path to java>          
export PATH=$JAVA_HOME/bin:$PATH
export _JAVA_OPTIONS=-Xmx4096m
export JVM_ARGS="-Xms4096m -Xmx4096m"
export MAVEN_OPTS="-Xms4096m -Xmx8192m -Xss4096k"

1.6) Download the source code

cd /<source_root>/
git clone https://github.com/apache/storm.git  
cd storm  
git checkout tags/v1.2.2

1.7) Build Apache Storm

mvn clean install

Note: If you wish to skip the unit tests you can do this by adding -DskipTests to the command line.

Step 2: Testing(Optional)

2.1) Run unit test cases

To run Clojure and Java unit tests but no integration tests execute the command

cd /<source_root>/storm
mvn test

Note:

• storm-core Clojure tests might fail intermittently. Setting the environment variable STORM_TEST_TIMEOUT_MS=150000 and re-run individual test to avoid test failures. To re-run tests in storm-core, use the script ./dev-tools/test-ns.py [test name], for example, ./dev-tools/test-ns.py org.apache.storm.metrics-test.
• Integration tests require that you activate the profile integration-test and that you specify the maven-failsafe-plugin in the module pom file.
• JmsSpoutTest fails intermittently.

Step 3: Create a Storm distribution (packaging)

cd /<source_root>/storm/storm-dist/binary
mvn package  

Note:

• After running mvn package you may be asked to enter your GPG/PGP credentials (once for each binary file, in fact). This happens because the packaging step will create *.asc digital signatures for all the binaries, and in the workflow above your GPG private key will be used to create those signatures.
• You can disbale GPG signing by adding -Dgpg.skip=true
• The binaries can be found at /<source_root>/storm/storm-dist/binary/target/.

References:

• http://storm.apache.org/releases/current/index.html
• https://github.com/apache/storm/blob/v1.2.2/DEVELOPER.md