Release prep:

-Version bump 0.2.0
-Update Docs
-Update Sample Configuration
-Minor changes to docker scripts
Pull Request:
  Closes: #72

README.md

@@ -37,7 +37,8 @@ Please keep checking this section for updates.
 - [x] Framework checkpointing
 - [x] Launch Job History Server
 - [x] Constraints based Node Manager placement
-- [ ] Support multi-tenancy for Node Managers
+- [x] Docker support
+- [x] Support multi-tenancy for Node Managers
 - [ ] Configuration store for storing rules and policies for clusters managed by Myriad
 
 ## Mailing Lists

build.gradle

@@ -64,7 +64,7 @@ subprojects {
     mainClassName = "org.apache.myriad.Main"
 
     group = "org.apache.myriad"
-    version = "0.1.0"
+    version = "0.2.0"
 
     ext {
         mesosVer = "0.28.1"

docker/Dockerfile

@@ -22,12 +22,17 @@
 FROM debian
 MAINTAINER Apache Myriad [email protected]
 
-ENV HADOOP_USER="yarn"
+ENV YARN_USER="yarn"
+ENV YARN_UID="108"
+ENV HADOOP_GROUP="hadoop"
+ENV HADOOP_GID="112"
+ENV YARN_GROUP="yarn"
+ENV YARN_GID="113"
 ENV HADOOP_VER="2.7.0"
 ENV HADOOP_HOME=/usr/local/hadoop
 ENV JAVA_HOME=/usr
 ENV MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
-
+ENV MESOS_VER=0.28.1
 
 # Setup mesosphere repositories
 RUN apt-get -y update
@@ -37,7 +42,7 @@ RUN DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_releas
 RUN apt-get -y update
 
 # Install Mesos
-RUN apt-get install -y mesos curl tar
+RUN apt-get install -y mesos=`apt-cache madison mesos | grep $MESOS_VER | head -1 | awk {'print $3'}` curl tar
 # Run local YARN install
 ADD myriad-bin/install-yarn.sh /install-yarn.sh
 RUN sh /install-yarn.sh

docker/README.md

@@ -1,41 +1,63 @@
 # Using Myriad with Docker #
 Docker is the easiest way to from 0 to Yarn on Mesos within minutes. 
 
-## Building the Resource Manager Docker
+# Building the Resource Manager Docker
 
-#Configuration Guide#
+YARN runs as a user and group, and expects consistent uid and gids in HDFS and accross the cluster.  Hence it is necessary 
+to edit docker/Dockerfile and modify the following:
+```
+ENV YARN_USER="yarn"
+ENV YARN_UID="107"
+ENV HADOOP_GROUP="hadoop"
+ENV HADOOP_GID="112"
+ENV YARN_GROUP="yarn"
+ENV YARN_UID="113"
+```
+
+The run the following commands:
+```bash
+./gradlew -P dockerTag=username/myriad buildRMDocker
+docker push username/myriad
+```
+
+This will build the ResourceManager from src, save, and push the image as *username/myriad*.
+
+# Configuration Guide
+
+It is assumed you already have hdfs and Mesos running.  For more information about Apache Mesos visit the [website](http://mesos.apache.org). 
+If you need to setup hdfs, consider using the [hdfs-mesos framework](https://github.com/mesosphere/hdfs).
 
 In order for the ResourceManager to operate correctly, you will need to provide 5 configuration files. These files will 
-need to mounted from a directory into `./myriad-etc` of this folder.
+need to mounted from a directory into `/usr/local/hadoo/etc` of the docker container.
 
-* [myriad-config-default.yml](https://github.com/mesos/myriad/blob/phase1/myriad-scheduler/src/main/resources/myriad-config-default.yml) (template provided)
-* [yarn-site.xml](https://github.com/mesos/myriad/blob/phase1/docs/myriad-dev.md) (template provided)
+* [myriad-config-default.yml](https://github.com/apacher/incubator-myriad/blob/phase1/myriad-scheduler/src/main/resources/myriad-config-default.yml) (template provided)
+* [yarn-site.xml](https://github.com/apache/incubator-myriad/blob/phase1/docs/myriad-dev.md) (template provided)
 * mapred-site.xml (template provided)
 * hdfs-site.xml (used for hdfs)
 * core-site.xml (used for hdfs)
 
-It is assumed you already have hdfs and Mesos running.  For more information about Apache Mesos visit the [website](http://mesos.apache.org). 
-If you need to setup hdfs, consider using the [hdfs-mesos framework](https://github.com/mesosphere/hdfs).  Copy the hdfs.xml and 
-core-site.xml from your hdfs configuration into myriad-etc.  Additional, files maybe necessary such as rack.sh, log4j.properties, 
-hadoop-env.sh, and yarn-env.sh depending on your configuration.
+Edit and copy the file templates along with the hdfs.xml and core-site.xml from your hdfs configuration into a directory named `config`.  Additional, files maybe necessary such as rack.sh, log4j.properties, 
+hadoop-env.sh, and yarn-env.sh depending on your configuration. Create a second directoy called `dist`. Then tar the files using `tar -zcf dist\config.tgz config`.
 
-Run the following commands:
+# Running the Resource Manager Docker
+
+From the directories containing `dist` and `config` execute:
 ```bash
-./gradlew -P username/myriad buildRMDocker
-docker push username/myriad
+docker run --net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop --name='myriad-resourcemanager' -t \
+  <username>/myriad
 ```
-This will build the ResourceManager from src, save, and push the image as *username/myriad*.
+# Updating configuration
 
-## Running the Resource Manager Docker
+There is no need to to rebuild the docker image to update YARN's configuration, simply edit the files in config and repackage using 
+`tar -zcf dist\config.tgz config`.  To update the configuration on the Nodemanagers they will need to be flexed down and flexed up, to 
+ update the resource manager the docker container will need to be restarted:
 
 ```bash
-docker run --net=host --name='myriad-resourcemanager' -t \
+docker kill myriad-resourcemanager
+docker run --net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop --name='myriad-resourcemanager' -t \
   <username>/myriad
 ```
-
-#Environment Variables#
-* *ALTERNATE_HADOOP_URL* : Optional - Allows user to override the hadoop distribution used by Myriad. This will download the *.tar.gz file to be used as the hadoop distribution of choice for Myriad. 
-
+
 ---
 <sub>
 Licensed to the Apache Software Foundation (ASF) under one

docker/myriad-bin/configure-yarn.sh

@@ -22,20 +22,20 @@ under the License.
 ##
 
 # Put in env defaults if they are missing
-export HADOOP_GROUP=${HADOOP_GROUP:='hadoop'}
-
 export YARN_USER=${YARN_USER:='yarn'}
-export USER_UID=${USER_UID:='107'}
+export YARN_UID=${YARN_UID:='107'}
 export YARN_GROUP=${YARN_GROUP:='yarn'}
+export HADOOP_GROUP=${HADOOP_GROUP:='hadoop'}
 export HADOOP_GID=${HADOOP_GID:='112'}
 export YARN_GID=${YARN_GID:='113'}
 export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
+export MESOS_NATIVE_LIBRARY=${MESOS_NATIVE_LIBRARY:='/usr/local/lib/libmesos.so'}
 
 # Add hduser user
 groupadd ${HADOOP_GROUP} -g ${HADOOP_GID}
 groupadd ${YARN_GROUP} -g ${YARN_GID}
-useradd ${YARN_USER} -g ${YARN_GROUP} -G ${HADOOP_GROUP} -u ${USER_UID} -s /bin/bash
-mkdir /home/${HADOOP_USER}
+useradd ${YARN_USER} -g ${YARN_GROUP} -G ${HADOOP_GROUP} -u ${YARN_UID} -s /bin/bash
+mkdir /home/${YARN_USER}
 chown -R ${YARN_USER}:${YARN_GROUP} /home/${YARN_USER}
 
 #set permissions
@@ -45,16 +45,16 @@ chown root:${YARN_GROUP} ${HADOOP_HOME}/bin/container-executor
 chmod 6050 ${HADOOP_HOME}/bin/container-executor
 
 # Init bashrc with hadoop env variables
-sh -c 'echo export JAVA_HOME=/usr >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/bin >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/sbin >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_MAPRED_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_COMMON_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_HDFS_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export YARN_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_COMMON_LIB_NATIVE_DIR=\$\{HADOOP_HOME\}/lib/native >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_OPTS=\"-Djava.library.path=\${HADOOP_HOME}/lib\" >> /home/${HADOOP_USER}/.bashrc'
-
+sh -c 'echo export JAVA_HOME=/usr >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export MESOS_NATIVE_LIBRARY=/usr >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/bin >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/sbin >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_MAPRED_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_COMMON_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_HDFS_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export YARN_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_COMMON_LIB_NATIVE_DIR=\$\{HADOOP_HOME\}/lib/native >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_OPTS=\"-Djava.library.path=\${HADOOP_HOME}/lib\" >> /home/${YARN_USER}/.bashrc'
 
 echo "end of configure-yarn.sh script"

docker/myriad-bin/install-yarn.sh

@@ -20,7 +20,8 @@ under the License.
 ##
 # YARN installer script for Apache Myriad Deployment
 ##
-HADOOP_VER="2.7.0"
+export HADOOP_VER=${HADOOP_VER:='2.7.0'}
+export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
 
 echo "Installing Yarn...."
 if [ ! -z $1 ];then
@@ -30,7 +31,6 @@ else
 fi
 HADOOP_TGZ=`basename ${HADOOP_URL}`
 HADOOP_BASENAME=`basename ${HADOOP_URL} .tar.gz`
-export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
 
 # Extract Hadoop
 echo "Downloading ${HADOOP_TGZ} from ${HADOOP_URL}"

docker/myriad-etc/myriad-config-default.yml.template

@@ -18,12 +18,12 @@
 ##
 
 # Address of the mesos master - <IP:port> or ZooKeeper path
-mesosMaster: zk://CHANGEME:2181/mesos
+mesosMaster: zk://<ZOOKEEPER>:2181/mesos
 #Container information for the node managers
 containerInfo:
     type: DOCKER
     dockerInfo:
-        image: CHANGEME/myriad
+        image: <IMAGE>/myriad
     volume:
         -
           containerPath: /tmp
@@ -35,19 +35,13 @@ checkpoint: false
 frameworkFailoverTimeout: 60000 #10 minutes
 # Myriad's mesos framework name.
 frameworkName: MyriadAlpha
-# Myriad's mesos framework role.
-#frameworkRole: myriad
 # User the Node Manager will run as (Defaults to user running the resource manager if absent,  necessary for remote distribution).
 frameworkUser: yarn
-# User that gets the nodeManagerUri and sets up the directories for Node Manager, must have passwordless sudo (Necessary only for remote distribution, otherwise ignored).
 # frameworkSuperUser: root
 # Myriad's REST-ful services port mapping.
 restApiPort: 8192
-# Address of the ZK ensemble (separate by comma, if multiple zk servers are used)
-#zkServers: localhost:2181
-# ZK Session timeout
-zkTimeout: 20000
 haEnabled: True
+servedConfigPath: /dist/config.tgz
 # The node manager profiles. The REST API to flex up expects one of the profiles defined here.
 # Admin can define custom profiles (requires restart of Resource Manager)
 nmInstances:
@@ -75,37 +69,18 @@ rebalancer: false
 nodemanager:
   jvmMaxMemoryMB: 1024  # Xmx for NM JVM process.
   cpus: 0.2             # CPU needed by NM process.
-#  cgroups: true         # Whether NM should support CGroups. If set to 'true', myriad automatically
-                        # configures yarn-site.xml to attach YARN's cgroups under Mesos' cgroup hierarchy.
+
 executor:
   jvmMaxMemoryMB: 256   # Xmx for myriad's executor that launches Node Manager.
-  #path: hdfs://172.31.2.176:54310/dist/myriad-executor-runnable-x.x.x.jar # Path for the myriad's executor binary.
-  nodeManagerUri: hdfs://172.31.2.167:54310/dist/hadoop-2.6.0.tgz # the uri to d/l hadoop from   # Path to the Hadoop tarball
+  configUri: http://<RESOURCEMANAGER>:8192/api/config.tgz
 # Environment variables required to launch Node Manager process. Admin can also pass other environment variables to NodeManager.
 yarnEnvironment:
   YARN_HOME: /usr/local/hadoop # Or /usr/local/hadoop if using MapR's Hadoop
-  #YARN_HOME: hadoop-2.6.0 # Should be relative nodeManagerUri is set
-  #YARN_NODEMANAGER_OPTS: -Dnodemanager.resource.io-spindles=4.0 # Required only if using MapR's Hadoop
   JAVA_HOME: /usr/ # System dependent, but sometimes necessary
-services:
-   jobhistory:
-     command: $YARN_HOME/bin/mapred historyserver
-     jvmMaxMemoryMB: 1024
-     cpus: 1
-     maxInstances: 1
-     #ports:
-      # myriad.mapreduce.jobhistory.admin.address: -1
-      # myriad.mapreduce.jobhistory.address: -1
-      # myriad.mapreduce.jobhistory.webapp.address: -1
-     envSettings: -Dcluster.name.prefix=/mycluster
-     taskName: jobhistory
-  timelineserver:
-     command: $YARN_HOME/bin/yarn timelineserver
-     jvmMaxMemoryMB: 1024
-     cpus: 1
-     envSettings: -Dcluster.name.prefix=/mycluster2
-     taskName: timelineserver
-
+  HADOOP_CONF_DIR: /mnt/mesos/sandbox/config
+  HADOOP_TMP_DIR: /tmp
+  HADOOP_LOG_DIR: /mnt/mesos/sandbox
+  #YARN_NODEMANAGER_OPTS: -Dnodemanager.resource.io-spindles=4.0 # Required only if using MapR's Hadoop
 
 # Authentication principal for Myriad's mesos framework
 #mesosAuthenticationPrincipal: some_principal

docker/myriad-etc/yarn-site.xml.template

@@ -17,8 +17,6 @@
 -->
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
-
-
 <configuration>
 
   <!--The hostname of the Resource Manager, this MUST be changed! -->
@@ -76,6 +74,7 @@
     <name>yarn.nodemanager.resource.cpu-vcores</name>
     <value>${nodemanager.resource.cpu-vcores}</value>
    </property>
+
    <property>
      <name>yarn.scheduler.minimum-allocation-vcores</name>
      <value>0</value>
@@ -85,6 +84,7 @@
     <name>yarn.nodemanager.resource.memory-mb</name>
     <value>${nodemanager.resource.memory-mb}</value>
   </property>
+
   <property>
     <name>yarn.scheduler.minimum-allocation-mb</name>
     <value>0</value>
@@ -114,49 +114,4 @@
     <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value>
   </property>
 
-  <!-- Cgroups configuration -->
-  <!--
-  <property>
-    <description>who will execute(launch) the containers.</description>
-    <name>yarn.nodemanager.container-executor.class</name>
-    <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
-  </property>
-
-  <property>
-    <description>The class which should help the LCE handle resources.</description>
-    <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
-    <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value>
-  </property>
-
-  <property>
-    <description>The class which should help the LCE handle resources.</description>
-    <name>yarn.nodemanager.linux-container-executor.group</name>
-    <value>yarn</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
-    <value></value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
-    <value>true</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
-    <value>${yarn.nodemanager.linux-container-executor.cgroups.mount-path}</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.group</name>
-    <value>root</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.path</name>
-    <value>${yarn.home}/bin/container-executor</value>
-  </property>
--->
 </configuration>

docs/cgroups.md

@@ -1,5 +1,5 @@
 # Configuring Cgroups
-
+** CGroups are currently not supported when using Docker, we are aware this feature may be of interest and will add soon ** 
 A node within a cluster is the location where tasks are run. The tasks are launched with help of a daemon which resides inside the node. This daemon, in the case of Mesos, is Mesos Slave, and in case of YARN, is NodeManager.
 
 The Cgroups Linux kernel feature allows aggregating or partitioning a set of tasks and their future children into hierarchical groups with respect to one or more subsystems. For example, when Cgroups is enabled for the CPU subsystem, and a task is launched by Mesos Slave, it goes under the following hierarchy:

docs/myriad-multi-tenancy.md

@@ -0,0 +1,10 @@
+# Myriad Multi-tenacy
+
+Myriad supports multi-tenancy in the following ways:
+- Myriad uses Mesos to assign ports for the Node Managers.  This means if to Node Managers run on the same host, they will both be able to 
+bind to a port and communicate
+- Myriad provides mechanisms to distribute the hadoop binaries and configurations using [tarballs](https://github.com/apache/incubator-myriad/blob/master/docs/myriad-remote-distribution-configuration.md) or [Docker](https://github.com/apache/incubator-myriad/blob/master/docker/README.md).
+
+This means it's entirely possible to run multiple Myriad Clusters on the same hardware. One potential issue to be aware of however is it 
+is often important when configuring yarn to use multiple directories.  If one needs to do this one should be careful and not use the directories 
+for to different Myriad frameworks.