[LIMINAL-86] Enable Black - Python code formatter

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,23 @@
+## Changes Title (replace this with a logical title for your changes)
+
+*Issue #, if available:*
+
+### *Description of changes:*
+
+Replace this with the PR description (mention the changes you have made, why you have made them, provide some background and any references to the provider documentation if needed, etc.).
+
+For more information on contributing, please see [Contributing](../CONTRIBUTING.md)
+section of our documentation.
+
+### Status
+
+Replace this: describe the PR status. Examples:
+
+- WIP - work in progress
+- DONE - ready for review
+
+### Checklist (tick everything that applies)
+
+- [ ] [PreCommitChecks - Code linting](../CONTRIBUTING.md#InstallRunPreCommit) (required)
+- [ ] [Tests](../CONTRIBUTING.md#RunningTests)
+

.github/workflows/pre_commits.yml

@@ -0,0 +1,57 @@
+name: PreCommitChecks
+
+on:
+  push:
+    branches:
+      - master
+      # - '!release*'
+  pull_request:
+    branches:
+      - master
+      # - '!release*'
+jobs:
+  linting:
+    name: "Run pre-commit hooks on py3.6"
+    runs-on: ubuntu-latest
+    steps:
+      #----------------------------------------------
+      # Checkout, SetUp Python, Load Cache and Run PreCommitChecks
+      #----------------------------------------------
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      # Install Python
+      - uses: actions/setup-python@v2
+        with:
+            python-version: 3.6
+
+      - name: Check Python ${{ matrix.python-version }} version
+        run: python -V
+
+      - uses: actions/cache@v2
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip
+          restore-keys: ${{ runner.os }}-pip
+
+      - name: Install python requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install pre-commit
+
+      # load cached venv if cache exists
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ hashFiles('**/requirements.txt') }}
+
+      - run: pre-commit install
+      - name: Run pre-commit hooks on all the files
+        run: pre-commit run --all-files --show-diff-on-failure --color always --verbose
+      # - uses: pre-commit/[email protected]
+      #   with:
+      #     token: ${{ secrets.GITHUB_TOKEN }}

.pre-commit-config.yaml

@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+minimum_pre_commit_version: "2.16.0"
+exclude: >
+  (?x)^(
+      .+/.venv/.+|.+/dist/.+|.+/.autovenv|.+/docs/|.github
+  )$
+fail_fast: true
+default_language_version:
+  python: python3
+default_stages:
+  - prepare-commit-msg
+  - commit
+  # - push
+
+repos:
+  - repo: https://github.com/psf/black
+    rev: 21.12b0
+    hooks:
+      - id: black
+        args: [ --safe, --quiet ]
+        files: \.pyi?$
+        exclude: .github/
+        # override until resolved: https://github.com/psf/black/issues/402
+        types: [ ]

CONTRIBUTING.md

@@ -24,10 +24,10 @@ Apache Liminal use Sphinx and readthedocs.org to create and publish documentatio
 ## Basic setup
 Here is what you need to do in order to work and create docs locally
 
-## Installing sphinx and documentation 
+## Installing sphinx and documentation
 
 More details  about sphinx and readthedocs can be found [here:](https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html)
- 
+
 Here is what you need to do:
 ```
 pip install sphinx
@@ -44,15 +44,58 @@ pip install sphinx-autobuild
 cd docs
 sphinx-autobuild source/ build/html/
 ```
-Now open a browser on 
+Now open a browser on
 http://localhost:8000/
 Your docs will automatically update
 
 ## Rebuilding the pythondoc liminal code documentation
+
 ```
 cd docs
 sphinx-apidoc -o source/ ../liminal
 ```
 
+## RunningTests
+
+When doing local development and running Liminal unit-tests, make sure to set LIMINAL_STAND_ALONE_MODE=True
+
+1. Setup Minikube
+2.Install python requirements: `pip install -r requirements.txt`
+3. Run tests: `./run_tests.sh`
+
+## InstallRunPreCommit
+
+[pre-commit](https://pre-commit.com/) is used to install Python code linting and formatting tools:
+
+### Getting started
+
+**Requires `python >=3.6`, `pre-commit>=1.14` and a `git` repository**
+
+1. Run `pip install pre-commit` or `pip install  -r requirements.txt`
+2. Install the hooks:
+  Simple install without post hooks: `pre-commit install`
+
+  OR
+  Install the hooks: `pre-commit install --install-hooks`
+
+  Optional:
+  Install the post commit hooks: `pre-commit install --hook-type post-commit`
+
+1. To run pre commit hooks:
+   1. Either run `git commit` the new configuration files
+
+   1. Run `pre-commit run -a` to lint and format your entire project
+
+Now on every commit, `pre-commit` will use a git hook to run the tools.
+**Warning: the first commit will take some time because the tools are being installed by
+`pre-commit`**
+
+`pre-commit run -a` to lint and format your entire project
+
+### Resolving failed commits
+
+* If `black` fail, they have reformatted your code. `git add` and `git commit` the changes.
 
+Example:
 
+![Fixing black failed commits](https://user-images.githubusercontent.com/16241795/146011210-7bc11b24-2033-43f7-8150-5ece4fe7bfea.png)

README.md

@@ -35,15 +35,19 @@ Using simple YAML configuration, create your own schedule data pipelines (a sequ
 perform), application servers,  and more.
 
 ## Getting Started
+
 A simple getting stated guide for Liminal can be found [here](docs/getting_started.md)
 
 ## Apache Liminal Documentation
+
 Full documentation of Apache Liminal can be found [here](docs/liminal)
 
 ## High Level Architecture
+
 High level architecture documentation can be found [here](docs/architecture.md)
 
 ## Example YAML config file
+
 ```yaml
 ---
 name: MyLiminalStack
@@ -104,13 +108,16 @@ services:
     image: my_server_image
 ```
 
-
 # Installation
+
 1. Install this repository (HEAD)
+
 ```bash
    pip install git+https://github.com/apache/incubator-liminal.git
 ```
+
 2. Optional: set LIMINAL_HOME to path of your choice (if not set, will default to ~/liminal_home)
+
 ```bash
 echo 'export LIMINAL_HOME=</path/to/some/folder>' >> ~/.bash_profile && source ~/.bash_profile
 ```
@@ -119,34 +126,36 @@ echo 'export LIMINAL_HOME=</path/to/some/folder>' >> ~/.bash_profile && source ~
 
 This involves at minimum creating a single file called liminal.yml as in the example above.
 
-If your pipeline requires custom python code to implement tasks, they should be organized 
+If your pipeline requires custom python code to implement tasks, they should be organized
 [like this](https://github.com/apache/incubator-liminal/tree/master/tests/runners/airflow/liminal)
 
-If your pipeline  introduces imports of external packages which are not already a part 
-of the liminal framework (i.e. you had to pip install them yourself), you need to also provide 
+If your pipeline  introduces imports of external packages which are not already a part
+of the liminal framework (i.e. you had to pip install them yourself), you need to also provide
 a requirements.txt in the root of your project.
 
 # Testing the pipeline locally
 
 When your pipeline code is ready, you can test it by running it locally on your machine.
 
 1. Ensure you have The Docker engine running locally, and enable a local Kubernetes cluster:
-![Kubernetes configured](https://raw.githubusercontent.com/apache/incubator-liminal/master/images/k8s_running.png)
 
-And allocate it at least 3 CPUs (under "Resources" in the Docker preference UI).
+  ![Kubernetes configured](https://raw.githubusercontent.com/apache/incubator-liminal/master/images/k8s_running.png)
+
+  And allocate it at least 3 CPUs (under "Resources" in the Docker preference UI).
+
+  If you want to execute your pipeline on a remote kubernetes cluster, make sure the cluster is configured using:
+
+  ```bash
+  kubectl config set-context <your remote kubernetes cluster>
+  ```
 
-If you want to execute your pipeline on a remote kubernetes cluster, make sure the cluster is configured
-using :
-```bash
-kubectl config set-context <your remote kubernetes cluster>
-``` 
 2. Build the docker images used by your pipeline.
 
-In the example pipeline above, you can see that tasks and services have an "image" field - such as 
-"my_static_input_task_image". This means that the task is executed inside a docker container, and the docker container 
+In the example pipeline above, you can see that tasks and services have an "image" field - such as
+"my_static_input_task_image". This means that the task is executed inside a docker container, and the docker container
 is created from a docker image where various code and libraries are installed.
 
-You can take a look at what the build process looks like, e.g. 
+You can take a look at what the build process looks like, e.g.
 [here](https://github.com/apache/incubator-liminal/tree/master/liminal/build/image/python)
 
 In order for the images to be available for your pipeline, you'll need to build them locally:
@@ -161,34 +170,41 @@ You'll see that a number of outputs indicating various docker images built.
 3. Create a kubernetes local volume \
 In case your Yaml includes working with [volumes](https://github.com/apache/incubator-liminal/blob/6253f8b2c9dc244af032979ec6d462dc3e07e170/docs/getting_started.md#mounted-volumes)
 please first run the following command:
+
 ```bash
-cd </path/to/your/liminal/code> 
+cd </path/to/your/liminal/code>
 liminal create
 ```
 
 4. Deploy the pipeline:
+
 ```bash
-cd </path/to/your/liminal/code> 
+cd </path/to/your/liminal/code>
 liminal deploy
 ```
-Note: after upgrading liminal, it's recommended to issue the command 
+
+Note: after upgrading liminal, it's recommended to issue the command
+
 ```bash
 liminal deploy --clean
-``` 
+```
 
 This will rebuild the airlfow docker containers from scratch with a fresh version of liminal, ensuring consistency.
 
 5. Start the server
+
 ```bash
 liminal start
 ```
 
 6. Stop the server
+
 ```bash
 liminal stop
 ```
 
 7. Display the server logs
+
 ```bash
 liminal logs --follow/--tail
 
@@ -205,8 +221,9 @@ liminal logs --follow
 The pipeline is scheduled to run according to the ```json schedule: 0 * 1 * *``` field in the .yml file you provided.
 
 10. To manually activate your pipeline:
-Click your pipeline and then click "trigger DAG"
-Click "Graph view"
+
+- Click your pipeline and then click "trigger DAG"
+- Click "Graph view"
 You should see the steps in your pipeline getting executed in "real time" by clicking "Refresh" periodically.
 
 ![Pipeline activation](https://raw.githubusercontent.com/apache/incubator-liminal/master/images/airflow_trigger.png)
@@ -215,5 +232,13 @@ You should see the steps in your pipeline getting executed in "real time" by cli
 
 More information on contributing can be found [here](CONTRIBUTING.md)
 
+# Community
+
+The Liminal community holds a public call every Monday
+
+- [Liminal Community Calendar](https://calendar.google.com/calendar/u/0/[email protected])
+- [Dev-Mailing-List](https://lists.apache.org/[email protected])
+
 ## Running Tests (for contributors)
+
 When doing local development and running Liminal unit-tests, make sure to set LIMINAL_STAND_ALONE_MODE=True