[DCOS-51637] Add tooling support for submodule-based projects. (#3046)

* Abide to shellcheck.

* Improve formatting.

* Update pre-commit to 1.16.1 (latest).

* Silence deprecation warning.

* Adapt tooling to allow running from projects with dcos-commons as a submodule.

* Improvements to bash scripts.

* Default Docker image tag to latest stable (0.55.5).

* Inspect AWS credentials file.

* Use test_runner from git submodule.

* Add 'run_container.sh' alias for 'test.sh'.

* Handle wiring of dcos-commons submodule git directory.

* Change dcos-commons Docker image used by tooling to "latest".

* Pull Docker image it explicitly.

* Improve help output.

* Cause formatting issues to actually return a non-zero exit code.

Since version 18.4b3 the '--diff' and '--check' aren't mutually exclusive. We
were only setting the '--diff' flag in our black hook, which means no formatting
errors were being raised. Adding back the '--check' flag fixes that.

https://github.com/python/black#184a3

* Handle PYTHONPATH setup in test.sh.

* Remove unused pre-commit hook.

* Don't need to run this in interactive mode.

.pre-commit-config.yaml

@@ -1,23 +1,19 @@
 repos:
--   repo: https://github.com/ambv/black
-    rev: stable
-    hooks:
-    - id: black
-      args: [--diff]
-      language_version: python3
-    - id: black
-      name: black-format
-      language_version: python3
-      stages: [manual]
--   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v1.3.0
-    hooks:
-    - id: flake8
--   repo: local
-    hooks:
-    - id: pylint
-      name: pylint
-      entry: pylint
-      args: [-E, -j4, --disable=invalid-sequence-index, --disable=no-member, --disable=no-name-in-module]
-      language: system
-      types: [python]
+- repo: https://github.com/python/black
+  rev: stable
+  hooks:
+  - id: black
+    args: [--diff, --check]
+    language_version: python3.6
+- repo: https://gitlab.com/pycqa/flake8
+  rev: 3.7.7
+  hooks:
+  - id: flake8
+- repo: local
+  hooks:
+  - id: pylint
+    name: pylint
+    entry: pylint
+    args: [-E, -j4, --disable=invalid-sequence-index, --disable=no-member, --disable=no-name-in-module]
+    language: system
+    types: [python]

frozen_requirements.txt

@@ -82,7 +82,7 @@ pathlib2==2.3.3
 pefile==2018.8.8
 pkginfo==1.2.1
 pluggy==0.8.1
-pre-commit==1.14.4
+pre-commit==1.16.1
 prettytable==0.7.2
 py==1.7.0
 pyasn1==0.4.5

pyproject.toml

@@ -1,3 +1,3 @@
 [tool.black]
 line-length = 100
-py36 = true
+target-version = ["py36"]

run_container.sh

@@ -0,0 +1 @@
+test.sh