Merge pull request #155 from scipp/pre-commit-in-ci

Use pre-commit in ci

Author: jl-wynen

.github/workflows/pr_and_main.yml

@@ -27,7 +27,7 @@ jobs:
         with:
           python-version: 3.8
       - run: python -m pip install --upgrade pip
-      - run: python -m pip install -r requirements/static.txt
+      - run: python -m pip install -r requirements/ci.txt
       - run: tox -e static
       - uses: stefanzweifel/git-auto-commit-action@v4
         with:

.pre-commit-config.yaml

@@ -0,0 +1,63 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.3.0
+    hooks:
+      - id: check-added-large-files
+      - id: check-json
+        exclude: asv.conf.json
+      - id: check-toml
+      - id: check-yaml
+        exclude: conda/meta.yaml
+      - id: detect-private-key
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+        args: [ --markdown-linebreak-ext=md ]
+        exclude: '\.svg'
+  - repo: https://github.com/pycqa/isort
+    rev: 5.10.1
+    hooks:
+      - id: isort
+        name: isort (python)
+  - repo: https://github.com/pre-commit/mirrors-yapf
+    rev: v0.32.0
+    hooks:
+      - id: yapf
+        args: [ "-i", "-r" ]
+        types: [ "python" ]
+        additional_dependencies: [ "toml" ]
+  - repo: https://github.com/kynan/nbstripout
+    rev: 0.6.0
+    hooks:
+      - id: nbstripout
+        types: [ "jupyter" ]
+        args: [ "--drop-empty-cells",
+                "--extra-keys 'metadata.language_info.version cell.metadata.jp-MarkdownHeadingCollapsed cell.metadata.pycharm'" ]
+  - repo: https://github.com/pycqa/flake8
+    rev: 4.0.1
+    hooks:
+      - id: flake8
+        types: ["python"]
+        additional_dependencies: ["flake8-bugbear==22.10.27"]
+  - repo: https://github.com/pycqa/bandit
+    rev: 1.7.4
+    hooks:
+      - id: bandit
+        additional_dependencies: ["bandit[toml]"]
+        args: ["-c", "pyproject.toml"]
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.2.2
+    hooks:
+      - id: codespell
+        additional_dependencies:
+          - tomli
+  - repo: https://github.com/pre-commit/pygrep-hooks
+    rev: v1.9.0
+    hooks:
+      - id: python-no-eval
+        exclude: "object_list.py"
+      - id: python-no-log-warn
+      - id: python-use-type-annotations
+      - id: rst-backticks
+      - id: rst-directive-colons
+      - id: rst-inline-touching-normal
+      - id: text-unicode-replacement-char

docs/about/release-notes.rst

@@ -154,7 +154,7 @@ Breaking changes
 * A major rewrite of the reflectometry code was undertaken `#61 <https://github.com/scipp/ess/pull/61>`_:
 
   * The reflectometry and Amor Data classes were removed in favour of a formalism consisting of free-functions that accept Scipp DataArrays and Datasets.
-  * Moved the chopper class from the `wfm` submodule to free functions in its own ``choppers`` module.
+  * Moved the chopper class from the ``wfm`` submodule to free functions in its own ``choppers`` module.
   * The unit conversion (computing wavelength and Q) now use ``transform_coords``.
   * The Amor reduction notebook from ``ess-notebooks`` has been imported into the ``ess`` repository.
 

docs/conf.py

@@ -2,16 +2,16 @@
 # Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
 
 import doctest
-from datetime import date
-import ess
-
 import os
 import sys
-
+from datetime import date
 from typing import Any, Dict, Optional
+
+import sphinx_book_theme
 from docutils.nodes import document
 from sphinx.application import Sphinx
-import sphinx_book_theme
+
+import ess
 
 sys.path.insert(0, os.path.abspath('.'))
 

docs/instruments/amor/amor_reduction.ipynb

@@ -611,7 +611,7 @@
    "metadata": {},
    "source": [
     "The reference is assumed to be a perfect scatterer, therefore where there is no reflectivity in the reference measurement is it taken to be a region of `'Q'` space that cannot be accessed by the instrument. \n",
-    "This leads to the number of detectors feeding data into each $Q$-bin being variable, this is particularly noticable at low-$Q$, there there are only a few pixels detecting neutrons. \n",
+    "This leads to the number of detectors feeding data into each $Q$-bin being variable, this is particularly noticeable at low-$Q$, there there are only a few pixels detecting neutrons. \n",
     "Therefore, in order to account for this variability as a function of $Q$, we mask those pixels (performed in `normalize_by_supermirror`) where no neutrons were detected and perform an average over the remaining `'detector_id'` to reduce the data. "
    ]
   },
@@ -630,7 +630,7 @@
    "id": "d5b8ab01-4924-451e-95fb-a04365373c54",
    "metadata": {},
    "source": [
-    "To obtain the final resolution, the three components of the resolution function are combined and multipled by the midpoints of the $Q$-bins. "
+    "To obtain the final resolution, the three components of the resolution function are combined and multiplied by the midpoints of the $Q$-bins. "
    ]
   },
   {

docs/instruments/loki/sans2d_reduction.ipynb

@@ -148,7 +148,7 @@
     "# Custom SANS2D position offsets\n",
     "sample_pos_z_offset = 0.053 * sc.units.m\n",
     "bench_pos_y_offset = 0.001 * sc.units.m\n",
-    "# There is some uncertainity here\n",
+    "# There is some uncertainty here\n",
     "monitor4_pos_z_offset = -6.719 * sc.units.m\n",
     "\n",
     "# Geometry transformation\n",
@@ -189,7 +189,7 @@
     "The next step is to mask noisy and saturated pixels,\n",
     "as well as a time-of-flight range that contains spurious artifacts from the beamline components.\n",
     "\n",
-    "**Note:** We use programatic masks here and not those stored in xml files.\n",
+    "**Note:** We use programmatic masks here and not those stored in xml files.\n",
     "\n",
     "### Mask bad pixels\n",
     "\n",

docs/instruments/loki/sans2d_to_I_of_Q.ipynb

@@ -174,7 +174,7 @@
     "# Custom SANS2D position offsets\n",
     "sample_pos_z_offset = 0.053 * sc.units.m\n",
     "bench_pos_y_offset = 0.001 * sc.units.m\n",
-    "# There is some uncertainity here\n",
+    "# There is some uncertainty here\n",
     "monitor4_pos_z_offset = -6.719 * sc.units.m\n",
     "\n",
     "# Geometry transformation based on the found beam center position \n",
@@ -216,7 +216,7 @@
     "The next step is to mask noisy and saturated pixels,\n",
     "as well as a time-of-flight range that contains spurious artifacts from the beamline components.\n",
     "\n",
-    "**Note:** We use programatic masks here and not those stored in xml files.\n",
+    "**Note:** We use programmatic masks here and not those stored in xml files.\n",
     "\n",
     "### Mask bad pixels\n",
     "\n",

docs/techniques/reflectometry/reference.rst

@@ -21,10 +21,10 @@ Corrections
    corrections.normalize_by_counts
    corrections.beam_on_sample
 
-IO 
+IO
 --
 
-.. autosummary:: 
+.. autosummary::
    :toctree: ../../generated
-   
-   io.save_ort
+
+   io.save_ort

docs/techniques/wfm/reducing-wfm-data.ipynb

@@ -104,7 +104,7 @@
     "\n",
     "For the events in both `sample` and `vanadium`,\n",
     "we define a wavelength for the neutrons as well as a birth time,\n",
-    "which will be a random time between the pulse $t_0$ and the end of the useable pulse $t_0$ + pulse_length."
+    "which will be a random time between the pulse $t_0$ and the end of the usable pulse $t_0$ + pulse_length."
    ]
   },
   {

docs/version.py

@@ -1,11 +1,12 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
 
-import git
+import argparse
 import sys
 from typing import List
-from packaging.version import parse, Version, InvalidVersion
-import argparse
+
+import git
+from packaging.version import InvalidVersion, Version, parse
 
 
 def _get_releases() -> List[Version]:

pyproject.toml

@@ -7,6 +7,23 @@ build-backend = "setuptools.build_meta"
 
 [tool.setuptools_scm]
 
+[tool.bandit]
+# Excluding tests because bandit doesn't like `assert`.
+exclude_dirs = ["docs/conf.py", "docs/make_docs.py", "tests", "install", "tools"]
+
+[tool.codespell]
+ignore-words-list = "elemt"
+skip = "./.git,./install,./build,./.tox,*/*_cache,*/.virtual_documents,*/.ipynb_checkpoints,*.pdf,*.svg"
+
+[tool.isort]
+skip_gitignore = true
+line_length = 88
+multi_line_output = 2  # for compatibility with yapf
+
+[tool.mypy]
+mypy_path = "src"
+ignore_missing_imports = true
+
 [tool.pytest.ini_options]
 minversion = "7.0"
 testpaths = "tests"
@@ -22,8 +39,5 @@ filterwarnings = [
   'ignore:\n            Sentinel is not a public part of the traitlets API:DeprecationWarning',
   'ignore:Keyword `trait` is deprecated in traitlets 5.0, use `value_trait` instead:DeprecationWarning',
   'ignore:Keyword `traits` is deprecated in traitlets 5.0, use `per_key_traits` instead:DeprecationWarning',
-  'ignore:Passing None into shape arguments as an alias for ():DeprecationWarning',]
-
-[tool.mypy]
-mypy_path = "src"
-ignore_missing_imports = true
+  'ignore:Passing None into shape arguments as an alias for ():DeprecationWarning',
+]

requirements/static.in

@@ -1,5 +1 @@
-flake8
-nbstripout
-toml
-tox
-yapf
+pre-commit

requirements/static.txt

@@ -1,70 +1,27 @@
-# SHA1:1ae14843fff4b22dd16da520170445443705c399
 #
-# This file is autogenerated by pip-compile-multi
+# This file is autogenerated by pip-compile with python 3.8
 # To update, run:
 #
-#    pip-compile-multi
+#    pip-compile requirements/static.in
 #
-attrs==22.2.0
-    # via jsonschema
-cachetools==5.2.0
-    # via tox
-chardet==5.1.0
-    # via tox
-colorama==0.4.6
-    # via tox
+cfgv==3.3.1
+    # via pre-commit
 distlib==0.3.6
     # via virtualenv
-fastjsonschema==2.16.2
-    # via nbformat
 filelock==3.9.0
-    # via
-    #   tox
-    #   virtualenv
-flake8==6.0.0
-    # via -r requirements/static.in
-jsonschema==4.17.3
-    # via nbformat
-jupyter-core==5.1.2
-    # via nbformat
-mccabe==0.7.0
-    # via flake8
-nbformat==5.7.1
-    # via nbstripout
-nbstripout==0.6.1
-    # via -r requirements/static.in
-packaging==22.0
-    # via
-    #   pyproject-api
-    #   tox
+    # via virtualenv
+identify==2.5.13
+    # via pre-commit
+nodeenv==1.7.0
+    # via pre-commit
 platformdirs==2.6.2
-    # via
-    #   jupyter-core
-    #   tox
-    #   virtualenv
-pluggy==1.0.0
-    # via tox
-pycodestyle==2.10.0
-    # via flake8
-pyflakes==3.0.1
-    # via flake8
-pyproject-api==1.2.1
-    # via tox
-pyrsistent==0.19.3
-    # via jsonschema
-toml==0.10.2
-    # via -r requirements/static.in
-tomli==2.0.1
-    # via
-    #   pyproject-api
-    #   tox
-tox==4.1.3
+    # via virtualenv
+pre-commit==2.21.0
     # via -r requirements/static.in
-traitlets==5.8.0
-    # via
-    #   jupyter-core
-    #   nbformat
+pyyaml==6.0
+    # via pre-commit
 virtualenv==20.17.1
-    # via tox
-yapf==0.32.0
-    # via -r requirements/static.in
+    # via pre-commit
+
+# The following packages are considered to be unsafe in a requirements file:
+# setuptools

src/ess/__init__.py

@@ -9,5 +9,5 @@
 except ImportError:
     __version__ = "0.0.0-unknown"
 
-from .logging import get_logger
 from . import logging
+from .logging import get_logger

src/ess/amor/__init__.py

@@ -1,12 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
 # flake8: noqa: F401
-from . import data
-from . import conversions
-from . import calibrations
-from . import normalize
-from . import resolution
-from .beamline import make_beamline, instrument_view_components
+from . import calibrations, conversions, data, normalize, resolution, tools
+from .beamline import instrument_view_components, make_beamline
 from .instrument_view import instrument_view
 from .load import load
-from . import tools

src/ess/amor/beamline.py

@@ -2,6 +2,7 @@
 # Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
 import scipp as sc
 from scipp.constants import g
+
 from ..choppers import make_chopper
 from ..logging import log_call
 

src/ess/amor/calibrations.py

@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
 import scipp as sc
+
 from ..reflectometry import orso
 
 

src/ess/amor/conversions.py

@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
 import scipp as sc
+
 from ..reflectometry.conversions import specular_reflection as spec_relf_graph
 
 

src/ess/amor/instrument_view.py

@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright (c) 2023 Scipp contributors (https://github.com/scipp)
 import scipp as sc
-from scipp.plotting.objects import Plot
 import scippneutron as scn
+from scipp.plotting.objects import Plot
+
 from .beamline import instrument_view_components