Split the docs (#1963)

* generate individual conf.py files

* Generate individual html documentation for each nimi-python package

* Clean up index page

* Add TODO for release process

* Fix flake8 error

* Delete now-unused Root VERSION file

* Split LATEST_RELEASE file used for generating example.rst

* Tweak wording; nimi-python is not a package, but a repo.

* Add another TODO

* Forgot to regenerate in a previous commit

* Update nidcpower readthedocs link

* Update readthedocs links

* Delete variable that I don't need after all

* Remove TODO about leaving project name unchanged.
That value controls what appears in the top left corner of the built html docs.

* Remove CHANGELOG todo in build_release.py
CHANGELOG split is being handled in a separate PR

* Remove update links todo
We should use redirects in the ReadTheDocs project for that.

* Use mako comments so that TODO only appears in one place

* Use api-specific start year for docs copyright date

* FIgure out new release process and update build_release.py

* Undo all build_release.py changes

* Add .readthedocs.yaml
- Gives us better control over how documentation is built.
- Allows contributors who don't have the readthedocs maintenance permission to update the configuration (with permission from repo maintainers)

* Restore original conf.py until we've updated the documentation for the original ReadTheDocs projct.

* Update the original index.rst to help users find documentation

* don't delete docs/conf.py during clean; we no longer codegen it

* Update clean recipe to also delete .readthedocs.yaml
Any codegen'd file should be delete by make clean

* For new projects, add link to original project.

* Update intersphinx_mapping format

* Fix "Read the Docs" spelling

* lead with spaces on continued lines in modified GNU Make recipes

* Add empty line to end of conf.py

* Reword support.inc to not visibly mention nimi-python

Author: ni-jfitzger

README.rst

@@ -419,17 +419,17 @@ NI-TClk Python API Status
 Installation
 ============
 
-Driver specific installation instructions can be found on Read The Docs:
+Driver specific installation instructions can be found on **Read the Docs**:
 
-* `nidcpower <http://nimi-python.readthedocs.io/en/master/nidcpower.html#installation>`_
-* `nidigital <http://nimi-python.readthedocs.io/en/master/nidigital.html#installation>`_
-* `nidmm <http://nimi-python.readthedocs.io/en/master/nidmm.html#installation>`_
-* `nifgen <http://nimi-python.readthedocs.io/en/master/nifgen.html#installation>`_
-* `nimodinst <http://nimi-python.readthedocs.io/en/master/nimodinst.html#installation>`_
-* `niscope <http://nimi-python.readthedocs.io/en/master/niscope.html#installation>`_
-* `nise <http://nimi-python.readthedocs.io/en/master/nise.html#installation>`_
-* `niswitch <http://nimi-python.readthedocs.io/en/master/niswitch.html#installation>`_
-* `nitclk <http://nimi-python.readthedocs.io/en/master/nitclk.html#installation>`_
+* `nidcpower <https://nidcpower.readthedocs.io/en/latest/nidcpower.html#installation>`_
+* `nidigital <https://nidigital.readthedocs.io/en/latest/nidigital.html#installation>`_
+* `nidmm <https://nidmm.readthedocs.io/en/latest/nidmm.html#installation>`_
+* `nifgen <https://nifgen.readthedocs.io/en/latest/nifgen.html#installation>`_
+* `nimodinst <https://nimodinst.readthedocs.io/en/latest/nimodinst.html#installation>`_
+* `niscope <https://niscope.readthedocs.io/en/latest/niscope.html#installation>`_
+* `nise <https://nise.readthedocs.io/en/latest/nise.html#installation>`_
+* `niswitch <https://niswitch.readthedocs.io/en/latest/niswitch.html#installation>`_
+* `nitclk <https://nitclk.readthedocs.io/en/latest/nitclk.html#installation>`_
 
 
 Contributing
@@ -456,22 +456,20 @@ The following is a basic example of using the **nidmm** module to open a session
 Support / Feedback
 ==================
 
-The packages included in **nimi-python** package are supported by NI. For support, open
-a request through the NI support portal at `ni.com <http://www.ni.com>`_.
+For support specific to the Python API, follow the processs in `Bugs / Feature Requests`_.
+For support with hardware, the driver runtime or any other questions not specific to the Python API, please visit `NI Community Forums <https://forums.ni.com/>`_.
 
 .. _bugs-section:
 
 Bugs / Feature Requests
 =======================
 
-To report a bug or submit a feature request specific to NI Modular Instruments Python bindings (nimi-python), please use the
+To report a bug or submit a feature request specific to Python API, please use the
 `GitHub issues page <https://github.com/ni/nimi-python/issues>`_.
 
 Fill in the issue template as completely as possible and we will respond as soon
 as we can.
 
-For hardware support or any other questions not specific to this GitHub project, please visit `NI Community Forums <https://forums.ni.com/>`_.
-
 
 .. _documentation-section:
 

VERSION

@@ -23,8 +23,6 @@ endif
 ifeq (,$(MAKECMDGOALS))
 ROOT_FILES := \
 	$(ROOT_DIR)/README.rst \
-	$(ROOT_DIR)/VERSION \
-	$(DOCS_DIR)/conf.py \
 
 endif
 
@@ -79,9 +77,9 @@ clean: start
 	-$(_hide_cmds)find $(ROOT_DIR)/build -name __pycache__ -exec rmdir {} \; $(SUPPRESS_ERROR_OUTPUT) ||:
 	-$(_hide_cmds)find $(ROOT_DIR)/build -name '*.pyc' -exec rm {} \; $(SUPPRESS_ERROR_OUTPUT) ||:
 	-$(_hide_cmds)rm $(foreach d,$(DRIVERS), $(ROOT_DIR)/docs/$(d)/*) $(SUPPRESS_ERROR_OUTPUT) ||:
+	-$(_hide_cmds)rm $(foreach d,$(DRIVERS), $(ROOT_DIR)/docs/$(d)/.readthedocs.yaml) $(SUPPRESS_ERROR_OUTPUT) ||:
 	-$(_hide_cmds)rm -Rf $(ROOT_DIR)/.coverage $(SUPPRESS_ERROR_OUTPUT) ||:
 	-$(_hide_cmds)rm -Rf $(ROOT_DIR)/README.rst $(SUPPRESS_ERROR_OUTPUT) ||:
-	-$(_hide_cmds)rm -Rf $(ROOT_DIR)/docs/conf.py $(SUPPRESS_ERROR_OUTPUT) ||:
 	-$(_hide_cmds)mkdir $(GENERATED_DIR) $(SUPPRESS_ERROR_OUTPUT) ||:
 
 start:
@@ -147,16 +145,15 @@ endef
 
 $(ROOT_DIR)/README.rst: $(GLOBAL_FILES_STARTED_FILE)
 	$(call trace_to_console, "Creating Root",$(notdir $@))
-	$(_hide_cmds)$(call global_log_command,cat $(STATIC_DOCS_DIR)/status_project.inc $(STATIC_DOCS_DIR)/about.inc $(DOCS_DIR)/*/status.inc $(STATIC_DOCS_DIR)/installation.inc $(STATIC_DOCS_DIR)/contributing.inc $(STATIC_DOCS_DIR)/nidmm_usage.inc $(STATIC_DOCS_DIR)/support.inc $(STATIC_DOCS_DIR)/documentation.inc $(STATIC_DOCS_DIR)/license.inc > $@)
-
-$(ROOT_DIR)/VERSION: $(GLOBAL_FILES_STARTED_FILE)
-	$(call trace_to_console, "Creating Root",$(notdir $@)) tools/update_version_file.py
-	$(_hide_cmds)$(call global_log_command,$(PYTHON_CMD) tools/update_version_file.py --output-file $@ $(foreach d,$(DRIVERS),--input-file $(GENERATED_DIR)/$(d)/$(d)/VERSION ))
-
-$(DOCS_DIR)/conf.py: build/templates/conf.py.mako $(ROOT_DIR)/VERSION tools/simple_mako.py $(GLOBAL_FILES_STARTED_FILE)
-	$(call trace_to_console, "Creating Root",$(notdir $@))
-	$(_hide_cmds)$(call global_log_command,$(PYTHON_CMD) tools/simple_mako.py --output-file $@ --template $<)
-
+	$(_hide_cmds)$(call global_log_command,cat $(STATIC_DOCS_DIR)/status_project.inc \
+                                               $(STATIC_DOCS_DIR)/about.inc \
+                                               $(DOCS_DIR)/*/status.inc \
+                                               $(STATIC_DOCS_DIR)/installation.inc \
+                                               $(STATIC_DOCS_DIR)/contributing.inc \
+                                               $(STATIC_DOCS_DIR)/nidmm_usage.inc \
+                                               $(STATIC_DOCS_DIR)/support.inc \
+                                               $(STATIC_DOCS_DIR)/documentation.inc \
+                                               $(STATIC_DOCS_DIR)/license.inc > $@)
 
 # Any step that any driver build does that would invalidate unit testing, flake8 or generated html
 # needs to delete this file. This will trigger a tox run.

build/Makefile

@@ -32,7 +32,7 @@ GRPC_SUPPORTED := $(if $(wildcard $(PROTO_FILE)),$(true))
 PYTHON_CMD ?= python
 GRPC_SUPPORT_PARAM := $(if $(GRPC_SUPPORTED),--include-grpc-support)
 define GENERATE_SCRIPT
-$(PYTHON_CMD) -m build --template $1 --dest-dir $2 --metadata $3 $(if $(PRINT),-v,) $(GRPC_SUPPORT_PARAM)
+$(PYTHON_CMD) -m build --template $1 --dest-dir $2 --metadata $3 $(if $4,--dest-file $4) $(if $(PRINT),-v,) $(GRPC_SUPPORT_PARAM)
 endef
 
 ifeq (,$(PRINT))
@@ -75,6 +75,8 @@ DEFAULT_PY_FILES_TO_COPY := \
     _visatype.py \
 
 DEFAULT_RST_FILES_TO_GENERATE := \
+    index.rst \
+    $(DRIVER).rst \
     enums.rst \
     examples.rst \
     installation.inc \
@@ -87,6 +89,8 @@ DEFAULT_RST_FILES_TO_GENERATE := \
         grpc_session_options.rst \
     ) \
 
+DEFAULT_SPHINX_CONF_PY := $(DRIVER_DOCS_DIR)/conf.py
+DEFAULT_READTHEDOCS_CONFIG := $(DRIVER_DOCS_DIR)/.readthedocs.yaml
 
 # Files for tracking parts of the build
 SDIST_WHEEL_BUILD_DONE := $(LOG_DIR)/sdist_wheel_build_done

build/defines.mak

@@ -30,8 +30,6 @@ endif # ifneq (,$(EXAMPLE_FILES))
 
 MKDIR: $(MKDIRECTORIES)
 
-CURRENT_DIR := $(shell pwd)
-
 define mkdir_rule
 $1:
 	$(call trace_to_console, "Making dir",$1)
@@ -82,10 +80,22 @@ $(DRIVER_DOCS_DIR)/%.rst: %.rst.mako $(BUILD_HELPER_SCRIPTS) $(METADATA_FILES)
 	$(call trace_to_console, "Generating",$@)
 	$(_hide_cmds)$(call log_command,$(call GENERATE_SCRIPT, $<, $(dir $@), $(METADATA_DIR)))
 
+$(DRIVER_DOCS_DIR)/$(DRIVER).rst: driver.rst.mako $(BUILD_HELPER_SCRIPTS) $(METADATA_FILES)
+	$(call trace_to_console, "Generating",$@)
+	$(_hide_cmds)$(call log_command,$(call GENERATE_SCRIPT, $<, $(dir $@), $(METADATA_DIR), $(notdir $@)))
+
 $(DRIVER_DOCS_DIR)/%.inc: %.inc.mako $(BUILD_HELPER_SCRIPTS) $(METADATA_FILES)
 	$(call trace_to_console, "Generating",$@)
 	$(_hide_cmds)$(call log_command,$(call GENERATE_SCRIPT, $<, $(dir $@), $(METADATA_DIR)))
 
+$(SPHINX_CONF_PY): $(TEMPLATE_DIR)/conf.py.mako $(BUILD_HELPER_SCRIPTS) $(METADATA_FILES)
+	$(call trace_to_console, "Generating",$@)
+	$(_hide_cmds)$(call log_command,$(call GENERATE_SCRIPT, $<, $(dir $@), $(METADATA_DIR)))
+
+$(READTHEDOCS_CONFIG): $(TEMPLATE_DIR)/.readthedocs.yaml.mako $(BUILD_HELPER_SCRIPTS) $(METADATA_FILES)
+	$(call trace_to_console, "Generating",$@)
+	$(_hide_cmds)$(call log_command,$(call GENERATE_SCRIPT, $<, $(dir $@), $(METADATA_DIR)))
+
 $(DRIVER_EXAMPLES_ZIP_FILE): $(EXAMPLE_FILES)
 	$(call trace_to_console, "Zipping",$@)
 	$(_hide_cmds)$(call log_command,cd src/$(DRIVER)/examples && zip -u -r -9 $@ * || ([ $$? -eq 12 ] && exit 0) || exit)
@@ -101,7 +111,7 @@ clean:
 
 .PHONY: module doc_files sdist wheel installers
 module: $(MODULE_FILES) $(UNIT_TEST_FILES)
-doc_files: $(RST_FILES)
+doc_files: $(RST_FILES) $(SPHINX_CONF_PY) $(READTHEDOCS_CONFIG)
 installers: sdist wheel
 
 $(UNIT_TEST_FILES): $(MODULE_FILES)
@@ -131,7 +141,15 @@ else
 # We piece together the readme files instead of relying on the rst include directive because we need these files to be standalone and not require any additional files that are in specific locations.
 $(README): $(RST_FILES) $(wildcard $(STATIC_DOCS_DIR)/*)
 	$(call trace_to_console, "Creating",$@)
-	$(_hide_cmds)$(call log_command,cat $(STATIC_DOCS_DIR)/status_project.inc $(STATIC_DOCS_DIR)/about.inc $(DRIVER_DOCS_DIR)/status.inc $(DRIVER_DOCS_DIR)/installation.inc $(STATIC_DOCS_DIR)/contributing.inc $(STATIC_DOCS_DIR)/$(DRIVER)_usage.inc $(STATIC_DOCS_DIR)/support.inc $(STATIC_DOCS_DIR)/documentation.inc $(STATIC_DOCS_DIR)/license.inc > $@)
+	$(_hide_cmds)$(call log_command,cat $(STATIC_DOCS_DIR)/status_project.inc \
+                                        $(STATIC_DOCS_DIR)/about.inc \
+                                        $(DRIVER_DOCS_DIR)/status.inc \
+                                        $(DRIVER_DOCS_DIR)/installation.inc \
+                                        $(STATIC_DOCS_DIR)/contributing.inc \
+                                        $(STATIC_DOCS_DIR)/$(DRIVER)_usage.inc \
+                                        $(STATIC_DOCS_DIR)/support.inc \
+                                        $(STATIC_DOCS_DIR)/documentation.inc \
+                                        $(STATIC_DOCS_DIR)/license.inc > $@)
 
 endif
 

build/rules.mak

@@ -0,0 +1,60 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+<%
+    config        = template_parameters['metadata'].config
+    module_name   = config['module_name']
+    # All of the files used to configure and build docs and readthedocs are in these 2 folders
+    build_trigger_paths = f'docs/_static/ docs/{module_name}/'
+    conf_py_path = f'docs/{module_name}/conf.py'
+%>\
+
+# Why Use A Configuration File?
+# https://docs.readthedocs.io/en/stable/config-file/index.html
+# The main advantages of using a configuration file over the web interface are:
+# * Settings are per version rather than per project.
+# * Settings live in your VCS.
+# * They enable reproducible build environments over time.
+# * Some settings are only available using a configuration file
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+  jobs:
+    # pre_build:
+    #   # Check for broken external links
+    #   - python -m sphinx -b linkcheck -D linkcheck_timeout=1 docs/ _build/linkcheck
+    post_checkout:
+      # https://docs.readthedocs.io/en/stable/build-customization.html#cancel-build-based-on-a-condition
+      # Build-cancellation rules are recommended for monorepos.
+      # Cancel building pull requests when there aren't changes in any of these paths: ${build_trigger_paths}.
+      #
+      # If there are no changes (git diff exits with 0) we force the command to return with 183.
+      # This is a special exit code on Read the Docs that will cancel the build immediately.
+      - |
+        if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/master -- ${build_trigger_paths};
+        then
+          exit 183;
+        fi
+
+# Have Read the Docs build documentation with Sphinx
+sphinx:
+  builder: html
+  configuration: ${conf_py_path}
+
+# If using Sphinx, optionally build your docs in additional formats such as PDF
+formats:
+  - epub
+  - pdf
+
+# Optionally declare the Python requirements required to build your docs
+## TODO(ni-jfitzger): Create requirements file for docs to make builds reproducible. See https://github.com/ni/nimi-python/issues/1968
+## Note: Our nimi-python readthedocs project used the defaults here: https://docs.readthedocs.io/en/stable/build-default-versions.html#external-dependencies
+# python:
+#   install:
+#   - requirements: docs/requirements.txt

build/templates/.readthedocs.yaml.mako

@@ -2,14 +2,18 @@
 import datetime
 current_year = datetime.datetime.today().year
 
-with open('./VERSION') as vf:
-    global_version = vf.read().strip()
+config = template_parameters['metadata'].config
+copyright_start_year = config['initial_release_year']
+version = config['module_version']
 
+api_name = f"{config['driver_name']} Python API"
+api_name_no_spaces_or_hyphens = api_name.replace(" ", "").replace("-", "")
+api_name_no_spaces_or_hyphens_lower = api_name_no_spaces_or_hyphens.lower()
 %>\
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #
-# NI Modular Instruments Python API documentation build configuration file, created by
+# ${api_name} documentation build configuration file, created by
 # sphinx-quickstart on Fri Jul 14 13:04:36 2017.
 #
 # This file is execfile()d with the current directory set to its
@@ -62,16 +66,16 @@ source_suffix = '.rst'
 master_doc = 'index'
 
 # General information about the project.
-project = 'NI Modular Instruments Python API'
-copyright = '2017-${current_year}, National Instruments Corporation'
+project = '${api_name}'
+copyright = '${copyright_start_year}-${current_year}, National Instruments Corporation'
 author = 'NI'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The full version, including alpha/beta/rc tags.
-release = '${global_version}'
+release = '${version}'
 # The short X.Y version.
 version = release[:3]
 
@@ -112,7 +116,7 @@ html_theme_options = {
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ['../_static']
 
 # Fix wide tables of RTD per https://github.com/rtfd/sphinx_rtd_theme/issues/117#issuecomment-41571653
 def setup(app):
@@ -164,7 +168,7 @@ latex_elements = {
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'NIModularInstrumentsPythonAPI.tex', 'NI Modular Instruments Python API Documentation',
+    (master_doc, '${api_name_no_spaces_or_hyphens}.tex', '${api_name} Documentation',
      'NI', 'manual'),
 ]
 
@@ -174,7 +178,7 @@ latex_documents = [
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
-    (master_doc, 'nimodularinstrumentspythonapi', 'NI Modular Instruments Python API Documentation',
+    (master_doc, '${api_name_no_spaces_or_hyphens_lower}', '${api_name} Documentation',
      [author], 1)
 ]
 
@@ -185,11 +189,12 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'NIModularInstrumentsPythonAPI', 'NI Modular Instruments Python API Documentation',
-     author, 'NIModularInstrumentsPythonAPI', 'One line description of project.',
+    (master_doc, '${api_name_no_spaces_or_hyphens}', '${api_name} Documentation',
+     author, '${api_name_no_spaces_or_hyphens}', 'One line description of project.',
      'Miscellaneous'),
 ]
 
 # Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'https://docs.python.org/': None}
-
+## TODO(ni-jfitzger): Add mappings for nimi-python APIs that reference other nimi-python APIs.
+## We can probably just list all of the mappings (other than maybe the current module, I think)
+intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}

build/templates/conf.py.mako

@@ -0,0 +1,16 @@
+<%
+    '''This is a template for the module-specific .rst'''
+
+    config        = template_parameters['metadata'].config
+    module_name   = config['module_name']
+    doc_header    = f"{module_name} module"
+%>\
+${doc_header}
+${"=" * len(doc_header)}
+
+.. include:: installation.inc
+
+.. include:: ../_static/${module_name}_usage.inc
+
+.. include:: toc.inc
+

build/templates/driver.rst.mako

@@ -25,7 +25,7 @@
     #       - (1) will link to the zip file for the current release
     #       - (2) will include current code snippet and URL will point to release version
 
-    with open('./LATEST_RELEASE') as vf:
+    with open(f'./src/{module_name}/LATEST_RELEASE') as vf:
         latest_release_version = vf.read().strip()
     released_zip_url = 'https://github.com/ni/nimi-python/releases/download/{0}/{1}_examples.zip'.format(latest_release_version, module_name)
 

build/templates/examples.rst.mako

@@ -0,0 +1,37 @@
+<%
+    config        = template_parameters['metadata'].config
+    doc_header    = f"{config['driver_name']} Python API Documentation"
+    module_name   = config['module_name']
+    driver_name   = config['driver_name']
+%>\
+
+${doc_header}
+${"=" * len(doc_header)}
+
+.. include:: ../_static/about_driver.inc
+
+.. include:: ../_static/contributing.inc
+
+.. include:: ../_static/support.inc
+
+.. toctree::
+   :maxdepth: 3
+   :caption: Documentation
+
+   ${module_name}
+
+Additional Documentation
+------------------------
+
+Refer to your driver documentation for device-specific information and detailed API documentation.
+
+Refer to the `nimi-python Read the Docs project <https://nimi-python.readthedocs.io/en/stable/>`_ for documentation of versions 1.4.4 of the module or earlier.
+
+.. include:: ../_static/license.inc
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`