Merge pull request #49 from OpenDataServices/schema

Rewrite docs/development/schema.md and related components and patterns

Author: duncandewhurst

codelist-schema.json

@@ -0,0 +1,14 @@
+{
+    "$defs": {
+        "Row": {
+            "properties": {
+                "Deprecated": {
+                    "$ref": "https://codelist-schema.readthedocs.io/1__0__0/codelist-schema.json#/$defs/Row/properties/Deprecated"
+                },
+                "Deprecation note": {
+                    "$ref": "https://codelist-schema.readthedocs.io/1__0__0/codelist-schema.json#/$defs/Row/properties/Deprecation note"
+                }
+            }
+        }
+    }    
+}

docs/components/index.md

@@ -515,6 +515,12 @@ Allowing conversion between serialization formats (e.g. CSV -> XML; JSON -> XLS)
 
 Data standards often use structured data formats such as JSON or XML to give more flexbility in modelling and to allow validation against schema. Typically, developers prefer to work with structured data formats as they are easier to work with in programs. However, JSON and XML aren't very human-friendly, and people working with data in many domains prefer to use flat representations of data such as CSV and XLSX spreadsheets, both for publishing and manipulating data. Conversion tools allow conversion between the formats, to allow the standard and developers to retain the benfits of a structured data format and users to continue to be able to engage with the data in a way that they're comfortable with.
 
+#### Examples
+
+##### Flatten Tool
+
+We maintain [Flatten Tool](http://flatten-tool.readthedocs.io), a Python library and command-line interface for converting data between structured formats like JSON and XML and tabular formats like CSV and XLSX. It can use a standard's schema to handle data types correctly, to produce human-readable column headings, and to structure tabular data helpfully.
+
 #### Prioritisation Factors
 
 - If the standard uses a structured data format, while data publishers and/or users prefer flat representations.

docs/conf.py

@@ -30,7 +30,7 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['sphinxcontrib.mermaid', 'myst_parser', 'sphinxcontrib.mermaid']
+extensions = ['sphinxcontrib.mermaid', 'myst_parser', 'sphinxcontrib.mermaid', 'sphinx_togglebutton', 'sphinx_design', 'sphinxcontrib.jsonschema']
 
 # Myst parser configuration
 
@@ -359,6 +359,6 @@
 
 
 linkcheck_ignore = [
-    # The ODI is now behind a Clouflare challenge that we can't check.
+    # The ODI is now behind a Cloudflare challenge that we can't check.
     'http://www.theodi.org'
 ]

docs/development/schema.md

@@ -1,75 +1,136 @@
-# Schema development
+# Data modelling and schema development
 
-This section outlines a number of the pattens we commonly use to develop the schema for an open data standard.
+This page provides an overview of the steps involved in data modelling and schema development.
 
-```{admonition} Learning / reflection
----
-class: note
----
-We currently jump straight from the conceptual framework document, to working up the data model and schema for a standard in a schema language. 
+## Document a data model
 
-This differs [from the approach proposed here](https://github.com/open-contracting-archive/technical-approach#data-model) of maintaining the **data model** as a narrative document, only then given form by a schema as a reference implementation. 
+A data model is an abstract model that organizes elements of data and standardises how they relate to one another and to the properties of real-world entities. A data model focuses on what data represents rather than how it is stored or exchanged.
+
+Before authoring the schema for a standard and committing to specific implementation details, it is recommended to document a data model to help stakeholders align on definitions and relationships.
+
+The data model for a standard should be based on [research](research.md) into the related policy area and a thorough understanding of the concepts which underpin it (a conceptual model). Documenting the data model for a standard involves identifying and defining the entities (classes), attributes (properties), relationships and permissable values (codelists) needed to satisfy the requirements, user stories and use cases for the standard.
+
+Developing a good data model is an art as much as a science. It requires sensitivity to the needs of both data producers and data users, and an understanding of the incentive structures that will drive adoption of a standard.
+
+The recommended approach is to document the data model using the [standard development template](../tools.md#standard-development-template-airtable), which ensures that the data model is grounded through explicit links to the requirements, user stories and use cases.
+
+```{admonition} History
+:class: dropdown
+
+Previously, we moved straight from documenting a conceptual framework to documenting a schema. The reasons for documenting a data model are explored in the [technical scoping](https://github.com/open-contracting-archive/technical-approach?tab=readme-ov-file#data-model) for the Open Contracting Data Standard.
 
 ```
 
-## Schema language
+## Choose your publication formats
+
+A publication format is a format in which data can be published by implementers of a standard. Common publication formats include:
+
+* [JSON](https://www.json.org/json-en.html)
+* [GeoJSON](https://datatracker.ietf.org/doc/html/rfc7946)
+* [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) and other tabular formats, such as XLSX and ODS.
+* [XML](https://www.w3.org/TR/xml/)
+
+Based on your [research](research.md), you need to decide which publication formats to support.
 
-Our preferred schema language is [JSON Schema v0.4](https://tools.ietf.org/html/draft-zyp-json-schema-04).
+It is [best practice](https://www.w3.org/TR/dwbp/#MultipleFormats) for data publishers to provide data in multiple formats, so that as many users as possible can use the data without first having to transform it to their preferred format. Therefore, you should consider how to support publication in multiple formats.
 
-This allows us to provide field structures and definitions. Although less expressive than other schema languages, the constraints of JSON Schema enable us to focus on keeping data simple enough for a wide range of users.
+On a technical level, the recommended approach is to use JSON as the primary format around which a standard's tools are built, and to provide support for other formats through conversion tooling. Depending on the user needs identified in your research, a standard's documentation site and tooling might present an alternative format, such as CSV, as the primary format.
 
-We generally use simple CSV files to represent codelists.
+Open Data Services' reusable tools for documenting, converting and validating data are built around JSON. If your research surfaces demand for a different primary format that cannot be supported through conversion to JSON, you should consider the potential costs associated with authoring new tooling.
 
-We have a number of extensions to JSON Schema 0.4 we use (documented below).
+```{admonition} Example: The Open Contracting Data Standard
+:class: note
 
-## Serializations
+The primary publication format of the Open Contracting Data Standard is JSON, but CSV and spreadsheet formats are also supported via conversion tooling. For more information, see [Serialization (Open Contracting Data Standard Documentation)](https://standard.open-contracting.org/latest/en/guidance/build/serialization/#serialization).
 
-We design with a range of serializations in mind, and, where possible, to enable round-tripping of data between different serializations.
+```
 
-In particular, through [flatten-tool](http://flatten-tool.readthedocs.io) we design with support for:
+```{admonition} Example: 360Giving
+:class: note
 
-- Structured JSON serialization;
-- Excel serialization;
-- CSV serialization.
+The 360Giving Data Standard supports both spreadsheet and JSON formats, but most 360Giving data is published in spreadsheet format. Therefore, the documentation for the standard is primarily focussed on the spreadsheet format. For more information, see [Choosing your file format (360Giving Data Standard Documentation)](https://standard.threesixtygiving.org/en/latest/guidance/prepare-data/#choosing-your-file-format).
 
-[Flatten-tool](http://flatten-tool.readthedocs.io/en/latest/unflatten/#human-friendly-headings-using-a-json-schema-with-titles) can use the titles in a schema to provide 'friendly' column headings, and with use of a [metatab](http://flatten-tool.readthedocs.io/en/latest/unflatten/#metadata-tab) also supports packaging meta-data and options to control how spreadsheets are parsed.
+```
 
-## Extended JSON schema
+```{seealso}
 
-We use a number of custom properties in our JSON Schema implementation. A [patch against JSON Schema 0.4 to include these is found here](https://github.com/open-contracting/standard/blob/6e538252dd08344222b5cd16b864ed0a2a866197/standard/schema/metaschema/meta-schema-patch.json).
+* 🧩 [Conversion tools](../components/index.md#conversion-tools)
+* 💡 [Spreadsheet first schema design](../patterns/schema.md#spreadsheet-first)
 
-### Codelist properties
+```
 
-- `codelist` - the filename of a .csv file that contains at least a `Code` column. Used by the CoVE validator to check for acceptable values.
-- `openCodelist` - a boolean value to indicate whether values can **only** come from the codelist, or whether additional values not on the codelist are permitted. When `openCodelist` = 'true' then encountering a value not on the codelist should generate a warning. When `openCodelist` = 'false' then encountering a value not on the codelist should generate an error.
+## Choose a schema language
 
-### Deprecation properties
+A schema defines the meaning, structure and format of data.
 
-> "Deprecation is the discouragement of use of some terminology, feature, design, or practice; typically because it has been superseded or is no longer considered efficient or safe – but without completely removing it or prohibiting its use."
+Based on your chosen publication formats, you need to decide on a language in which to document the schema for a standard, 
 
-See: [Deprecation (Wikipedia)](https://en.wikipedia.org/wiki/Deprecation)
+For standards that support JSON as a publication format, the preferred approach is to use [JSON Schema](https://json-schema.org/) to document the canonical schema for the standard, specifically [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12). Although less expressive than other schema languages, the constraints of JSON Schema enable a focus on keeping data simple enough for a wide range of users.
 
-- `deprecated` - and object to indicate that the field is deprecated, consisting of fields for:
-  - `description` - a message that explains the deprecation, and that should be presented by validators to any publisher using this field.
-  - `deprecatedVersion` - a string indicating the version in which the field was first deprecated.
+If you choose to support other publication formats alongside JSON, you should consider whether to provide secondary, derived schema for those formats.
 
-We also use the column title `Deprecated` with a version number as the cell value in codelist CSV files when a code has been deprecated.
+```{admonition} Example: Open Referral
+:class: note
 
-### Merge strategies
+The canonical schema for the Open Referral Data Specifications is documented using JSON Schema. However, a secondary schema is provided for the Tabular Data Package format, which is derived from the canonical schema. For more information, see [Serialization and Publication Formats (Open Referral Data Specifications Documentation)](http://docs.openreferral.org/en/latest/hsds/serialization.html).
 
-The Open Contracting Data Standard describes an approach to merge together releases of data from different point in time. We add a number of properties to indicate how merging should be approached.
+```
 
-- `omitWhemMerged`
-- `wholeListMerge`
-- `versionId`
+```{admonition} History
+:class: dropdown
+Previously, the recommended approach was to use [JSON Schema Draft 4](https://json-schema.org/draft-04/draft-zyp-json-schema-04). However, Draft 2020-12 contains several useful features not available in Draft 4.
+```
 
-Behaviour for these is [described in the OCDS documentation](http://standard.open-contracting.org/1.1/en/schema/merging/#merging-rules).
+## Choose a codelist format
 
-## Design patterns
+A codelist defines a set of permissable values for a field.
 
-Developing a good schema is an art as much as a science. It requires sensitivity to the needs of both data producers and data users, and an understanding of the incentive structures that will drive adoption of a standard.
+The recommended approach is to document codes, titles and descriptions in a CSV file, according to the [Open Data Services Codelist Schema](https://codelist-schema.readthedocs.io/).
 
 ```{seealso}
-[Schema patterns](../patterns/schema.md)
-The following section provides links to a non-exhaustive set of design patterns that can be drawn upon when developing a schema. 
+* 💡 [CSV codelists](../patterns/schema.md#csv-codelists)
 ```
+
+## Choose your packaging formats
+
+A packaging format is structued way of bundling together data and, sometimes, metadata. You can think of a packaging format as a container for multiple records, texts or documents.
+
+Packaging formats aid interoperability and reuse by providing tool developers and analysts with predicatable and consistent approaches to grouping, streaming and pagination.
+
+Based on your chosen publication formats and the requirements identified in your research, you need to decide on a packaging format or formats for each publication format.
+
+The recommended approach is to consider providing:
+
+* A small file and API response format for files that are small enough to fit into memory or are published via API.
+* A bulk download format for files that are too large to fit into memory.
+
+```{admonition} Example: The Open Fibre Data Standard
+:class: note
+
+The Open Fibre Data Standard supports publication in JSON, GeoJSON and CSV formats. For the JSON and GeoJSON formats, it provides containers for publishing one or more networks and options to support pagination and streaming:
+
+Format | Small files and API responses | Streaming
+--- | --- | ---
+JSON | A JSON object with an embedded array of `Network` objects, with an optional `.links` object for pagination | A [JSON Lines](https://jsonlines.org/) file in which each line is an `Network` object.
+GeoJSON | GeoJSON [feature collections](https://datatracker.ietf.org/doc/html/rfc7946#section-3.3), with an optional `.links` object for pagination | [Newline-delimited GeoJSON](https://stevage.github.io/ndgeojson/) files
+
+```
+
+```{seealso}
+* 💡 [Packaging](../patterns/schema.md#packaging)
+* 💬 [Packaging multiple networks · Issue #51 · Open-Telecoms-Data/open-fibre-data-standard](https://github.com/Open-Telecoms-Data/open-fibre-data-standard/issues/51)
+* 💬 [Deprecate remaining package metadata and add bulk data format · Issue #1084 · open-contracting/standard](https://github.com/open-contracting/standard/issues/1084)
+* 💬 [Add a metadata package schema · Issue #200 · GFDRR/rdl-standard](https://github.com/GFDRR/rdl-standard/issues/200)
+```
+
+## Author your schema and codelists
+
+Authoring the schema and codelists for a standard involves documenting the standard's data model in your chosen schema language and codelist formats.
+
+JSON Schema specifies a number of keywords to describe and constrain JSON data. For example, the `type` keyword is used to restrict a field to a specific type, like "string" or "number", whilst the `title` keyword is used to provide a human-readable title for a field.
+
+As well as the keywords specified in JSON Schema, the [Open Data Services JSON Schema Extension](https://json-schema-extension.readthedocs.io/) specifies additional keywords for linking fields to [CSV codelists](../patterns/schema.md#csv-codelists),  and providing information about [deprecated fields](../patterns/schema.md#deprecated-fields).
+
+```{seealso}
+💡 [Schema patterns](../patterns/schema.md)
+```

docs/index.md

@@ -41,6 +41,7 @@ adoption/index
 learning/index
 components/index
 patterns/index
+tools
 about/index
 meta/index