Update schema.json to split development and production data

.gitignore

@@ -1,18 +1,15 @@
-backend/.venv
+backend/*venv
 __pycache__/
-.env
+.idea
+backend/.env
+frontend/.env
 *.pyc
-
+.vscode
 node_modules
 *.log*
 .nuxt
 .nitro
 .cache
 .output
-.env
 dist
-
-# deployment dir
-deployment
-gitlab-templates
-.gitlab-ci.yml
+.coverage

.prettierrc.json

@@ -0,0 +1,5 @@
+{
+  "semi": false,
+  "singleQuote": true,
+  "trailingComma": "es5"
+}

LICENSE.md

@@ -285,4 +285,4 @@ the rights granted in Article 2 of this Licence and protect the covered Source
 Code from exclusive appropriation.
 
 All other changes or additions to this Appendix require the production of a new
-EUPL version.
+EUPL version.

README.md

@@ -1,14 +1,18 @@
-# About the Algorithm Register 
-The Ministry of the Interior and Kingdom Relations is developing the Algorithm Register. The team does this using an open development style, through open source software. Anyone can view the team’s progress on the GitHub of the Ministry of the Interior and Kingdom Relations. The Algorithm Register is under development. The content changes over time. New algorithms are added and algorithms are updated. 
-The Algorithm Register (algoritmes.overheid.nl) is being developed with open source software. The source code can be viewed here on GitHub. It can be downloaded and reused. Github is a platform with transparent version control, where technology in development can be easily shared. 
-## Motivation 
-The government increasingly works digitally and uses more and more algorithms when doing so. The government wants to work towards transparent and responsible use of these algorithms. Therefore, the government is making information about the algorithms used by the government available to everyone: citizens, their representatives, the media and (government) professionals. 
-See the Frequently Asked Questions on the Algorithm Register for more information. 
-### Open source software 
-Will you join us and add your thoughts? To contribute, create an account on GitHub (https://github.com/signup ) and read this readme and the code of conduct to get started. 
-Once signed-up, you can provide feedback on the code of this website, on the algorithm 'standard', on the published description or on the user-friendliness of the website, etc. 
-Get started: join us and add your thoughts! 
+# About the Algorithm Register
 
+The Ministry of the Interior and Kingdom Relations is developing the Algorithm Register. The team does this using an open development style, through open source software. Anyone can view the team’s progress on the GitHub of the Ministry of the Interior and Kingdom Relations. The Algorithm Register is under development. The content changes over time. New algorithms are added and algorithms are updated.
+The Algorithm Register (algoritmes.overheid.nl) is being developed with open source software. The source code can be viewed here on GitHub. It can be downloaded and reused. Github is a platform with transparent version control, where technology in development can be easily shared.
+
+## Motivation
+
+The government increasingly works digitally and uses more and more algorithms when doing so. The government wants to work towards transparent and responsible use of these algorithms. Therefore, the government is making information about the algorithms used by the government available to everyone: citizens, their representatives, the media and (government) professionals.
+See the Frequently Asked Questions on the Algorithm Register for more information.
+
+### Open source software
+
+Will you join us and add your thoughts? To contribute, create an account on GitHub (https://github.com/signup ) and read this readme and the code of conduct to get started.
+Once signed-up, you can provide feedback on the code of this website, on the algorithm 'standard', on the published description or on the user-friendliness of the website, etc.
+Get started: join us and add your thoughts!
 
 # Technical information
 
@@ -25,11 +29,11 @@ The documentation below is tested on Ubuntu, but it should work on Windows as we
 
 ### Environment variables
 
-Environment variables are stored in `.env`, but because they can contain secrets (passwords), it is not in the repository. Therefore, copy and paste `.env.dummy` to `.env`. For local development, no changes are needed.
+Environment variables are stored in `.env` in the frontend and backend, but because they can contain secrets (passwords), it is not in the repository. Therefore, copy and paste `.env.dummy` to `.env`.
 
 ### Database
 
-1. Start database. Run from root folder: `docker compose up -d`.
+1. Start database. Run from `/backend` folder: `docker compose up -d`.
 
 Validate the backend is running by navigating to the GUI (DBgate) on `http://localhost:8093`. There you should be able to open the database `algreg_db`.
 
@@ -39,26 +43,24 @@ All commnands below should be run from the `/backend` directory.
 
 2. Specify Python version to use for virtual environment: `poetry env use <python_version>`. (The required Python version is specified in `backend/.python-version`.)
 3. Install packages: `poetry install`.
-4. Run database migrations: `poetry run uvicorn upgrade head`.
+4. Run database migrations: `poetry run alembic upgrade head`.
 5. Start backend: `poetry run uvicorn app.main:app --reload`
 
-Validate the backend is running by navigating to the documentation on `http://localhost:8000/api-docs`.
+Validate the backend is running by navigating to the documentation on `http://localhost:8000/api/api-docs`.
 
 ### Frontend
 
 1.  Install dependencies. Run from '`/frontend`: `npm install`.
-2.  Start local server. Run from '`/frontend`: `npm run dev -- -o`.
+2.  Start local server. Run from '`/frontend`: `npm run dev -- --port 3000`.
 
 Validate the frontend is running by navigating to `http://localhost:3000`.
 
-## Run locally
+There is also a second frontend: 'frontend-beheer' (management frontend). This frontend is used to manage the algorithms. To run it, follow these steps:
 
-1. Start database from backend. Run from `/backend`: `docker compose up -d`
-2. Ensure latest migration has been completed. Run from `/backend`: `poetry run alembic upgrade head`
-3. Navigate to `/frontend` and run `npm run dev`.
-4. Navigate to `/backend` and run `poetry run uvicorn app.main:app --reload`
+1. Install dependencies. Run from '`/frontend-beheer`: `npm install`.
+2. Start local server. Run from '`/frontend-beheer`: `npm run dev -- --port 3001`.
 
-## Other
+Validate that frontend-beheer is running by navigating to `http://localhost:3001`.
 
 ### Linting
 
@@ -73,6 +75,16 @@ This project uses Volar. Install Volar in Visual Studio Code and enable [take ov
 Note:
 Vue linting config inspired by a repository on [Github](https://github.com/weicheng2138/nuxt3-eslint-starter).
 
+### Versioning
+
+This register allows use of multiple versions of the metadatastandard. To add a new version (`v0_x_x`), follow these steps:
+
+1. in `/backend/app/schemas/config`, you see the metadatastandard in JSON format. Create a new file with your file version name, e.g. `v0_x_x.json`. Change the standard as you see fit.
+2. in `/backend/app/routers`, duplicate one of the version folders. Rename it the same as your version in step 1, e.g. `v0_x_x`. Do not change the content.
+3. add the name of this file in the \_\_init\_\_.py file in the same folder, e.g. `from . import v0_1_0, v0_x_x  # noqa`.
+4. If your new schema involves changes to the database (notably for new columns) you'll have to do a SQLAlchemy model change (in `/backend/app/models/algoritme_version.py`) as well as a database migration.
+5. The frontend needs a configuration file in `/frontend/public/layouts` named similar to the other files, e.g. `v0_x_x.json`. The configuration file defines how the data should be displayed in the tabs.
+
 ### Icons
 
 Icons library can be found at `https://icones.js.org/`.

algoritmeregister.code-workspace

@@ -8,21 +8,37 @@
     },
     {
       "path": "backend"
+    },
+    {
+      "path": "frontend-beheer"
     }
   ],
   "settings": {
     "python.linting.flake8Enabled": true,
     "python.linting.enabled": true,
-    "prettier.enable": true,
-    "editor.formatOnSave": true,
+    "editor.defaultFormatter": "rvest.vs-code-prettier-eslint",
+    "editor.formatOnPaste": false, // required 
+    "editor.formatOnType": false, // required
+    "editor.formatOnSave": true, // optional 
+    "editor.formatOnSaveMode": "file", // required to format on save
+    "files.autoSave": "onFocusChange", // optional but recommended
+    "vs-code-prettier-eslint.prettierLast": false, // set as "true" to run 'prettier' last not first
+
     "search.exclude": {
+      "*.yaml": true,
       "*.mjs": true,
       "*.sql": true,
       "**/.nuxt": true,
       "**/.output": true,
       "**/dist/_nuxt": true,
       "**/dist/": true,
       "**/alembic/scripts": true
+    },
+    "[yaml]": {
+      "editor.defaultFormatter": "redhat.vscode-yaml"
+    },
+    "[typescript]": {
+      "editor.defaultFormatter": "esbenp.prettier-vscode"
     }
   }
 }

backend/.coveragerc

@@ -0,0 +1,12 @@
+[run]
+branch=True
+source = app
+omit =
+    */__init__.py
+[report]
+include_namespace_packages = False
+exclude_lines =
+    import
+    from
+    logger
+    logging

backend/.env.dummy

@@ -1,6 +1,8 @@
 POSTGRES_SERVER=127.0.0.1
 POSTGRES_PORT=5432
-POSTGRES_USER=postgres
-POSTGRES_PASSWORD=postgres
-POSTGRES_DB=algreg_db
-DEV_DOCKER_CONTAINER_NAME_PREFIX=Algoritme_register
+POSTGRES_USER=
+POSTGRES_PASSWORD=
+POSTGRES_DB=
+DEV_DOCKER_CONTAINER_NAME_PREFIX=Algoritme_register
+
+TYPE=DEV

backend/.vscode/settings.json

@@ -1,3 +1,4 @@
 {
-    "python.formatting.provider": "black"
+    "python.formatting.provider": "black",
+    "python.analysis.typeCheckingMode": "basic",
 }

backend/Dockerfile

@@ -1,5 +1,4 @@
-# FROM harbor.cicd.s15m.nl/ictu-devops-pub/rid_base:3.11_alpine_3.16
-FROM harbor.cicd.s15m.nl/ictu-devops-pub/rid_base:3.11-slim2-0.1
+FROM harbor.cicd.s15m.nl/ictu-devops-pub/algreg_base as buildstep
 
 ENV YOUR_ENV=${YOUR_ENV} \
   PYTHONFAULTHANDLER=1 \
@@ -13,30 +12,60 @@ ENV YOUR_ENV=${YOUR_ENV} \
 RUN apt-get update && \
   apt-get -y install sudo
 
-ADD ./backend/app /backend/app
-ADD ./backend/alembic /backend/alembic
-ADD ./backend/common /backend/common
-COPY ./backend/alembic.ini /backend/alembic.ini
-COPY ./backend/poetry.lock /backend/poetry.lock
-COPY ./backend/pyproject.toml /backend/pyproject.toml
-COPY ./backend/start.sh /backend/start.sh
+# #TODO
+# RUN apt-get -y purge libpython3.9* python3.9*
+# RUN apt-get install --upgrade libtasn1-6 linux-libc-dev
+# #TODO
+
+ADD /backend/app /backend/app
+ADD /backend/alembic /backend/alembic
+ADD /backend/common /backend/common
+COPY /backend/alembic.ini /backend/alembic.ini
+COPY /backend/poetry.lock /backend/poetry.lock
+COPY /backend/pyproject.toml /backend/pyproject.toml
+COPY /backend/start.sh /backend/start.sh
 
 WORKDIR /backend
 
-RUN sudo apt-get -y install gcc g++ && \
-  pip install "poetry==$POETRY_VERSION" && \
-  poetry config virtualenvs.create false && \
-  poetry install --without dev --no-interaction --no-ansi && \
-  apt-get remove --purge -y gcc g++ && \
-  pip uninstall -y "poetry==$POETRY_VERSION" && \
-  sudo apt-get clean
+# RUN sudo apt-get -y install gcc g++ && \
+#   pip install "poetry==$POETRY_VERSION" && \
+#   poetry config virtualenvs.create false && \
+#   poetry install --without dev --no-interaction --no-ansi && \
+#   apt-get remove --purge -y gcc g++ && \
+#   pip uninstall -y "poetry==$POETRY_VERSION" && \
+#   sudo apt-get clean
 
-# Expose ports
-EXPOSE 8000:8000
+RUN sudo apt-get -y install gcc g++
+RUN pip install "poetry==$POETRY_VERSION"
+RUN poetry config virtualenvs.create false
+RUN poetry install --without dev --no-interaction --no-ansi
+RUN apt-get remove --purge -y gcc g++
+RUN pip uninstall -y "poetry==$POETRY_VERSION"
+RUN sudo apt-get clean
+
+FROM harbor.cicd.s15m.nl/ictu-devops-pub/python:3.11.1-lunar-devops_base_0.2
+WORKDIR /backend
+COPY --from=buildstep /backend /backend
 
-ENV PYTHONPATH="$PYTHONPATH:/:/backend"
+# Copy python binaries ensuring a clean install without overhead
+COPY --from=buildstep /usr/local/lib /usr/local/lib
+RUN ln -f -s /usr/bin/python3.11 /usr/bin/python3
+ENV PYTHONPATH="${PYTHONPATH}:/usr/local/lib/python3.11/site-packages"
 
-# Run the application
-RUN sudo chmod 777 /backend/start.sh
+# remove unused dist-packages to prevent false positives for trivy
+RUN rm -rf /usr/local/lib/python3.11/dist-packages
 
-CMD ["./start.sh"]
+# add testing tools
+RUN pip install pytest==7.3.1 coverage==7.2.7
+
+# # add non root user for security with home directory for test files
+RUN groupadd -r nonroot &&\
+  useradd -r -g nonroot nonroot
+RUN chown -R nonroot:nonroot /backend
+
+# make start.sh executable
+RUN chmod 777 /backend/start.sh
+
+USER nonroot
+EXPOSE 8000:8000
+CMD ["./start.sh"]

backend/alembic/env.py

@@ -5,6 +5,8 @@
 
 from alembic import context
 from common.database_url import get_database_url
+from app.database.database import Base
+from app.models import *
 
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
@@ -19,7 +21,7 @@
 # for 'autogenerate' support
 # from myapp import mymodel
 # target_metadata = mymodel.Base.metadata
-target_metadata = None
+target_metadata = Base.metadata
 
 # other values from the config, defined by the needs of env.py,
 # can be acquired:
@@ -67,9 +69,7 @@ def run_migrations_online() -> None:
     )
 
     with connectable.connect() as connection:
-        context.configure(
-            connection=connection, target_metadata=target_metadata
-        )
+        context.configure(connection=connection, target_metadata=target_metadata)
 
         with context.begin_transaction():
             context.run_migrations()

backend/alembic/versions/04b904159980_rename_schema_to_standard_version.py

@@ -0,0 +1,27 @@
+"""rename schema to standard_version
+
+Revision ID: 04b904159980
+Revises: 845b0b6e654f
+Create Date: 2023-04-05 11:57:40.883818
+
+"""
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision = "04b904159980"
+down_revision = "845b0b6e654f"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.alter_column(
+        "algoritme_version", "schema", nullable=True, new_column_name="standard_version"
+    )
+
+
+def downgrade() -> None:
+    op.alter_column(
+        "algoritme_version", "standard_version", nullable=True, new_column_name="schema"
+    )