morcus.net

Source code for morcus.net, a collection of digital tools for Latin.

---

First Time Setup

morcus.net development is currently done only on Linux machines. Before you get started, install git.

Typescript and Rust Code (Client, Server, Workers)

To start, download and run the setup script. This will clone this (the morcus-net) repository, and several repositories with raw data that is processed and served by the web server. It will also prompt you to install Rust and Node, if you do not already have these on your machine.

curl https://raw.githubusercontent.com/nkprasad12/morcus-net/dev/set_up_or_update.sh | bash

In the future, you can run ./morcus.sh web from morcus-net to build the client and start the server. Common arguments for this script (run ./morcus.sh --help for full options):

Flag	Explanation
--build_ls	This processes Lewis and Short for use by the server. This needs to be run the first time and whenever you modify the Lewis and Short raw XML file. This is slow.
--build_sh	This processes Smith and Hall for use by the server. This needs to be run the first time and whenever you modify the Smith and Hall raw text file or any of the processing code. This is slow.
--minify	Builds a minified client. This is used in production builds. This skips React's StrictMode warnings as well.
--transpile_only	Skips typescript type checking. This is faster.

Python Code (ML)

From the morcus-net root directory, set up a Python virtual environment and install Python dependencies.

1. python3.12 -m venv venv
2. source venv/bin/activate && python3.12 -m pip install -r requirements.txt
3. Run npm run setup-alatius to set up the macronizer for local use.
4. To run NLP models, you may also need to install stanza and cltk from pip.

---

Contribution Guidelines

1. File an issue (or accept an existing one) for tracking purposes
2. Make sure to write unit tests verifying your change, if applicable
3. Run npm run pre-commit to run all formatting checks, tests, and builds.
4. Send a Pull Request to merge to dev

---

License Details

Work in this repository is provided under the terms of the LICENSE file in the root directory. Work in the texts and libs directories are provided under their original licenses.

---

VS Code

Setting up formatting

After initial setup, run npm install to ensure you have the latest prettier config. Then, install the Prettier VS Code plugin by by pressing Ctrl+P and entering ext install esbenp.prettier-vscode. The project settings are already configured to use prettier as the default formatting for Typescript and Javascript, and black as the default formatter for Python. This will also turn on format on save.

Integrating Jest

After initial setup, install the Jest Runner VS Code plugin by pressing Ctrl+P and entering ext install firsttris.vscode-jest-runner. This will allow you to run and debug unit tests from within the VS Code UI when browsing a test file.

ESLint Plugin

Install the ESLint plugin. This will add warnings within VSCode for any ESLint errors in the code.

---

Deployment

Heroku

There is a Procfile in the root directory for use with Heroku. There are some environment variables that can be set, but all are optional.

1. MONGODB_URI - The URI to a MongoDB database. If set, system health and usage metrics will be written to this database. Otherwise, they will be logged locally.
2. DB_SOURCE - Only used if MONGODB_URI is present. If set, this will be used to disambiguate records from different deployments (e.g. use dev for staging).
3. GITHUB_TOKEN - A GitHub token with Issue creation permissions. If set, issue reports will be written to GitHub Issues. Otherwise, they will be logged locally.
4. APPNAME - set to Dev for a staging / dev deployment.
5. NPM_CONFIG_OMIT - set to optional as an optimization to avoid installing integration test dependencies.

Docker Images

Docker images are generated for each push. These can be downloaded from the registry at ghcr.io/nkprasad12/morcus. There are four tags of note:

1. main-latest is the most up-to-date image of the main branch.
2. main-previous is a backup of the previous main-latest.
3. dev-latest is the most up-to-date image of the dev branch.
4. dev-previous is a backup of the previous dev-latest.

Please contact Morcus if you are interested in images built for arm64.

Note that the latest tag is not used.

Self-hosted Setup

This guide assumes you are using a Linux machine with the following prerequisites:

1. The Docker CLI has been installed.
2. Your SSL certificate and private key files are on the machine.

Run the following command and follow the resulting directions to:

1. Create or download all required configuration files
2. Start containers for a prod Morcus instance, a dev Morcus instance, a reverse proxy, and a job that updates all containers.
3. (Optionally) add a cron job to restart all containers on reboot.

MORCUS_REPO=https://raw.githubusercontent.com/nkprasad12/morcus-net/dev/src/devops && \
curl $MORCUS_REPO/docker-compose.yaml -o docker-compose.yaml && \
curl $MORCUS_REPO/vm_setup.sh -o vm_setup.sh && chmod +x vm_setup.sh && \
curl $MORCUS_REPO/.env.template -o .env && \
echo -e "\n\nSetup files downloaded. Open '.env' and follow the instructions." ; \
unset MORCUS_REPO

By default, the main-latest tag will be used for the production instance. For ARM-based systems, you should specify PROD_TAG=main-latest-arm in the .env file to use the ARM image.

Please contact Morcus if you are interested in images built for arm64. These builds are currently not made, but can be added easily.

Local testing

If you don't have an SSL certificate and key, you can generate one for local testing with the following:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -nodes -subj "/CN=your.domain"

By default, the reverse proxy will only proxy traffic coming from the hosts specified by PROD_HOST_NAMES and DEV_HOST_NAMES (as specified in the .env file - see instructions there for more details) and will error on all other requests. For local testing, you can map these hosts to the loopback address in your hosts file (generally /etc/hosts on a Unix-like OS). For example:

in your .env file:

PROD_HOST_NAMES=foo.morcus.net

in your hosts file:

127.0.0.1 foo.morcus.net

Updating the Reverse Proxy image

This is updated so infrequently that doing it as part of the CI workflow (where it would rebuild with every commit) was not really worth it. In order to update:

1. cd src/devops/reverse_proxy
2. docker buildx build --platform linux/amd64,linux/arm64 -t ghcr.io/nkprasad12/morcus-proxy:latest --push .

New VM Checklist

A list of things that may be good to do when setting up a new VM.

1. (On a machine with SSH keys generated) ssh-copy-id to copy the credentials
2. Disable password login via SSH: a. Set UsePAM no and PasswordAuthentication no in /etc/ssh/sshd_config.
3. Set up a firewall to block unused ports. For ufw a. sudo apt update && sudo apt install ufw b. sudo ufw enable (turns it on) c. sudo ufw default deny incoming d. sudo ufw default allow outgoing e. sudo ufw allow 22/tcp && sudo ufw allow 443/tcp f. sudo ufw status verbose to double check

Play Store

There is a Play Store app at https://play.google.com/store/apps/details?id=net.morcus.pwa. This is packaged using PWA builder - see instructions at https://docs.pwabuilder.com/#/builder/android?id=update-existing-pwa to update.

• Current version: 1.0.1
• Current version code: 10
• Signing key information is in a file called signing-key-info.txt

Code Guide

Overview

Morcus Latin Tools is a web application (it is also available in the Play Store, but this is just a packaged PWA). There are four high level components to the code:

1. A client, which is developed in Typescript. This is currently a single page application written using vanilla React.
2. A backend, which is also developed in Typescript. This is a Node.js based HTTP server that uses Express.
3. preprocessing code, which is also developed in Typescript. This is responsible for transforming raw files (often XML or plain text files) into structured data that can be served by the backend.
4. NLP code, which is developed in Python. This is currently used for the macronizer.

TODO: add subsections for all of these, and explain the build step.

Misc

Running the Morceus analyzer

To run the analyzer on the given word (e.g. habes), use:

npm run ts-node src/morceus/cli.ts analyzeWord habes

(if you have bun installed, you can replace npm run ts-node with bun run, which is much faster)

Analyzing the library for unknown words

To run the analyzer and determine unknown words for the current library, run

DEBUG_OUT=./ ./morcus.sh build --build_latin_library

This will create files in the specified directory called <name of work>.debug.txt which contains lists of words that were not known to the analyzer.

You can update the list ALL_WORKS in src/common/process_library.ts to add or remove a work. Currently only DBG and Phaedrus run by default as they are checked in to the repo as test data, but Amores, Germania, and Satires are also known to work.

Expanding all Morceus tables

npm run ts-node src/morceus/cli.ts buildTables

Python Sadness

Updating packages

This may be required when updating Python versions.

1. sed -i 's/==/>=/g' requirements.txt (don't pin to exact versions so the upgrade can work)
2. venv/bin/pip install -r requirements.txt --upgrade
3. venv/bin/pip freeze > requirements.txt

Manual Maintenance Tasks

The following should be done at regular intervals:

• Update the fast-forward token on GitHub: required for pushing to main. Due before 2026/12/01.
  • Go to account settings, and rotate the token.
  • Copy the rotated value into morcus-net settings (under Secrets currently).
• Update the token used on the server for writing to GitHub issues. Due before 2026/12/01.
  • Go to account settings, and rotate the token.
  • Update the value of GITHUB_TOKEN in the .env file on the VM.
  • Restart the server (docker compose down ; docker compose up -d).
• Update the VM OS (currently: Debian 12). Due before it goes EOL in 2028/06.
  • Standard instructions. Since we have a backup machine now, we can route traffic to the backup machine while this is in progress.
• Update the Python version (currently: 3.12). Due before it goes EOL in 2028/10.
  • Search for 3.12 and replace with the latest version everywhere. Currently this just means in the CI workflow, documentations, and in one script.
  • Update the requirements.txt (see above).
• Update the Docker image OS version (current: debian-trixie). Due before it goes EOL in 2030/06
  • Check the Dockerfile
  • Update the version.
• Update the node version, (currently: 22). Due before it goes EOL in 2027/04.
  • Check the package.json
  • Update the version used in the CI workflows
  • Update the version used in the Dockerfile
• Update the nginx version, (currently 1.28.0). Due before it goes EOL in 2026/03ish.
  • Update the nginx version in src/devops/reverse-proxy/Dockerfile
  • Follow the build and push instructions above

Following a reboot

1. Check the firewall is up: sudo ufw status verbose
2. Check the services are up (docker ps)