Skip to content

Commit ae66f7b

Browse files
Initial Repair Services Cron Job Documentation
1 parent c38a293 commit ae66f7b

File tree

1 file changed

+18
-3
lines changed

1 file changed

+18
-3
lines changed

_docs/sysadmin/installation/system_customization.md

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,25 @@ You may want to back up more of `/var/local/submitty` to save configurations and
2828

2929
## Capture cron error messages
3030

31-
The `submitty_daemon` user runs the [sbin/send_email.py](https://github.com/Submitty/Submitty/blob/master/sbin/send_email.py)
32-
script. Console output from this script can be emailed to a sysadmin to help ensure that errors can be reported and addressed.
31+
To ensure the reliability of the various Submitty services, such as the WebSocket server, their health status is monitored and restarted hourly via the [sbin/repair_services.sh](https://github.com/Submitty/Submitty/blob/master/sbin/repair_services.sh) script run by the submitty_daemon user. This script leverages `systemctl` along with various health-check utility scripts to verify the active state of these core services, triggering a restart if an inactive state is detected.
3332

34-
The first line should be set as `MAILTO=` with a valid email address. For example:
33+
Service failures can occur for various reasons, including unhandled exceptions, memory leaks, port binding issues, or OS-level disruptions such as resource exhaustion. All failures are logged with their relevant timestamp, source, and last output within the `/var/logs/services` directory for the given day in the format `YYYYMMDD.txt`.
34+
35+
To disable this auto-repair mechanism, comment out the relevant line in the source `.setup/submitty_crontab` file within your repository. Since the crontab is auto-generated during installation, any changes must be followed by a re-run of `submitty_install` to persist them.
36+
37+
**Note: This mechanism should only be disabled with caution in production environments.**
38+
39+
```bash
40+
# In .setup/submitty_crontab, comment out the repair_services.sh line:
41+
# 0 * * * * submitty_daemon sudo /usr/local/submitty/sbin/repair_services.sh
42+
43+
# Then re-apply the configuration:
44+
submitty_install
45+
```
46+
47+
The `submitty_daemon` user runs a variety of other scripts, such as [sbin/send_email.py](https://github.com/Submitty/Submitty/blob/master/sbin/send_email.py) to send pending emails every minute. Console output from these scripts can be emailed to a sysadmin to help ensure that errors can be reported and addressed.
48+
49+
The first line of the relevant script should be set as `MAILTO=` with a valid email address, as shown below.
3550
```
3651
3752
* * * * * python3 /usr/local/submitty/sbin/send_email.py

0 commit comments

Comments
 (0)