meso360

Real-time data system for NSSL Mobile Mesonet vehicles. → Field Guide for full installation, dashboard, and troubleshooting reference. Three components work together — mesoingest, mesoview, and mesosync — orchestrated by a single process supervisor that also hosts a web control panel.

---

Components

supervisor — supervisor.py

The only script you need to run. Hosts a Flask web app with two interfaces:

• Control panel at / — start/stop/restart/enable/disable components, view live logs, system stats, current observations, and GPS position
• Mesoview dashboard at /view — the real-time data dashboard (see below)

Also manages mesoingest and mesosync as child subprocesses, restarts any that exit unexpectedly, rotates the log file at midnight, and handles SIGINT/SIGTERM for clean shutdown. Works cross-platform (Windows/macOS/Linux).

python supervisor.py              # live mode (opens browser automatically)
python supervisor.py --test       # replay test_data/test.txt at 1 Hz
python supervisor.py --no-browser # skip opening browser on startup

mesoingest — mesoingest.py

Fetches the newest observation from the Campbell Scientific datalogger at 1 Hz via its HTTP interface, formats it as a CSV row, and appends it to a daily .txt file. Writes a header on the first record of each day. Detects and backfills gaps using the datalogger's lastrecords.html endpoint. Corrects duplicate GPS timestamps by inferring +1 s. Retries automatically on network errors and exits after a configurable number of consecutive failures so the supervisor can restart it.

mesoview — mesoview.py

Flask Blueprint registered by supervisor at /view. Streams live data to any browser on the local network over SSE (Server-Sent Events). Reads the daily .txt files written by mesoingest and pushes one JSON record per second to connected browsers. Serves an interactive dashboard with real-time wind, temperature, pressure, and map charts built with uPlot. Also exposes /view/initial to preload the last 2 hours of history from an in-memory cache on page load.

Not runnable standalone — started automatically by supervisor.

mesosync — SSH reverse tunnel (managed by supervisor.py)

Maintains a persistent SSH reverse tunnel to clamps@remote.bliss.science so operators can reach the vehicle machine from outside the vehicle network. Uses ~/.ssh/clamps_rsa for authentication. Each vehicle uses a unique rtun_port to avoid conflicts. The supervisor probes the tunnel every 30 seconds and restarts it if it drops. Mesosync is optional — if rtun_port is not set in the config it is shown as "not configured" in the control panel.

---

Requirements

• Python 3.10 or later
• Miniforge (recommended) or any Python installation with pip
• OpenSSH client — built into macOS, Linux, and Windows 10/11 (no WSL required)
• ~/.ssh/clamps_rsa — the NSSL/BLISS RSA key must be present on each vehicle machine before mesosync will connect

---

Setup

1 — Clone the repo

git clone <repo-url>
cd meso360

---

2 — Set up the Python environment

Option A — conda (recommended)

conda env create -f environment.yml
conda activate meso360

To update later after a git pull:

conda env update -f environment.yml --prune

Option B — pip / venv

python -m venv .venv

# macOS / Linux
source .venv/bin/activate

# Windows
.venv\Scripts\activate

pip install -r requirements.txt

---

3 — Configure

Copy the example config and edit it:

cp meso360.config.example.json meso360.config.json

meso360.config.json is git-ignored so your local settings won't be overwritten by git pull.

{
  "data_dir":           "~/data/raw/mesonet",
  "log_dir":            "~/mesoview_logs",
  "logger_ip":          "192.168.4.6",
  "http_port":          8080,
  "mdns_hostname":      "mesoview",
  "ingest_retry_max":   100,
  "ingest_retry_delay": 5,
  "rtun_port":          2222
}

Key	Component	Description	Default
data_dir	mesoingest / mesoview	Where daily .txt data files are written and read	~/data/raw/mesonet
log_dir	supervisor	Where daily log files are written (meso360.YYYYMMDD.log)	~/mesoview_logs
logger_ip	mesoingest	IP address of the Campbell datalogger on the vehicle network	192.168.4.6
http_port	supervisor	Port the web server (control panel + dashboard) listens on	8080
mdns_hostname	supervisor	mDNS hostname advertised as <hostname>.local on the LAN	mesoview
ingest_retry_max	mesoingest	Max consecutive fetch failures before mesoingest exits	100
ingest_retry_delay	mesoingest	Seconds between retry attempts	5
rtun_port	mesosync	Port for the SSH reverse tunnel — unique per vehicle to avoid conflicts	(optional)

---

4 — Run

# Make sure your environment is active first
conda activate meso360   # or: source .venv/bin/activate

python supervisor.py

On startup, supervisor runs a set of preflight checks before launching any child processes and logs the results:

[supervisor] === Preflight checks ===
[supervisor]   PASS  config file found: .../meso360.config.json
[supervisor]   PASS  data directory writable: ~/data/raw/mesonet
[supervisor]   PASS  SSH key found: ~/.ssh/clamps_rsa
[supervisor]   PASS  software up to date (a1b2c3d)
[supervisor] ========================

The fourth check looks for updates with git fetch and compares local vs upstream commit SHAs. If the worktree is clean and the remote is ahead, supervisor runs git pull before children start. If local uncommitted changes are present, supervisor logs a WARN and skips the automatic pull to avoid overwriting local work.

Any issue that needs user action appears as WARN with an explanation and the exact command or step to resolve it. All warnings are non-blocking — supervisor still starts, and each component handles its own retry logic.

A browser window opens automatically on startup. To skip this, use --no-browser.

Open a browser on any device connected to the same network and go to:

http://<host-machine-ip>:8080        # control panel
http://<host-machine-ip>:8080/view   # mesoview dashboard

Or, if your network supports mDNS/Bonjour:

http://mesoview.local:8080

---

Control Panel (/)

The control panel is the primary operator interface. It shows:

• Components — status dot, uptime, restart count, and start/stop/restart/enable/disable buttons for mesoingest and mesosync. Mesoview (in-process) is always shown as running.
• Quick Links — direct links to the mesoview dashboard and the datalogger rack web interface.
• Config — current values from meso360.config.json.
• System — supervisor URL, UTC clock, uptime, cache stats, Python version, and live/test mode indicator.
• Position — latest GPS lat/lon with a Google Maps link.
• Resources — CPU, memory, and disk usage (via psutil).
• Recent Log — live-streaming tail of the current log file via SSE.
• Latest Observation sidebar — compass rose (vehicle heading + wind direction), current met values, 30-second averages, max wind speed tracker, and a status card.

---

Dashboard (/view)

Status indicator

The colored dot in the top-left of the dashboard shows the current data feed health:

Color	Meaning
Green	Connected and receiving data normally
Orange	Connected but no data received in the last 30 seconds — mesoingest may be down, the datalogger may be unreachable, or the rack may not be powered on
Red	SSE connection to mesoview lost — the mesoview process may be down or the network between your browser and the host machine is interrupted

If the feed is stale or disconnected, the header also shows a short text message describing the issue.

Status card

The Status card at the bottom of the right sidebar shows additional system health details:

Field	Description
Last update	Time of the most recently received data point
Gaps (UTC day)	Number of data gaps detected since UTC midnight
Completeness	Percentage of expected 1 Hz readings received in the last 10 minutes
3‑min ΔP	Pressure change over the last 3 minutes (hPa) — useful for detecting rapid pressure falls
GPS fix	Fix (green) = valid GPS position; No Fix (orange) = GPS reporting nan, position unavailable
Record age	Seconds since the latest GPS timestamp — turns orange when >30 s, matching the connection dot

Logs

All output from all components is written to ~/mesoview_logs/meso360.YYYYMMDD.log (configurable via log_dir). To watch it live:

tail -f ~/mesoview_logs/meso360.$(date +%Y%m%d).log

Test / replay mode

To run with no datalogger (replays test_data/test.txt at 1 Hz):

python supervisor.py --test

---

Updating

At startup: supervisor checks for updates during preflight. If the repo is clean and the remote is ahead, it runs git pull before launching the child processes. If the repo has local uncommitted changes, it skips the pull and logs a warning instead.

Mid-operations: if an update is available while the system is running, an update available button appears in the top-right corner of both the control panel and dashboard. The button is only shown when the repo is clean, and the /update endpoint only accepts requests from the host machine running supervisor. Clicking it runs git pull and restarts mesoingest and mesosync within a few seconds. The page reconnects automatically. If supervisor.py itself changed, a banner prompts you to restart the process manually.

Local changes: if the repo has uncommitted local changes, a dev badge appears instead of the update button.

New dependencies: if a pull adds new Python packages, update the environment manually:

conda env update -f environment.yml --prune

The config file (meso360.config.json) is never modified by a git update.

---

Utilities

analyze_gaps.py

Scans a daily data file and reports all timestamp gaps with size breakdown and a per-gap table.

python analyze_gaps.py                  # today's file
python analyze_gaps.py 20260403.txt     # specific file
python analyze_gaps.py --all            # all files in data dir

---

Desktop icon / launcher

The repo includes a graphical launch dialog (launch_meso360.pyw) and one-time shortcut-creation scripts for Windows and macOS. The scripts search common conda install locations automatically — no path editing needed.

---

Windows

Run create_shortcut.ps1 once from PowerShell to create a meso360 shortcut on your Desktop:

cd C:\path\to\meso360
.\create_shortcut.ps1

The shortcut uses pythonw.exe so no console window flashes on launch. Double-click meso360 on the Desktop to open the launch dialog, then click Launch (or press Enter). The supervisor runs in a new console window; close it to stop.

If PowerShell blocks the script, run: Set-ExecutionPolicy -Scope CurrentUser RemoteSigned

---

macOS

Step 1 — make the launcher executable (once):

chmod +x /path/to/meso360/launch_meso360.command

Step 2 — create the Desktop app bundle (once):

bash /path/to/meso360/create_shortcut.sh

This creates ~/Desktop/meso360.app with the mesonet icon. Double-click it to open the launch dialog. The supervisor starts as a background process; use the control panel's stop button or kill to stop it.

Gatekeeper prompt: the first time macOS may ask you to confirm opening an unnotarized app. Open System Settings → Privacy & Security and click Open Anyway, or right-click the app and choose Open.

If you just want a plain terminal launcher without the app bundle, double-click launch_meso360.command directly from Finder (after the chmod +x step above).

---

Run on startup

The goal is to have supervisor.py launch automatically when the host machine boots, using the Python executable from your environment. Find that path first:

# conda
conda activate meso360
which python          # macOS / Linux
where python          # Windows — copy the full path shown

# venv
# macOS / Linux: /path/to/meso360/.venv/bin/python
# Windows:       C:\path\to\meso360\.venv\Scripts\python.exe

---

macOS — launchd

Create ~/Library/LaunchAgents/com.mesoview.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>             <string>com.mesoview</string>
  <key>ProgramArguments</key>
  <array>
    <string>/Users/YOUR_USER/miniforge3/envs/meso360/bin/python</string>
    <string>/path/to/meso360/supervisor.py</string>
  </array>
  <key>WorkingDirectory</key>  <string>/path/to/meso360</string>
  <key>RunAtLoad</key>         <true/>
  <key>KeepAlive</key>         <true/>
</dict>
</plist>

Replace the Python path and repo path, then load it:

launchctl load ~/Library/LaunchAgents/com.mesoview.plist

To stop / unload:

launchctl unload ~/Library/LaunchAgents/com.mesoview.plist

---

Linux — systemd (recommended)

Create ~/.config/systemd/user/mesoview.service:

[Unit]
Description=meso360 supervisor
After=network.target

[Service]
WorkingDirectory=/path/to/meso360
ExecStart=/path/to/conda/envs/meso360/bin/python supervisor.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Enable and start:

systemctl --user enable mesoview
systemctl --user start mesoview
systemctl --user status mesoview   # check logs

Linux — cron (alternative)

crontab -e

Add this line (replace paths):

@reboot cd /path/to/meso360 && /path/to/conda/envs/meso360/bin/python supervisor.py

---

Windows — Task Scheduler

1. Open Task Scheduler → Create Task
2. General tab: give it a name (e.g. Mesoview); check Run whether user is logged on or not
3. Triggers tab: New → At startup
4. Actions tab: New →
   • Program/script: full path to your env's Python, e.g. C:\Users\YOU\miniforge3\envs\mesoview\python.exe
   • Add arguments: supervisor.py
   • Start in: full path to the repo directory, e.g. C:\path\to\meso360
5. Settings tab: check If the task fails, restart every 1 minute
6. Click OK and enter your Windows password when prompted

To test without rebooting: right-click the task → Run.

---

File layout

meso360/
├── supervisor.py               # start here — Flask app + process manager (control panel at /, dashboard at /view)
├── mesoingest.py               # fetches data from the datalogger at 1 Hz; writes daily .txt files
├── mesoview.py                 # Flask Blueprint for the real-time dashboard (registered by supervisor)
├── common.py                   # shared config loading, paths, and CSV header
├── analyze_gaps.py             # offline utility: scan daily data files for timestamp gaps
├── meso360.config.example.json # copy to meso360.config.json and edit
│
├── launch_meso360.pyw          # graphical launch dialog (cross-platform tkinter GUI)
├── launch_meso360.command      # macOS: double-click wrapper that opens the dialog via conda
├── create_shortcut.ps1         # Windows: one-time script — creates Desktop shortcut with icon
├── create_shortcut.sh          # macOS:   one-time script — creates meso360.app on Desktop
├── meso360.ico                 # app icon (Windows) — used by launch dialog window chrome
├── meso360.png                 # app icon (dialog logo + macOS window icon)
├── mesonet_launcher.ico        # desktop shortcut icon (separate file avoids Windows icon cache conflicts)
│
├── environment.yml             # conda environment spec
├── requirements.txt            # pip fallback
├── templates/
│   ├── index.html              # mesoview dashboard (served at /view)
│   └── control.html            # control panel (served at /)
├── FIELD_GUIDE.md              # full reference: installation, dashboard guide, variables, troubleshooting
├── static/                     # uPlot chart library (auto-downloaded on first run if missing)
├── test_data/
│   └── test.txt                # sample data for --test mode