Restructure userguide

README.md

@@ -6,7 +6,7 @@
 
 https://jabs-tutorial.readthedocs.io/en/latest/index.html
 
-[User Guide (Markdown)](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/main/src/jabs/resources/docs/user_guide/user_guide.md)
+[User Guide (Markdown)](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/main/src/jabs/resources/docs/user_guide/user-guide.md)
 
 ## Copyright
 

src/jabs/resources/docs/user_guide/cli-tools.md

@@ -0,0 +1,101 @@
+# Command Line Tools
+
+## jabs-classify
+
+JABS includes a script called `jabs-classify`, which can be used to classify a single video from the command line.
+
+```text
+usage: jabs-classify COMMAND COMMAND_ARGS
+
+commands:
+ classify   classify a pose file
+ train      train a classifier that can be used to classify multiple pose files
+
+See `jabs-classify COMMAND --help` for information on a specific command.
+```
+
+### Classify Command
+
+```text
+usage: jabs-classify classify [-h] [--random-forest | --xgboost]
+                            (--training TRAINING | --classifier CLASSIFIER) --input-pose
+                            INPUT_POSE --out-dir OUT_DIR [--fps FPS]
+                            [--feature-dir FEATURE_DIR]
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --fps FPS             frames per second, default=30
+  --feature-dir FEATURE_DIR
+                        Feature cache dir. If present, look here for features before computing.
+                        If features need to be computed, they will be saved here.
+
+required arguments:
+  --input-pose INPUT_POSE
+                        input HDF5 pose file (v2, v3, or v4).
+  --out-dir OUT_DIR     directory to store classification output
+
+optionally override the classifier specified in the training file:
+ Ignored if trained classifier passed with --classifier option.
+ (the following options are mutually exclusive):
+  --random-forest       Use Random Forest
+  --xgboost             Use XGBoost
+
+Classifier Input (one of the following is required):
+  --training TRAINING   Training data h5 file exported from JABS
+  --classifier CLASSIFIER
+                        Classifier file produced from the `jabs-classify train` command
+```
+
+### Train Command
+
+```text
+usage: jabs-classify train [-h] [--random-forest | --xgboost]
+                         training_file out_file
+
+positional arguments:
+  training_file        Training h5 file exported by JABS
+  out_file             output filename
+
+optional arguments:
+  -h, --help           show this help message and exit
+
+optionally override the classifier specified in the training file:
+ (the following options are mutually exclusive):
+  --random-forest      Use Random Forest
+  --xgboost            Use XGBoost
+```
+
+> Note: XGBoost may be unavailable on macOS if `libomp` isn't installed. See `jabs-classify classify --help` output for list of classifiers supported in the current execution environment.
+
+> Note: fps parameter is used to specify the frames per second (used for scaling time unit for speed and velocity features from "per frame" to "per second").
+
+## jabs-features
+
+JABS includes a script called `jabs-features`, which can be used to generate a feature file for a single video from the command line.
+
+```text
+usage: jabs-features [-h] --pose-file POSE_FILE --pose-version POSE_VERSION
+                            --feature-dir FEATURE_DIR [--use-cm-distances]
+                            [--window-size WINDOW_SIZE] [--fps FPS]
+
+options:
+  -h, --help            show this help message and exit
+  --pose-file POSE_FILE
+                        pose file to compute features for
+  --pose-version POSE_VERSION
+                        pose version to calculate features
+  --feature-dir FEATURE_DIR
+                        directory to write output features
+  --use-cm-distances    use cm distance units instead of pixel
+  --window-size WINDOW_SIZE
+                        window size for features (default none)
+  --fps FPS             frames per second to use for feature calculation
+```
+
+## jabs-cli
+
+`jabs-cli` is a command line interface that provides access to JABS utilities that did not warrant a full command line tool. To get a listing of current commands, run:
+
+```bash
+jabs-cli --help
+```

src/jabs/resources/docs/user_guide/file-formats.md

@@ -0,0 +1,97 @@
+# File Formats
+
+This section documents the format of JABS output files that may be needed for downstream analysis.
+
+## Prediction File
+
+An inference file represents the predicted classes for each identity present in one video file.
+
+### Location
+
+The prediction files are saved in `<JABS project dir>/jabs/predictions/<video_name>.h5` if they were generated by the JABS GUI.
+
+The `jabs-classify` script saves inference files in `<out-dir>/<video_name>_behavior.h5`
+
+### Contents
+
+The H5 file contains one group, called `predictions`. This group contains one or more behavior prediction groups. Each behavior prediction group contains 3 datasets and 1 new group.
+
+```text
+predictions/
+  behavior_1/
+    predicted_class
+    probabilities
+    identity_to_track
+  behavior_2/
+    ...
+```
+
+### Attributes
+
+The root file contains the following attributes:
+
+- pose_file: filename of the pose file used during prediction
+- pose_hash: blake2b hash of pose file
+- version: prediction output version
+
+Each behavior prediction group contains the following attributes:
+
+- classifier_file: filename of the classifier file used to predict
+- classifier_hash: blake2b hash of the classifier file
+- app_version: JABS application version used to make predictions
+- prediction_date: date when predictions were made
+
+### predicted_class
+
+- dtype: 8-bit integer
+- shape: #identities x #frames
+
+This dataset contains the predicted class. Each element contains one of three values:
+
+- 0: "not behavior"
+- 1: "behavior"
+- -1: "identity not present in frame".
+
+### probabilities
+
+- dtype: 32-bit floating point
+- shape: #identities x #frames
+
+This dataset contains the probability (0.0-1.0) of each prediction. If there is no prediction (the identity doesn't exist at a given frame) then the prediction probability is 0.0.
+
+### identity_to_track
+
+- dtype: 32-bit integer
+- shape: #identities x #frames
+
+This dataset maps each JABS-assigned identity (Pose version 3) back to the original track ID from the pose file at each frame. -1 indicates the identity does not map to a track for that frame. For Pose File Version 4 and greater, JABS uses the identity assignment contained in the pose file. For pose version 2, there will be exactly one identity (0).
+
+## Feature File
+
+A feature file represents features calculated by JABS for a single animal in a video file.
+
+### Location
+
+Feature files are saved per identity at `<JABS project dir>/jabs/features/<video_name>/<identity>/features.h5`.
+
+### Contents
+
+The H5 file contains feature data described in the [feature documentation](features.md). Features used in JABS classifiers are located within the `features` group, further separated by `per_frame` and `window_features_<window_size>` groups. Features not used in JABS classifiers are located outside the `features` group.
+
+All features are a vector of data containing the feature value for each frame in the video.
+
+The root file contains the following attributes:
+
+- distance_scale_factor: scale factor used when converting from pixel space to cm space
+- identity: identity value from the original pose value
+- num_frames: number of frames in the video
+- pose_hash: blake2b hash of pose file
+- version: feature version used when generating this feature file
+
+### Per-Frame Feature Names
+
+Per frame features are named `<feature module> <feature name>`. Feature modules cannot contain spaces, while feature names can.
+
+### Window Feature Names
+
+Window features are named `<feature module> <window operation> <feature name>`. Feature modules and window operations cannot contain spaces, while feature names can.

src/jabs/resources/docs/user_guide/gui.md

@@ -0,0 +1,131 @@
+# GUI
+
+## Main Window
+
+<img src="imgs/main_window.png" alt="JABS Main Window" width=1150 />
+
+- **Behavior Selection:** Select current behavior to label
+- **Add New Behavior Button:** Add new behavior label to project
+- **Identity Selection:** Select subject mouse to label (subject can also be selected by clicking on mouse in the video)
+- **Classifier Controls:** Configure and train classifier. Use trained classifier to infer classes for unlabeled frames. See "Classifier Controls" section for more details.
+- **Label Summary:** Counts of labeled frames and bouts for the subject identity in the current video and across the whole project.
+- **Label "Behavior" Button:** Label current selection of frames as showing behavior. This button is labeled with the current behavior name.
+- **Label "Not Behavior" Button:** Label current selection of frames as not showing behavior. This button is labeled with `Not <current behavior name>`.
+- **Clear Selection Button:** remove labels from current selection of frames
+- **Toggle Select Mode Button:** toggle select mode on/off (turning select mode on will begin selecting frames starting from that point)
+- **Video Playlist:** list of videos in the current project. Click a video name to make it the active video.
+- **Video Player:** Displays the current video. See "Video Player" section for more information.
+- **Manual Label and Predicted Label Visualizations:** see "Label Visualizations" for more information.
+- **Status Bar:** Displays periodic status messages.
+
+## Classifier Controls
+
+<img src="imgs/classifier_controls.png" alt="JABS Classifier Controls" width=900 />
+
+- **Train Button:** Train the classifier with the current parameters. This button is disabled until minimum number of frames have been labeled for a minimum number of mice (increasing the cross validation k parameter increases the minimum number of labeled mice)
+- **Classify Button:** Infer class of unlabeled frames. Disabled until classifier is trained. Changing classifier parameters may require retraining before the Classify button becomes active again.
+- **Classifier Type Selection:** Users can select from a list of supported classifiers.
+- **Window Size Selection:** Number of frames on each side of the current frame to include in window feature calculations for that frame. A "window size" of 5 means that 11 frames are included into the window feature calculations for each frame (5 previous frames, current frame, 5 following frames).
+- **New Window Size:** Add a new window size to the project.
+- **Label Balancing Toggle:** Balances the training data by downsampling the class with more labels such that the distribution is equal.
+- **Symmetric Behavior Toggle:** Tells the classifier that the behavior is symmetric. A symmetric behavior is when left and right features are interchangeable.
+- **All k-fold Toggle:** Uses the maximum number of cross validation folds. Useful when you wish to compare classifier performance and may have an outlier that can be held-out.
+- **Cross Validation Slider:** Number of "Leave One Out" cross validation iterations to run while training.
+
+## Timeline Visualizations
+
+<img src="imgs/label_viz.png" alt="JABS Label Visualizations" width=900 />
+
+- **Manual Labels (sliding window):** Displays manually assigned labels for a sliding window of frames. The window range is the current frame +/-50 frames. Orange indicates frames labeled as showing the behavior, blue indicates frames labeled as not showing the behavior. Unlabeled frames are colored gray.
+- **Manual Labels (global view):** Displays a zoomed out view of the manual labels for the entire video
+- **Predicted Classes (sliding window):** Displays predicted classes (if the classifier has been run). Color opacity indicates prediction probability for the predicted class. Manually assigned labels are also displayed with probability of 100%.
+- **Predicted Class (global view):** Displays a zoomed out view of the predicted classes for the entire video.
+- **Sliding Window Indicator:** highlights the section of the global views that correspond to the frames displayed in the "sliding window" views.
+
+By default, the Timeline shows manual labels and predicted behaviors for the current subject animal. The Timeline can be toggled to show all subjects by selecting View->Timeline->All Animals in the menu bar. The Timeline can also be configured to show only manual labels or only predicted labels. If "All Animals" is selected, the Timeline will show which set of labels and predictions belong to the subject animal by drawing a colored border around them.
+
+### Timeline Menu
+
+<img src="imgs/timeline_menu.png" alt="Timeline visualization options" />
+
+<br /><br />
+
+### Example Timeline with "Labels & Predictions" and "All Animals" selected
+
+<img src="imgs/stacked_timeline.png" alt="Timeline with all animals" width=900 />
+
+## Video Controls
+
+<img src="imgs/video-control-overlay.png" alt="Video Control Overlay" width=900 />
+
+Mousing over the video player will display a control overlay with the following controls:
+
+- **Video Playback Speed Controls:** Controls the speed of video playback. Clicking this control will open a menu with options for playback speed. The default speed is 1x.
+- **Video Cropping:** Allows the user to crop the video to a specific region of interest. After clicking the cropping control, the user can click and drag a rectangular selecting tool to select the region of interest. The video will be cropped to the selection, and scaled to fill the available player area. If the video is currently cropped, the cropping control will be replaced with a "Reset Cropping" control, which will reset the cropping to the original video size.
+- **Brightness Adjustment:** Allows the user to adjust the brightness of the video. Clicking this control will open a slider that can be used to adjust the brightness.
+- **Contrast Adjustment:** Allows the user to adjust the contrast of the video. Clicking this control will open a slider that can be used to adjust the contrast.
+
+Clicking the Brightness or Contrast controls will reset the brightness or contrast to the default value before displaying the slider control. Clicking the video or moving the mouse off the video frame will dismiss the slider control.
+
+## Menu
+
+- **JABS→About:** Display About Dialog
+- **JABS→User Guide:** Display User Guide
+- **JABS→Quit JABS:** Quit Program
+- **File→Open Project:** Select a project directory to open. If a project is already opened, it will be closed and the newly selected project will be opened.
+- **File→Export Training Data:** Create a file with the information needed to share a classifier. This exported file is written to the project directory and has the form `<Behavior_Name>_training_<YYYYMMDD_hhmmss>.h5`. This file is used as one input for the `jabs-classify` script.
+- **File→Archive Behavior:** Remove behavior and its labels from project. Labels are archived in the `jabs/archive` directory.
+- **File→Prune Project:** Remove videos and pose files that are not labeled.
+- **View:** Menu to control various display options.
+  - **View→View Playlist:** can be used to hide/show video playlist
+  - **View→Timeline:** Menu to control the timeline display.
+  - **View→Label Overlay:** Control the floating display of manual labels or predicted classes.
+  - **View→Identity Overlay:** Configure the identity overlay mode.
+  - **View→Show Track:** show/hide track overlay for the subject. The track overlay shows the nose position for the previous 5 frames and the next 10 frames. The nose position for the next 10 frames is colored red, and the previous 5 frames it is a shade of pink.
+  - **View→Overlay Pose:** toggle the overlay of the pose on top of the subject mouse
+  - **View→Overlay Landmarks:** toggle the overlay of arena landmarks over the video.
+- **Features:** Menu item for controlling per-behavior classifier settings. Menu items are disabled when at least 1 pose file in the project does not contain the data to calculate features.
+  - **Features→CM Units:** toggle using CM or pixel units (Warning! Changing this will require features to be re-calculated)
+  - **Features→Enable Window Features:** toggle using statistical window features
+  - **Features→Enable Signal Features:** toggle using fft-based window features
+  - **Features→Enable Social Features:** toggle using social features (v3+ projects)
+  - **Features→Enable Corners Features:** toggle using arena corner features (v5+ projects with arena corner static object)
+  - **Features→Enable Lixit Features:** toggle using lixit features (v5+ projects with lixit static object)
+  - **Features→Enable Food_hopper Features:** toggle using food hopper features (v5+ projects with food hopper static object)
+  - **Features→Enable Segmentation Features:** toggle using segmentation features (v6+ projects)
+
+## Overlays
+
+### Track Overlay Example
+
+<img src="imgs/track_overlay.png" alt="Track Overlay" width=400 />
+
+### Pose Overlay Example
+
+<img src="imgs/pose_overlay.png" alt="Pose Overlay" />
+
+### Pose Overlay Keypoint Legend
+
+<img src="imgs/keypoint_legend.png" alt="Pose Keypoint Legend" width="500"/>
+
+### Identity Overlay
+
+JABS offers several ways to overlay mouse identities on the video. Choose a mode from View → Identity Overlay.
+
+In all modes, you can select the subject directly in the video: click inside the convex hull of body keypoints (excluding the tail) to select that animal. You can also click the floating identity label. With the Bounding Box overlay, clicking the tab selects the animal. These options are in addition to the Identity dropdown in the main window and the Shift+↑ / Shift+↓ keyboard shortcuts.
+
+#### Floating
+
+<img src="imgs/floating-identity-overlay.png" alt="Floating Identity Overlay" />
+
+#### Centroid
+
+<img src="imgs/centroid-identity-overlay.png" alt="Centroid Identity Overlay" />
+
+#### Minimalist
+
+<img src="imgs/minimalist-identity-overlay.png" alt="Minimalist Identity Overlay" />
+
+#### Bounding Box
+
+<img src="imgs/bbox-identity-overlay.png" alt="Bounding Box Identity Overlay" width="600"/>

src/jabs/resources/docs/user_guide/keyboard-shortcuts.md

@@ -0,0 +1,41 @@
+# All Keyboard Shortcuts
+
+## Application
+
+| Action                      | Shortcut       |
+| --------------------------- | -------------- |
+| Quit JABS                   | Ctrl+Q / Cmd+Q |
+| Export training data        | Ctrl+T / Cmd+T |
+| Open behavior search dialog | Ctrl+F / Cmd+F |
+
+## Video Navigation
+
+| Action                        | Shortcut  |
+| ----------------------------- | --------- |
+| Play / Pause video            | Space bar |
+| Previous / Next frame         | ← / →     |
+| Move forward / back 10 frames | ↑ / ↓     |
+| Previous / Next video         | , / .     |
+
+## Labeling
+
+| Action                                   | Shortcut       |
+| ---------------------------------------- | -------------- |
+| Start select mode                        | z, x, c        |
+| Label selection as behavior              | z              |
+| Clear labels from selection              | x              |
+| Label selection as not behavior          | c              |
+| Exit select mode without making a change | Esc            |
+| Select all frames                        | Ctrl+A / Cmd+A |
+
+## Identity & Overlays
+
+| Action                            | Shortcut          |
+| --------------------------------- | ----------------- |
+| Switch to next / previous subject | Shift+↑ / Shift+↓ |
+| Toggle track overlay              | t                 |
+| Toggle pose overlay               | p                 |
+| Toggle landmark overlay           | l                 |
+| Toggle Minimalist Identity Labels | Ctrl+I / Cmd+I    |
+
+> **Note:** Next/previous subject order follows the "Identity Selection" dropdown ordering.