Update documentation

_sources/docs/classifier_evaluate.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "To evaluate our trained classifier model, we compute a few metrics on the test dataset. We use the [TorchMetrics](https://lightning.ai/docs/torchmetrics/stable/) library to generate class-wise reports. TorchMetrics is a library in PyTorch for computing metrics for evaluation tasks. The following metrics are reported:\n",
     "\n",
-    "1. **Precision**: Precision measures the proportion of true positive predictions among all positive predictions made by the model. It indicates how many of the predicted positive cases are actually positive. As a areminder, precision ranges from 0 to 1, where higher values indicate better performance. See the page \"Detector evaluate\" for the equation to calculate precision.\n",
+    "1. **Precision**: Precision measures the proportion of true positive predictions among all positive predictions made by the model. It indicates how many of the predicted positive cases are actually positive. As a reminder, precision ranges from 0 to 1, where higher values indicate better performance. See the page \"Detector evaluate\" for the equation to calculate precision.\n",
     "\n",
     "2. **Recall**: Recall, also known as sensitivity or true positive rate, measures the proportion of true positive predictions among all actual positive cases in the dataset. It indicates how many of the actual positive cases the model correctly identifies. As a reminder, recall also ranges from 0 to 1, with higher values indicating better performance. See the page \"Detector evaluate\" for the equation to calculate recall.\n",
     "\n",

_sources/docs/classifier_train.ipynb

@@ -153,7 +153,11 @@
     "   - `OUT_DIR`: Directory to write the \"ground-truth-box-clipped-and-buffered\" images to, along with the csv file containing annotations.\n",
     "\n",
     "   Execute the script using the command:\n",
-    "   `python classifier_train.py <IMG_DIR> <ANN_JSON> <SIDE> <OUT_DIR>`\n",
+    "   `python prep_classifier_training_data.py <IMG_DIR> <ANN_JSON> <SIDE> <OUT_DIR>`\n",
+    "\n",
+    "   For example:\n",
+    "   `python prep_classifier_training_data.py mapillary_images/ coco/annotations/instances_default_left.json left clipped_images/`\n",
+    "\n",
     "\n",
     "2. **Merge the left and right side csv files and add weights column**: \n",
     "\n",
@@ -170,9 +174,21 @@
     "prefix_to_add = \"\" # Whatever the out_dir was e.g. \"clipped_images/\"\n",
     "df_combined['file_name'] = prefix_to_add + df_combined['file_name']\n",
     "df_combined[\"weights\"] = 1\n",
-    "df_combined.to_csv(\"data/raw/data_fixed.csv\")\n",
+    "df_combined.to_csv(\"data/raw/data.csv\")\n",
     "```\n",
     "\n",
+    "3. Now you'll need to split `data.csv` into training, validation, and test partitions. We use stratified sampling to ensure that classes are evenly represented across the partitions. To do this, run `src/split_dataset.py`.\n",
+    "\n",
+    "**Command-Line Arguments**: Specify the following command-line arguments:\n",
+    "   - `CSV_PATH`: Directory containing the dataset images.\n",
+    "   - `OUTPUT_PATH`: Directory to write the partitioned data to.\n",
+    "\n",
+    "   Execute the script using the command:\n",
+    "   `python src/split_dataset.py <CSV_PATH> <OUTPUT_PATH>`\n",
+    "\n",
+    "   For example:\n",
+    "   `python src/split_dataset.py data/raw/data.csv data/intermediate/` which will output files named `train.csv`, `valid.csv` and `test.csv` in `data/intermediate/`.\n",
+    "\n",
     "### Now for training\n",
     "\n",
     "The script can be executed directly, taking command-line arguments for specifying experiment name, image directory, and data directory. It is designed to support a modular and configurable implementation of a neural network model for multi-class building property classification tasks, utilizing pre-trained backbone architectures and PyTorch Lightning for training and evaluation.\n",
@@ -186,14 +202,7 @@
     "    - `LearningRateMonitor`: Monitors and logs the learning rate during training.\n",
     "    - `ModelCheckpoint`: Saves the best model based on validation performance.\n",
     "    - `BackboneFreezeUnfreeze`: Unfreezes the model backbone after a certain epoch.\n",
-    "6. Initialize the Lightning Trainer with required configurations:\n",
-    "    - `devices`: Automatic device selection.\n",
-    "    - `accelerator`: Automatic accelerator selection.\n",
-    "    - `max_epochs`: Maximum number of training epochs.\n",
-    "    - `precision`: Mixed-precision training for improved efficiency.\n",
-    "    - `logger`: AIM logger for experiment tracking.\n",
-    "    - `callbacks`: List of callback functions.\n",
-    "7. Train the model using the `fit` method with training and validation data loaders.\n",
+    "6. Train the model using the `fit` method with training and validation data loaders.\n",
     "    - We unfreeze the backbone architecture after 10 epochs.\n",
     "\n",
     "Here are the final model's loss curve and F1 score curve:\n",
@@ -213,7 +222,9 @@
     "\n",
     "`python classifier_train.py <EXPERIMENT_NAME> <IMG_DIR> <DATA_DIR>`\n",
     "\n",
-    "Replace `<EXPERIMENT_NAME>`, `<IMG_DIR>`, and `<DATA_DIR>` with the appropriate values."
+    "Replace `<EXPERIMENT_NAME>`, `<IMG_DIR>`, and `<DATA_DIR>` with the appropriate values.\n",
+    "\n",
+    "For example `python classifier_train.py hp_classify_exp clipped_images/ data/intermediate/`"
    ]
   },
   {

_sources/docs/detector_train.ipynb

@@ -30,7 +30,7 @@
     "\n",
     "## Architecture\n",
     "\n",
-    "We have implemented the `RetinaNet_R_50_FPN_3x` architecture supplied in Detectron2's model zoo. RetinaNet is a popular single-stage object detection architecture introduced by Facebook AI Research. It is designed to address the problem of class imbalance in object detection by introducing a novel focal loss function, which focuses training on hard examples. RetinaNet uses a single convolutional neural network (CNN) to simultaneously predict object bounding boxes and their corresponding class probabilities. This variant of the implements a ResNet-50 backbone network. ResNet-50 is a convolutional neural network architecture that consists of 50 layers. It is a popular choice for many computer vision tasks due to its balance between model complexity and computational efficiency. \n",
+    "We have implemented the `RetinaNet_R_50_FPN_3x` architecture supplied in Detectron2's model zoo. RetinaNet is a popular single-stage object detection architecture introduced by Facebook AI Research. It is designed to address the problem of class imbalance in object detection by introducing a novel focal loss function, which focuses training on hard examples. RetinaNet uses a single convolutional neural network (CNN) to simultaneously predict object bounding boxes and their corresponding class probabilities. This variant implements a ResNet-50 backbone network. ResNet-50 is a convolutional neural network architecture that consists of 50 layers. It is a popular choice for many computer vision tasks due to its balance between model complexity and computational efficiency. \n",
     "\n",
     "FPN stands for Feature Pyramid Network, which is a type of architecture used to build feature pyramids in convolutional neural networks. FPN enhances the ability of the network to detect objects at different scales by combining features from multiple levels of the network hierarchy. It achieves this by adding lateral connections and upsampling layers to produce a set of feature maps at different resolutions.\n",
     "\n",
@@ -45,9 +45,9 @@
     "\n",
     "## Dataset prep\n",
     "\n",
-    "To train the mode, we combined the left and right side images into a single COCO annotation files from `s3://hp-deliverables-v2/coco_format_12_4/`, shuffled the combined annotation file and partitioned that into training, validation and testing splits using a 70 percent, 20 percent and 10 percent ratio, respectively. The results of this were 3 COCO annotation files, one for each partition.\n",
+    "To train the model, we combined the left and right side images into a single COCO annotation files from `s3://hp-deliverables-v2/coco_format_12_4/`, shuffled the combined annotation file and partitioned that into training, validation and testing splits using a 70 percent, 20 percent and 10 percent ratio, respectively. The results of this were 3 COCO annotation files, one for each partition.\n",
     "\n",
-    "You can find this partitioned data here: `s3://hp-deliverables-v2/partitions/`.\n",
+    "You can find the partitioned data here: `s3://hp-deliverables-v2/partitions/`.\n",
     "\n",
     "## Usage\n",
     "\n",
@@ -111,6 +111,11 @@
     "    - Replace `ANN_DIR` with the path to the directory containing COCO-formatted JSON annotations.\n",
     "    - Replace `IMG_DIR` with the path to the directory containing images.\n",
     "\n",
+    "    For example:\n",
+    "    ```\n",
+    "    python detector_train.py coco/ mapillary_images/\n",
+    "    ```\n",
+    "\n",
     "4. **Training:** The script will initialize and train the Detectron2 model based on the provided dataset and configuration. The trained model will be saved in the output directory specified in the configuration. By default, this directory is `./output/`.\n"
    ]
   }

docs/classifier_evaluate.html

@@ -438,7 +438,7 @@ <h2> Contents </h2>
 <h1>Classifier Evaluation<a class="headerlink" href="#classifier-evaluation" title="Permalink to this heading">#</a></h1>
 <p>To evaluate our trained classifier model, we compute a few metrics on the test dataset. We use the <a class="reference external" href="https://lightning.ai/docs/torchmetrics/stable/">TorchMetrics</a> library to generate class-wise reports. TorchMetrics is a library in PyTorch for computing metrics for evaluation tasks. The following metrics are reported:</p>
 <ol class="arabic">
-<li><p><strong>Precision</strong>: Precision measures the proportion of true positive predictions among all positive predictions made by the model. It indicates how many of the predicted positive cases are actually positive. As a areminder, precision ranges from 0 to 1, where higher values indicate better performance. See the page “Detector evaluate” for the equation to calculate precision.</p></li>
+<li><p><strong>Precision</strong>: Precision measures the proportion of true positive predictions among all positive predictions made by the model. It indicates how many of the predicted positive cases are actually positive. As a reminder, precision ranges from 0 to 1, where higher values indicate better performance. See the page “Detector evaluate” for the equation to calculate precision.</p></li>
 <li><p><strong>Recall</strong>: Recall, also known as sensitivity or true positive rate, measures the proportion of true positive predictions among all actual positive cases in the dataset. It indicates how many of the actual positive cases the model correctly identifies. As a reminder, recall also ranges from 0 to 1, with higher values indicating better performance. See the page “Detector evaluate” for the equation to calculate recall.</p></li>
 <li><p><strong>F1-Score</strong>: F1-score is the harmonic mean of precision and recall. It provides a single metric that balances both precision and recall. F1-score is calculated as:</p>
 <p><code class="docutils literal notranslate"><span class="pre">F1-score</span> <span class="pre">=</span> <span class="pre">2</span> <span class="pre">x</span> <span class="pre">((Precision</span> <span class="pre">x</span> <span class="pre">Recall)</span> <span class="pre">/</span> <span class="pre">(Precision</span> <span class="pre">+</span> <span class="pre">Recall))</span></code></p>

docs/classifier_train.html

@@ -599,7 +599,9 @@ <h3>Prepare the data<a class="headerlink" href="#prepare-the-data" title="Permal
 <li><p><code class="docutils literal notranslate"><span class="pre">OUT_DIR</span></code>: Directory to write the “ground-truth-box-clipped-and-buffered” images to, along with the csv file containing annotations.</p></li>
 </ul>
 <p>Execute the script using the command:
-<code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">classifier_train.py</span> <span class="pre">&lt;IMG_DIR&gt;</span> <span class="pre">&lt;ANN_JSON&gt;</span> <span class="pre">&lt;SIDE&gt;</span> <span class="pre">&lt;OUT_DIR&gt;</span></code></p>
+<code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">prep_classifier_training_data.py</span> <span class="pre">&lt;IMG_DIR&gt;</span> <span class="pre">&lt;ANN_JSON&gt;</span> <span class="pre">&lt;SIDE&gt;</span> <span class="pre">&lt;OUT_DIR&gt;</span></code></p>
+<p>For example:
+<code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">prep_classifier_training_data.py</span> <span class="pre">mapillary_images/</span> <span class="pre">coco/annotations/instances_default_left.json</span> <span class="pre">left</span> <span class="pre">clipped_images/</span></code></p>
 </li>
 <li><p><strong>Merge the left and right side csv files and add weights column</strong>:</p></li>
 </ol>
@@ -615,9 +617,21 @@ <h3>Prepare the data<a class="headerlink" href="#prepare-the-data" title="Permal
 <span class="n">prefix_to_add</span> <span class="o">=</span> <span class="s2">&quot;&quot;</span> <span class="c1"># Whatever the out_dir was e.g. &quot;clipped_images/&quot;</span>
 <span class="n">df_combined</span><span class="p">[</span><span class="s1">&#39;file_name&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">prefix_to_add</span> <span class="o">+</span> <span class="n">df_combined</span><span class="p">[</span><span class="s1">&#39;file_name&#39;</span><span class="p">]</span>
 <span class="n">df_combined</span><span class="p">[</span><span class="s2">&quot;weights&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
-<span class="n">df_combined</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s2">&quot;data/raw/data_fixed.csv&quot;</span><span class="p">)</span>
+<span class="n">df_combined</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s2">&quot;data/raw/data.csv&quot;</span><span class="p">)</span>
 </pre></div>
 </div>
+<ol class="arabic simple" start="3">
+<li><p>Now you’ll need to split <code class="docutils literal notranslate"><span class="pre">data.csv</span></code> into training, validation, and test partitions. We use stratified sampling to ensure that classes are evenly represented across the partitions. To do this, run <code class="docutils literal notranslate"><span class="pre">src/split_dataset.py</span></code>.</p></li>
+</ol>
+<p><strong>Command-Line Arguments</strong>: Specify the following command-line arguments:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">CSV_PATH</span></code>: Directory containing the dataset images.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">OUTPUT_PATH</span></code>: Directory to write the partitioned data to.</p></li>
+</ul>
+<p>Execute the script using the command:
+<code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">src/split_dataset.py</span> <span class="pre">&lt;CSV_PATH&gt;</span> <span class="pre">&lt;OUTPUT_PATH&gt;</span></code></p>
+<p>For example:
+<code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">src/split_dataset.py</span> <span class="pre">data/raw/data.csv</span> <span class="pre">data/intermediate/</span></code> which will output files named <code class="docutils literal notranslate"><span class="pre">train.csv</span></code>, <code class="docutils literal notranslate"><span class="pre">valid.csv</span></code> and <code class="docutils literal notranslate"><span class="pre">test.csv</span></code> in <code class="docutils literal notranslate"><span class="pre">data/intermediate/</span></code>.</p>
 </section>
 <section id="now-for-training">
 <h3>Now for training<a class="headerlink" href="#now-for-training" title="Permalink to this heading">#</a></h3>
@@ -636,16 +650,6 @@ <h4>Script Workflow<a class="headerlink" href="#script-workflow" title="Permalin
 <li><p><code class="docutils literal notranslate"><span class="pre">BackboneFreezeUnfreeze</span></code>: Unfreezes the model backbone after a certain epoch.</p></li>
 </ul>
 </li>
-<li><p>Initialize the Lightning Trainer with required configurations:</p>
-<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">devices</span></code>: Automatic device selection.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">accelerator</span></code>: Automatic accelerator selection.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">max_epochs</span></code>: Maximum number of training epochs.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">precision</span></code>: Mixed-precision training for improved efficiency.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">logger</span></code>: AIM logger for experiment tracking.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">callbacks</span></code>: List of callback functions.</p></li>
-</ul>
-</li>
 <li><p>Train the model using the <code class="docutils literal notranslate"><span class="pre">fit</span></code> method with training and validation data loaders.</p>
 <ul class="simple">
 <li><p>We unfreeze the backbone architecture after 10 epochs.</p></li>
@@ -671,6 +675,7 @@ <h4>How to run training<a class="headerlink" href="#how-to-run-training" title="
 </ol>
 <p><code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">classifier_train.py</span> <span class="pre">&lt;EXPERIMENT_NAME&gt;</span> <span class="pre">&lt;IMG_DIR&gt;</span> <span class="pre">&lt;DATA_DIR&gt;</span></code></p>
 <p>Replace <code class="docutils literal notranslate"><span class="pre">&lt;EXPERIMENT_NAME&gt;</span></code>, <code class="docutils literal notranslate"><span class="pre">&lt;IMG_DIR&gt;</span></code>, and <code class="docutils literal notranslate"><span class="pre">&lt;DATA_DIR&gt;</span></code> with the appropriate values.</p>
+<p>For example <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">classifier_train.py</span> <span class="pre">hp_classify_exp</span> <span class="pre">clipped_images/</span> <span class="pre">data/intermediate/</span></code></p>
 </section>
 </section>
 <section id="visual-display-of-training-data">