Skip to content

Feat/preprocessing #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
janvis11 wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Feat/preprocessing #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d31648a
update
janvis11 Mar 29, 2026
10fca92
janvi's work and implementation
janvis11 Mar 30, 2026
c60a56a
janvi's work and implementation
janvis11 Mar 30, 2026
382ffc6
dem integration update
janvis11 Mar 30, 2026
2f54a46
generates 8-bit quality flags, filtering.
janvis11 Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
213 changes: 56 additions & 157 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,159 +1,58 @@
### Automated Coastline Extraction for Erosion Modeling in Alaska

The primary goal of this project is to enhance the accuracy of coastline extraction, particularly for erosion modeling in Deering, Alaska, using high-resolution Planet imagery with a 3-meter resolution. The project focuses on creating reliable ground truth data and labels that will be used to train the [DeepWaterMap algorithm](https://github.com/isikdogan/deepwatermap), a deep convolutional neural network designed to segment surface water on multispectral imagery. Originally trained on 30-meter resolution Landsat data, DeepWaterMap will be adapted to work with higher-resolution data in this project.

One of the key challenges in coastline extraction is the application of the Normalized Difference Water Index (NDWI), a widely used remote sensing index for identifying water bodies. However, using a single threshold across an entire image often results in suboptimal accuracy. To address this, I implemented a sliding window approach combined with Otsu thresholding, which dynamically adjusts thresholds over localized regions of the image. This method has shown promising improvements in accuracy.

The newly generated labeled data, derived from this approach, will be used to retrain the [DeepWaterMap algorithm](https://github.com/isikdogan/deepwatermap), replacing the original Global Surface Water data. This project aims to produce a more accurate and reliable tool for coastline detection, which is crucial for monitoring and mitigating coastal erosion in vulnerable areas like Alaska.

## Installation

### Prerequisites

Before installing this project, ensure you have the following requirements:

- **Python**
- **Project version**: Tested and developed with **Python 3.13.5**
- **Conda environment**: Recommended to use **Python 3.10** for best compatibility with dependencies

- **Miniconda** (for managing conda environments)

- **Git** (for cloning the repository)

- **GDAL** (install via `conda-forge` for easier setup)

- **Rasterio 1.4.3+** (for geospatial data processing)

---

### Clone the Repository

Clone the project using the `dev` branch (this branch contains the latest development features):

```bash
git clone -b dev https://github.com/your-username/coastline-extraction.git
cd coastline-extraction
```

---

### Environment Setup

1. **Create a virtual environment**:

```bash
python -m venv coastline_env
source coastline_env/bin/activate # On Windows: coastline_env\Scripts\activate
```

2. **Install required dependencies**:

```bash
# Core deep learning libraries
pip install torch torchvision

# Geospatial data processing
pip install rasterio gdal

# Data manipulation and visualization
pip install numpy pandas matplotlib

# Image processing
pip install scikit-image opencv-python

# Utilities
pip install tqdm pillow

# Additional dependencies for data preprocessing
pip install shapely fiona geopandas
```
# Automated Coastline Extraction for Erosion Modeling in Alaska

Work of - Janvi Singh

**Google Summer of Code (GSoC) 2026 Project**

A comprehensive deep learning pipeline for extracting accurate coastlines from high-resolution PlanetLabs satellite imagery, specifically designed for coastal erosion monitoring in Arctic Alaska communities.

## Overview

The rapidly warming Arctic is leading to increased rates of coastal erosion, placing hundreds of Alaska communities at the frontline of climate change. This project provides an automated solution for extracting vectorized coastlines from 3-meter resolution PlanetLabs imagery to support erosion modeling and forecasting.

## What has been implemented in this workspace
1. Unified pipeline entrypoint
- `coastline_pipeline.py` orchestrates steps: UDM masking, NDWI, DEM integration, quality flags, validation.
2. UDM/QA60 cloud/shadow/snow masking
- `data_preprocessing/udm_masking.py` supports PlanetLabs UDM2 bit flags and Sentinel-2 QA60.
3. Adaptive NDWI pipeline with windowing + majority voting
- `data_preprocessing/ndwi_with_udm.py` does local thresholding, mask application, and subscene vector output.
4. DEM/slope/cliff-aware terrain integration
- `data_preprocessing/dem_integration.py` computes slope/aspect/TRI and cliff masks, influences classification.
5. Shadow/artifact / quality flags
- `data_preprocessing/shadow_artifact_detection.py` generates 8-bit quality flags, filtering.
6. Data expansion pipeline
- `data_preprocessing/data_expansion.py` supports 2017-2026 ingestion, seasonal filtering, incremental catalog.
7. DeepWaterMap training and inference
- `training_pipeline/deepwatermap_train.py`, `train_unet.py`, `predict.py` support U-Net training with Planet imagery.
8. Validation framework
- `validation_framework.py` with transect RMSE, comparison to ground truth coastlines.

## Status vs. “Potential areas of improvement”
| Area | Status | Notes / Action required |
|---|---|---|
| Improve training data with PlanetLabs UDM | in progress | UDM masking is implemented, verify dataset-level integration for training label creation in `deepwatermap_train.py` and `data_expansion.py`.
| Data expansion beyond 2017-2019 | ✓ done | `data_expansion.py` has 2020-2026 config and pipeline.
| Improved cliff area segmentation | ✓ done | `dem_integration.py` handles cliff detection (slope >30°, elevation-stratified thresholding).
| Handling shadows/buildings/artifacts | ✓ done | `shadow_artifact_detection.py` intended for this; check empirical results and refine thresholds.
| SWIR + elevation integration | partially done | elevation/DEM done; SWIR support likely not yet explicit (only RGBN). Add SWIR path if available.


### Key Features

- **UDM-Based Cloud Masking**: Integrated QA60/UDM2 quality band processing for reliable cloud and shadow removal
- **Sliding Window NDWI**: Localized Otsu thresholding with majority voting for adaptive water detection
- **DEM Integration**: Terrain analysis for improved cliff/steep slope segmentation
- **Quality Flag System**: 8-bit comprehensive pixel-level quality assessment
- **DeepWaterMap Training**: U-Net based deep learning model adapted for 4-band PlanetLabs imagery
- **Data Expansion Pipeline**: Automated ingestion of imagery from 2016-2026 with seasonal filtering
- **Validation Framework**: Transect-based RMSE and raster-based accuracy metrics

### Project Links

- **Source Code**: https://github.com/fwitmer/CoastlineExtraction
- **My Fork**:https://github.com/janvis11/CoastlineExtraction
- **Discussion Forum**: https://github.com/fwitmer/CoastlineExtraction/discussions
- **Mentors**: Frank Witmer (fwitmer@alaska.edu), Ritika Kumari (rkjane333@gmail.com)

---

## Configuration

This project uses a centralized configuration system to manage file paths and parameters.
Configuration is handled through `config_template.json` and the `load_config.py` module.

### Setting Up Configuration

1. **Copy the template**:

```bash
cp config_template.json config.json
```

2. **Edit the configuration**: Open `config.json` and modify the paths according to your setup:

```json
{
"data_dir": "data",
"image_folder": "sample_data/PlanetLabs",
"raw_data_folder": "raw_data",
"shapefile_folder": "USGS_Coastlines",
"ground_truth_folder": "ground_truth",
"processed_data_folder": "processed_data",
"training": {
"model_save_path": "training_pipeline/unet_model.pth",
"batch_size": 8,
"epochs": 30,
"learning_rate": 1e-4,
"image_size": [256, 256],
"train_split": 0.8,
"device": "auto"
}
}
```

### Using the Configuration System

The `load_config.py` module provides convenient functions to access your data files:

```python
from load_config import load_config, get_image_path, get_shapefile_path

# Load configuration
config = load_config()

# Get specific file paths
image_path = get_image_path(config, 0) # First image file
shapefile_path = get_shapefile_path(config, 0) # First shapefile
```
---


## Contributing

### Working with the Dev Branch

This project uses the `dev` branch for active development. When contributing:

1. **Fork the repository on GitHub**

2. **Clone your fork using the `dev` branch**:

```bash
git clone -b dev https://github.com/your-username/coastline-extraction.git
cd coastline-extraction
```

3. **Create a feature branch from `dev`**:

```bash
git checkout -b feature/your-feature-name
```

4. **Make your changes and commit them**:

```bash
git add .
git commit -m "Add your feature description"
```

5. **Push to your fork**:

```bash
git push origin feature/your-feature-name
```

6. **Create a Pull Request** targeting the `dev` branch (not `main`)
Loading