end-to-end-gee.Rmd

---
title: "End-to-End Google Earth Engine (Full Course)"
subtitle: "A hands-on introduction to applied remote sensing using Google Earth Engine."
author: "Ujaval Gandhi"
fontsize: 12pt
output:
  # pdf_document:
  #   latex_engine: xelatex
  #   toc: yes
  #   toc_depth: 3
  #   fig_caption: false
  # word_document:
  #   toc: false
  #   fig_caption: false
  html_document:
    df_print: paged
    highlight: pygments
    includes:
      after_body: comment.html
    toc: yes
    toc_depth: 3
header-includes:
- \usepackage{fancyhdr}
- \pagestyle{fancy}
- \renewcommand{\footrulewidth}{0.4pt}
- \fancyhead[LE,RO]{\thepage}
- \geometry{left=1in,top=0.75in,bottom=0.75in}
- \fancyfoot[CE,CO]{{\includegraphics[height=0.5cm]{images/cc-by-nc.png}} Ujaval Gandhi http://www.spatialthoughts.com}
---
\newpage

***

```{r echo=FALSE, fig.align='center', out.width='75%', out.width='250pt'}
knitr::include_graphics('images/spatial_thoughts_logo.png')
```

***

\newpage

# Introduction 

Google Earth Engine is a cloud-based platform that enables large-scale processing of satellite imagery to detect changes, map trends, and quantify differences on the Earth’s surface. This course covers the full range of topics in Earth Engine to give the participants practical skills to master the platform and implement their remote sensing projects.


[![Watch the video](https://img.youtube.com/vi/jGuCnu1I2qw/mqdefault.jpg)](https://www.youtube.com/watch?v=jGuCnu1I2qw&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=1){target="_blank"}

[Watch the Video ↗](https://www.youtube.com/watch?v=jGuCnu1I2qw&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=1){target="_blank"}

[Access the Presentation ↗](https://docs.google.com/presentation/d/1q8HRDTqgQEp3Hmi8IG0T7djPLTC1wRig3jXrwFTmoVE/edit?usp=sharing){target="_blank"}


# Sign-up for Google Earth Engine

If you already have a Google Earth Engine account, you can skip this step.

Visit our [GEE Sign-Up Guide](gee-sign-up.html){target="_blank"} for step-by-step instructions.

# Complete the Class Pre-Work

This class needs about 2-hours of pre-work. Please watch the following videos to get a good understanding of remote sensing and how Earth Engine works.

* [Introduction to Remote Sensing](https://www.youtube.com/watch?v=xAyNu9HbK8s){target="_blank"}: This video introduces the remote sensing concepts, terminology and techniques. 
* [Introduction to Google Earth Engine](https://www.youtube.com/watch?v=kpfncBHZBto){target="_blank"}: This video gives a broad overview of Google Earth Engine with selected case studies and application. The video also covers the Earth Engine architecture and how it is different than traditional remote sensing software.

# Get the Course Videos

The course is accompanied by a set of videos covering the all the modules. These videos are recorded from our live instructor-led classes and are edited to make them easier to consume for self-study. We have 2 versions of the videos:

## YouTube

We have created a YouTube Playlist with separate videos for each module to enable effective online-learning. [Access the YouTube Playlist ↗](https://www.youtube.com/playlist?list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha){target="_blank"}

## Vimeo

We are also making the module videos available on Vimeo. These videos can be downloaded for offline learning. [Access the Vimeo Playlist ↗](https://vimeo.com/showcase/11467828?share=copy){target="_blank"}


# Get the Course Materials

The course material and exercises are in the form of Earth Engine scripts shared via a code repository.

1. [Click this link](https://code.earthengine.google.co.in/?accept_repo=users/ujavalgandhi/End-to-End-GEE) to open Google Earth Engine code editor and add the repository to your account.
2. If successful, you will have a new repository named `users/ujavalgandhi/End-to-End-GEE` in the *Scripts* tab in the *Reader* section.
3. Verify that your code editor looks like below

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Code Editor After Adding the Class Repository'}
knitr::include_graphics('images/end_to_end_gee/repository.png')
```

If you do not see the repository in the *Reader* section, click *Refresh repository cache* button in your *Scripts* tab and it will show up.

```{r echo=FALSE, fig.align='center', out.width='50%', fig.cap='Refresh repository cache'}
knitr::include_graphics('images/common/repository_cache.png')
```

There are several slide decks containing useful information and references. You can access all the presentations used in the course from the links below.

* Introduction and course overview [[Presentation ↗](https://docs.google.com/presentation/d/1q8HRDTqgQEp3Hmi8IG0T7djPLTC1wRig3jXrwFTmoVE/edit?usp=sharing)]
* Map/Reduce Programming Concepts [[Presentation ↗](https://docs.google.com/presentation/d/10qOyxhubkwnsAVjniW54ETgwUHq3DXYKo3HGb6Gemi0/edit?usp=sharing)]
* Introduction to Machine Learning & Supervised Classification [[Presentation ↗](https://docs.google.com/presentation/d/19L1b5vsxb38xS8GlHNKOjvPZ0IGqDhv93681btMEL5w/edit?usp=sharing)]
* Introduction to Change Detection [[Presentation ↗](https://docs.google.com/presentation/d/1vdFTWJ61yDuVfbfhpnumQ8zuMPGwGcHpHsBTRgo_o5I/edit?usp=sharing)]
* Introduction to Earth Engine Apps [[Presentation ↗](https://docs.google.com/presentation/d/1u4Q91OqT9_OS4m1OWMm3uRUgu_oseqDUxHV-8mpzGz4/edit?usp=sharing)]
* Introduction to Google Earth Engine Python API [[Presentation ↗](https://docs.google.com/presentation/d/1hPVRnxp2Vp1VHXBtu36SH_UtEOjPz70KcDV-zGIin3U/edit?usp=sharing)]


\newpage

# Module 1: Earth Engine Basics

Module 1 is designed to give you basic skills to be able to find datasets you need for your project, filter them to your region of interest, apply basic processing and export the results. Mastering this will allow you to start using Earth Engine for your project quickly and save a lot of time pre-processing the data.

[![Watch the Video](https://img.youtube.com/vi/daYhxVVIpJU/mqdefault.jpg)](https://www.youtube.com/watch?v=daYhxVVIpJU&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=2){target="_blank"}

[Watch the Video ↗](https://www.youtube.com/watch?v=daYhxVVIpJU&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=2){target="_blank"}

## 01. Hello World

This script introduces the basic Javascript syntax and the video covers the programming concepts you need to learn when using Earth Engine. To learn more, visit [Introduction to JavaScript for Earth Engine](https://developers.google.com/earth-engine/tutorials/tutorial_js_01) section of the Earth Engine User Guide. 

The *Code Editor* is an Integrated Development Environment (IDE) for Earth Engine Javascript API.. It offers an easy way to type, debug, run and manage code. Type the code below and click *Run* to execute it and see the output in the *Console* tab.

> Tip: You can use the keyboard shortcut *Ctrl+Enter* to run the code in the Code Editor

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Hello World'}
knitr::include_graphics('images/end_to_end_gee/hello_world.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F01b_Hello_World_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/01b_Hello_World_(complete)')}
```



### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F01c_Hello_World_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/01c_Hello_World_(exercise)')}
```

### Saving Your Work

When you modify any script for the course repository, you may want to save a copy for yourself. If you try to click the *Save* button, you will get an error message like below

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/setup1.png')
```

This is because the shared class repository is a *Read-only* repository. You can click *Yes* to save a copy in your repository. If this is the first time you are using Earth Engine, you will be prompted to choose a *Earth Engine username*. Choose the name carefully, as it cannot be changed once created.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/end_to_end_gee/setup2.png')
```

After entering your username, your home folder will be created. After that, you will be prompted to enter a new repository. A repository can help you organize and share code. Your account can have multiple repositories and each repository can have multiple scripts inside it. To get started, you can create a repository named *default*. Finally, you will be able to save the script.

## 02. Working with Image Collections

Most datasets in Earth Engine come as a `ImageCollection`. An ImageCollection is a dataset that consists of images takes at different time and locations - usually from the same satellite or data provider. You can load a collection by searching the [Earth Engine Data Catalog](https://developers.google.com/earth-engine/datasets) for the *ImageCollection ID*. Search for the *Sentinel-2 Level 1C* dataset and you will find its id `COPERNICUS/S2_SR`. Visit the [Sentinel-2, Level 1C page](https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S2) and see *Explore in Earth Engine* section to find the code snippet to load and visualize the collection. This snippet is a great starting point for your work with this dataset. Click the **Copy Code Sample** button and paste the code into the code editor. Click *Run* and you will see the image tiles load in the map.


```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/image_collection1.png')
```

In the code snippet, You will see a function `Map.setCenter()` which sets the viewport to a specific location and zoom level. The function takes the X coordinate (longitude), Y coordinate (latitude) and Zoom Level parameters. Replace the X and Y coordinates with the coordinates of your city and click *Run* to see the images of your city.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/image_collection2.png')
```

### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F02c_Image_Collections_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/02c_Image_Collections_(exercise)')}
```

## 03. Filtering Image Collections

The collection contains all imagery ever collected by the sensor. The entire collections are not very useful. Most applications require a subset of the images. We use **filters** to select the appropriate images. There are many types of filter functions, look at `ee.Filter...` module to see all available filters. Select a filter and then run the `filter()` function with the filter parameters. 

We will learn about 3 main types of filtering techniques

* **Filter by metadata**: You can apply a filter on the image metadata using filters such as `ee.Filter.eq()`, `ee.Filter.lt()` etc. You can filter by PATH/ROW values, Orbit number, Cloud cover etc.
* **Filter by date**: You can select images in a particular date range using filters such as `ee.Filter.date()`.
* **Filter by location**: You can select the subset of images with a bounding box, location or geometry using the `ee.Filter.bounds()`. You can also use the drawing tools to draw a geometry for filtering.

After applying the filters, you can use the `size()` function to check how many images match the filters.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/filters.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F03b_Filtering_Image_Collection_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/03b_Filtering_Image_Collection_(complete)')}
```


### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F03c_Filtering_Image_Collection_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/03c_Filtering_Image_Collection_(exercise)')}
```

## 04. Creating Mosaics and Composites from ImageCollections

The default order of the collection is by date. So when you display the collection, it implicitly creates a mosaic with the latest pixels on top. You can call `.mosaic()` on a ImageCollection to create a mosaic image from the pixels at the top.

We can also create a composite image by applying selection criteria to each pixel from all pixels in the stack. Here we use the `median()` function to create a composite where each pixel value is the median of all pixels from the stack.

> Tip: If you need to create a mosaic where the images are in a specific order, you can use the `.sort()` function to sort your collection by a property first.



```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='Mosaic vs. Composite'}
knitr::include_graphics('images/end_to_end_gee/mosaic_composite.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F04b_Mosaics_and_Composites_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/04b_Mosaics_and_Composites_(complete)')}
```



### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F04c_Mosaics_and_Composites_(exercise)){target="_blank"}

```{js eval=FALSE}
// Create a median composite for the year 2020 and load it to the map

// Compare both the composites to see the changes in your city
```



## 05. Working with Feature Collections

Feature Collections are similar to Image Collections - but they contain *Features*, not images. They are equivalent to Vector Layers in a GIS. We can load, filter and display Feature Collections using similar techniques that we have learned so far. 

Search for *GAUL Second Level Administrative Boundaries* and load the collection. This is a global collection that contains all Admin2 boundaries. We can apply a filter using the `ADM1_NAME` property to get all Admin2 boundaries (i.e. Districts) from a state.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/feature_collection.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F05b_Feature_Collections_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/05b_Feature_Collections_(complete)')}
```


### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F05c_Feature_Collections_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/05c_Feature_Collections_(exercise)')}
```

## 06. Importing Data

You can import vector or raster data into Earth Engine. We will now import a shapefile of [Urban Centres](https://bit.ly/ghs-ucdb-shapefile) from JRC's GHS Urban Centre Database (GHS-UCDB). Unzip the `ghs_urban_centers.zip` into a folder on your computer. In the Code Editor, go to *Assets → New → Table Upload → Shape Files*. Select the `.shp`, `.shx`, `.dbf` and .`prj` files. Enter `ghs_urban_centers` as the *Asset Name* and click  *Upload*. Once the upload and ingest finishes, you will have a new asset in the *Assets* tab. The shapefile is imported as a Feature Collection in Earth Engine. Select the `ghs_urban_centers` asset and click *Import*.  You can then visualize the imported data.


```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Importing a Shapefile'}
knitr::include_graphics('images/end_to_end_gee/import.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F06b_Import_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/06b_Import_(complete)')}
```



### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F06c_Import_(exercise)){target="_blank"}


```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/06c_Import_(exercise)')}
```

## 07. Clipping Images

It is often desirable to clip the images to your area of interest. You can use the `clip()` function to mask out an image using a geometry.

> While in a Desktop software, clipping is desirable to remove unnecessary portion of a large image and save computation time, in Earth Engine clipping can actually increase the computation time. As described in the [Earth Engine Coding Best Practices](https://developers.google.com/earth-engine/guides/best_practices?hl=en#if-you-dont-need-to-clip,-dont-use-clip) guide, avoid clipping the images or do it at the end of your script.


```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='Original vs. Clipped Image'}
knitr::include_graphics('images/end_to_end_gee/clipping.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F07b_Clipping_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/07b_Clipping_(complete)')}
```

### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F07c_Clipping_(exercise)){target="_blank"}


```{js eval=FALSE}
// Add the imported table to the Map
// Use the Inspector to find the id of your home city or any urban area of your choice
// Change the filter to use the id of the selected feature
```

## 08. Exporting Data

Earth Engine allows for exporting both vector and raster data to be used in an external program. Vector data can be exported as a `CSV` or a `Shapefile`, while Rasters can be exported as `GeoTIFF` files. We will now export the Sentinel-2 Composite as a GeoTIFF file.

> Tip: Code Editor supports autocompletion of API functions using the combination *Ctrl+Space*. Type a few characters of a function and press *Ctrl+Space* to see autocomplete suggestions. You can also use the same key combination to fill all parameters of the function automatically.

Once you run this script, the *Tasks* tab will be highlighted. Switch to the tab and you will see the tasks waiting. Click *Run* next to each task to start the process.  

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/export_task01.png')
```

On clicking the *Run* button, you will be prompted for a confirmation dialog. Verify the settings and click *Run* to start the export.

```{r echo=FALSE, fig.align='center', out.width='40%'}
knitr::include_graphics('images/end_to_end_gee/export_task02.png')
```

Once the Export finishes, a GeoTiff file for each export task will be added to your Google Drive in the specified folder. You can download them and use it in a GIS software.

```{r echo=FALSE, fig.align='center', out.width='85%', fig.cap='Visualized vs. Raw Composite'}
knitr::include_graphics('images/end_to_end_gee/exporting_data.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F08b_Export_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/01-Earth-Engine-Basics/08b_Export_(complete)')}
```

### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A01-Earth-Engine-Basics%2F08c_Export_(exercise)){target="_blank"}

```{js eval=FALSE}

// Write the export function to export the results for your chosen urban area
```

## Assignment 1

Load the Night Lights Data for May 2015 and May 2020. Compare the imagery for your region and find the changes in the city due to COVID-19 effect.

```{r echo=FALSE, fig.align='center', out.width='85%', fig.cap='Assignment1 Expected Output'}
knitr::include_graphics('images/end_to_end_gee/assignment1.png')
```

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3AAssignments%2FAssignment1){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/Assignments/Assignment1')}
```



\newpage

# Module 2: Earth Engine Intermediate

Module 2 builds on the basic Earth Engine skills you have gained. This model introduces the parallel programming concepts using Map/Reduce - which is key in effectively using Earth Engine for analyzing large volumes of data. You will learn how to use the Earth Engine API for calculating various spectral indices, do cloud masking and then use map/reduce to do apply these computations to collections of imagery. You will also learn how to take long time-series of data and create charts.

[![Watch the Video](https://img.youtube.com/vi/UW3jinsnLko/mqdefault.jpg)](https://www.youtube.com/watch?v=UW3jinsnLko&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=3){target="_blank"}

[Watch the Video ↗](https://www.youtube.com/watch?v=UW3jinsnLko&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=3){target="_blank"}


## 01. Earth Engine Objects

This script introduces the basics of the Earth Engine API. When programming in Earth Engine, you must use the Earth Engine API so that your computations can use the Google Earth Engine servers. To learn more, visit [Earth Engine Objects and Methods](https://developers.google.com/earth-engine/tutorial_js_02) section of the Earth Engine User Guide. 

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F01b_Earth_Engine_Objects_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/01b_Earth_Engine_Objects_(complete)')}
```


> As a general rule, you should always use Earth Engine API methods in your code, there is one exception where you will need to use client-side Javascript method. If you want to get the current time, the server doesn't know your time. You need to use javascript method and cast it to an Earth Engine object.
  ```
  var now = Date.now()
  print(now)
  var now = ee.Date(now)
  print(now)
  ```

### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F01c_Earth_Engine_Objects_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/01c_Earth_Engine_Objects_(exercise)')}
```

## 02. Calculating Indices

Spectral Indices are central to many aspects of remote sensing. Whether you are studying vegetation or tracking fires - you will need to compute a pixel-wise ratio of 2 or more bands. The most commonly used formula for calculating an index is the *Normalized Difference* between 2 bands. Earth Engine provides a helper function `normalizedDifference()` to help calculate normalized indices, such as Normalized Difference Vegetation Index (NDVI). For more complex formulae, you can also use the `expression()` function to describe the calculation.

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='MNDWI, SAVI and NDVI images'}
knitr::include_graphics('images/end_to_end_gee/indices.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F02b_Calculating_Indices_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/02b_Calculating_Indices_(complete)')}
```



### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F02c_Calculating_Indices_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/02c_Calculating_Indices_(exercise)')}
```

## 03. Computation on ImageCollections

So far we have learnt how to run computation on single images. If you want to apply some computation - such as calculating an index - to many images, you need to use `map()`. You first define a function that takes 1 image and returns the result of the computation on that image. Then you can `map()` that function over the ImageCollection which results in a new ImageCollection with the results of the computation. This is similar to a *for-loop* that you maybe familiar with - but using `map()` allows the computation to run in parallel. Learn more at [Mapping over an ImageCollection](https://developers.google.com/earth-engine/guides/ic_mapping)

```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='NDVI computation on an ImageCollection'}
knitr::include_graphics('images/end_to_end_gee/imagecollection_computation.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F03b_Computation_on_Image_Collections_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/03b_Computation_on_Image_Collections_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F03c_Computation_on_Image_Collections_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/03c_Computation_on_Image_Collections_(exercise)')}
```

## 04. Cloud Masking

Masking pixels in an image makes those pixels transparent and excludes them from analysis and visualization. To mask an image, we can use the `updateMask()` function and pass it an image with 0 and 1 values. All pixels where the mask image is 0 will be masked.

Most remote sensing datasets come with a QA or Cloud Mask band that contains the information on whether pixels is cloudy or not. Your *Code Editor* contains pre-defined functions for masking clouds for popular datasets under *Scripts Tab → Examples → Cloud Masking*. To understand how cloud-masking functions work and learn advanced techniques for bitmasking, please refer to our article on [Working with QA Bands and Bitmasks in Google Earth Engine
](https://spatialthoughts.com/2021/08/19/qa-bands-bitmasks-gee/).

The script below takes the Sentinel-2 masking function and shows how to apply it on an image.


```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='Applying pixel-wise QA bitmask'}
knitr::include_graphics('images/end_to_end_gee/cloud_masking.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F04b_Cloud_Masking_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/04b_Cloud_Masking_(complete)')}
```

### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F04c_Cloud_Masking_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/04c_Cloud_Masking_(exercise)')}
```

> Learn more about the [Cloud Score+](https://medium.com/google-earth/all-clear-with-cloud-score-bd6ee2e2235e) project.

## 05. Reducers

When writing parallel computing code, a *Reduce* operation allows you to compute statistics on a large amount of inputs. In Earth Engine, you need to run reduction operation when creating composites, calculating statistics, doing regression analysis etc. The Earth Engine API comes with a large number of built-in reducer functions (such as `ee.Reducer.sum()`, `ee.Reducer.histogram()`, `ee.Reducer.linearFit()` etc.) that can perform a variety of statistical operations on input data. You can run reducers using the `reduce()` function. Earth Engine supports running reducers on all data structures that can hold multiple values, such as Images (reducers run on different bands), ImageCollection, FeatureCollection, List, Dictionary etc. The script below introduces basic concepts related to reducers.

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F05b_Reducers_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/05b_Reducers_(complete)')}
```


### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F05c_Reducers_(exercise)){target="_blank"}


```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/05c_Reducers_(exercise)')}
```

## 06. Time-Series Charts

Now we can put together all the skills we have learnt so far - filter, map, reduce, and cloud-masking to create a chart of average NDVI values for a given farm over 1 year. Earth Engine API comes with support for charting functions based on the Google Chart API. Here we use the `ui.Chart.image.series()` function to create a time-series chart.


```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='Computing NDVI Time-series for a Farm'}
knitr::include_graphics('images/end_to_end_gee/charts1.png')
```

```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='NDVI Time-series showing Dual-Cropping Cycle'}
knitr::include_graphics('images/end_to_end_gee/charts2.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F06b_Time_Series_Charts_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/02-Earth-Engine-Intermediate/06b_Time_Series_Charts_(complete)')}
```



### Exercise


[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A02-Earth-Engine-Intermediate%2F06c_Time_Series_Charts_(exercise)){target="_blank"}

```{js eval=FALSE}
// Delete the farm boundary from the previous script 
// and add another farm at a location of your choice

// Print the chart.
```

## Assignment 2

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Assignment2 Expected Output'}
knitr::include_graphics('images/end_to_end_gee/assignment2.png')
```

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3AAssignments%2FAssignment2){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/Assignments/Assignment2')}
```


\newpage


# Module 3: Supervised Classification

## Introduction to Machine Learning and Supervised Classification

Supervised classification is arguably the most important classical machine learning techniques in remote sensing. Applications range from generating Land Use/Land Cover maps to change detection. Google Earth Engine is unique suited to do supervised classification at scale. The interactive nature of Earth Engine development allows for iterative development of supervised classification workflows by combining many different datasets into the model. This module covers basic supervised classification workflow, accuracy assessment, hyperparameter tuning and change detection.

[![Watch the Video](https://img.youtube.com/vi/lULwcRpkMv8/mqdefault.jpg)](https://www.youtube.com/watch?v=lULwcRpkMv8&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=4){target="_blank"}

[Watch the Video ↗](https://www.youtube.com/watch?v=lULwcRpkMv8&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=4){target="_blank"}


## 01. Basic Supervised Classification

We will learn how to do a basic land cover classification using training samples collected from the Code Editor using the High Resolution basemap imagery provided by Google Maps. This method requires no prior training data and is quite effective to generate high quality classification samples anywhere in the world. The goal is to classify each source pixel into one of the following classes - urban, bare, water or vegetation. Using the drawing tools in the code editor, you create 4 new feature collection with points representing pixels of that class. Each feature collection has a property called `landcover` with values of 0, 1, 2 or 3 indicating whether the feature collection represents urban, bare, water or vegetation respectively. We then train a *Random Forest* classifier using these training set to build a model and apply it to all the pixels of the image to create a 4 class image.

> Fun fact: The classifiers in Earth Engine API have names starting with **smile** - such as `ee.Classifier.smileRandomForest()`. The *smile* part refers to the [Statistical Machine Intelligence and Learning Engine (SMILE)](https://haifengl.github.io/index.html) JAVA library which is used by Google Earth Engine to implement these algorithms.

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Supervised Classification Output'}
knitr::include_graphics('images/end_to_end_gee/classified.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F01d_Basic_Supervised_Classification_(noimport)){target="_blank"}


```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/01d_Basic_Supervised_Classification_(noimport)')}
```



### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F01c_Basic_Supervised_Classification_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/01c_Basic_Supervised_Classification_(exercise)')}
```

## 02. Accuracy Assessment

It is important to get a quantitative estimate of the accuracy of the classification. To do this, a common strategy is to divide your training samples into 2 random fractions - one used for *training* the model and the other for *validation* of the predictions. Once a classifier is trained, it can be used to classify the entire image. We can then compare the classified values with the ones in the validation fraction. We can use the `ee.Classifier.confusionMatrix()` method to calculate a *Confusion Matrix* representing expected accuracy.

Classification results are evaluated based on the following metrics

* **Overall Accuracy**: How many samples were classified correctly.
* **Producer’s Accuracy**: How well did the classification predict each class.
* **Consumer’s Accuracy (Reliability)**: How reliable is the prediction in each class.
* **Kappa Coefficient**: How well the classification performed as compared to random assignment.

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Accuracy Assessment'}
knitr::include_graphics('images/end_to_end_gee/accuracy_assessment.png')
```

> Don't get carried away tweaking your model to give you the highest validation accuracy. You must use both qualitative measures (such as visual inspection of results) along with quantitative measures to assess the results. 


[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F02b_Accuracy_Assessment_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/02b_Accuracy_Assessment_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F02c_Accuracy_Assessment_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/02c_Accuracy_Assessment_(exercise)')}
```

## 03. Improving the Classification

The Earth Engine data-model is especially well suited for machine learning tasks because of its ability to easily incorporate data sources of different spatial resolutions, projections and data types together By giving additional information to the classifier, it is able to separate different classes easily. Here we take the same example and augment it with the following techniques

* *Apply Cloud Masking*
* *Add Spectral Indices*: We add bands for different spectral indices such as - NDVI, NDBI, MNDWI and BSI. 
* *Add Elevation and Slope*: We also add slope and elevation bands from the ALOS DEM. 
* *Normalize the Inputs*: Machine learning models work best when all the inputs have the same scale. We will divide each band with the maximum value. This method ensures that all input values are between 0-1. A more [complete and robust technique](#image-normalization-and-standardization) for image normalization is provided in the course Supplement.

Our training features have more parameters and contain values of the same scale. The result is a much improved classification.


```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Improved Classification Accuracy with use of Spectral Indices and Elevation Data'}
knitr::include_graphics('images/end_to_end_gee/improving_classification.png')
```


[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F03b_Improving_the_Classification_(complete)){target="_blank"}


```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/03b_Improving_the_Classification_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F03c_Improving_the_Classification_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/03c_Improving_the_Classification_(exercise)')}
```

## 04. Exporting Classification Results

When working with complex classifiers over large regions, you may get a *User memory limit exceeded* or *Computation timed out* error in the Code Editor. The reason for this is that there is a fixed time limit and smaller memory allocated for code that is run with the *On-Demand Computation* mode. For larger computations, you can use the *Batch* mode with the `Export` functions. Exports run in the background and can run longer than 5-minutes time allocated to the computation code run from the Code Editor. This allows you to process very large and complex datasets. Here's an example showing how to export your classification results to Google Drive.


```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Exported Classification Outputs'}
knitr::include_graphics('images/end_to_end_gee/export_classification.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F04b_Exporting_Classification_Results_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/04b_Exporting_Classification_Results_(complete)')}
```

### Exercise


It is also a good idea to export the classified image as an Asset. This will allows you to import the classified image in another script without running the whole classification workflow. Use the Export.image.toAsset() function to export the classified image as an asset.

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F04c_Exporting_Classification_Results_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/04c_Exporting_Classification_Results_(exercise)')}
```

## 05. Calculating Area

Now that we have the results of our classification, we will learn how to calculate the area for pixels in each class. The functions used for area computations are different for vectors and raster data.

* **Area of Polygons**: Calculating area for polygons is done using the `area()` function. It computes area on a sphere (ignoring the ellipsoid flattening) and gives you the area in square meters. You can optionally supply `proj` and a non-zero `maxError` parameters to calculate area in a specific projected CRS. For example, `area({proj:'EPSG:32643', maxError: 1})` will calculate the area of the polygon after reprojecting it to the *WGS 84/UTM Zone 43* CRS with a tolerance of 1 meter.
* **Area of Image Pixels**: Area of image pixels is computed using the `ee.Image.pixelArea()` function. This function computes the area inside the 4 corners of each pixel using the WGS84 ellipsoid. The `ee.Image.pixelArea()` function uses a custom equal-area projection for area calculation. The result is area in square meters regardless of the projection of the input image. [Learn more](https://groups.google.com/g/google-earth-engine-developers/c/Ccaorx-obVw/m/_ZQdP2wVAgAJ).


```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Calculating Green Cover from Classified Image'}
knitr::include_graphics('images/end_to_end_gee/area_calculation.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F05b_Calculating_Area_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/03-Supervised-Classification/05b_Calculating_Area_(complete)')}
```

> If you want to compute area covered by each class, you can use a [Grouped Reducer](https://developers.google.com/earth-engine/reducers_grouping). See the [Supplement](end-to-end-gee-supplement.html#calculating-area-by-class) to see a code snippet.

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A03-Supervised-Classification%2F05c_Calculating_Area_(exercise)){target="_blank"}

```{js eval=FALSE}
// Exercise
// Compute and print the percentage green cover of the city
```

## Assignment 3

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3AAssignments%2FAssignment3){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/Assignments/Assignment3')}
```

# Module 4: Change Detection

## Introduction to Change Detection

Many earth observation datasets are available at regular intervals over long periods of time. This enables us to detect changes on the Earth's surface. Change detection technique in remote sensing fall in the following categories

- **Single Band Change**: Measuring change in a single band image or a spectral index using a threshold
- **Multi Band Change**: Measuring spectral distance and spectral angle between two multiband images
- **Classification of Change**: One-pass classification using stacked image containing bands from before and after an event
- **Post Classification Comparison**: Comparing two classified images and computing class transitions

[![Watch the Video](https://img.youtube.com/vi/ybHxiygVVz0/mqdefault.jpg)](https://www.youtube.com/watch?v=ybHxiygVVz0&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=5){target="_blank"}

[Watch the Video ↗](https://www.youtube.com/watch?v=ybHxiygVVz0&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=5){target="_blank"}


## 01. Spectral Index Change

Many types of change can be detected by measuring the change in a spectral index and applying a threshold. This technique is suitable when there is a suitable spectral index is available for the type of change you are interested in detecting.

Here we apply this technique to map the extent and severity of a forest fire. The **Normalized Burn Ratio (NBR)** is an index that is designed to highlight burnt vegetation areas. We compute the NBR for before and after images. Then we apply a suitable threshold to find burnt areas.

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Spectral Index Change Detection'}
knitr::include_graphics('images/end_to_end_gee/spectral_index_change.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F01b_Spectral_Index_Change_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/04-Change-Detection/01b_Spectral_Index_Change_(complete)')}
```

### Exercise

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Classifying the Change Image'}
knitr::include_graphics('images/end_to_end_gee/spectral_index_change_exercise.png')
```

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F01c_Spectral_Index_Change_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/04-Change-Detection/01c_Spectral_Index_Change_(exercise)')}
```

## 02. Spectral Distance Change

When you want to detect changes from multi-band images, a useful technique is to compute the Spectral Distance and Spectral Angle between the two images. Pixels that exhibit a large change will have a larger distance compared to those that did not change. This technique is particularly useful when there are no suitable index to detect the change. It can be applied to detect change after natural disasters or human conflicts.

Here we use this technique to detect landslides using before/after composites. You may learn more about this technique at [Craig D'Souza's Change Detection](https://goo.gl/xotYhk) presentation.

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Spectral Distance Change Detection'}
knitr::include_graphics('images/end_to_end_gee/spectral_distance.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F01b_Spectral_Distance_Change_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/04-Change-Detection/02b_Spectral_Distance_Change_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F01c_Spectral_Distance_Change_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/04-Change-Detection/02c_Spectral_Distance_Change_(exercise)')}
```

## 03. Direct Classification of Change

This technique of change detection is also known as *One-pass Classification* or *Direct Multi-date Classification*. Here we create a single stacked image containing bands from before and after images. We train a classifier with training data sampled from the stacked image and apply the classifier on the stacked image to find all change pixels.

```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='All pixels that changed from bare ground to built-up'}
knitr::include_graphics('images/end_to_end_gee/change_classification.png')
```

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F03b_Classifying_Change_(complete)){target="_blank"}


```{js eval=FALSE, code=readLines('code/end_to_end_gee/04-Change-Detection/03b_Classifying_Change_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?accept_repo=users%2Fujavalgandhi%2FEnd-to-End-GEE&scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F03c_Classifying_Change_(exercise)){target="_blank"}

```{js eval=FALSE}
// Add an NDBI band to improve the detection of changes.

var addNDBI = function(image) {
  var ndbi = image.normalizedDifference(['B11', 'B8']).rename(['ndbi']);
  return image.addBands(ndbi)
}

// use addNDBI() function to add the NDBI band to both 2019 and 2020 composite images
// Hint1: You can save the resulting image in the same variable to avoid changing 
// a lot of code.
// var image = addNDBI(image)

```

## 04. Post-classification Comparison

We dealing with multi-class images, a useful metric for change detection is to know how many pixels from class X changed to class Y. This can be accomplished using the `ee.Reducer.frequencyHistogram()` reducer as shown below.


[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F04b_Post_Classification_Comparison_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/04-Change-Detection/04b_Post_Classification_Comparison_(complete)')}
```

### Exercise


```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='Lost water pixels between 2019 and 2020'}
knitr::include_graphics('images/end_to_end_gee/post_classification.png')
```

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A04-Change-Detection%2F04c_Post_Classification_Comparison_(exercise)){target="_blank"}

```{js eval=FALSE}
// Exercise
// Show all areas where water became other classes and display the result
// Hint1: Select class 3 pixels from before image and NOT class 3 pixels from after image
// Hint2: use the .and() operation to select pixels matching both conditions
```


# Module 5: Earth Engine Apps

This module is focused the concepts related to client vs. server that will help you in creating web apps. We will be building an app using the Earth Engine User Interface API and publishing it to Google Cloud.

[![Watch the Video](https://img.youtube.com/vi/Y4lM7Wtckhs/mqdefault.jpg)](https://www.youtube.com/watch?v=Y4lM7Wtckhs&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=6){target="_blank"}

[Watch the Video ↗](https://www.youtube.com/watch?v=Y4lM7Wtckhs&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=6){target="_blank"}


## 01. Client vs. Server

The User Interface elements in your Code Editor - Map View, Drawing Tools etc. are 'client-side' elements. They run in YOUR browser. Image Collections, feature collections, calculations on Earth Engine objects etc. are 'server-side' elements. They run in Google's data center. You cannot mix both these objects. To learn more, visit the [Client vs. Server](https://developers.google.com/earth-engine/guides/client_server) section of the Earth Engine User Guide.

* To convert client-side objects to server-side objects, you can use the appropriate API function. Server-side functions start with `ee.`, such `ee.Date()`, `ee.Image()` etc.
* To convert server-side objects to client-side objects, you can call `.getInfo()` on am Earth Engine object. For the Python API, this is the only way to extract information from a server-side object, but the Javascript API provides a better (and preferred) - method for bring server-side objects to client-side using the `evaluate()` method. This method asynchronously retrieves the value of the object, without blocking the user interface - meaning it will let your code continue to execute while it fetches the value. 

> Tip: You can use `ee.Algorithms.ObjectType()` to get the type of a server-side object

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F01b_Client_vs_Server_(complete)){target="_blank"}
```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/01b_Client_vs_Server_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F01c_Client_vs_Server_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/01c_Client_vs_Server_(exercise)')}
```

## 02. Using UI Elements

Earth Engine comes with a User Interface API that allows you to build an interactive web application powered by Earth Engine.

The Earth Engine API provides a library of User Interface (UI) widgets - such as Buttons, Drop-down Menus, Sliders etc. - that can be used to create interactive apps. All the user interface functions are contained in the `ui.` package - such as `ui.Select()`, `ui.Button()`. You can create those elements by calling these functions with appropriate parameters. Learn more in the [Earth Engine User Interface API](https://developers.google.com/earth-engine/guides/ui) section of the Earth Engine User Guide.

This section shows how to build a drop-down selector using the `ui.Select()` widget.

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F02b_Using_UI_Elements_(complete)){target="_blank"}
```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/02b_Using_UI_Elements_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F02c_Using_UI_Elements_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/02c_Using_UI_Elements_(exercise)')}
```


## 03. Building and Publishing an App

Building a web mapping application typically requires the skills of a full stack developer and are out of reach for most analysts and scientists. But the Earth Engine User Interface API makes this process much more accessible by providing ready-to-use widgets and free cloud hosting to allow anyone to publish an app with just a few lines of code.  The main container object is the `ui.Panel()` which can contain different types of widgets. 

The code below shows how to build an app called [Night Lights Explorer](https://santhosh-m.users.earthengine.app/view/night-lights-explorer) that allows anyone to pick a year/month and load the *VIIRS Nighttime Day/Night Band Composite* for the selected month. Copy/paste the code below to your Code Editor and click *Run*.


```{r echo=FALSE, fig.align='center', out.width='100%'}
knitr::include_graphics('images/end_to_end_gee/app1.png')
```

You will see a panel on the right-hand side with 2 drop-down boxes and a button. These are User Interface (UI) widgets provided by the Earth Engine API that allows the user to interactively select the values. You can select the values for *year* and *month* and click *Load* button to see the image for the selected month. 

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F03b_Building_an_App_with_UI_Widgets_(complete)){target="_blank"}


```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/03b_Building_an_App_with_UI_Widgets_(complete)')}
```


### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F03c_Building_an_App_with_UI_Widgets_(exercise)){target="_blank"}

```{js eval=FALSE}
// Exercise
// Add a button called 'Reset'
// Clicking the button should remove all loaded layers

// Hint: Use Map.clear() for removing the layers
```

## 04. Publishing the App

We will now publish this app. Click on the *Apps* button.

```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='App with UI Elements'}
knitr::include_graphics('images/end_to_end_gee/app2.png')
```


In the *Manage Apps* window, click *New App*.

```{r echo=FALSE, fig.align='center', out.width='60%'}
knitr::include_graphics('images/end_to_end_gee/app3.png')
```

Select the existing project or create a new project. The app will be hosted on Google Cloud, so you will need to create and link a Google Cloud project with the app. If you don't have a Google Cloud account, you can select the *Register a New Project* option to create a new project. You can provide an edit access based on the project selection.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/end_to_end_gee/app4.png')
```

Give the name of your app and see the URL created for your app.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/end_to_end_gee/app5.png')
```

Select code to use for the app. It can be from the current content or choose any repository path where the code is saved. We will go ahead with *Current contents of editor* 

```{r echo=FALSE, fig.align='center', out.width='60%'}
knitr::include_graphics('images/end_to_end_gee/app6.png')
```

Click next and in the *Publish New App* dialog, leave all other settings to default and click *Publish*.


```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/end_to_end_gee/app7.png')
```

The app will be hosted on Google Cloud and you can access it by clicking on the *App Name* of your app in the *Manage Apps* dialog.

```{r echo=FALSE, fig.align='center', out.width='60%'}
knitr::include_graphics('images/end_to_end_gee/app8.png')
```

You will see your Earth Engine powered app running in the browser. Anyone can access and interact with the app by just visiting the App URL.

> The app publishing process takes a few minutes. So if you get an error that your app is not yet ready, check back in a few minutes.

[Explore The App ↗](https://ujavalgandhi.users.earthengine.app/view/night-lights-explorer){target="_blank"}

```{r echo=FALSE, fig.align='center', out.width='100%'}
knitr::include_graphics('images/end_to_end_gee/app9.png')
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F04c_Publishing_the_App_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/04c_Publishing_the_App_(exercise)')}
```

## 05. Create a Split Panel App

Another useful widget that can be used in Apps is `ui.SplitPanel()`. This allows you to create an app that can display 2 different images of the same region that can be explored interactively by swiping. Here we create an app to explore the [ESA WorldCover 10m](https://developers.google.com/earth-engine/datasets/catalog/ESA_WorldCover_v100) global classification dataset. 

On the left-hand panel, we will load a Sentinel-2 composite for the year 2020. On the right-hand panel, we will load the 11-class landcover classification of the same region.

[Open in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F05b_Split_Panel_App_(complete)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/05b_Split_Panel_App_(complete)')}
```

### Exercise

[Try in Code Editor ↗](https://code.earthengine.google.co.in/?scriptPath=users%2Fujavalgandhi%2FEnd-to-End-GEE%3A05-Earth-Engine-Apps%2F05c_Split_Panel_App_(exercise)){target="_blank"}

```{js eval=FALSE, code=readLines('code/end_to_end_gee/05-Earth-Engine-Apps/05c_Split_Panel_App_(exercise)')}
```

# Module 6: Google Earth Engine Python API

## Introduction to the Python API

Till this point in the course, we have used the Earth Engine Javascript API for all our analysis. Earth Engine also provides a Python API. If you are a Python programmer, you may prefer to use the Python API to integrate Earth Engine in your spatial analysis workflow. There are many options for running Python code that uses the Google Earth Engine API. We will use the following two methods in this course.

[![Watch the Video](https://img.youtube.com/vi/34yNkLmEHAI/mqdefault.jpg)](https://www.youtube.com/watch?v=34yNkLmEHAI&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=7){target="_blank"}

[Watch the Video ↗](https://www.youtube.com/watch?v=34yNkLmEHAI&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=7){target="_blank"}


### Google Colab

An easy way to start using the Google Earth Engine Python API is via [Google Colab](https://colab.research.google.com/). Google Colaboratory provides a hosted environment to run Python notebooks without having to install Python locally. It also comes pre-installed with many useful packages - including the Google Earth Engine Python API. You can simply visit https://colab.research.google.com/ and start a new notebook. 


### Local Development Environment

An advantage of Python API is that you can use it in your own development environment - so you get a lot more flexibility to automate as well as combine other analysis and visualization libraries with Earth Engine. This requires installing Python and the Earth Engine Python API on your machine or server. You also need to do a one-time authentication and save the token on the machine. The preferred method for installing the Earth Engine Python API is via Anaconda. Please follow our [Google Earth Engine Python API Installation Guide](install-gee-python-api.html) for step-by-step instructions.


## 01. Python API Syntax

[Open in Google Colab ↗](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/end_to_end_gee/01_python_api_syntax.ipynb){target="_blank"}

```{r child='end-to-end-gee-output/01_python_api_syntax.md'}
```


## 02. Automatic Conversion of Javascript Code to Python

[Open in Google Colab ↗](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/end_to_end_gee/02_automatic_conversion_of_scripts.ipynb){target="_blank"}


```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='Interactive leaflet map created by geemap'}
knitr::include_graphics('images/end_to_end_gee/automatic_conversion.png')
```


```{r child='end-to-end-gee-output/02_automatic_conversion_of_scripts.md'}
```


## 03. Batch Exports

[Open in Google Colab ↗](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/end_to_end_gee/03_export_a_collection.ipynb){target="_blank"}

```{r child='end-to-end-gee-output/03_export_a_collection.md'}
```

```{r echo=FALSE, fig.align='center', out.width='100%', fig.cap='Launching multiple tasks using the  Python API'}
knitr::include_graphics('images/end_to_end_gee/exporting_a_collection.png')
```

## 04. Using Earth Engine with XArray

[Open in Google Colab ↗](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/end_to_end_gee/04_time_series_processing_xee.ipynb){target="_blank"}

```{r child='end-to-end-gee-output/04_time_series_processing_xee.md'}
```

## 05. Automating Downloads

Another common use of the GEE Python API is to automate data processing and export. You can create a Python script that can be called from a server or launched on a schedule using tools such as [Windows Scheduler](https://medium.com/@roddyjaques/how-to-run-anaconda-programs-with-a-bat-file-5f6dd7675508) or [crontab](https://donny-son.github.io/posts/cronjob-with-conda/). 

This script below provides a complete example of automating a download using Google Earth Engine API. It uses the Google Earth Engine API to compute the average soil moisture for the given time period over all districts in a state. The result is then downloaded as a CSV file and saved locally.

> Before running the script, please install the Earth Engine Python Client Library and commplete the authentication workflow on your machine using our [step-by-step instructions](#install-gee-python-api.html).

Once you have finished the authentication, follow the steps below to create a script to download data from GEE. 


1. Create a new file named `download_data.py` with the content shown below.

```{py eval=FALSE, code=readLines('code/end_to_end_gee/download_data.py')}
``` 

2. From the terminal, navigate to the directory where you have created the file and type the command below to run the script.

```
python download_data.py
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/download2.png')
```

3. The script will download the data from GEE and save a file to your current directory.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/download3.png')
```

## 06. Automating Exports

If you want to automate Earth Engine jobs on a server, it is preferable to use a [Service Account](https://developers.google.com/earth-engine/guides/service_account) for authentication. A service account is a virtual account linked to a Google Cloud project. Once you create a service account and give necessary permissions, it can be used to authenticate your Earth Engine account instead of your own account. Each service account is identified by an email address in the form of `<account_name>@<project>.iam.gserviceaccount.com`. Along with them, the credentials are stored in a separate `.json` file. You need to keep this `.json` file safe on the system with appropriate permissions so external users do not have access to it.

> Before running the script, please install the Earth Engine Python Client Library on the server using our [step-by-step instructions](#install-gee-python-api.html).

The below code example shows how to authenticate an Earth Engine Export job using a service account. The code snippet assumes that the private key file is in the same directory as the scirpt and saved as a file named `.private_key.json`.

```{py eval=FALSE, code=readLines('code/end_to_end_gee/export_data.py')}
```

## 07. Using the Google Earth Engine QGIS Plugin

The [QGIS Google Earth Engine plugin](https://gee-community.github.io/qgis-earthengine-plugin/) allows GEE Python API code to be run from QGIS and visualize the results directly in QGIS. This allows users to integrate QGIS's rich cartographic features with the cloud data processing capabilities. After installing the plugin, you will be prompted to authenticate to GEE using your credentials. Once authenticated, you are able to run EE Python API code along with other PyQGIS code within QGIS.


> If you get an authentication error, [follow these instructions](https://courses.spatialthoughts.com/install-gee-python-api.html#authentication) to install the Python API client and run `earthengine authenticate`

We will run a script that processes and visualizes a future climate scenario using the [CMIP6 Climate Models](https://www.carbonbrief.org/cmip6-the-next-generation-of-climate-models-explained/).

Open the QGIS Python Console from **Plugins &rarr; Python Console** and click the **Show Editor** button. Paste the following code and click the **Run** button. Once the code runs, the resulting image computed by Earth Engine and will be streamed to QGIS as a new layer.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/ee_qgis1.png')
```

```{py eval=FALSE, code=readLines('code/end_to_end_gee/ee_qgis.py')}
```

You can now use this layer in your QGIS project or Print Layout. Here's an example of visualizing the layer on a custom globe using the [Globe Builder Plugin](https://plugins.qgis.org/plugins/GlobeBuilder/).

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/end_to_end_gee/ee_qgis2.png')
```

\newpage


# Supplement

We have a large collection of scripts that accompany this course. Visit the [Supplement](https://courses.spatialthoughts.com/end-to-end-gee-supplement.html).

# Guided Projects

Below are step-by-step video-based walkthrough of implementing real-world projects using Earth Engine. You can continue their learning journey by implementing these projects for their region of interest after the class.

[![Watch the Video](https://img.youtube.com/vi/uoOvOPK0iro/mqdefault.jpg)](https://www.youtube.com/watch?v=uoOvOPK0iro&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=8){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=uoOvOPK0iro&list=PLppGmFLhQ1HJuIb7qMiKIv11HiEQhy3ha&index=8){target="_blank"}


## Get the Code

1. [Click this link](https://code.earthengine.google.co.in/?accept_repo=users/ujavalgandhi/End-to-End-Projects) to open Google Earth Engine code editor and add the repository to your account.
2. If successful, you will have a new repository named `users/ujavalgandhi/End-to-End-Projects` in the *Scripts* tab in the *Reader* section.

```{r echo=FALSE, fig.align='center', out.width='75%', fig.cap='Code Editor After Adding the Projects Repository'}
knitr::include_graphics('images/end_to_end_gee/projects.png')
```

If you do not see the repository in the *Reader* section, click *Refresh repository cache* button in your *Scripts* tab and it will show up.

```{r echo=FALSE, fig.align='center', out.width='50%', fig.cap='Refresh repository cache'}
knitr::include_graphics('images/end_to_end_gee/repository_cache.png')
```

## Project 1: Calculating Rainfall Deviation

Calculating Rainfall Deviation from the 30-year mean using CHIRPS Gridded Rainfall Data

[![Watch the Video](https://img.youtube.com/vi/zHUCM3XLc6k/mqdefault.jpg)](https://www.youtube.com/watch?v=zHUCM3XLc6k&list=PLppGmFLhQ1HJ5VhW6BZfhPX6spUcTY7SR){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=zHUCM3XLc6k&list=PLppGmFLhQ1HJ5VhW6BZfhPX6spUcTY7SR){target="_blank"}

## Project 2: Flood Mapping

Rapid mapping of a flood using Sentinel-1 SAR Data.

[![Watch the Video](https://img.youtube.com/vi/jYsK9Y4ICrY/mqdefault.jpg)](https://www.youtube.com/watch?v=jYsK9Y4ICrY&list=PLppGmFLhQ1HJzzKVS_4v8nBiXLYxAu100){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=jYsK9Y4ICrY&list=PLppGmFLhQ1HJzzKVS_4v8nBiXLYxAu100){target="_blank"}

## Project 3: Extracting Time-Series

Extracting a 10-year NDVI time-series over multiple polygons using MODIS data.

[![Watch the Video](https://img.youtube.com/vi/LqSClCXrMl4/mqdefault.jpg)](https://www.youtube.com/watch?v=LqSClCXrMl4&list=PLppGmFLhQ1HJV1CctqanQvXQI1JmqGDDD){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=LqSClCXrMl4&list=PLppGmFLhQ1HJV1CctqanQvXQI1JmqGDDD){target="_blank"}

## Project 4: LandCover Analysis

Use existing land cover products to extract specific classes and compute statistics across many regions.

[![Watch the Video](https://img.youtube.com/vi/B0E_dzO1J4g/mqdefault.jpg)](https://www.youtube.com/watch?v=B0E_dzO1J4g&list=PLppGmFLhQ1HLl0St2wiOPePr58sKu0Vh1){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=B0E_dzO1J4g&list=PLppGmFLhQ1HLl0St2wiOPePr58sKu0Vh1){target="_blank"}

## Project 5: Extracting Nighttime Lights Statistics

Calculate the sum of Nighttime Lights (NTL) for urban and agriculture land use in each Admin1 region in a country for each year from 2013-2021.

[![Watch the Video](https://img.youtube.com/vi/_c-_UU9J6s0/mqdefault.jpg)](https://www.youtube.com/watch?v=_c-_UU9J6s0&list=PLppGmFLhQ1HLN6ivbJJHLZbEYLdq5zgns){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=_c-_UU9J6s0&list=PLppGmFLhQ1HLN6ivbJJHLZbEYLdq5zgns){target="_blank"}

\newpage

# Learning Resources

* [Cloud-Based Remote Sensing with Google Earth Engine: Fundamentals and Applications](https://www.eefabook.org/): A free and open-access book with 55-chapters covering fundamentals and applications of GEE. Also includes YouTube videos summarizing each chapter.
* [Awesome Earth Engine](https://github.com/giswqs/Awesome-GEE): A curated list of Google Earth Engine resources.
* [Google Earth Engine for Water Resources Management](https://courses.spatialthoughts.com/gee-water-resources-management.html): Application-focused Introduction to Google Earth Engine.
* [Creating Publication Quality Charts with GEE](https://courses.spatialthoughts.com/gee-charts.html): A comprehesive guide on creating high-quality data visualizations with Google Earth Engine and Google Charts.

# Useful Public Repositories

Please visit [Awesome Earth Engine](https://github.com/giswqs/Awesome-GEE) to see a curated list of Google Earth Engine resources. 

We also have a few recommendations of a few selected packages, which have very useful functions to make you productive in Earth Engine.

**General Purpose Packages**

* [eepackages](https://github.com/gee-community/ee-packages-py): A set of Google Earth Engine utilities by maintained by Gennadii Donchyts for both Javascript and Python API.
* [geetools](https://github.com/fitoprincipe/geetools-code-editor/wiki): Tools for cloud masking, batch processing, and more
* [ee-palettes](https://github.com/gee-community/ee-palettes): Module for generating color palettes
* [spectral](https://github.com/awesome-spectral-indices/spectral): A javascript module that provides ready-to-use curated list of spectral indices for GEE.
* [eemont](https://github.com/davemlz/eemont): Python package that provides utility methods to create a more fluid code by being friendly with the Python method chaining.

**Application Specific Packages**

* [LEAF-Toolbox](https://github.com/rfernand387/LEAF-Toolbox/wiki): Google Earth Engine application that produces various Vegetation Biophysical Products, including Leaf Area Index (LAI).
* [RivWidthCloud](https://github.com/seanyx/RivWidthCloudPaper): Package to automate extracting river centerline and width for both Javascript and Python API.


# Debugging Errors and Scaling Your Analysis

As you start implementing more complex workflows and analyze large regions - you are bound to run into scaling issues and errors such as below:

* *User memory-limit exceeded*
* *Computation timed-out*
* *Computed value is too large*


These are usually the result of inefficient code and structure of your script. Below are my recommendations for improving your coding style and utilizing the full power of the Earth Engine infrastructure. 

* If/else statements and for-loops should be avoided completely and replaced with filter/map/reduce. The former are sequential and will be slow. Refer to the [Function Programming Concepts](https://developers.google.com/earth-engine/tutorials/tutorial_js_03) guide on how to restructure your code to utilize the parallel computing infratructure provided by GEE.
* Remove any reprojection or resampling calls fro your script. Barring a few exceptions, you should not be reprojecting or resampling data. Earth Engine is designed to take care of it internally.
* For complex analysis involving large volumes of data, you should break down your workflow into logical steps and export intermediate results. For example, if you are implementing a complex supervised classification workflow - do it in multiple stages. A) Create your composite, add required bands, normalize them and export it as an Asset. B) Import the exported composite into another script and train a classifier. Export the classified image as an Asset. C) Import the exported classified image and do accuracy assessment and visualize the results. You will be able to run your analysis much faster, experiment easily and avoid scaling errors. Refer to the [Export Intermediate Result](https://developers.google.com/earth-engine/guides/best_practices?hl=en#export_intermediate_results) guide to learn more.
* You may also break your export region into smaller regions (typically by admin units) and Export each region separately. For example, if you want to export results for an entire country at 30m resolution - rather than a running a large Export - you can export each Admin1 unit in your country separately. This helps with computation timed-out errors and improves overall processing time. You can start as many exports from your account as you want and they will be executed in a queue. But do not split your exports over multiple Earth Engine accounts. This is a [violation of GEE Terms of Service](https://developers.google.com/earth-engine/batch-task-restrictions) and may result in your account getting blocked.

The Earth Engine User Guide also has tips and examples of best practices. You can review the following articles to learn them.

* [Coding Best Practices](https://developers.google.com/earth-engine/guides/best_practices?hl=en)
* [Debugging Guide](https://developers.google.com/earth-engine/guides/debugging?hl=en)

The following videos are highly recommended and contain good advice on debugging and scaling workflows. 

[![Earth Engine Scaling and Debugging](https://img.youtube.com/vi/3AtnpkZTvnk/mqdefault.jpg)](https://www.youtube.com/watch?v=3AtnpkZTvnk){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=3AtnpkZTvnk){target="_blank"}

[![Using Earth Engine for Very Large Computations](https://img.youtube.com/vi/fsez4HiOc8k/mqdefault.jpg)](https://www.youtube.com/watch?v=fsez4HiOc8k){target="_blank"}

[Watch the Video &#8599;](https://www.youtube.com/watch?v=fsez4HiOc8k){target="_blank"}


\newpage
# Data Credits

* **Sentinel-2 Level-1C, Level-2A** and **Sentinel-1 SAR GRD**: Contains Copernicus Sentinel data.
* **TerraClimate: Monthly Climate and Climatic Water Balance for Global Terrestrial Surfaces, University of Idaho**: Abatzoglou, J.T., S.Z. Dobrowski, S.A. Parks, K.C. Hegewisch, 2018, Terraclimate, a high-resolution global dataset of monthly climate and climatic water balance from 1958-2015, Scientific Data 5:170191, doi:10.1038/sdata.2017.191
* **VIIRS Stray Light Corrected Nighttime Day/Night Band Composites Version 1**: C. D. Elvidge, K. E. Baugh, M. Zhizhin, and F.-C. Hsu, “Why VIIRS data are superior to DMSP for mapping nighttime lights,” Asia-Pacific Advanced Network 35, vol. 35, p. 62, 2013.
* **FAO GAUL 500m: Global Administrative Unit Layers 2015, Second-Level Administrative Units**: Source of Administrative boundaries: The Global Administrative Unit Layers (GAUL) dataset,
implemented by FAO within the CountrySTAT and Agricultural Market Information System
(AMIS) projects.
* **CHIRPS Pentad: Climate Hazards Group InfraRed Precipitation with Station Data (version 2.0 final)**: Funk, Chris, Pete Peterson, Martin Landsfeld, Diego Pedreros, James Verdin, Shraddhanand Shukla, Gregory Husak, James Rowland, Laura Harrison, Andrew Hoell & Joel Michaelsen. "The climate hazards infrared precipitation with stations—a new environmental record for monitoring extremes". Scientific Data 2, 150066. doi:10.1038/sdata.2015.66 2015.
* **MOD13Q1.006 Terra Vegetation Indices 16-Day Global 250m**: Didan, K. (2015). <i>MOD13Q1 MODIS/Terra Vegetation Indices 16-Day L3 Global 250m SIN Grid V006</i> [Data set]. NASA EOSDIS Land Processes DAAC. Accessed 2021-05-06 from https://doi.org/10.5067/MODIS/MOD13Q1.006
* **GHS Urban Centre Database 2015  multitemporal and multidimensional attributes R2019A**: Florczyk A., Corbane C,. Schiavina M., Pesaresi M., Maffenini L., Melchiorri, M., Politis P., Sabo F., Freire S., Ehrlich D., Kemper T., Tommasi P., Airaghi D., Zanchetta L. (2019) European Commission, Joint Research Centre (JRC) [PID](https://data.jrc.ec.europa.eu/dataset/53473144-b88c-44bc-b4a3-4583ed1f547e) 
* **ESA WorldCover v100**: Zanaga, D., Van De Kerchove, R., De Keersmaecker, W., Souverijns, N., Brockmann, C., Quast, R., Wevers, J., Grosu, A., Paccini, A., Vergnaud, S., Cartus, O., Santoro, M., Fritz, S., Georgieva, I., Lesiv, M., Carter, S., Herold, M., Li, Linlin, Tsendbazar, N.E., Ramoino, F., Arino, O., 2021. ESA WorldCover 10 m 2020 v100. [doi:10.5281/zenodo.5571936](https://doi.org/10.5281/zenodo.5571936)



# References

## Composites

* Phan, T.N.; Kuch, V.; Lehnert, L.W. Land Cover Classification using Google Earth Engine and Random Forest Classifier—The Role of Image Composition. Remote Sens. 2020, 12, 2411. https://doi.org/10.3390/rs12152411
* D. Simonetti, U. Pimple, A. Langner, A. Marelli, Pan-tropical Sentinel-2 cloud-free annual composite datasets, Data in Brief, Volume 39, 2021, 107488, ISSN 2352-3409,https://doi.org/10.1016/j.dib.2021.107488.

## Supervised Classification

* Shetty, S.; Gupta, P.K.; Belgiu, M.; Srivastav, S.K. Assessing the Effect of Training Sampling Design on the Performance of Machine Learning Classifiers for Land Cover Mapping Using Multi-Temporal Remote Sensing Data and Google Earth Engine. Remote Sens. 2021, 13, 1433. https://doi.org/10.3390/rs13081433
* Kelley, L.C.; Pitcher, L.; Bacon, C. Using Google Earth Engine to Map Complex Shade-Grown Coffee Landscapes in Northern Nicaragua. Remote Sens. 2018, 10, 952. https://doi.org/10.3390/rs10060952
* Arsalan Ghorbanian, Mohammad Kakooei, Meisam Amani, Sahel Mahdavi, Ali Mohammadzadeh, Mahdi Hasanlou, Improved land cover map of Iran using Sentinel imagery within Google Earth Engine and a novel automatic workflow for land cover classification using migrated training samples, ISPRS Journal of Photogrammetry and Remote Sensing, Volume 167, 2020, https://doi.org/10.1016/j.isprsjprs.2020.07.013
* Tassi, A.; Vizzari, M. Object-Oriented LULC Classification in Google Earth Engine Combining SNIC, GLCM, and Machine Learning Algorithms. Remote Sens. 2020, 12, 3776. https://doi.org/10.3390/rs12223776
* Cristina Gómez, Joanne C. White, Michael A. Wulder, Optical remotely sensed time series data for land cover classification: A review, ISPRS Journal of Photogrammetry and Remote Sensing, Volume 116, 2016, Pages 55-72, ISSN 0924-2716, https://doi.org/10.1016/j.isprsjprs.2016.03.008
* Sherrie Wang, George Azzari, David B. Lobell, Crop type mapping without field-level labels: Random forest transfer and unsupervised clustering techniques, Remote Sensing of Environment, Volume 222, 2019, Pages 303-317, ISSN 0034-4257, https://doi.org/10.1016/j.rse.2018.12.026

# License

The course material (text, images, presentation, videos) is licensed under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).

The code (scripts, Jupyter notebooks) is licensed under the MIT License. For a copy, see https://opensource.org/licenses/MIT

Kindly give appropriate credit to the original author as below:

Copyright &copy; 2022 Ujaval Gandhi [www.spatialthoughts.com](https://spatialthoughts.com)


# Citing and Referencing

You can cite the course materials as follows

* Gandhi, Ujaval, 2021. *End-to-End Google Earth Engine* Course. Spatial Thoughts. https://courses.spatialthoughts.com/end-to-end-gee.html

***

**This course is offered as an instructor-led online class. Visit [Spatial Thoughts](https://spatialthoughts.com/events/) to know details of upcoming sessions.**