python-remote-sensing.Rmd

---
title: "Cloud Native Remote Sensing with Python (Full Course)"
author: "Ujaval Gandhi"
subtitle: A structured introduction to XArray, DuckDB, STAC and Dask for cloud-based remote sensing applications.
output:
  # pdf_document:
  #   toc: yes
  #   toc_depth: '3'
  #   latex_engine: xelatex 
  html_document:
    df_print: paged
    highlight: pygments
    toc: yes
    toc_depth: 2
    includes:
      in_header: 
        - copy-code.html
        - GA_Script.html
      after_body: comment.html
fontsize: 12pt
header-includes:
- \usepackage{fancyhdr}
- \pagestyle{fancy}
- \renewcommand{\footrulewidth}{0.4pt}
- \fancyhead[LE,RO]{\thepage}
- \geometry{left=1in,top=0.75in,bottom=0.75in}
- \fancyfoot[CE,CO]{{\includegraphics[height=0.5cm]{images/cc-by-nc.png}} Ujaval Gandhi
  http://www.spatialthoughts.com}
classoption: a4paper
---
\newpage

***

```{r echo=FALSE, fig.align='center', out.width='75%', out.width='250pt'}
knitr::include_graphics('images/spatial_thoughts_logo.png')
```

***

\newpage

# Introduction 

This is an intermediate-level course that covers tools and techniques for working with climate and earth observation datasets using modern cloud-native approach. With the growing ecosystem of cloud native data formats, open data catalogs and powerful open-source packages - remote sensing practitioners are now able to adopt open and vendor agnostic cloud-based data processing workflows. This class will cover how to implement these workflows using Python-based tooling with hands-on examples. 


[![View Presentation](images/python_remote_sensing/introduction.png){width="400px"}](https://docs.google.com/presentation/d/1YhT8OdrOm0JkkoTJ-eyQC89V2q6QiFXpA_57QW6caec/edit?usp=sharing){target="_blank"}

[View the Presentation ↗](https://docs.google.com/presentation/d/1YhT8OdrOm0JkkoTJ-eyQC89V2q6QiFXpA_57QW6caec/edit?usp=sharing){target="_blank"}


# Installation and Setting up the Environment 

All the notebooks in this course are structured so they can be run in any Jupyter-based notebook environment. We will run them in the cloud using Google Colab and on your own machine using Jupyter Lab. 

## Cloud Notebook Environment

We will be using Google Colab as the main cloud-based environment for executing the notebooks in this course.
[Google Colab](https://research.google.com/colaboratory/faq.html) provides a cloud-hosted Jupyter notebook environment. 

**This does not require any setup and can be used with a Google account.**

The notebooks in this course can be accessed by clicking on the ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg) buttons at the beginning of each section.

## Local Development Environment

To run the notebooks on your own machine, we first create a new conda environment and install the required packages. Then using Jupyter Lab, you can execute the notebook.

### Install Conda

Follow our step-by-step [Conda Installation Guide](install-conda.html) to install Miniconda for your operating system.

### Create an Environment and Install Packages

We will use conda to install the required Python packages and manage local development environment. 

1. *(Windows users)*, search for Anaconda Powershell Prompt and launch it. *(Mac/Linux users)*: Launch a Terminal window. Run the following commands to create a fresh environment and activate it.

```{bash eval=FALSE}
conda create --name python_remote_sensing -y
conda activate python_remote_sensing
```

2. Now your environment is ready. We will install the required packages from `conda-forge`. Copy/paste the platform-appropriate code from below.

*Windows Users*

```{bash eval=FALSE}
conda install -c conda-forge -y `
  botocore `
  bottleneck `
  coiled `
  dask `
  distributed `
  duckdb `
  earthengine-api `
  exactextract `
  geopandas `
  jupyterlab `
  jupyter-server-proxy `
  lonboard `
  matplotlib `
  netcdf4 `
  numpy `
  odc-algo `
  odc-stac `
  openpyxl `
  pandas `
  planetary-computer `
  pyproj `
  pystac-client `
  python-graphviz `
  rioxarray `
  scikit-learn `
  xarray `
  xarray-spatial `
  xee `
  xvec
```

*Mac/Linux Users*

```{bash eval=FALSE}
conda install -c conda-forge -y \
  botocore \
  bottleneck \
  coiled \
  dask \
  distributed \
  duckdb \
  earthengine-api \
  exactextract \
  geopandas \
  jupyterlab \
  jupyter-server-proxy \
  lonboard \
  matplotlib \
  netcdf4 \
  numpy \
  odc-algo \
  odc-stac \
  openpyxl \
  pandas \
  planetary-computer \
  pyproj \
  pystac-client \
  python-graphviz \
  rioxarray \
  scikit-learn \
  xarray \
  xarray-spatial \
  xee \
  xvec
```

3. Some packages are not available on conda-forge, so we install them from PyPI using `pip`.

*Windows/Mac/Linux Users*

```{bash eval=FALSE}
conda install -c conda-forge -y pip
pip install aef-loader xrscipy 
```

Your local development environment is now ready.

# Development Tools

## Introduction to Google Colab

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/00_hello_colab.ipynb)


```{r child='python-remote-sensing-output/00_hello_colab.md'}
```

## Introduction to GeoLibre

[GeoLibre](https://geolibre.app/) is an open-source cloud-native GIS platform that supports loading and visualizing a wide-variety of cloud-native geospatial data formats. It is well suited for cloud-native data work - exploring large datasets, collecting training samples, interactively visualizing outputs and more. 

We will be using the [GeoLibre Viewer](https://viewer.geolibre.app/) in this course. Let's take a quick tour to get familiar with the interface and the capabilities.

1. Visit https://viewer.geolibre.app/ to open the GeoLibre Viewer. Select the *Advanced* interface.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre1.png')
```

2. The viewer is a static web app that runs on your browser.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre2.png')
```

3. GeoLibre can load data from your local machine as well as cloud locations. Let's load a vector layer. Go to **Add Data → Vector Layer**.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre3.png')
```

4. In the *Add Vector Data* panel, click the *Drop file or click to browse* button. Locate the `capitals.gpkg` we downloaded in the previous section and click *Open*. A new layer `capitals` will be added to the viewer. Close the panel.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre4.png')
```

5. Let's add a raster layer next. Go to **Add Data → Raster Layer**. We will load a large Cloud Optimized GeoTiff (COG) of the VIIRS Nighttiem Lights dataset hosted in a cloud bucket. Paste the following URL and click *Load*.

```
https://storage.googleapis.com/spatialthoughts-public-data/ntl/viirs/viirs_ntl_2021_global.tif
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre5.png')
```

6. A new layer `viirs_ntl_2021_global.tif` will be added. You can change the default colormap to any other colormap of your choice and adjust the visualization settings. You can also insepct the pixel values by first enabling the *Inspect* button and then clicking on the map. When you are done, close the panel.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre6.png')
```

7. GeoLibre comes with many plugins that extend its core functionality. We will add a basemap layer next. Go to **Plugins → Basemaps → Activate**.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre7.png')
```

8. Select the **Google Satellite** basemap and it will be added to the viewer. Adjust the *Opacity* slider to see the layer below.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/geolibre8.png')
```

In this course, we will also use GeoLibre for collecting samples for supervised classification. Detailed workflow for data creation is explained in Module 4.

## AI Coding Agents

### Google Gemini in Colab

We recommend using the built-in Gemini integration in Google Colab for writing, modifying and updating code in the provided notebooks.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gemini_colab.png')
```

### Claude Code

We have provided a Claude Code skill [cloud-native-remote-sensing](https://github.com/spatialthoughts/spatialthoughts-ai-skills/) that captures the best-practices and workflows taught in this course. This approach is recommended when creating new notebooks or working on more complex updates. See the repository [README.md](https://github.com/spatialthoughts/spatialthoughts-ai-skills/blob/main/README.md) for instructions on how to install and use this skill.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/claude_skill.png')
```


# Module 1: Cloud Native Geospatial Fundamentals

[![View Presentation](images/python_remote_sensing/cng_basics.png){width="400px"}](https://docs.google.com/presentation/d/1FPQ4ZQVqTXW5hQ2FVZG9knwZNDnR3aUoeJvpUOoRC6Y/edit?usp=sharing){target="_blank"}

[View the Presentation ↗](https://docs.google.com/presentation/d/1FPQ4ZQVqTXW5hQ2FVZG9knwZNDnR3aUoeJvpUOoRC6Y/edit?usp=sharing){target="_blank"}


## 1.1 XArray Basics

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_01/01_xarray_basics.ipynb)


```{r child='python-remote-sensing-output/module_01/01_xarray_basics.md'}
```

## 1.2 STAC and Dask Basics

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_01/02_stac_dask_basics.ipynb)


```{r child='python-remote-sensing-output/module_01/02_stac_dask_basics.md'}
```

## 1.3 DuckDB Basics

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_01/03_duckdb_basics.ipynb)


```{r child='python-remote-sensing-output/module_01/03_duckdb_basics.md'}
```

## 1.4 Creating a Median Composite

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_01/04_median_composite.ipynb)


```{r child='python-remote-sensing-output/module_01/04_median_composite.md'}
```

## Assignment 1

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_01/assignment1.ipynb)


```{r child='python-remote-sensing-output/module_01/assignment1.md'}
```


# Module 2: Remote Sensing Fundamentals


[![View Presentation](images/python_remote_sensing/remote_sensing.png){width="400px"}](https://docs.google.com/presentation/d/1_LlMfxZ54QESEb0iKpclYWLqruzGXVW_d3w9q5epenA/edit?usp=sharing){target="_blank"}

[View the Presentation ↗](https://docs.google.com/presentation/d/1_LlMfxZ54QESEb0iKpclYWLqruzGXVW_d3w9q5epenA/edit?usp=sharing){target="_blank"}


## 2.1 Calculating Spectral Indices

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_02/01_calculating_indices.ipynb)


```{r child='python-remote-sensing-output/module_02/01_calculating_indices.md'}
```

## 2.2 Masking Clouds

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_02/02_masking_clouds.ipynb)


```{r child='python-remote-sensing-output/module_02/02_masking_clouds.md'}
```

## 2.3 Extracting and Processing Time-Series

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_02/03_extracting_time_series.ipynb)


```{r child='python-remote-sensing-output/module_02/03_extracting_time_series.md'}
```

## Assignment 2

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_02/assignment2.ipynb)


```{r child='python-remote-sensing-output/module_02/assignment2.md'}
```


# Module 3: Computation and Data Processing

[![View Presentation](images/python_remote_sensing/computation.png){width="400px"}](https://docs.google.com/presentation/d/1rC3wh7hqZzWRmb5AP7hlteCJf3KbS5fL7csndGE7bNo/edit?usp=sharing){target="_blank"}

[View the Presentation ↗](https://docs.google.com/presentation/d/1rC3wh7hqZzWRmb5AP7hlteCJf3KbS5fL7csndGE7bNo/edit?usp=sharing){target="_blank"}


## 3.1 Working with Landcover Data

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_03/01_working_with_landcover.ipynb)


```{r child='python-remote-sensing-output/module_03/01_working_with_landcover.md'}
```

## 3.2 Analyzing Landcover Change

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_03/02_analyzing_landcover_change.ipynb)


```{r child='python-remote-sensing-output/module_03/02_analyzing_landcover_change.md'}
```


## 3.3 Computing Zonal Statistics

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_03/03_zonal_statistics.ipynb)


```{r child='python-remote-sensing-output/module_03/03_zonal_statistics.md'}
```

## 3.4 Interoperability with Google Earth Engine

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_03/04_working_with_xee.ipynb)


```{r child='python-remote-sensing-output/module_03/04_working_with_xee.md'}
```

# Module 4: Machine Learning and AI

[![View Presentation](images/python_remote_sensing/machine_learning.png){width="400px"}](https://docs.google.com/presentation/d/1AHlQq75wm3G5H-KRDhnaAdjtfmVW1Gbm3tBQLCBx2Uk/edit?usp=sharing){target="_blank"}

[View the Presentation ↗](https://docs.google.com/presentation/d/1AHlQq75wm3G5H-KRDhnaAdjtfmVW1Gbm3tBQLCBx2Uk/edit?usp=sharing){target="_blank"}


## 4.1 Preparing Data for Machine Learning

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_04/01_preparing_composites.ipynb)


```{r child='python-remote-sensing-output/module_04/01_preparing_composites.md'}
```

## 4.2 Unsupervised Classification

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_04/02_unsupervised_classification.ipynb)


```{r child='python-remote-sensing-output/module_04/02_unsupervised_classification.md'}
```

## 4.3 Collecting Training Samples

For training a supervised classification model, we need to collect training samples with labels. We will be using visual interpretation of the Sentinel-2 composite image created in the [previous section](#preparing-data-for-machine-learning) to mark locations and label them with appropriate landcover class.

<table style="border-collapse: collapse;">
<tr><th style="border: 1px solid black; padding: 6px;">Class</th><th style="border: 1px solid black; padding: 6px;">Description</th><th style="border: 1px solid black; padding: 6px;">Value</th><th style="border: 1px solid black; padding: 6px;">Color</th></tr>
<tr><td style="border: 1px solid black; padding: 6px;">urban</td><td style="border: 1px solid black; padding: 6px;">All built surfaces - buildings, bridges, roads etc.</td><td style="border: 1px solid black; padding: 6px;">0</td><td style="border: 1px solid black; padding: 6px;">#cc6d8f</td></tr>
<tr><td style="border: 1px solid black; padding: 6px;">bare</td><td style="border: 1px solid black; padding: 6px;">All bare surfaces - exposed soil, sand, rock etc.</td><td style="border: 1px solid black; padding: 6px;">1</td><td style="border: 1px solid black; padding: 6px;">#ffc107</td></tr>
<tr><td style="border: 1px solid black; padding: 6px;">water</td><td style="border: 1px solid black; padding: 6px;">All surface water - lake, ponds, rivers, ocean etc.</td><td style="border: 1px solid black; padding: 6px;">2</td><td style="border: 1px solid black; padding: 6px;">#1e88e5</td></tr>
<tr><td style="border: 1px solid black; padding: 6px;">vegetation</td><td style="border: 1px solid black; padding: 6px;">All types of vegetation - trees, crops, grass etc.</td><td style="border: 1px solid black; padding: 6px;">3</td><td style="border: 1px solid black; padding: 6px;">#004d40</td></tr>
</table>

<br>

We will use the [GeoLibre Web](https://web.geolibre.app/) for training data collection.

1. Open [GeoLibre Web](https://web.geolibre.app/). Go to **Add Data &rarr; Raster Layer**.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gcp1.png')
```

2. Browse to the downloaded `multiband_composite.tif` created in the previous section and select *Open*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gcp2.png')
```

3. This will be the reference image for the sample collection. Next go to **Controls &rarr; Field Collection...**.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gcp3.png')
```

4. Enter the *Layer name* as `gcps`. Add a field named `landcover` and select `Number` type. Check the `Required` box and click *Create layer*.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/gcp4.png')
```

5. Now the layer is ready. Click on the *Pick on map* button. Zoom to a location on the map and click on a point you want to add.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/gcp5.png')
```

6. Enter the *landcover* value for the location and click *Save point*.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/gcp6.png')
```

7. Collect approximately 5-10 samples for each class - ensuring good spatial distribution across your region of interest. If you need to edit or delete a point, you can use the *GeoEditor* plugin. Go to **Plugins &rarr; GeoEditor &rarr; Activate**.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gcp7.png')
```

8. Toggle the editing mode by clicking on the layer menu for the `gcps` layer and selecting *Edit geometry*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gcp8.png')
```

9. Select the *Delete* button from the GeoEditor and click on the point you want to delete. Similarly, you may use the *Drag* button to move the point. Click *Save* once you are done editing.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gcp9.png')
```

10. Once you are done with sample collection, select on the layer menu for the `gcps` layer and go to **Export &rarr; GeoJSON**. Save the resulting file as `gcps.geojson`. 

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/gcp10.png')
```

Upload this file to your Google Drive in the data folder so we can use it in our next notebook.


## 4.4 Supervised Classification

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_04/03_supervised_classification.ipynb)

```{r child='python-remote-sensing-output/module_04/03_supervised_classification.md'}
```

## 4.5 Supervised Classification with Embeddings

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_04/04_supervised_classification_embeddings.ipynb)

```{r child='python-remote-sensing-output/module_04/04_supervised_classification_embeddings.md'}
```


# Module 5: Computation Environments

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_05/cloudfree_s2_composite.ipynb)

```{r child='python-remote-sensing-output/module_05/cloudfree_s2_composite.md'}
```

## 5.1 Running Computation on Your Hardware

Please following our [Installation and Setting up the Environment](#installation-and-setting-up-the-environment) instructions to install and configure your system. Once you have a working conda environment, you can use it to run the notebook.

### Run Your Notebook.

1. Open the notebook in Google Colab. Go to **File &rarr; Download &rarr; Download .ipynb** and download the notebook to your computer.

2. *(Windows users)*, search for Windows Powershell and launch it. *(Mac/Linux users)*: Launch a Terminal window. Run the following commands to activate your conda environment.

```{bash eval=FALSE}
conda activate python_remote_sensing
```

3. Launch Jupyter Lab.

```{bash eval=FALSE}
jupyter-lab
```

4. Locate the notebook in Jupyter Lab and click *Run*.

## 5.2 Using Google Cloud Runtime

Google Colab offers a seamless way to setup a VM in Google Cloud with a Colab Runtime and use it for executing your Colab Notebooks. This section requires a Google Cloud account. Follow our [Google Cloud Sign-up Guide](google-cloud-sign-up.html) to setup your cloud project.

### Connect to a Cloud Runtime


1. In Google Colab, click the  *Additional connection options* dropdown next to the *Connect* button and select **Connect to Google Cloud Runtime**.

```{r echo=FALSE, fig.align='center', out.width='60%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime1.png')
```

2. Select a project in the *Google Cloud project*. If you do not have any VMs available, click *Manage runtimes*.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime2.png')
```

3. This will open the *Colab Enterprise* section in Google Cloud Console. You will be prompted to enable the APIs required for this service. Click on *Enable*.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime3.png')
```

4. Once the APIs are enabled, click on *Runtime templates*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime4.png')
```

5. In the *Runtime templates* tab, click on *+ New Template*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime5.png')
```

6. Enter a name for your template and click *Configure compute*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime6.png')
```

7. Select the type of machine you need. Higher number of CPU cores will allow more parallel computing and faster processing. Once selected, click *Create*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime7.png')
```

8. Once the template is created, click on the overflow menu *:* and select *Create runtime*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime8.png')
```

9. Confirm the details and click *Create*.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime9.png')
```

10. It will take a few minutes for the new runtime to be created. Wait till the *Status* shows **Healthy**.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime10.png')
```

11. Switch back to Google Colab and click the *Refresh* button. Your newly created VM should be listed. Select it and click *Connect*.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/colab_cloudruntime11.png')
```

### Run Your Notebook

Run your notebook as usual. Your analysis will run on the cloud VM.

### Delete the Runtime

If you do not delete the VM after you are done with the analysis, you will incur recurring charges for the compute and storage. It is advisable to delete any Runtimes that you are not using actively.

Once you are done with the analysis, go back to the *Colab Enterprise* section and select *Runtimes*. Click the *:* menu for any Runtime that are running and select *Delete*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/colab_deleteruntime1.png')
```

## 5.3 Scaling Analysis in Cloud with Coiled

[Coiled](https://coiled.io/) is a platform that allows you to easily setup cloud infrastructure for distributed processing. It provides a Python-package and a Notebook service that makes it very easy to scale your Xarray + Dask workflows to process large volumes of data. In this section, you will learn how to setup coiled and run a notebook in a cloud-hosted machine for creating a median cloud-free annual composite from Sentinel-2 imagery.

### Installation and Setting up the Environment

You need to install the `coiled` along with other required packages in your local Python environment and configure your account. Visit the [Coiled Documentation](https://docs.coiled.io/user_guide/setup/index.html) for detailed setup instructions. 

#### Create an Account

[Sign-up](https://cloud.coiled.io/signup) for a free coiled account.

#### Login to Coiled

We will use our existing conda environment to run the coiled notebook. Activate the environment and login to coiled.

```
conda activate python_remote_sensing
coiled login
```

#### Connect your Cloud Account

Next, you will need to configure it with your own cloud account. All popular cloud services (GCP, AWS, Azure) are supported. Below is the command required to configure it with your GCP account.

```
coiled setup gcp
```

Once setup, you can visit the [Coiled Dashboard](https://cloud.coiled.io/) to verify the setup.

#### Setup Mutagen (Optional)

A great feature of the coiled notebook service is the ability to sync your local Python environment and files with the cloud machine. This requires installing the [Mutagen](https://docs.coiled.io/user_guide/notebooks.html#notebook-file-sync) utility. On MacOS and Linux, this is straightforward using Homebrew.

```
brew install mutagen-io/mutagen/mutagen
```

#### Start a Notebook on a Cloud Machine

1. Change your local directory to a folder where you have your code.

```
cd Desktop/coiled
```

2. Start a notebook on a cloud machine. The default machine you get on GCP is the `e2-standard-4` VM with 4 vCPU and 16 GB memory. This is a decent and cheap option for small to medium sized workloads. You can always get bigger machine with more memory of more vCPUs. See how to specify [VM Size and Type](https://docs.coiled.io/user_guide/clusters/size-and-type.html).

```
coiled notebook start --sync
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/coiled_notebook_start.png')
```

### Run Your Notebook

1. Download the example notebook to your machine. Visit [coiled_s2_composite.ipynb](https://github.com/spatialthoughts/courses/blob/master/code/python_remote_sensing/module_05/cloudfree_s2_composite.ipynb) and click the *Download raw file* button. Copy the file to your preferred directory where you started the coiled notebook. The file will appear in the Jupyter Lab instance that was launched by Coiled. Double-click to open the notebook.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/coiled_notebook_view.png')
```

2. Run the notebook to start the data processing. The notebook is running on a cloud machine and you can see the progress on the Dask Dashboard.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/coiled_notebook_run.png')
```

3. Once the processing finishes, the resulting composite `rgb_composite_2024.tif` will be available in the `output` folder which will be automatically synced to your machine.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/coiled_output.png')
```

4. The resulting Sentinel-2 RGB composite is now available on your machine and can be viewed using QGIS.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_remote_sensing/coiled_composite.png')
```

5. Once you are done with processing, remember to stop the notebook server. This will stop the cloud instance.

> If you forget to stop the server, you will continue getting charged for the running intance in the cloud. You can always visit the [Coiled Dashboard](https://cloud.coiled.io/) to verify that there are no running clusters.

```{r echo=FALSE, fig.align='center', out.width='50%'}
knitr::include_graphics('images/python_remote_sensing/coiled_notebook_stop.png')
```

# Supplement

## Using Planetary Computer Data Catalog

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/supplement/pc_extracting_time_series.ipynb)

```{r child='python-remote-sensing-output/supplement/pc_extracting_time_series.md'}
```

## Downloading Sentinel-2 Cloud Free Mosaics

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/supplement/s2_cloudfree_mosaics.ipynb)

```{r child='python-remote-sensing-output/supplement/s2_cloudfree_mosaics.md'}
```

## Extracting Building Heights from GlobalBuildingAtlas

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/supplement/extracting_building_heights.ipynb)

```{r child='python-remote-sensing-output/supplement/extracting_building_heights.md'}
```

## Working with GLC-FCS30 LandCover Data

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/supplement/glc_fcs30d_landcover.ipynb)

```{r child='python-remote-sensing-output/supplement/glc_fcs30d_landcover.md'}
```


## Calculating Zonal Stats for Landcover Area

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/spatialthoughts/courses/blob/master/code/python_remote_sensing/supplement/s2_cloudfree_mosaics.ipynb)

```{r child='python-remote-sensing-output/supplement/zonal_histogram.md'}
```


----

# Learning Resources

## Tutorials

* [EOPF Xarray Notebooks](https://eopf-toolkit.github.io/eopf-101/06_eopf_zarr_in_action/64_flood_mapping_valencia.html): Case studies and applications of EOPF Sentinel Zarr products - including flood mapping, vegetation monitoring, fire mapping and more.
* [Copernicus Data Space Ecosystem Notebooks](https://github.com/eu-cdse/notebook-samples/tree/main): Notebooks showing how to use access the data from the [CDSE STAC Catalog](https://documentation.dataspace.copernicus.eu/APIs/STAC.html) having data from all Sentinel missions as well as landcover data.
* [Earth Data Hub Tutorials](https://github.com/bopen/edh-learning/): Jupyter notebooks showing how to access and process climate and Earth observation Zarr data (ERA5, Sentinel-1 ARD, CMIP6, Copernicus DEM) via Python and Xarray.
* [Building an EVI Timeseries with ODC STAC](https://github.com/nasa/HLS-Data-Resources/blob/main/python/tutorials/EVI_timeseries_with_odc_stac.ipynb): A tutorial showing how to access Harmonized Landsat Sentinel-2 (HLS) data from NASA CMR-STAC and earthaccess.
* [Project Pythia Cookbooks](https://cookbooks.projectpythia.org/): A community learning resource for Python-based computing in the geosciences. Includes examples of remote sensing data such as [Landsat ML Cookbook](https://projectpythia.org/landsat-ml-cookbook/).
* [Exploring HLS data](https://github.com/auspatious/hls-experiments/tree/main): Loading and processing Harmonized Landsat and Sentinel-2 data with advanced cloud-masking using morphological operations.


# Data Credits

* Sentinel-2 Level 2A Scenes: Contains modified Copernicus Sentinel data (2025-02)
* Bangalore Ward Maps Provided by Spatial Data of Municipalities (Maps) Project by Data{Meet}.
* Admin boundaries: FieldMaps, geoBoundaries, U.S. Department of State, OpenStreetMap
* TerraClimate: Monthly Climate and Climatic Water Balance for Global Terrestrial Surfaces, University of Idaho: Abatzoglou, J.T., S.Z. Dobrowski, S.A. Parks, K.C. Hegewisch, 2018, Terraclimate, a high-resolution global dataset of monthly climate and climatic water balance from 1958-2015, Scientific Data 5:170191, doi:10.1038/sdata.2017.191
* Climate Hazards Center (2025). Climate Hazards Center Infrared Precipitation with Stations (CHIRPS) Data Repository. https://doi.org/10.15780/G2JQ0P. Accessed 2026-05-01.
* Zhu, X. X., Chen, S., Zhang, F., Shi, Y., and Wang, Y.: GlobalBuildingAtlas: an open global and complete dataset of building polygons, heights and LoD1 3D models, Earth Syst. Sci. Data, 17, 6647–6668, https://doi.org/10.5194/essd-17-6647-2025, 2025.
* Chen, J., Jönsson, P., Tamura, M., Gu, Z., Matsushita, B., & Eklundh, L. (2004). A simple method for reconstructing a high-quality NDVI time-series data set based on the Savitzky–Golay filter. Remote Sensing of Environment, 91(3–4), 332–344. https://doi.org/10.1016/j.rse.2004.03.014
* Potapov P, Hansen MC, Pickens A, Hernandez-Serna A, Tyukavina A, Turubanova S, Zalles V, Li X, Khan A, Stolle F, Harris N, Song X-P, Baggett A, Kommareddy I and Kommareddy A (2022) The Global 2000-2020 Land Cover and Land Use Change Dataset Derived From the Landsat Archive: First Results. Front. Remote Sens. 3:856903. doi: 10.3389/frsen.2022.856903
* Karra, Kontgis, et al. "Global land use/land cover with Sentinel-2 and deep learning." IGARSS 2021-2021 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2021.

# References

* Cordeiro, M. C. R.; Martinez, J.-M.; Peña-Luque, S. Automatic Water Detection from Multidimensional Hierarchical Clustering for Sentinel-2 Images and a Comparison with Level 2A Processors. *Remote Sensing of Environment* 2021, 253, 112209.

# License

This course material is licensed under a [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). You are free to re-use and adapt the material but are required to give appropriate credit to the original author as below:

*Cloud Native Remote Sensing with Python* course by Ujaval Gandhi [www.spatialthoughts.com](https://spatialthoughts.com)


***

&copy; 2025 Spatial Thoughts [www.spatialthoughts.com](https://spatialthoughts.com)