{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Application Package reproducibility\n", "\n", "### Personas\n", "\n", "* **Alice** developed a Water Body detection Earth Observation application and package it as an EO Application Package\n", "* **Bob** scripts the execution of application\n", "\n", "### Scenario\n", "\n", "Alice included in the water bodies detection Application Package software repository a Continuous Integration configuration relying on Github Actions to:\n", "\n", "* build the containers\n", "* push the built containers to Github container registry\n", "* update the Application Package with these new container references\n", "* push the updated Application Package to Github's artifact registry\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alice sent an email to Bob:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "from: alice@acme.io\n", "\n", "to: bob@acme.io\n", "\n", "subject: Detecting water bodies with NDWI and the Otsu threshold\n", "\n", "\n", "Hi Bob!\n", "\n", "checkout my new application package for detecting water bodies using NDWI and the Ostu threshold.\n", "\n", "I've ran it over our test site bounding box and preliminary result look promising.\n", "\n", "The github repo is https://github.com/eoap/quickwin and I've just released version 1.0.0.\n", "\n", "Let me know!\n", "\n", "Cheers\n", "\n", "Alice\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With this information, Bob scripts the Application Execution in a Jupyter Notebook.\n", "\n", "His environment has a container engine (e.g. podman or docker) and the cwltool CWL runner." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running the Scenario" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "ename": "ImportError", "evalue": "cannot import name 'ModelField' from 'pydantic.fields' (/Users/simonevaccari/Documents/repos/eoap/open-reproducible-app-package/env_reproducible_app/lib/python3.12/site-packages/pydantic/fields.py)", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[1], line 17\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpystac_client\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Client\n\u001b[1;32m 15\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mrasterio\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mfeatures\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m dataset_features, sieve\n\u001b[0;32m---> 17\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mhelpers\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Params, get_param_model_fields, get_release_assets, stage_in\n\u001b[1;32m 19\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mshutil\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m which\n\u001b[1;32m 21\u001b[0m nest_asyncio\u001b[38;5;241m.\u001b[39mapply()\n", "File \u001b[0;32m~/Documents/repos/eoap/open-reproducible-app-package/helpers.py:6\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Any, Dict, Optional, List\n\u001b[1;32m 5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpydantic\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m BaseModel, DirectoryPath\n\u001b[0;32m----> 6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpydantic\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mfields\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ModelField, FieldInfo\n\u001b[1;32m 7\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mjson\u001b[39;00m\n\u001b[1;32m 8\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mrequests\u001b[39;00m\n", "\u001b[0;31mImportError\u001b[0m: cannot import name 'ModelField' from 'pydantic.fields' (/Users/simonevaccari/Documents/repos/eoap/open-reproducible-app-package/env_reproducible_app/lib/python3.12/site-packages/pydantic/fields.py)" ] } ], "source": [ "import argparse\n", "import asyncio\n", "import json\n", "import os\n", "from datetime import datetime\n", "from io import StringIO\n", "\n", "import nest_asyncio\n", "import pystac\n", "import rasterio\n", "from cwltool.main import main\n", "from ipyleaflet import GeoJSON, Map\n", "from pydantic_yaml import to_yaml_str\n", "from pystac_client import Client\n", "from rasterio.features import dataset_features, sieve\n", "\n", "from helpers import Params, get_param_model_fields, get_release_assets, stage_in\n", "\n", "from shutil import which\n", "\n", "nest_asyncio.apply()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Check the container engine" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "if which(\"podman\"):\n", " podman = True\n", "elif which(\"docker\"):\n", " podman = False\n", "else:\n", " raise ValueError(\"No container engine\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Application Package releases\n", "\n", "Bob uses Github API to list the artifacts published by Alice in the release" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'1.0.0': [{'url': 'https://github.com/Terradue/app-package-training-bids23/releases/download/1.0.0/app-water-bodies-cloud-native.1.0.0.cwl',\n", " 'cwl': ,\n", " 'label': 'Water bodies detection based on NDWI and otsu threshold',\n", " 'doc': 'Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 COG STAC items'},\n", " {'url': 'https://github.com/Terradue/app-package-training-bids23/releases/download/1.0.0/app-water-body-cloud-native.1.0.0.cwl',\n", " 'cwl': ,\n", " 'label': 'Water bodies detection based on NDWI and the otsu threshold',\n", " 'doc': 'Water bodies detection based on NDWI and otsu threshold applied to a single Sentinel-2 COG STAC item'},\n", " {'url': 'https://github.com/Terradue/app-package-training-bids23/releases/download/1.0.0/app-water-body.1.0.0.cwl',\n", " 'cwl': ,\n", " 'label': 'Water body detection based on NDWI and the otsu threshold',\n", " 'doc': 'Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 or Landsat-9 staged acquisitions'}]}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "assets = get_release_assets(\n", " user=\"eoap\",\n", " repo=\"quickwin\",\n", " token=os.environ[\"GH_PAT\"],\n", ")\n", "\n", "assets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running the Application Package to detect water bodies on Sentinel-2 data\n", "\n", "Alice published three Application Packages.\n", "\n", " Bob selects the one processing several Sentinel-2 acquisitions provided as STAC Items\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 COG STAC items\n", "https://github.com/Terradue/app-package-training-bids23/releases/download/1.0.0/app-water-bodies-cloud-native.1.0.0.cwl\n" ] } ], "source": [ "app_package = assets[\"1.0.0\"][0]\n", "\n", "print(app_package[\"doc\"])\n", "\n", "print(app_package[\"url\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Application Package parameters are discovered and a pydantic model is created" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'aoi': ModelField(name='aoi', type=str, required=True),\n", " 'bands': ModelField(name='bands', type=List[str], required=False, default=['green', 'nir']),\n", " 'epsg': ModelField(name='epsg', type=str, required=False, default='EPSG:4326'),\n", " 'stac_items': ModelField(name='stac_items', type=List[str], required=True)}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Params.set_fields(**get_param_model_fields(cwl_obj=app_package[\"cwl\"]))\n", "\n", "Params.get_fields()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Application Package takes as inputs:\n", "- one or more STAC Items\n", "- a list of the bands for the normalized difference\n", "- an area of interest\n", "- the EPSG code used for the area of interest coordinates\n", "\n", " Bob uses a STAC API endpoint to discover Sentinel-2 acquisitions over an area of interest and time of interest " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", "
    \n", " \n", " \n", " \n", "
  • \n", " type\n", " \"Catalog\"\n", "
    • \n", " \n", " \n", " \n", " \n", " \n", "
  • \n", " id\n", " \"earth-search-aws\"\n", "
    • \n", " \n", " \n", " \n", " \n", " \n", "
  • \n", " stac_version\n", " \"1.0.0\"\n", "
    • \n", " \n", " \n", " \n", " \n", " \n", "
  • \n", " description\n", " \"A STAC API of public datasets on AWS\"\n", "
    • \n", " \n", " \n", " \n", " \n", "
  • \n", " \n", " links\n", " [] 19 items\n", " \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 0\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"self\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 1\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"root\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 2\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"conformance\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/conformance\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 3\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"data\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 4\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"search\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/search\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " method\n", " \"GET\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 5\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"search\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/search\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " method\n", " \"POST\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 6\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"aggregate\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/aggregate\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " method\n", " \"GET\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 7\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"aggregations\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/aggregations\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 8\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"service-desc\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/api\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/vnd.oai.openapi\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 9\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"service-doc\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/api.html\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"text/html\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 10\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"http://www.opengis.net/def/rel/ogc/1.0/queryables\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/queryables\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/schema+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 11\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"server\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://stac-utils.github.io/stac-server/\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"text/html\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 12\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"child\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections/cop-dem-glo-30\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 13\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"child\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections/naip\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 14\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"child\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 15\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"child\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections/sentinel-2-l1c\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 16\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"child\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections/landsat-c2-l2\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 17\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"child\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections/cop-dem-glo-90\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 18\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"child\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"https://earth-search.aws.element84.com/v1/collections/sentinel-1-grd\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/geo+json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
    • \n", " \n", " \n", " \n", "
  • \n", " \n", " conformsTo\n", " [] 14 items\n", " \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 0\n", " \"https://api.stacspec.org/v1.0.0/core\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 1\n", " \"https://api.stacspec.org/v1.0.0/collections\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 2\n", " \"https://api.stacspec.org/v1.0.0/ogcapi-features\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 3\n", " \"https://api.stacspec.org/v1.0.0/item-search\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 4\n", " \"https://api.stacspec.org/v1.0.0-rc.3/ogcapi-features#fields\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 5\n", " \"https://api.stacspec.org/v1.0.0-rc.2/ogcapi-features#sort\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 6\n", " \"https://api.stacspec.org/v1.0.0-rc.2/ogcapi-features#query\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 7\n", " \"https://api.stacspec.org/v1.0.0-rc.3/item-search#fields\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 8\n", " \"https://api.stacspec.org/v1.0.0-rc.2/item-search#sort\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 9\n", " \"https://api.stacspec.org/v1.0.0-rc.2/item-search#query\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 10\n", " \"https://api.stacspec.org/v0.3.0/aggregation\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 11\n", " \"http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/core\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 12\n", " \"http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/oas30\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 13\n", " \"http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/geojson\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
    • \n", " \n", " \n", " \n", " \n", "
  • \n", " title\n", " \"Earth Search by Element 84\"\n", "
    • \n", " \n", " \n", " \n", "
\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "URL = \"https://earth-search.aws.element84.com/v1/\"\n", "\n", "headers = []\n", "\n", "cat = Client.open(URL, headers=headers)\n", "cat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bod defines the search parameter and get the results:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# Collection\n", "collections = [\"sentinel-2-l2a\"]\n", "\n", "# Start and end dates\n", "start_date = datetime.fromisoformat(\"2021-07-08T00:00:00\")\n", "stop_date = datetime.fromisoformat(\"2021-07-08T23:59:59\")\n", "\n", "bbox = [-121.399, 39.834, -120.74, 40.472]\n", "\n", "# Other metadata\n", "cloud_cover = 5\n", "\n", "# Query by AOI, start and end date and other params\n", "query = cat.search(\n", " collections=collections,\n", " datetime=(start_date, stop_date),\n", " bbox=bbox,\n", " query={\"eo:cloud_cover\": {\"lt\": cloud_cover}},\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob plots the Sentinel-2 discovered STAC Items footprint:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6bd48dd034bc4ca1971a7247e682ca40", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map(center=[40.153000000000006, -121.0695], controls=(ZoomControl(options=['position', 'zoom_in_text', 'zoom_i…" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "center = ((bbox[1] + bbox[3]) / 2, (bbox[0] + bbox[2]) / 2)\n", "\n", "m = Map(center=center, zoom=8)\n", "\n", "for item in list(query.item_collection()):\n", " geo_json = GeoJSON(\n", " name=item.id,\n", " data=item.geometry,\n", " style={\n", " \"opacity\": 1,\n", " \"dashArray\": \"9\",\n", " \"fillOpacity\": 0.1,\n", " \"weight\": 1,\n", " \"color\": \"blue\",\n", " },\n", " hover_style={\"color\": \"white\", \"dashArray\": \"0\", \"fillOpacity\": 0.5},\n", " )\n", " m.add_layer(geo_json)\n", "\n", "m" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob lists the STAC Items self link, these are the URLs to the Sentinel-2 STAC Items to process:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_0_L2A',\n", " 'https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_1_L2A']" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[item.get_self_href() for item in list(query.item_collection())]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And creates the parameters for running the Application Package (the epsg and bands input parameters have default values)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'aoi': '-121.399,39.834,-120.74,40.472',\n", " 'bands': ['green', 'nir'],\n", " 'epsg': 'EPSG:4326',\n", " 'stac_items': ['https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_0_L2A',\n", " 'https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_1_L2A']}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "params = Params(\n", " aoi=\",\".join([str(elem) for elem in bbox]),\n", " stac_items=[item.self_href for item in query.item_collection()],\n", " epsg=\"EPSG:4326\",\n", " bands=[\"green\", \"nir\"],\n", ")\n", "\n", "params.dict()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob writes a YAML file with the parameters and their values:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "with open(\"params-s2.yaml\", \"w\") as file:\n", " print(to_yaml_str(params), file=file)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The file `params.yaml` contains:\n", "\n", "```yaml\n", "aoi: -121.399,39.834,-120.74,40.472\n", "bands:\n", "- green\n", "- nir\n", "epsg: EPSG:4326\n", "stac_items:\n", "- https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_0_L2A\n", "- https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_1_L2A\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob uses the CWL runner `cwltool` Python API to script the Application Package execution " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1;30mINFO\u001b[0m /data/work/open-reproducible-app-package/env_reproducible_app/lib/python3.9/site-packages/ipykernel_launcher.py 3.1.20231020140205\n", "\u001b[1;30mINFO\u001b[0m [workflow ] starting step node_water_bodies\n", "\u001b[1;30mINFO\u001b[0m [workflow ] start\n", "\u001b[1;30mINFO\u001b[0m [step node_water_bodies] start\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies] start\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies] starting step node_crop\n", "\u001b[1;30mINFO\u001b[0m [step node_crop] start\n", "\u001b[1;30mINFO\u001b[0m [step node_crop] start\n", "\u001b[1;30mINFO\u001b[0m [step node_water_bodies] start\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies_2] starting step node_crop_2\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies_2] start\n", "\u001b[1;30mINFO\u001b[0m [step node_crop_2] start\n", "\u001b[1;30mINFO\u001b[0m [step node_crop_2] start\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_crop] /tmp/h0vl2wbo$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/h0vl2wbo,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/6gt147fu,target=/tmp \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/g2e089lm/20231109085606-963229.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/crop@sha256:c4ef31e7eef8acb6be8565636396f8e4548585a6609b4a2d10e38ad4c3004ff4 \\\n", " python \\\n", " -m \\\n", " app \\\n", " --aoi \\\n", " -121.399,39.834,-120.74,40.472 \\\n", " --band \\\n", " green \\\n", " --epsg \\\n", " EPSG:4326 \\\n", " --input-item \\\n", " https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_0_L2A\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop_2] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop_2] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_2] /tmp/x0jy_qzp$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/x0jy_qzp,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/2l51rpij,target=/tmp \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/8xpevz0r/20231109085606-974964.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/crop@sha256:c4ef31e7eef8acb6be8565636396f8e4548585a6609b4a2d10e38ad4c3004ff4 \\\n", " python \\\n", " -m \\\n", " app \\\n", " --aoi \\\n", " -121.399,39.834,-120.74,40.472 \\\n", " --band \\\n", " nir \\\n", " --epsg \\\n", " EPSG:4326 \\\n", " --input-item \\\n", " https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_0_L2A\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop_3] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop_3] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_3] /tmp/ya93vr56$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/ya93vr56,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/0lx8xk93,target=/tmp \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/13jqrvdf/20231109085607-006913.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/crop@sha256:c4ef31e7eef8acb6be8565636396f8e4548585a6609b4a2d10e38ad4c3004ff4 \\\n", " python \\\n", " -m \\\n", " app \\\n", " --aoi \\\n", " -121.399,39.834,-120.74,40.472 \\\n", " --band \\\n", " green \\\n", " --epsg \\\n", " EPSG:4326 \\\n", " --input-item \\\n", " https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_1_L2A\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop_4] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_crop_4] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_4] /tmp/m_8409by$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/m_8409by,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/2iw5wgb8,target=/tmp \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/uunytv58/20231109085607-013230.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/crop@sha256:c4ef31e7eef8acb6be8565636396f8e4548585a6609b4a2d10e38ad4c3004ff4 \\\n", " python \\\n", " -m \\\n", " app \\\n", " --aoi \\\n", " -121.399,39.834,-120.74,40.472 \\\n", " --band \\\n", " nir \\\n", " --epsg \\\n", " EPSG:4326 \\\n", " --input-item \\\n", " https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_1_L2A\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_2] Max memory used: 370MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_crop] Max memory used: 377MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_crop] completed success\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_2] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_crop] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies] starting step node_normalized_difference\n", "\u001b[1;30mINFO\u001b[0m [step node_normalized_difference] start\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_normalized_difference] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_normalized_difference] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_normalized_difference] /tmp/pmtmq7pj$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/pmtmq7pj,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/yp0s6lib,target=/tmp \\\n", " --mount=type=bind,source=/tmp/h0vl2wbo/crop_green.tif,target=/var/lib/cwl/stg931251cb-1cd7-449e-89a1-da5765391a4d/crop_green.tif,readonly \\\n", " --mount=type=bind,source=/tmp/x0jy_qzp/crop_nir.tif,target=/var/lib/cwl/stgae72f928-5065-4107-bd3a-23a5ddea34ff/crop_nir.tif,readonly \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/z6df5m1f/20231109085700-056209.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/norm_diff@sha256:ff26cb57b3cfa1085f5e263b62ae2586ed12370b1447e11bebdebcdc195f1cfb \\\n", " python \\\n", " -m \\\n", " app \\\n", " /var/lib/cwl/stg931251cb-1cd7-449e-89a1-da5765391a4d/crop_green.tif \\\n", " /var/lib/cwl/stgae72f928-5065-4107-bd3a-23a5ddea34ff/crop_nir.tif\n", "\u001b[1;30mINFO\u001b[0m [job node_normalized_difference] Max memory used: 594MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_normalized_difference] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_normalized_difference] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies] starting step node_otsu\n", "\u001b[1;30mINFO\u001b[0m [step node_otsu] start\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_otsu] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_otsu] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_otsu] /tmp/0oj2e9n_$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/0oj2e9n_,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/03k16vh0,target=/tmp \\\n", " --mount=type=bind,source=/tmp/pmtmq7pj/norm_diff.tif,target=/var/lib/cwl/stg0d09a10d-3357-4f49-9375-31dff776edd4/norm_diff.tif,readonly \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/8jvwzzpe/20231109085707-404364.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/otsu@sha256:72b5579eae5cb993b32102705f9075cabfba940edabd1c921b7e025e9fb6c31e \\\n", " python \\\n", " -m \\\n", " app \\\n", " /var/lib/cwl/stg0d09a10d-3357-4f49-9375-31dff776edd4/norm_diff.tif\n", "\u001b[1;30mINFO\u001b[0m [job node_otsu] Max memory used: 326MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_otsu] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_otsu] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies] completed success\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_4] Max memory used: 383MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_4] completed success\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_3] Max memory used: 377MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_crop_3] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_crop_2] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies_2] starting step node_normalized_difference_2\n", "\u001b[1;30mINFO\u001b[0m [step node_normalized_difference_2] start\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_normalized_difference_2] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_normalized_difference_2] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_normalized_difference_2] /tmp/d180qrbz$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/d180qrbz,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/xfy9lihz,target=/tmp \\\n", " --mount=type=bind,source=/tmp/ya93vr56/crop_green.tif,target=/var/lib/cwl/stgec3986f5-72b2-4b6f-9bce-0f404005317f/crop_green.tif,readonly \\\n", " --mount=type=bind,source=/tmp/m_8409by/crop_nir.tif,target=/var/lib/cwl/stg100fb3d1-f66d-41ad-9d84-89220adc63ce/crop_nir.tif,readonly \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/oxu7geey/20231109085802-373086.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/norm_diff@sha256:ff26cb57b3cfa1085f5e263b62ae2586ed12370b1447e11bebdebcdc195f1cfb \\\n", " python \\\n", " -m \\\n", " app \\\n", " /var/lib/cwl/stgec3986f5-72b2-4b6f-9bce-0f404005317f/crop_green.tif \\\n", " /var/lib/cwl/stg100fb3d1-f66d-41ad-9d84-89220adc63ce/crop_nir.tif\n", "\u001b[1;30mINFO\u001b[0m [job node_normalized_difference_2] Max memory used: 594MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_normalized_difference_2] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_normalized_difference_2] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies_2] starting step node_otsu_2\n", "\u001b[1;30mINFO\u001b[0m [step node_otsu_2] start\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_otsu_2] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_otsu_2] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_otsu_2] /tmp/ntx5jf_c$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/ntx5jf_c,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/e0xjecvq,target=/tmp \\\n", " --mount=type=bind,source=/tmp/d180qrbz/norm_diff.tif,target=/var/lib/cwl/stg2f748878-c01b-4cef-94f3-c1f77ce98c5c/norm_diff.tif,readonly \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/ea9b5aex/20231109085809-340818.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/otsu@sha256:72b5579eae5cb993b32102705f9075cabfba940edabd1c921b7e025e9fb6c31e \\\n", " python \\\n", " -m \\\n", " app \\\n", " /var/lib/cwl/stg2f748878-c01b-4cef-94f3-c1f77ce98c5c/norm_diff.tif\n", "\u001b[1;30mINFO\u001b[0m [job node_otsu_2] Max memory used: 492MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_otsu_2] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_otsu_2] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow node_water_bodies_2] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_water_bodies] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow ] starting step node_stac\n", "\u001b[1;30mINFO\u001b[0m [step node_stac] start\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_stac] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mWARNING\u001b[0m \u001b[33m[job node_stac] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.\u001b[0m\n", "\u001b[1;30mINFO\u001b[0m [job node_stac] /tmp/i6jddzuk$ docker \\\n", " run \\\n", " -i \\\n", " --mount=type=bind,source=/tmp/i6jddzuk,target=/EqECVR \\\n", " --mount=type=bind,source=/tmp/utwvpo3e,target=/tmp \\\n", " --mount=type=bind,source=/tmp/0oj2e9n_/otsu.tif,target=/var/lib/cwl/stg36233271-07fb-44d3-abf8-211d0546c425/otsu.tif,readonly \\\n", " --mount=type=bind,source=/tmp/ntx5jf_c/otsu.tif,target=/var/lib/cwl/stg9cb27243-e8e8-41d7-8e70-07b5d24bae1f/otsu.tif,readonly \\\n", " --workdir=/EqECVR \\\n", " --read-only=true \\\n", " --user=1000:1000 \\\n", " --rm \\\n", " --cidfile=/tmp/u7gbbkw7/20231109085812-474326.cid \\\n", " --env=TMPDIR=/tmp \\\n", " --env=HOME=/EqECVR \\\n", " --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\\n", " --env=PYTHONPATH=/app \\\n", " ghcr.io/terradue/app-package-training-bids23/stac@sha256:0b23e5cbd67aa2030a88da6d987bf5863e33bbd6f2492d3d12bc9b7f5bafc3cc \\\n", " python \\\n", " -m \\\n", " app \\\n", " --input-item \\\n", " https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_0_L2A \\\n", " --water-body \\\n", " /var/lib/cwl/stg36233271-07fb-44d3-abf8-211d0546c425/otsu.tif \\\n", " --input-item \\\n", " https://earth-search.aws.element84.com/v1/collections/sentinel-2-l2a/items/S2A_10TFK_20210708_1_L2A \\\n", " --water-body \\\n", " /var/lib/cwl/stg9cb27243-e8e8-41d7-8e70-07b5d24bae1f/otsu.tif\n", "\u001b[1;30mINFO\u001b[0m [job node_stac] Max memory used: 38MiB\n", "\u001b[1;30mINFO\u001b[0m [job node_stac] completed success\n", "\u001b[1;30mINFO\u001b[0m [step node_stac] completed success\n", "\u001b[1;30mINFO\u001b[0m [workflow ] completed success\n", "\u001b[1;30mINFO\u001b[0m Final process status is success\n" ] } ], "source": [ "parsed_args = argparse.Namespace(\n", " podman=podman,\n", " parallel=True,\n", " debug=False,\n", " outdir=\"./runs\",\n", " workflow=app_package[\"url\"],\n", " job_order=[\"params-s2.yaml\"],\n", ")\n", "\n", "stream_out = StringIO()\n", "stream_err = StringIO()\n", "\n", "res = main(\n", " args=parsed_args,\n", " stdout=stream_out,\n", ")\n", "\n", "assert res == 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This execution generates as output a JSON file listing all files produced.\n", "\n", "The JSON contains the output defined in the CWL workflow that can be accessed with: \n", "\n", "```python\n", "os.path.basename(app_package[\"cwl\"].outputs[0].id)\n", "```" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk',\n", " 'basename': 'i6jddzuk',\n", " 'class': 'Directory',\n", " 'listing': [{'class': 'Directory',\n", " 'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_1_L2A',\n", " 'basename': 'S2A_10TFK_20210708_1_L2A',\n", " 'listing': [{'class': 'File',\n", " 'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_1_L2A/otsu.tif',\n", " 'basename': 'otsu.tif',\n", " 'checksum': 'sha1$eac01b59127e027758b6b4add8c4b7cb6475ba42',\n", " 'size': 286925,\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_1_L2A/otsu.tif'},\n", " {'class': 'File',\n", " 'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_1_L2A/S2A_10TFK_20210708_1_L2A.json',\n", " 'basename': 'S2A_10TFK_20210708_1_L2A.json',\n", " 'checksum': 'sha1$34642536318a6ca807ab038cc55685082d22825c',\n", " 'size': 4889,\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_1_L2A/S2A_10TFK_20210708_1_L2A.json'}],\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_1_L2A'},\n", " {'class': 'Directory',\n", " 'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A',\n", " 'basename': 'S2A_10TFK_20210708_0_L2A',\n", " 'listing': [{'class': 'File',\n", " 'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A/otsu.tif',\n", " 'basename': 'otsu.tif',\n", " 'checksum': 'sha1$210598cf619d49eabfdc24414bd445f59668f3bd',\n", " 'size': 289288,\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A/otsu.tif'},\n", " {'class': 'File',\n", " 'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A/S2A_10TFK_20210708_0_L2A.json',\n", " 'basename': 'S2A_10TFK_20210708_0_L2A.json',\n", " 'checksum': 'sha1$d2ec2f86148cdf8d2fe6681a462d3c680f141d3c',\n", " 'size': 4889,\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A/S2A_10TFK_20210708_0_L2A.json'}],\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A'},\n", " {'class': 'File',\n", " 'location': 'file:///data/work/open-reproducible-app-package/runs/i6jddzuk/catalog.json',\n", " 'basename': 'catalog.json',\n", " 'checksum': 'sha1$70a28aeddcbf6e8b18a81cd97761a18a4973daf9',\n", " 'size': 500,\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk/catalog.json'}],\n", " 'path': '/data/work/open-reproducible-app-package/runs/i6jddzuk'}" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = json.loads(stream_out.getvalue())\n", "\n", "results[os.path.basename(app_package[\"cwl\"].outputs[0].id)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob writes a simple code to find the STAC Catalog path and then list the contents of that STAC Catalog:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "* \n", " * \n", " * \n" ] } ], "source": [ "cat = pystac.read_file(\n", " [\n", " listing[\"path\"]\n", " for listing in results[os.path.basename(app_package[\"cwl\"].outputs[0].id)][\n", " \"listing\"\n", " ]\n", " if \"catalog.json\" in listing[\"path\"]\n", " ][0]\n", ")\n", "\n", "cat.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob uses the STAC Python library to open the first STAC Item produced:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", "
    \n", " \n", " \n", " \n", "
  • \n", " type\n", " \"Feature\"\n", "
    • \n", " \n", " \n", " \n", " \n", " \n", "
  • \n", " stac_version\n", " \"1.0.0\"\n", "
    • \n", " \n", " \n", " \n", " \n", " \n", "
  • \n", " id\n", " \"S2A_10TFK_20210708_0_L2A\"\n", "
    • \n", " \n", " \n", " \n", " \n", " \n", "
  • \n", " properties\n", "
      \n", " \n", " \n", " \n", "
    • \n", " proj:epsg\n", " 32610\n", "
      • \n", " \n", " \n", " \n", " \n", " \n", "
    • \n", " proj:geometry\n", "
        \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"Polygon\"\n", "
        • \n", " \n", " \n", " \n", " \n", "
      • \n", " \n", " coordinates\n", " [] 1 items\n", " \n", " \n", "
          \n", " \n", " \n", "
        • \n", " \n", " 0\n", " [] 5 items\n", " \n", " \n", "
            \n", " \n", " \n", "
          • \n", " \n", " 0\n", " [] 2 items\n", " \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 0\n", " 636990.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 1\n", " 4410550.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
            • \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", "
          • \n", " \n", " 1\n", " [] 2 items\n", " \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 0\n", " 691590.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 1\n", " 4410550.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
            • \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", "
          • \n", " \n", " 2\n", " [] 2 items\n", " \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 0\n", " 691590.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 1\n", " 4482600.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
            • \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", "
          • \n", " \n", " 3\n", " [] 2 items\n", " \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 0\n", " 636990.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 1\n", " 4482600.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
            • \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", "
          • \n", " \n", " 4\n", " [] 2 items\n", " \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 0\n", " 636990.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 1\n", " 4410550.0\n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
            • \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", " \n", "
        • \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", " \n", "
    • \n", " \n", " proj:bbox\n", " [] 4 items\n", " \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 0\n", " 636990.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 1\n", " 4410550.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 2\n", " 691590.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 3\n", " 4482600.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
      • \n", " \n", " \n", " \n", "
    • \n", " \n", " proj:shape\n", " [] 2 items\n", " \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 0\n", " 7205\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 1\n", " 5460\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
      • \n", " \n", " \n", " \n", "
    • \n", " \n", " proj:transform\n", " [] 9 items\n", " \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 0\n", " 10.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 1\n", " 0.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 2\n", " 636990.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 3\n", " 0.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 4\n", " -10.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 5\n", " 4482600.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 6\n", " 0.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 7\n", " 0.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
        \n", " \n", " \n", " \n", "
      • \n", " 8\n", " 1.0\n", "
        • \n", " \n", " \n", " \n", "
      \n", " \n", "
      • \n", " \n", " \n", " \n", " \n", "
    • \n", " proj:projjson\n", "
        \n", " \n", " \n", " \n", "
      • \n", " $schema\n", " \"https://proj.org/schemas/v0.4/projjson.schema.json\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"ProjectedCRS\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " name\n", " \"WGS 84 / UTM zone 10N\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " base_crs\n", "
          \n", " \n", " \n", " \n", "
        • \n", " name\n", " \"WGS 84\"\n", "
          • \n", " \n", " \n", " \n", " \n", " \n", "
        • \n", " datum\n", "
            \n", " \n", " \n", " \n", "
          • \n", " type\n", " \"GeodeticReferenceFrame\"\n", "
            • \n", " \n", " \n", " \n", " \n", " \n", "
          • \n", " name\n", " \"World Geodetic System 1984\"\n", "
            • \n", " \n", " \n", " \n", " \n", " \n", "
          • \n", " ellipsoid\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"WGS 84\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " semi_major_axis\n", " 6378137\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " inverse_flattening\n", " 298.257223563\n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", "
          • \n", " \n", " \n", " \n", " \n", " \n", "
        • \n", " coordinate_system\n", "
            \n", " \n", " \n", " \n", "
          • \n", " subtype\n", " \"ellipsoidal\"\n", "
            • \n", " \n", " \n", " \n", " \n", "
          • \n", " \n", " axis\n", " [] 2 items\n", " \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 0\n", "
                \n", " \n", " \n", " \n", "
              • \n", " name\n", " \"Geodetic latitude\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " abbreviation\n", " \"Lat\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " direction\n", " \"north\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " unit\n", " \"degree\"\n", "
                • \n", " \n", " \n", " \n", "
              \n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
              \n", " \n", " \n", " \n", "
            • \n", " 1\n", "
                \n", " \n", " \n", " \n", "
              • \n", " name\n", " \"Geodetic longitude\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " abbreviation\n", " \"Lon\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " direction\n", " \"east\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " unit\n", " \"degree\"\n", "
                • \n", " \n", " \n", " \n", "
              \n", "
              • \n", " \n", " \n", " \n", "
            \n", " \n", "
            • \n", " \n", " \n", "
          \n", "
          • \n", " \n", " \n", " \n", " \n", " \n", "
        • \n", " id\n", "
            \n", " \n", " \n", " \n", "
          • \n", " authority\n", " \"EPSG\"\n", "
            • \n", " \n", " \n", " \n", " \n", " \n", "
          • \n", " code\n", " 4326\n", "
            • \n", " \n", " \n", " \n", "
          \n", "
          • \n", " \n", " \n", " \n", "
        \n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " conversion\n", "
          \n", " \n", " \n", " \n", "
        • \n", " name\n", " \"UTM zone 10N\"\n", "
          • \n", " \n", " \n", " \n", " \n", " \n", "
        • \n", " method\n", "
            \n", " \n", " \n", " \n", "
          • \n", " name\n", " \"Transverse Mercator\"\n", "
            • \n", " \n", " \n", " \n", " \n", " \n", "
          • \n", " id\n", "
              \n", " \n", " \n", " \n", "
            • \n", " authority\n", " \"EPSG\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " code\n", " 9807\n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", "
          • \n", " \n", " \n", " \n", " \n", "
        • \n", " \n", " parameters\n", " [] 5 items\n", " \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 0\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"Latitude of natural origin\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " value\n", " 0\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " unit\n", " \"degree\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " id\n", "
                \n", " \n", " \n", " \n", "
              • \n", " authority\n", " \"EPSG\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " code\n", " 8801\n", "
                • \n", " \n", " \n", " \n", "
              \n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 1\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"Longitude of natural origin\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " value\n", " -123\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " unit\n", " \"degree\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " id\n", "
                \n", " \n", " \n", " \n", "
              • \n", " authority\n", " \"EPSG\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " code\n", " 8802\n", "
                • \n", " \n", " \n", " \n", "
              \n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 2\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"Scale factor at natural origin\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " value\n", " 0.9996\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " unit\n", " \"unity\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " id\n", "
                \n", " \n", " \n", " \n", "
              • \n", " authority\n", " \"EPSG\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " code\n", " 8805\n", "
                • \n", " \n", " \n", " \n", "
              \n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 3\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"False easting\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " value\n", " 500000\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " unit\n", " \"metre\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " id\n", "
                \n", " \n", " \n", " \n", "
              • \n", " authority\n", " \"EPSG\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " code\n", " 8806\n", "
                • \n", " \n", " \n", " \n", "
              \n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 4\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"False northing\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " value\n", " 0\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " unit\n", " \"metre\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " id\n", "
                \n", " \n", " \n", " \n", "
              • \n", " authority\n", " \"EPSG\"\n", "
                • \n", " \n", " \n", " \n", " \n", " \n", "
              • \n", " code\n", " 8807\n", "
                • \n", " \n", " \n", " \n", "
              \n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " coordinate_system\n", "
          \n", " \n", " \n", " \n", "
        • \n", " subtype\n", " \"Cartesian\"\n", "
          • \n", " \n", " \n", " \n", " \n", "
        • \n", " \n", " axis\n", " [] 2 items\n", " \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 0\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"Easting\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " abbreviation\n", " \"\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " direction\n", " \"east\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " unit\n", " \"metre\"\n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 1\n", "
              \n", " \n", " \n", " \n", "
            • \n", " name\n", " \"Northing\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " abbreviation\n", " \"\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " direction\n", " \"north\"\n", "
              • \n", " \n", " \n", " \n", " \n", " \n", "
            • \n", " unit\n", " \"metre\"\n", "
              • \n", " \n", " \n", " \n", "
            \n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " id\n", "
          \n", " \n", " \n", " \n", "
        • \n", " authority\n", " \"EPSG\"\n", "
          • \n", " \n", " \n", " \n", " \n", " \n", "
        • \n", " code\n", " 32610\n", "
          • \n", " \n", " \n", " \n", "
        \n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", " \n", " \n", "
    • \n", " datetime\n", " \"2021-07-08T19:03:25.410000Z\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", "
    • \n", " \n", " \n", " \n", " \n", " \n", "
  • \n", " geometry\n", "
      \n", " \n", " \n", " \n", "
    • \n", " type\n", " \"Polygon\"\n", "
      • \n", " \n", " \n", " \n", " \n", "
    • \n", " \n", " coordinates\n", " [] 1 items\n", " \n", " \n", "
        \n", " \n", " \n", "
      • \n", " \n", " 0\n", " [] 5 items\n", " \n", " \n", "
          \n", " \n", " \n", "
        • \n", " \n", " 0\n", " [] 2 items\n", " \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 0\n", " -121.39905410179915\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 1\n", " 39.833916743259095\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", " \n", "
          \n", " \n", " \n", "
        • \n", " \n", " 1\n", " [] 2 items\n", " \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 0\n", " -120.76135965075635\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 1\n", " 39.82336095080461\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", " \n", "
          \n", " \n", " \n", "
        • \n", " \n", " 2\n", " [] 2 items\n", " \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 0\n", " -120.73995321724426\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 1\n", " 40.471999341669175\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", " \n", "
          \n", " \n", " \n", "
        • \n", " \n", " 3\n", " [] 2 items\n", " \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 0\n", " -121.38373773482932\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 1\n", " 40.482798837728375\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", " \n", "
          \n", " \n", " \n", "
        • \n", " \n", " 4\n", " [] 2 items\n", " \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 0\n", " -121.39905410179915\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
            \n", " \n", " \n", " \n", "
          • \n", " 1\n", " 39.833916743259095\n", "
            • \n", " \n", " \n", " \n", "
          \n", " \n", "
          • \n", " \n", " \n", "
        \n", " \n", "
        • \n", " \n", " \n", "
      \n", " \n", "
      • \n", " \n", " \n", "
    \n", "
    • \n", " \n", " \n", " \n", " \n", "
  • \n", " \n", " links\n", " [] 3 items\n", " \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 0\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"root\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"/data/work/open-reproducible-app-package/runs/i6jddzuk/catalog.json\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 1\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"self\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A/S2A_10TFK_20210708_0_L2A.json\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 2\n", "
        \n", " \n", " \n", " \n", "
      • \n", " rel\n", " \"parent\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"/data/work/open-reproducible-app-package/runs/i6jddzuk/catalog.json\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"application/json\"\n", "
        • \n", " \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
    • \n", " \n", " \n", " \n", " \n", "
  • \n", " assets\n", "
      \n", " \n", " \n", " \n", "
    • \n", " data\n", "
        \n", " \n", " \n", " \n", "
      • \n", " href\n", " \"./otsu.tif\"\n", "
        • \n", " \n", " \n", " \n", " \n", " \n", "
      • \n", " type\n", " \"image/tiff; application=geotiff\"\n", "
        • \n", " \n", " \n", " \n", " \n", "
      • \n", " \n", " roles\n", " [] 2 items\n", " \n", " \n", "
          \n", " \n", " \n", " \n", "
        • \n", " 0\n", " \"data\"\n", "
          • \n", " \n", " \n", " \n", "
        \n", " \n", "
          \n", " \n", " \n", " \n", "
        • \n", " 1\n", " \"visual\"\n", "
          • \n", " \n", " \n", " \n", "
        \n", " \n", "
        • \n", " \n", " \n", "
      \n", "
      • \n", " \n", " \n", " \n", "
    \n", "
    • \n", " \n", " \n", " \n", " \n", "
  • \n", " \n", " bbox\n", " [] 4 items\n", " \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 0\n", " -121.39905410179915\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 1\n", " 39.82336095080461\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 2\n", " -120.73995321724426\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 3\n", " 40.482798837728375\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
    • \n", " \n", " \n", " \n", "
  • \n", " \n", " stac_extensions\n", " [] 1 items\n", " \n", " \n", "
      \n", " \n", " \n", " \n", "
    • \n", " 0\n", " \"https://stac-extensions.github.io/projection/v1.1.0/schema.json\"\n", "
      • \n", " \n", " \n", " \n", "
    \n", " \n", "
    • \n", " \n", " \n", "
\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "item = next(cat.get_items())\n", "item" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob gets the path of the ostu step asset:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'/data/work/open-reproducible-app-package/runs/i6jddzuk/S2A_10TFK_20210708_0_L2A/otsu.tif'" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "asset_href = item.get_assets()[\"data\"].get_absolute_href()\n", "\n", "asset_href" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob applies the sieve algorithm and then vectorizes the water bodies.\n", "\n", "Finally the water bodies are added to a map" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1c2f58872b9d499b83e398241791b6c7", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map(center=[40.153000000000006, -121.0695], controls=(ZoomControl(options=['position', 'zoom_in_text', 'zoom_i…" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Define the threshold size to remove small features (in pixels)\n", "threshold = 100 # Adjust this threshold as needed\n", "connectivity = 4 # Use 4-connected pixels for the sieve operation\n", "\n", "center = ((bbox[1] + bbox[3]) / 2, (bbox[0] + bbox[2]) / 2)\n", "\n", "m = Map(center=center, zoom=8)\n", "\n", "with rasterio.open(asset_href) as src:\n", " result = sieve(src, threshold, connectivity=8)\n", " for geom in dataset_features(src, band=True, as_mask=True):\n", " geo_json = GeoJSON(\n", " name=\"\",\n", " data=geom,\n", " style={\n", " \"opacity\": 1,\n", " \"fillOpacity\": 0.1,\n", " \"weight\": 1,\n", " \"color\": \"red\",\n", " },\n", " hover_style={\"color\": \"red\", \"dashArray\": \"0\", \"fillOpacity\": 0.5},\n", " )\n", " m.add_layer(geo_json)\n", "m" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bob sends an email to Alice\n", "\n", "
\n", "from: bob@acme.io\n", "\n", "to: alice@acme.io\n", "\n", "subject: RE:Detecting water bodies with NDWI and the Otsu threshold\n", "\n", "\n", "Hi Alice!\n", "\n", "The results look promising!\n", "\n", "Cheers,\n", "\n", "Bob\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.6" } }, "nbformat": 4, "nbformat_minor": 2 }