By Joey and Fancy from Cryosphere Lab, ESSC, CUHK.

• Geo SAM
  • Introduction
  • Installation
    • Install QGIS
    • Install Library Dependencies
      • For Windows Users
      • For Mac or Linux Users
    • Install the Geo SAM Plugin
      • Download the Plugin
      • Locate the QGIS Plugin folder
      • Activate the Geo SAM Plugin
  • Use the Geo SAM Segmentation Tool
    • Add Prompts
    • Save Current Results
    • Undo/Clear Prompts
    • Enable/Disable the Tool
    • Load Image Features
    • Shortcuts
    • Tips for Using the Segmentation Tool
  • Use the Geo SAM Encoding Tool
    • Download SAM Checkpoints
    • Select Bands and Value Range for Processing
    • Patch Sampling
    • Demo Animation
    • Tips for Making the Encoding Process Faster
  • Future Works
  • Citation
  • Acknowledgement

Geo SAM is a QGIS plugin that aims to help people segment, delineate or label landforms efficiently when using large-size geospatial raster images. Segment Anything Model (SAM) is a foundation AI model with the superpower, but the model size is huge, and using it to process images can take a long time, even with a modern GPU. Our tool uses the strategies of encoding image features in advance and trimming the SAM model. The interactive segmentation process can be run in real-time on a laptop by only using a CPU, making it a convenient and efficient tool for dealing with satellite images.

The Geo SAM plugin includes two separate tools, the encoding tool, and the segmentation tool. The encoding tool is designed to generate and save the image features using the SAM image encoder, and the encoding process only needs to run once per image. The segmentation tool is for interactively segmenting landforms, and it can only be used to segment preprocessed images (whose features have been generated in advance using the encoding tool, as the included demo image).

You are suggested to install the latest version of QGIS since the plugin has only been tested on versions newer than QGIS 3.30.

Open the OSGeo4W Shell application from the Start menu, which is a dedicated shell for the QGIS. Then run the following command to install the libraries.

pip3 install torch torchvision
pip3 install torchgeo
pip3 install segment-anything

Our encoder tool now supports using CUDA GPU to accelerate the encoding process. If your PC has dedicated CUDA GPUs, you can install the CUDA library first and then install the gpu-version pytorch using the following command (using CUDA version 11.7 as an example):

# add `--force-reinstall` if you installed the cpu version before.
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117

Open your own terminal application, and change the directory to where the QGIS Python binary file locates.

# Mac
cd /Applications/QGIS.app/Contents/MacOS/bin
# Linux (Please confirm the python env by "import qgis")
cd /usr/bin

To confirm the QGIS Python environment:

./python3
>>> import qgis

Then install the libraries.

# !important, add ./ to avoid using your default Python in the system
./pip3 install torch torchvision
./pip3 install torchgeo
./pip3 install segment-anything

For Linux users, if pip3 is not found in /usr/bin, try the following commands:

sudo apt-get update
sudo apt-get install python3-pip

For Linux users, if your computer got available CUDA GPUs and with CUDA library installed, the above commands should have helped you install the gpu-version pytorch. You can reach pytorch official website for more information.

Download the plugin zip file, unzip it, and rename the folder as Geo-SAM (be aware of undesired nested folders after unzipping).

In QGIS, Go to the menu Settings > User Profiles > Open active profile folder. You'll be taken straight to the profile directory. Under the profile folder, you may find a python folder; the plugins folder should be right inside the python folder (create the plugins folder if it does not exist). Put the entire Geo-SAM folder inside the plugins folder, then restart QGIS. The directory tree structure should be the same as the following.

python
└── plugins
    └── Geo-SAM
        ├── checkpoint
        ├── docs
        ├── ...
        ├── tools
        └── ui

Below are some general paths of the plugin folder for your reference.

# Windows
%APPDATA%\QGIS\QGIS3\profiles\default\python\plugins
# Mac
~/Library/Application\ Support/QGIS/QGIS3/profiles/default/python/plugins
# Linux
~/.local/share/QGIS/QGIS3/profiles/default/python/plugins

After restarting QGIS, go to the menu Plugins > Manage and Install Plugins, and under Installed, you may find the Geo SAM plugin; check it to activate the plugin.

After activating the Geo SAM plugin, you may find the Geo SAM tools under the Plugins menu,

You may also find a new toolbar, including two icons.

Click the Segmentation Tool icon to open the interactive segmentation widget. You will be shown a demo raster image with thaw slump and small pond landforms for you to try the tool. With a single click on the map, a segmentation result will be generated.

A user interface will be shown below.

Click the buttons to select between the Foreground(FG) and Background(BG) points. Use FG points to add areas you desire, and use BG points to remove areas you don't want.

Click the BBox button to choose to add a bounding box (BBox) on the canvas by pressing and dragging the mouse. BBox can be used together with adding points or independently.

After adding points and a BBox for segmenting a subject, you can save the segmentation results by clicking the Save button.

You can use the Undo button to undo the last point or BBox prompt.

You can use the Clear button to clear the added points and BBox.

You can uncheck the Enable button to temporally disable the tool and navigate on the map.

The plugin is initialized with features for demo purposes, and you can use the Feature Folder selection button to select the folder that includes the image features you need.

Then, press the Load button to load the selected image features. Remember to add the corresponding raster image to the QGIS project.

• Tab: loop between 3 prompt types (the cursor will also change to the corresponding types)
• C: clear all prompts in canvas [same as Clear button]
• Z: undo the last prompt in canvas [same as Undo button]
• S: save SAM output features into polygon [same as Save button]
• Ctrl+Z or command+Z: undo the last saved segmentation results

• Deal with only One Subject each time
• Use Background Points to exclude unwanted parts
• Use Bounding Box (BBox) to limit the segment polygon boundary
• The BBox should cover the entire subject
• Remember to press the Save button after the segmentation of the chosen subject

If you want to try your own images, you can use the Encoding Tool. This tool helps to preprocess geospatial images and generate image features using the SAM image encoder. The generated image features can then be used in our Geo-SAM tool to label the landforms by adding points and bounding box prompts.

SAM model checkpoints should be downloaded in advance, and three versions (huge, large, and base) are available. The large version "vit_l" is recommended to try first. You need to specify the model type that matches the checkpoint version. Using the following links to download the checkpoints.

• vit_h: ViT-H SAM model.
• vit_l: ViT-L SAM model.
• vit_b: ViT-B SAM model.

After selecting the raster layer or image file you want to process, you should also choose the proper bands. The SAM natively supports only three-band RGB images, but we have adapted the tool to support one or two-band images so that you can try grayscale images or NDVI spectral index images.

The values of the image input to the SAM should range from 0 to 255, and you may need to specify the value range (in Advanced Parameters) to be rescaled to [0, 255]. By default, the tool will help you to find the min and max values of the image data and rescale the value range of [min, max] to [0, 255].

Since SAM only supports input images with sizes of (1024, 1204), small images will be resized to match the input size, while large images will be sampled into overlapped patches (patch_size=1024) in a grid-like fashion. The stride parameter will determine the overlap behavior, overlap = patch_size - stride.

The following animation shows how to use the encoder tool.

After processing the image, by default, the generated features will automatically be loaded in the segmentation tool for you to start labeling. Or you can choose to load the image features manually afterward.

• Choose a smaller processing extent
• Reduce target resolution (in Advanced Parameters)
• Increase stride to minimize overlap
• Choose a smaller version of SAM model
• Use GPU
• Increase batch_size when using a GPU with sufficient GPU memory

• Existing polygon refinement

Zhao, Zhuoyi, Fan, Chengyan, & Liu, Lin. (2023). Geo SAM: A QGIS plugin using Segment Anything Model (SAM) to accelerate geospatial image segmentation (1.1.0). Zenodo. https://doi.org/10.5281/zenodo.8191039

@software{zhao_zhuoyi_2023_8191039,
  author       = {Zhao, Zhuoyi and Fan, Chengyan and Liu, Lin},
  title        = {{Geo SAM: A QGIS plugin using Segment Anything Model (SAM) to accelerate geospatial image segmentation}},
  month        = jul,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {1.1.0},
  doi          = {10.5281/zenodo.8191039},
  url          = {https://doi.org/10.5281/zenodo.8191039}
}

This repo benefits from Segment Anything and TorchGeo. Thanks for their wonderful work.