Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Interactive segmentation and extending SAM to video

Every workflow so far has run through code, one prompt at a time. This page covers two extensions: a map-based interface that needs no code at all, and video segmentation, which extends prompting and the streaming memory from earlier in this lesson across time.

1. Motivation

Not every use of SAM needs a script. Interactive, no-code exploration is often the fastest way to test whether a feature is segmentable at all, before committing to a coded workflow. Video segmentation, while more specialized, is directly useful for monitoring applications and makes the streaming-memory concept from earlier in this lesson concrete.

2. Core idea

show_map() combines a live map interface with SAM inference, letting you type a prompt or draw a box directly on the map and see results immediately, without writing a single line of segmentation code. SamGeo3Video extends the same prompting ideas, text and point prompts, object identity, confidence scores, from a single image across every frame of a video, using the streaming memory mechanism from earlier in this lesson.

3. Interactive map-based segmentation

from pathlib import Path

DATA_DIR = Path("data/raw")
DATA_DIR.mkdir(parents=True, exist_ok=True)

image_url = "https://data.source.coop/opengeos/geoai/uc-berkeley.tif"
image_path = download_file(image_url, output=str(DATA_DIR / Path(image_url).name))

sam3 = SamGeo3(backend="transformers", device=None, checkpoint_path=None, load_from_HF=True)
sam3.set_image(image_path)
sam3.generate_masks(prompt="building")
sam3.save_masks("masks.tif")

sam3.show_map(height="700px", min_size=10)

Note backend="transformers" here; the interactive interface specifically requires this backend, unlike the "meta" backend used in earlier pages. show_map() supports two interaction modes directly in the notebook: typing a text description (like “tree” or “parking lot”) and clicking Segment, or drawing a rectangle over an area of interest and clicking Segment. Results update without re-running the image encoder, the same cached-embedding efficiency from earlier in this lesson, now available through a point-and-click interface rather than code.

4. Video segmentation

A. Text-prompted video segmentation

video_url = "https://data.source.coop/opengeos/geoai/cars.mp4"
video_path = download_file(video_url, output=str(DATA_DIR / Path(video_url).name))

sam = SamGeo3Video()
sam.set_video(video_path)
sam.generate_masks("car")
sam.show_frames(frame_stride=20, ncols=3)

A text prompt on video finds all matching objects in the first frame, then propagates their masks through every remaining frame automatically, using the streaming memory mechanism from earlier in this lesson to maintain object identity as cars move.

B. Point-prompted video segmentation

sam = SamGeo3Video()
sam.set_video(video_path)
sam.init_tracker()
sam.show_frame(0, axis="on")

sam.add_point_prompts([[300, 200]], [1], obj_id=1, frame_idx=0)
sam.add_point_prompts([[420, 200]], [1], obj_id=2, frame_idx=0)
sam.propagate()

Each add_point_prompts() call assigns an instance identifier (obj_id) that persists across the entire video, the same “same ID means same object” idea from L08, now tracked through time rather than assigned once per static image. propagate() is the step that actually runs tracking across all frames using the accumulated prompts; nothing propagates until you call it.

Negative (background) points refine an individual object’s mask, the same idea from the point-prompt page, applied here to a specific tracked object:

sam.add_point_prompts(
    points=[[335, 195], [335, 220]],
    labels=[1, 0],
    obj_id=1,
    frame_idx=0,
)
sam.propagate()

C. Removing spurious detections

sam.remove_object(obj_id=[5, 8, 12, 13])
sam.propagate()

Text-prompted video segmentation can pick up spurious detections, an object that is not actually what you meant to track. Removing them by ID before the final propagation avoids wasting tracking effort on objects you do not need, more efficient than tracking everything and filtering afterward.

import os

os.makedirs("output", exist_ok=True)
sam.save_masks("output/masks")
sam.save_video("output/segmented.mp4", fps=25)
sam.close()

close() releases the video session’s resources; call it when you are done with a given video, the video-tracking equivalent of geoai.empty_cache() used elsewhere in this lesson to free GPU memory between operations.

5. Python reactivation

{i: f"Player {i}" for i in range(15)}-style dictionary construction (or the equivalent explicit loop) maps object IDs to display names, the same key-value pattern used for the stats and models dictionaries in earlier lessons, here applied to labeling tracked objects for display. remove_object(obj_id=[5, 8, 12, 13]) takes a list of IDs to remove at once, the same batch-removal idea as filtering multiple rows from a DataFrame in one operation rather than one at a time.

6. Common pitfalls

7. Mini task

You run text-prompted video segmentation with the prompt “player” on a sports video and notice object IDs 5, 8, 12, and 13 are clearly not players (perhaps a scoreboard graphic and stray background objects). What is the most efficient order of operations from here: propagate first and clean up after, or clean up first and then propagate?

Sample solution

Clean up first, then propagate. Removing the spurious object IDs before calling propagate() means tracking effort is only spent on the objects you actually want followed through the full video, rather than tracking every detection, including the unwanted ones, and discarding that wasted work afterward. This is exactly the order demonstrated in step C.

8. Key takeaways

Further reading

Spatial Data Analytics
Scaling segmentation
Spatial Data Analytics
Project transfer