VLM + SAM — Auto Masking for Video
Development
A pipeline that bridges a vision-language model (Gemini) with Segment Anything Model 3 to automatically detect, label, localize, and segment regions in video, exporting per-region masks for downstream use. I named it SAMhera for now.
How it works
The workflow selects a single frame from the video (set to the first frame by default) and uses a VLM to enumerate the visually distinct objects present — either freely, when no preset is given, or scoped to a category such as portrait, full body, or product.
For example, given Jennie's Travel Essentials video, Gemini 3.1 Pro Preview enumerates:
- two-tone blue background
- white tabletop
- white towel or cloth
- eyeglasses on table
- sunglasses on table
- black Adidas baseball cap
- navy blue 'Sleepy Jones' baseball cap
- woman with dark hair wearing black t-shirt and holding sunglasses
The workflow then sends those labels back to Gemini in a second call to localize each one, returning a bounding box per label. This two-call approach (discovery → localization) makes it well-suited for analyzing product and commercial content, where the items on screen aren't known in advance and can't be hardcoded as presets.
Video Masking
Unpack Bundle node acts as a bridge between SAMhera's SAM3_BOX_AND_POINT type and SAM3's native input types. (I bundled them for less noodling)
Auto Layer Detect outputs a SAM3_BOX_AND_POINT bundle per layer — containing the bounding box, positive points, and negative points all in one wire. Unpack Bundle splits this into the three separate types that SAM3VideoSegmentation already accepts: positive_boxes, positive_points, and negative_points.
From there, SAM3 propagates the segmentation mask across the full video.
This workflow eliminates a significant amount of hand-labor. No more manually drawing bounding boxes or placing positive/negative prompt points to refine object boundaries. The VLM handles discovery and localization automatically, passing clean inputs directly to SAM.
Products
Things Gemini considers products.. (phew it didn’t include her)
Output
More thoughts and tools coming..
While reading this research paper about SAM3: Segment Anything with Concepts, I found it fascinating how computer vision interprets what filmmakers (or anyone who captures the world with a camera) see — in their corporate way (… still interesting) .. and that ontological gap that reveals. It's like saying "I love you" from lovers versus insects (which I found brutal and computational).
For example, how these researchers deal with boundary ambiguity (page 3, Promptable Concept Segmentation) — their case of "mirror," which needs to be refined to stabilize their category definition — is exactly what filmmakers like Tarkovsky or Joyce Wieland uses to describe psychological state. (can’t be more precise!) What the model treats as ambiguity to be resolved, cinema uses as the very mechanism of meaning...
