> Documentation index for agents: https://docs.rocketride.ai/llms.txt

# detect_segment

A RocketRide image node that produces pixel-level instance or semantic segmentation, with a JSON result and an annotated image. Choose it when a bounding box alone is not enough and a downstream step needs the pixels belonging to each object or class region.

## About Hugging Face

This node's service definition identifies its Mask2Former engines as HuggingFace-native. Its configured profiles select pretrained Mask2Former models for object-instance and scene-region segmentation.

## What it does

The node consumes each image frame, runs the configured Mask2Former segmentation mode, and can emit both a JPEG overlay and a JSON mask payload. Instance mode returns separate masks, labels, scores, and boxes; semantic mode returns a per-pixel class map and class information. Use it instead of an object-detection node when the shape or extent of an object matters, rather than only its location.

## Lanes

| Lane in | Lane out | Description |
| ------- | -------- | ----------- |
| `image` | `image` | Annotated JPEG frame with translucent mask overlays; instance overlays also include boxes and labels. |
| `image` | `text` | JSON segmentation result: an instance list or a semantic-map object, according to the selected mode. |

## Profiles

Default: **Mask2Former — Object Instances (COCO, 80 classes, default)** (`mask2former-instance`).

| Profile | Mode | Default confidence threshold | Default max input edge |
| ------- | ---- | ---------------------------- | ---------------------- |
| Mask2Former — Object Instances (COCO, 80 classes, default) **(default)** | `instance` | `0.3` | `1024` px |
| Mask2Former — Scene Regions (ADE20K, 150 classes) | `semantic` | `0.0` | `1024` px |

## Configuration

Start with the profile that matches the kind of result a downstream node needs. The profile supplies a matching mode and engine; adjust the confidence and image-size controls only when the expected objects are missing, noisy, or too expensive to process.

### Mode and Engine

`instance` produces a separate mask for each detected object, while `semantic` produces a class map across the image. Keep the engine paired with its mode: the available instance and semantic engines are gated that way in the configuration. Pick instance mode for object-by-object handling, such as extracting individual people; pick semantic mode when classifying every pixel of a scene is more useful. An unrecognised mode falls back to the implementation default at startup.

### Confidence threshold

The threshold determines which masks are included, from `0.0` through `1.0`. The instance preset starts at `0.3`; raise it when low-confidence masks clutter the result, or lower it when a relevant object is being omitted. The semantic preset uses `0.0`, so its output retains all class assignments. Invalid threshold values are replaced with the implementation default rather than stopping the node.

### Max input edge (px)

Before inference, the node downscales an image whose long edge exceeds this value, then restores masks to the source dimensions for the overlay. The profiles use `1024` px. Lower it to reduce inference time and memory use on large frames; raise it when small boundaries need more detail, within the `256`–`4096` px range. Values outside that range are clamped at startup.

## Requirements

This node declares GPU capability and uses its configured transformer engine through a shared device lock when inference runs locally. Its profile metadata states that the engines can run through Transformers on CPU, MPS, or CUDA; use an accelerator when throughput or large images make CPU processing inadequate. The node also loads its declared dependencies before inference, including the mask decoder used to render returned masks.

## Notes

### Output and failures

The text and image outputs are produced only when those lanes have listeners. If a frame cannot be decoded, or segmentation or writing the outputs fails, the node raises an error that names the cause, and the engine marks the object as failed: the image, or the whole video the frame came from. The task goes on with the next object. Output already written is not withdrawn, such as the masks JSON when only the annotated image fails, or other frames of the same video. A mask or semantic map that cannot be decoded for the overlay is logged as a warning and left out of the annotated image; the JSON still carries it.

## Upstream docs

- [Hugging Face Transformers documentation](https://huggingface.co/docs/transformers/)

<!-- ROCKETRIDE:GENERATED:PARAMS START -->
<!-- Generated by nodes:docs-generate. Do not edit by hand. -->

## Schema

| Field | Type | Description | Default |
|---|---|---|---|
| `detect_segment.engine` | `string` | **Engine**<br/>Backing engine. Gated by mode: instance -> mask2former-instance; semantic -> mask2former-semantic. | `"mask2former-instance"` |
| `detect_segment.maxEdge` | `number` | **Max input edge (px)**<br/>Downscale source so long edge <= this value before inference; masks are upsampled back to the source size. Lower = faster + less VRAM. | `1024` |
| `detect_segment.mode` | `string` | **Mode**<br/>Segmentation mode. instance: per-instance masks (default). semantic: per-pixel class map. Both use Mask2Former under the hood. | `"instance"` |
| `detect_segment.profile` | `string` | **Profile**<br/>Segmentation preset. Runs on CPU/MPS/CUDA via transformers. | `"mask2former-instance"` |
| `detect_segment.threshold` | `number` | **Confidence threshold**<br/>Minimum score (0.0-1.0) to include a mask | `0.3` |

## Dependencies

- `pycocotools`

## Source

[<svg viewBox="0 0 16 16" width="15" height="15" fill="currentColor" aria-hidden="true" style="vertical-align:-0.15em;margin-right:0.35em"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"/></svg> View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/detect_segment)
<!-- ROCKETRIDE:GENERATED:PARAMS END -->
