Compare object detection and panoptic segmentation
This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Understand when to use bounding boxes versus pixel-level masks for image
analysis.What’s in this recipe:
Run object detection to get bounding boxes and labels
Run panoptic segmentation to get pixel-level masks
Visualize and compare outputs side-by-side
Problem
You need to analyze objects in images, but there are two approaches:Which should you use? Detection is faster but approximate. Segmentation
is slower but precise.
Solution
Run both approaches on the same images using DETR models and compare the
results.
import numpy as npimport pixeltable as pxtfrom pixeltable.functions.huggingface import ( detr_for_object_detection, detr_for_segmentation,)from pixeltable.functions.vision import bboxes_draw, overlay_segmentationpxt.drop_dir('detection_vs_seg', force=True)pxt.create_dir('detection_vs_seg')
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/pjlb/.pixeltable/pgdata
Created directory ‘detection_vs_seg’.
<pixeltable.catalog.dir.Dir at 0x145b43f90>
Added 2 column values with 0 errors in 0.03 s (58.89 rows/s)
2 rows updated.
images.select(images.detection_viz).collect()
Run panoptic segmentation
The detr_for_segmentation function returns pixel-level masks and
segment metadata.Parameters:
model_id: Segmentation model (facebook/detr-resnet-50-panoptic)
threshold: Confidence threshold for filtering segments
Output:
{ 'segmentation': np.ndarray, # (H, W) array where each pixel = segment ID 'segments_info': [{'id': 1, 'label_text': 'person', 'score': 0.98}, ...]}
Note: The full segmentation output contains a numpy array that
can’t be stored as JSON. We store just the segments_info metadata
and compute the pixel-level visualization inline.
# Store just the segments_info (JSON-serializable) as a computed column# The segmentation array will be computed inline for visualizationseg_expr = detr_for_segmentation( images.image, model_id='facebook/detr-resnet-50-panoptic', threshold=0.5,)images.add_computed_column(segments_info=seg_expr.segments_info)