What Is Garment Segmentation?
Garment segmentation is the first stage of every modern AI virtual try-on pipeline. A vision model isolates a clothing item from its background to produce a clean alpha mask the diffusion stage can paste onto a person.
What is garment segmentation in AI virtual try-on?
Garment segmentation is a computer-vision technique that isolates a clothing item from its background in a product photo. A vision transformer (typically a SAM-class model fine-tuned on apparel) classifies every pixel as either belonging to the garment or to the background, producing an alpha mask. This mask is the first artefact consumed by the rest of the virtual try-on pipeline. Without clean segmentation, the diffusion model that synthesises the final image will paint background fragments onto the wearer, ruining identity preservation.
- Segmentation = isolating garment pixels from background pixels.
- Output is a per-pixel alpha mask, typically a transparent PNG.
- Modern segmenters are vision transformers (SAM, MaskFormer derivatives).
- Works on flat-lays and on-model shots, with flat-lays producing cleaner masks.
- Quality of segmentation determines half of final try-on quality.
How it works under the hood
Modern garment segmentation runs in three steps. First, the input garment photo is passed through a vision transformer that produces per-pixel feature embeddings. Second, those embeddings are scored against an apparel-class head producing per-pixel probabilities (this pixel is shirt, this pixel is hanger, this pixel is wall). Third, a thresholding and morphological cleanup pass converts soft probabilities into a hard binary mask with smoothed edges.
The output is a single-channel image: white where the garment is, black where it is not. This mask plus the original RGB image gives the rest of the pipeline a clean transparent PNG of just the garment.
Why flat-lay photos work better
Garment segmentation models are trained predominantly on flat-lay photography — garment laid flat on a plain background. The training distribution is heavily skewed toward this format because catalog images dominate apparel datasets.
On-model photography (garment worn by a human) is harder. The segmenter must distinguish garment-from-body, garment-from-other-garment, and garment-from-shadow. Errors accumulate. The same garment photographed flat will segment perfectly while photographed on a model may bleed at the neckline, hem, or armhole.
What clean segmentation looks like
A high-quality segmentation has three properties. First, sharp edges that follow the garment outline without jaggies. Second, no holes inside the garment region (a common failure mode where dark print regions are misclassified as background). Third, no halos — a single-pixel-wide stripe of background colour clinging to the garment edge that will appear as a faint outline in the final try-on.
Definitions
- Alpha mask
- A single-channel image storing per-pixel transparency. White = fully opaque, black = fully transparent.
- Vision transformer
- A neural network architecture that processes images as sequences of patches, producing rich per-pixel embeddings.
- SAM
- Segment Anything Model from Meta AI — a general-purpose segmentation foundation model; many apparel segmenters are fine-tunes of SAM.
Frequently asked questions
Can I run garment segmentation myself?+
Why does my try-on result have background bleed?+
Is segmentation the same as background removal?+
Drape Editorial is the in-house research team behind Drape Try-On. We test virtual try-on models against real garment photography weekly and publish what we learn.