Most computer vision that survives contact with a fashion business reduces to a short list of jobs: find the garment, describe it, match it, cut it out, locate the body, render the garment onto that body, catch the fault, and normalise the picture. Each maps to a model class you can staff, benchmark and retrain, and each fails in a predictable way. What follows is the list genuinely running behind catalogue pipelines, search boxes, try-on widgets and cutting rooms.
Key takeaways
- Detection comes first, because nearly every other fashion vision task assumes the garment has already been located.
- Attribute tagging and visual search reach production fastest, since both plug into catalogue and search infrastructure a retailer already runs.
- Try-on is a chain — segmentation, pose, warping, compositing — and the weakest stage sets the ceiling for the feature.
- Defect detection is an anomaly problem, not a classification problem: balanced data on faults you are eliminating never arrives.
- Most of these tasks degrade on the same inputs — occlusion, unusual poses, black-on-black fabric, inconsistent supplier lighting.
Which computer vision tasks are actually in production at fashion companies?
Ordered roughly by how early each lands in a stack. Every item has the same shape: what the task is, the model class, how it works, where it breaks down.
1. Garment detection and localisation
Detection draws a box around each clothing item and assigns it a class. It is the cheapest task here and the precondition for most of the rest — the field's standard survey makes the same point, observing that most fashion vision work needs detection first and that virtual try-on cannot begin until the garment has been found (Fashion Meets Computer Vision: A Survey).
- Model class: single-stage or transformer-based detectors, fine-tuned on clean catalogue shots plus messy street imagery.
- How it works: a backbone produces multi-scale features, a head regresses boxes and class scores, and either non-maximum suppression or set-based matching resolves overlaps.
- Where it breaks down: layered outfits. A coat over a shirt over a tee produces heavily overlapping boxes, and overlap-based suppression quietly deletes the garment you cared about.
2. Attribute tagging
Tagging turns a picture into structured fields: neckline, sleeve length, closure, pattern, fit, heel height. Most retailers automate it first, because merchandising, filters and feeds all consume the same taxonomy, and retail AI platforms such as Vue.ai ship product tagging alongside on-model imagery and assortment visualisation as a productised service.
- Model class: a shared vision backbone with one classification head per attribute group, or an image-text embedding model scored against attribute prompts when labels are thin.
- How it works: each attribute group gets its own head over frozen or lightly tuned features, thresholds are calibrated per attribute, and low-confidence predictions route to human review.
- Where it breaks down: taxonomy drift. Merchandising renames a value or splits a category, every historical label silently changes meaning, and your accuracy metric looks stable while the filters go wrong.
3. Similar-item retrieval and visual search
Retrieval answers "show me more like this" from a catalogue photo, a user upload or a screenshot. It is the vision task tied most directly to revenue, since it feeds recommendation and personalisation — retailers fund vision largely to understand customers better and tailor the shopping experience to them (how computer vision is changing fashion retail).
- Model class: metric-learning embeddings trained with contrastive or triplet objectives, served through an approximate nearest-neighbour index.
- How it works: embed the catalogue offline, embed the query online, retrieve top matches by cosine distance, then re-rank with availability, size in stock and margin.
- Where it breaks down: index freshness. On a multi-brand European platform of the kind Zalando operates, the catalogue changes daily; retrain the encoder and every stored vector is stale, so you need a shadow index and a backfill plan.
4. Garment parsing and instance segmentation
Parsing produces pixel-level masks for garment regions — bodice, sleeve, collar — plus skin and hair. Anything that composites, recolours or measures needs it, and it is the quiet dependency behind most generative fashion imagery.
- Model class: mask-predicting instance segmentation networks, often bootstrapped with promptable segmentation used zero-shot, then distilled into a small task model.
- How it works: propose regions, predict a per-region mask, and resolve conflicts between overlapping garment classes with a parsing prior that knows a collar sits above a bodice.
- Where it breaks down: thin structures and transparency. Straps, lace and chiffon have no hard boundary, so mask losses trained on crisp edges either dissolve them or invent a solid one.
5. Body pose and dense correspondence
This task locates the person: joint keypoints at minimum, ideally a dense map from image pixels to a body-surface parameterisation. Try-on, size guidance and pose-conditioned generation all consume it.
- Model class: keypoint heatmap networks for the cheap version, parametric body-model regressors when you need surface correspondence rather than sticks.
- How it works: predict per-joint heatmaps or regress body-model parameters, enforce skeletal constraints, then lift to a mesh or UV map that warping can index into.
- Where it breaks down: self-occlusion and cropped frames. Shopper selfies cut off at the thigh, arms fold across the torso, and seated poses are barely represented in the datasets everyone trains on.
6. Try-on warping and compositing
Try-on takes a garment image and a person image and renders one wearing the other. It is a shipped consumer surface now rather than a conference demo: Google's try-on in Search and Shopping renders apparel onto a shopper's uploaded photo with its Gemini image model, and the real-time image-processing approach built by Zeekit, now part of Walmart, runs inside Walmart's online fashion experience.
- Model class: flow-based garment warping plus a diffusion or adversarial compositor, converging on one conditioned diffusion model that takes garment, pose and mask together.
- How it works: estimate correspondence between the flat garment and the posed body, warp the garment into place, then inpaint seams, occlusion boundaries and contact shadows.
- Where it breaks down: physics. There is no representation of drape, weight or bending stiffness, so a heavy wool coat renders with the fall of a jersey — hence our argument for evaluating try-on vendors against explicit physics constraints, then measuring the return-rate effect.
7. Fabric and seam defect detection
Line-side inspection: slubs, holes, misprints, skipped stitches, colour banding across a roll. It sits in operations rather than digital, which changes who owns the data and how tolerant the process is of false alarms.
- Model class: patch-level anomaly detection or one-class models trained on defect-free material, with a small supervised classifier over the few fault types you can genuinely collect.
- How it works: tile the frame, score each tile by distance from the normal manifold, threshold, then aggregate hits to a roll coordinate an operator can find.
- Where it breaks down: imbalance and illumination. Defects are rare by definition, and a mill that swaps a lamp shifts your score distribution enough to invalidate the threshold you just calibrated.
8. Catalogue image QC and standardisation
The least glamorous item is often the highest-yield one: gate images before they reach the site. Background removal, crop and alignment, colour normalisation, rejection of frames that break shot rules.
- Model class: matting and segmentation for cutouts, plus small binary classifiers as rule gates — model facing forward, ghost-mannequin interior clean, care label out of frame.
- How it works: run the cutout, score each gate, and fail the asset back to the studio naming the rule it violated rather than a generic rejection.
- Where it breaks down: colour. Perceptual accuracy under mixed lighting is a measurement problem, not a modelling one; without a physical reference chart in frame you keep shipping the wrong blue.
How should you sequence these on a roadmap?
Detection and tagging go first: they reuse assets you already hold and have an owner in merchandising. Retrieval follows on the same labelled images. Parsing and pose are infrastructure — build them once, because try-on and generative imagery inherit their errors. Defect detection is a separate programme with different data owners. Image QC is the task most teams postpone and most regret postponing.
Two decisions matter more than model choice: where product text and attributes live, since the taxonomy that feeds tagging also feeds retrieval-augmented assistants and belongs to one pipeline rather than two; and how you evaluate, which should follow the metric the business already argues about, not the one your benchmark reports.
FAQ
Which computer vision task should a fashion team ship first?
Attribute tagging or detection. Both reuse a catalogue you already own, both have a clear internal owner, and neither needs a new consumer surface to prove value. Retrieval is the natural second step.
Do you need a fashion-specific model or is a general vision model enough?
General backbones transfer well for detection and tagging. Fashion-specific data matters most for parsing and try-on, where garment structure and fabric behaviour drive the output. Fine-tune a general encoder on your catalogue.
What breaks first when a fashion vision pipeline scales?
Data plumbing, not models. Taxonomy changes, supplier images arriving in new formats, and retraining that invalidates a stored embedding index cause far more incidents than accuracy regressions.
Is virtual try-on a single model?
No. It is a chain: segment the garment, locate the body, warp, composite. Quality is capped by the weakest stage, which is why a convincing demo on a hoodie tells you little about a tailored jacket.
How do you evaluate defect detection without labelled defects?
Train on defect-free material and treat faults as anomalies, then hold back a small curated set of real defects purely for threshold calibration. Report recall at a false-alarm rate the cutting room will tolerate.
Further reading
- Computer Vision for Fashion: A Systematic Review of Design Applications
- Harnessing the potential of computer vision and AI in fashion