Attention in a garment attribute tagger scores pairs of image tokens and uses those scores to decide which regions each token reads from. The image is cut into a grid of fixed-size patches, every patch becomes a token, and the attribute head reads a representation already mixed across tokens by learned weights. Nothing in that mechanism knows what a collar is: the weights learn that tokens near the neckline carry the evidence that reduces collar-type loss.
Key takeaways
- Attention weights are computed over token pairs, not pixels, so tokenisation sets the ceiling on what a tagger can see.
- Per-attribute query tokens hand you localisation for free: each query's map shows which patches answered it.
- Fine texture and small trims fail below the patch grid, where the evidence is averaged away before attention runs.
- An attention map is a routing weight, not an explanation, and it can look right while a category prior drives the prediction.
- Layered and occluded garments need region assignment first, because a weighted sum cannot say which layer owns an attribute.
What does self-attention compute in an attribute tagger?
Each token is a linear projection of a flattened patch plus a positional embedding, projected again into a query, a key and a value. For a given query the layer takes the dot product against every key in the image, scales it, and softmaxes the score vector. The output for that token is the sum of all value vectors, weighted by that distribution.
The operation is content-based, not geometric: a query built from a sleeve patch scores highly against other sleeve patches wherever they sit, which is why one set of weights handles a flat shot and an on-body shot. Geometry enters only through the positional embedding, weak enough for texture similarity to overwrite.
Because softmax normalises across keys, attention is zero-sum per query: mass spent on the model's face is mass taken from the cuff. Heads let different routings specialise instead of averaging. Early blocks stay local, on neighbouring patches and colour transitions; later blocks route across the whole garment, which is where a relation such as "this placket implies a button-through front" can form.
Which tokens hold the answer to "collar type"?
The common setup pools the sequence into one vector, a class token or a mean over patch tokens, and hangs small classifiers off it, one per attribute group. It trains easily and it is opaque: every attribute reads the same pooled vector, so a neckline head and a hem head compete for capacity in one representation.
The alternative gives each attribute its own learned query, cross-attending to the patch tokens. Collar type queries the image, sleeve length queries the image, and each returns its own attended vector for its head to classify. What you gain:
- A weakly supervised localisation per attribute. You never labelled a collar box, but the collar query's mass lands on the neckline when training went well.
- No shared bottleneck, so a rare attribute is not squeezed by a frequent one.
- A debugging surface: a wrong answer from the right region is a class-boundary problem, from the wrong region a data problem.
The cost is a decoder pass, a longer schedule, and queries for rarely-present attributes collapsing onto whatever region is always there.
How does tokenisation limit what can be tagged?
Patching is lossy compression applied before any attention runs, and it is where garment taggers quietly fail.
- Fine texture. Rib against interlock, twill against plain weave, brushed against flat: sub-patch properties. The projection keeps dominant colour and a coarse gradient and discards the periodicity separating the weaves. Raise resolution and the distinction returns, but attention cost grows quadratically in token count.
- Small trims. A zip pull, a rivet, a stitched buttonhole: each often smaller than a patch and sitting inside a patch dominated by fabric. No attention weight recovers what the tokeniser threw away.
- Aspect ratio. Squashing a full-length coat into a square input compresses vertical structure, exactly the axis carrying hem length and layering order.
Windowed and hierarchical attention buys back resolution at fixed cost, at the price of a restricted receptive field early on. The pragmatic pattern is a staged read: tag globally at moderate resolution, then re-run a crop at native resolution for the attributes that need it. Trims and texture behave like a detection problem wearing a classification label.
Why does the attention map look right when the label is wrong?
Because the map shows where information was routed, not why the decision came out as it did. Attention weights sit inside a product with value vectors, residual streams and an MLP; rollout and gradient-weighted variants try to correct for that and remain approximations. Treat maps as triage, not evidence.
- Co-occurrence shortcuts. If tailored coats in your catalogue are almost always long-sleeved, sleeve length becomes predictable from category, and the model takes that route because it is cheaper than resolving the arm. Held-out accuracy looks fine until a sleeveless coat ships.
- Taxonomy noise. Labels come from merchandising, where the same neckline acquires different names across teams and seasons. The tagger learns the annotator, not the garment.
- Studio bias. Lighting, crop and pose become part of the representation, so supplier imagery moves the distribution and attention drifts to background.
What breaks on layered garments and occlusion?
Occlusion is the clean case. If a collar sits under a scarf, its tokens carry no evidence and the attended vector is filled in from context. The model does not abstain; it returns the most likely collar for that silhouette, with ordinary confidence. Unless you train an explicit "not visible" value and give the head a route to it, an occluded attribute is silently imputed.
Layering is worse, because the evidence is present and the assignment is ambiguous. A jacket over a shirt over a tee puts several necklines, more than one hem and a stack of sleeve terminations into one frame. A pooled vector cannot say which garment owns which, and per-attribute queries bind to a garment only if something in the input separates the garments first.
The survey literature sorts fashion computer vision into four task families, detection, analysis, synthesis and recommendation, and states the ordering plainly: most fashion tasks need detection to run first, virtual try-on included. For a tagger that means parse-then-tag is the honest architecture: segment into garment instances or parts, then predict attributes per region, with masks either cropping the input or biasing the attention scores directly.
How do you get this into a product pipeline?
Commercial tagging usually arrives as a module inside a broader retail platform rather than as a standalone model. Vue.ai is an example of that shape: product tagging sits alongside automated on-model imagery, virtual dressing rooms, personalised storefront journeys, demand forecasting and assortment visualisation in a composable enterprise platform. If you build rather than buy, most of the work is not the model.
- Calibrate per attribute. The same confidence value means different things for a neckline and a fabric finish, and a global threshold over-tags easy fields while under-tagging hard ones.
- Make abstention a first-class output. "Not visible" is cheaper than a wrong value written into a record downstream systems trust.
- Watch drift per attribute value: a new trim appears mid-season, the aggregate barely moves, and one value quietly collapses.
Whether the pipeline can write back into your product records is a systems question, which our comparison of PLM systems on AI-readiness for fashion product teams takes apart. And our checklist of signals that a vendor's AI is genuinely trained on domain data is the faster test of a fashion-specific claim.
What is still unsolved?
Fine-grained texture recognition from consumer-grade photographs, and compositional binding: telling a model reliably that this sleeve belongs to that garment. General representation work from research groups such as Meta AI Research, whose current programme centres on advanced AI systems and autonomous agents, reaches fashion tagging long after publication, and the drape and cloth-simulation side surfaces first on the cs.GR preprint feed.
FAQ
Is an attention map the same as a saliency map? No. Saliency estimates which input pixels changed the output; attention records where a query read from inside a layer. When they disagree on a garment photo, the attention map is the less trustworthy signal.
Do I need a vision transformer to tag garment attributes? No. Convolutional backbones stay competitive on clean catalogue imagery. Attention earns its cost when you need per-attribute localisation, relations across a whole garment, or a backbone shared with a text encoder.
Why does my tagger insist a coat has long sleeves when it does not? Almost always co-occurrence in the training catalogue: category predicts the attribute more cheaply than the arm region does, so the model learns the shortcut. Fix it with sampling, not a bigger backbone.
Can attention maps serve as segmentation labels? As weak supervision, sometimes; as ground truth, no. Maps localise the region a query trusted, not the garment boundary, and they degrade on layered outfits.
Further reading
- Computer Vision for Fashion: A Systematic Review of Design Applications - where computer vision attaches to fashion design work, useful for placing attribute tagging among its neighbouring tasks.