Fine-tuning a pre-trained vision transformer on garment attributes is mostly a data-engineering job with a short training run bolted on the end. You need a label schema that matches how your catalogue is queried, a split that never puts one garment on both sides, an input pipeline that preserves the signal you are predicting, and per-class metrics instead of one accuracy number. This walkthrough runs from schema to a prediction written into the product record.
Key takeaways
- Label schema design sets your ceiling: attribute groups with an explicit not-visible class beat one flat multi-label soup.
- Split by product identifier, never by image, or the same garment shot from three angles will inflate every score you report.
- Augmentation is label-dependent: colour jitter destroys colour labels and horizontal flips destroy asymmetric closures.
- Judge the model per class and on retrieval behaviour, because tail-attribute performance is what merchandising actually feels.
- Write predictions with model version, score and threshold attached, or nobody downstream can tell a guess from a decision.
What you need
- A single GPU with enough memory for a base-size vision transformer at a moderate batch size. Gradient accumulation covers the rest.
- Python with a current PyTorch build,
timmortransformersfor backbones, a streaming dataset library,scikit-learnfor metrics, and an experiment tracker you will actually open. - A pre-trained image backbone. Supervised classification checkpoints work; self-supervised transformer checkpoints transfer better to fine-grained texture.
- Catalogue images with product identifiers attached and the shot type recorded: on-model, flat-lay, detail crop.
- Attribute values exported from your product data system with those identifiers, plus access to whoever owns that taxonomy.
- A gold set: a few hundred images relabelled carefully by two people, held out of everything else.
- A staging destination for predictions that is not your production catalogue.
Step 1: Which labels do you actually need?
Start from queries, not from ontology. Read your site filters, your search logs and the facets merchandising asks for by email. Those are your attribute groups: category, silhouette, neckline, sleeve length, closure, pattern, fabric appearance.
Model each group as its own head with mutually exclusive classes plus an explicit not-visible class. A flat multi-label vector over every attribute value is easier to assemble and worse at everything: it happily predicts two necklines at once, and gives the model no way to say the closure is hidden behind an arm.
Fashion vision is a stack of tasks rather than one classifier, and a systematic review of computer vision for fashion covers that span from design through retail. Decide which task your schema serves before anyone argues about class names.
One line per image, product identifier on every row:
{
"image": "catalogue/AB1234/on-model-01.jpg",
"product_id": "AB1234",
"shot_type": "on_model",
"labels": {
"category": "blouse",
"neckline": "v_neck",
"sleeve_length": "long",
"closure": "button",
"pattern": "solid"
},
"label_source": "product_data_export"
}
Expected result: a versioned schema file listing every group, its classes and counts, plus an export that validates against it.
Count examples per class before training. A class with fewer examples than your batch size is a class you will not learn. Merge it, or accept that it lives only in the gold set.
Step 2: How do you curate and split the dataset?
Three jobs, in order: de-duplicate, split, balance.
De-duplicate with embeddings rather than filenames. Encode everything with the frozen backbone and cluster near neighbours: catalogues are full of one shot re-cropped for different placements, and duplicates straddling a split are the quietest way to fake a good result.
Split by product identifier, so every image of one garment lands on one side of the wall. Hashing the identifier keeps the assignment stable as the catalogue grows:
import hashlib
def bucket(product_id, salt="attr-v1"):
digest = hashlib.sha1(f"{salt}:{product_id}".encode()).hexdigest()
return int(digest, 16) % 100
def split_of(product_id):
b = bucket(product_id)
if b < 80:
return "train"
return "val" if b < 90 else "test"
Balance by capping: cap head classes per group, keep every tail example, store the distribution in the run metadata. Oversampling and loss weighting come later, once you can see which classes the model misses.
Expected result: three manifests with no shared product identifier, and a distribution table you can diff between runs.
Step 3: How should the input pipeline treat garment images?
Detection comes before classification. A full-frame on-model shot is mostly background, skin and pose, and the garment may hold a third of the pixels. Crop to the garment region first: a survey of fashion and computer vision makes the same ordering explicit, noting that most fashion tasks need detection before anything else runs. A person detector plus a region crop works; segmentation works better and costs more.
Resize to the backbone's native resolution, then treat augmentation as label-dependent:
- Random resized crop is safe within limits. Crop away the neckline and your neckline label becomes noise.
- Horizontal flip breaks asymmetric closures and side details. Turn it off for those groups.
- Colour jitter and grayscale conversion destroy colour and print labels. If colour is in your schema, leave hue and saturation alone.
- Keep mild blur and compression noise if you will ever score user-generated photos.
image_size: 224
train_transforms:
- random_resized_crop: {scale_min: 0.7, scale_max: 1.0}
- horizontal_flip: false
- jpeg_noise: {quality_min: 60}
- normalize: dataset_mean_std
eval_transforms:
- resize_shortest: 256
- center_crop: 224
- normalize: dataset_mean_std
Expected result: a batch you have looked at — dump a grid of augmented crops and confirm the labelled features are still visible.
If you cannot read the label off the augmented image, the model cannot either.
Step 4: What does the training configuration look like?
Freeze, then thaw. Train the heads on frozen features for a short warm-up, so randomly initialised classifiers stop pushing garbage gradients into the backbone, then unfreeze the upper blocks with a layer-wise learning-rate decay. Full fine-tuning pays off only on a large dataset visually far from the pretraining distribution.
backbone: vit_base_patch16_224
heads:
category: {classes: 24, loss: cross_entropy}
neckline: {classes: 12, loss: cross_entropy}
sleeve_length: {classes: 6, loss: cross_entropy}
pattern: {classes: 18, loss: cross_entropy}
optimizer: adamw
lr_head: 1.0e-3
lr_backbone: 3.0e-5
layer_decay: 0.75
schedule: cosine
warmup_epochs: 1
epochs: 12
batch_size: 64
precision: bf16
label_smoothing: 0.1
ema: true
Use per-group cross-entropy, masked where the label is absent from the export rather than folded into the not-visible class: a missing value is not evidence that the feature is missing. Sum the group losses, weight the groups you care about, and log each group separately. An aggregate curve hides a head that never converges.
Backbone choice matters less than label quality, but it is not free: self-supervised checkpoints hold the fine-grained texture that fabric and print attributes are made of. For primary work rather than benchmark write-ups, NVIDIA Research publishes rendering and generative-AI papers alongside open-source libraries you can build on, and Meta AI Research, operating as Meta Superintelligence Labs, works on the advanced AI systems side of the field.
Expected result: per-group curves, a checkpoint, and a config file committed next to the run identifier.
Step 5: Which metrics tell you it works?
Accuracy per group is a vanity metric on a long-tailed catalogue: predict the majority class everywhere and it still looks respectable. Track these instead.
- Macro-F1 per group, with per-class precision and recall sorted by support ascending, so the tail is the first thing you read.
- Confusion pairs. Persistent confusion between crew and scoop necklines means your annotators disagreed: the schema is the bug.
- Calibration per group. You will threshold on confidence in production, so a reliability curve beats a leaderboard position.
- Retrieval behaviour. Build query sets from real filter combinations, retrieve against the predicted attributes, and measure how often every filter is satisfied. That ranking layer is the subject of our piece on how the recommendation engine at Zalando ranks garments at scale.
from sklearn.metrics import classification_report
for group in heads:
print(group)
print(classification_report(y_true[group], y_pred[group],
target_names=names[group],
digits=3, zero_division=0))
Expected result: a per-class report stored with the run, and a decision per group: write to the catalogue, or stay in review.
Step 6: How do you ship predictions into the catalogue?
Export the model, then choose thresholds per class on the validation split. One global threshold is a decision to be wrong on the tail. Run batch inference across the catalogue, write nothing directly, and land output in a staging table carrying score and model version. Compare against existing values in shadow mode before promoting anything: the disagreements are your real error analysis, and some will be catalogue data that was already wrong.
Every promoted attribute should carry the model identifier, its version, the score, the threshold applied and the reviewer if a human touched it. Without that, nobody downstream can separate a model's guess from a merchandiser's decision — the gap our reality check on fashion AI governance and deployment describes at sector scale.
If you would rather not run a training loop, retail AI platforms cover this as a service: Vue.ai ships product tagging, automated on-model imagery and catalogue visualisation inside an enterprise orchestration platform. The trade is less control over the schema and the error profile.
Expected result: a staging table, a review queue for low-confidence predictions, and a promotion job that writes provenance with every value.
What goes wrong, and how do you fix it?
- Validation macro-F1 falls while accuracy climbs. The model is collapsing onto head classes. Weight the loss by inverse support, cap the head classes harder, and check the sampler is not showing tail classes once per epoch.
- Offline scores are excellent, production output is bad. Look for leakage first: one product identifier on both sides of the split, or a duplicate crop. Then look at shot type, because a model trained on flat-lays does not read on-model photography.
- The model learned the studio. If category prediction tracks background colour, the crops are too loose. Tighten the detection step and test on images from a different photographer.
- Colour predictions look random. Check the augmentation config for hue or saturation jitter, then check whether the colour field in your export is a marketing name rather than a colour.
- One head never converges, or its predictions fight the gold set. Usually schema or labels: overlapping classes, a group where most values are missing so the mask eats the batch, or annotators who disagreed. Measure agreement before blaming the model.
- Domain shift on user-generated photos. Adapt with a small in-domain set rather than retraining, and expect your photography assumptions to matter — the subject of our comparison of four technical approaches to e-commerce imagery.
FAQ
How many labelled images per attribute class do I need? Enough that the class survives the split: a few hundred clean examples is a workable floor for fine-grained groups, fewer for coarse categories. Balance matters more than volume.
Should I train one multi-task model or one model per attribute? Start multi-task, with per-group heads on a shared backbone. It is cheaper to serve and shared features help correlated groups. Split a group out only when its data or update cadence genuinely differs.
Do I need a vision transformer, or will a CNN do? A CNN is fine for coarse categories and cheaper to serve. Transformer backbones tend to pay off on fine-grained texture, print and construction detail, particularly with self-supervised pretraining.
Can I train on synthetic or rendered garment images? As augmentation for rare classes, yes, with real images in every evaluation set. Rendered data helps silhouette and geometry; it will not teach fabric behaviour under studio lighting.
How often should I retrain? When the catalogue's class distribution drifts or a new attribute group appears, not on a calendar. Track per-class recall against a frozen gold set and let that trigger the decision.
Further reading
- Applications of Computer Vision in Fashion and Textiles — a reference volume whose later sections cover recognition, modelling, retrieval and human shape modelling, for when you want the underlying methods rather than a framework tutorial.