A text-to-image diffusion model never draws a garment. It starts from a grid of random numbers in a compressed latent space, predicts the noise inside that grid, subtracts part of it, and repeats until what remains decodes into a photograph of a jacket. Every lever you have sits in the autoencoder, the denoising backbone, the conditioning path or the sampler — here they are in the order the tensor moves through them.
Key takeaways
- A diffusion model builds a garment by iteratively removing predicted noise from a latent tensor; no stroke and no pattern piece are involved.
- The autoencoder sets your detail ceiling: rib knits, topstitching and lace often die in latent compression before the backbone sees them.
- Cross-attention is the only place a prompt touches the image, so it is the highest-leverage fine-tuning target.
- Low-rank adapters teach a frozen backbone a house silhouette without the forgetting of a full fine-tune.
- Nothing in the stack simulates cloth mechanics, so drape is a learned correlation rather than physics.
What is a latent diffusion model actually doing?
Training has a fixed part and a learned part. The fixed part is the forward process: encode a real image to a latent, then add Gaussian noise according to a schedule indexed by a timestep. Early in the schedule the latent is barely disturbed; at the end it is indistinguishable from noise. No parameters are involved, so training pairs are free.
The learned part is the reverse process. The network takes a noised latent and its timestep and predicts the noise that was added, usually under a mean squared error loss — a better conditioned target than the clean image itself. That is what makes sampling work: estimate the noise in a latent and you can estimate a cleaner one, then chain those estimates from pure noise down to something decodable.
Why does it run in a latent space instead of on pixels?
Pixel-space diffusion works and is punishingly expensive: attention cost grows with the square of the number of spatial positions, and you pay it at every step of every sample. Latent diffusion moves the loop into the output space of an autoencoder. An encoder maps the image to a tensor with far fewer spatial positions; a decoder trained with reconstruction, perceptual and adversarial losses maps it back.
For fashion, that compression quietly decides the outcome, because it discards what a merchandiser inspects: the periodicity of a rib, the density of a seam, the specular behaviour of a zip pull. Before training anything, run the autoencoder alone — encode and decode your own product photography with no diffusion in the loop. Whatever the round trip loses, no downstream fine-tune puts back.
How is the denoising U-Net built, level by level?
The down path is a stack of residual convolution blocks; each stage shrinks the spatial size and widens the channel dimension. The timestep becomes a sinusoidal embedding, passes through a small MLP, and enters every residual block as a scale-and-shift on normalised activations. That injection is why one set of weights behaves like a structure model early in sampling and a texture model late in it.
A bottleneck of residual and attention blocks sits at the smallest spatial size, and the up path mirrors the down path, concatenating a skip connection from the matching stage before convolving. Those skips are why U-Nets hold high-frequency detail: fine information bypasses the bottleneck.
Garment information sorts itself across the levels. Stages near the bottleneck carry pose, garment class and layout; stages near the input and output carry weave, print scale and hardware. Adapt the coarse stages only, then the fine stages only, to see where your training signal lands.
Where does the transformer part live, and how does text steer the image?
Attention blocks are interleaved with the residual blocks, only at the lower spatial resolutions where sequence length is affordable. Self-attention lets distant image positions agree with each other, which keeps both sleeves the same colour and both patch pockets the same size. Cross-attention takes queries from the image features and keys and values from token embeddings produced by a frozen text encoder. That is the single point where the prompt reaches the image; the rest is unconditional machinery.
Guidance is the other half. Conditioning is randomly dropped during training, so the same weights model the conditional and unconditional distributions; at sampling you run both branches and extrapolate away from the unconditional one. Push the guidance weight too high on garments and you get oversaturated fabric, laser-cut edges and collapsing silhouette variety.
Newer backbones drop the U-Net: patchify the latent into a token sequence, run a homogeneous stack of transformer blocks, and condition by modulating layer-norm parameters. Scaling is more predictable, but without skip connections more of the fine-detail burden falls on the autoencoder.
How does one sampling run proceed?
- Tokenise the prompt and run the frozen text encoder once; those embeddings stay constant.
- Draw a latent of Gaussian noise at the target shape.
- Choose a timestep schedule — the sampler's discretisation of the reverse process, and your steps-against-fidelity knob.
- At each timestep, run the backbone on the conditional and unconditional branches.
- Combine both predictions with the guidance weight.
- Apply the sampler's update rule, deterministic or stochastic, to get a cleaner latent; repeat.
- Decode the final latent. Refiner and upscaler passes come after.
Which architectural choices matter when fine-tuning on garment imagery?
Freeze the autoencoder unless you have both a large corpus and a reason; retraining it destabilises everything conditioned on its latent statistics. Freeze the text encoder too, and put low-rank adapters on the cross-attention projections, where house vocabulary lands. Rank is a capacity dial: too low and the model cannot hold your silhouette, too high and it memorises your studio background with it.
Caption schema matters more than backbone choice. Fix a controlled vocabulary for garment class, sleeve, neckline, closure and fabric, because cross-attention can only bind to tokens it has seen. Bucket by aspect ratio instead of cropping: a centre crop on a full-length coat removes the hem, where the drape signal lives. Then read failures back to components: detail loss is the autoencoder, category bleed is the coarse stages, trim on the wrong panel is cross-attention, plasticised fabric is guidance set too high.
Where do the hosted image models fit into this?
Most teams meet these architectures through a hosted product rather than a checkpoint. DALL-E, the image generation model inside OpenAI, and Adobe Firefly, Adobe's studio of generative image and video models offered to enterprises through APIs, expose generation and editing without exposing weights. Midjourney runs its own image and video platform on the same basis. Behind an API your levers are the prompt, reference images and editing operations: you skip the training infrastructure and give up binding your own vocabulary into cross-attention, the argument our piece on proprietary data as the moat makes at length. Cloth simulation work from groups such as NVIDIA Research, part of NVIDIA Corporation, comes at garment behaviour from the other side, by modelling it.
What is still unsolved here?
Drape is the honest gap. The model has no notion of grain line, bias, seam allowance or fabric weight, and reproduces only the correlations it saw in training images. Change the fabric in the prompt and you change a texture, not a mechanical response — the subject of our explainer on where fabric simulation physics ends and approximation begins. Identity across views is the next gap: nothing ties one sample to another, so a front and a back of the same jacket are independent generations sharing a prompt — the same blind spot our piece on the physics gap in virtual try-on describes.
Then correspondence. The image has no link to a cartamodello, a marker or a spec, so nothing can be cut from it. The commercial case for generative imagery in fashion was mapped out early — the assessment McKinsey published on generative AI in fashion dates to March 2023 — and architecture alone has not closed the distance between a convincing render and a producible garment.
FAQ
Is a diffusion model generating an image or reconstructing one?
Generating. The only thing it reconstructs is the noise it was asked to predict; chaining those estimates from a random start yields a sample from the learned distribution.
Do I need to fine-tune the whole U-Net to teach a model my brand's garments?
Usually not. Low-rank adapters on the cross-attention projections capture garment vocabulary cheaply, keep the base model's general competence, and stay small enough to version.
Why does fabric texture look wrong when the silhouette is right?
Silhouette lives in the coarse stages; texture lives in the fine stages and in the autoencoder. If the encode-decode round trip loses your weave, the backbone cannot recover it.
Can a generated image be used for production?
Not on its own. There is no geometric correspondence between the pixels and a pattern, so the image is a communication artefact rather than a manufacturing input.
Further reading
- Archinect, December 2018 — how couture converges with technology and architecture.
- Daily Bruin, June 2019 — an account of printing techniques in experimental fashion design.