There is a category of thing you cannot write down. “Make it look like this” — where this is a photograph, a painting, a frame from a film — is not a prompt, because the words you would need do not exist, and the ones that do exist (“cinematic,” “painterly,” “moody”) have been diluted into noise by millions of prompts.
IP-Adapter takes the image instead.
The one thing to understand first
IP-Adapter does not look at your reference image. It looks at an embedding of it.
The reference goes through a CLIP image encoder, which produces a vector describing the image’s content in the same kind of space the text encoder uses for your prompt. IP-Adapter’s trained projection layers turn that vector into extra conditioning that is injected into the model’s cross-attention — the same place your text prompt enters — through a decoupled attention path, so image and text conditioning do not fight for the same weights.
Three consequences follow directly, and they explain nearly every confusing result you will get:
- It transfers what the encoder perceives, not what you noticed. If you hand it a portrait intending to copy the lighting, you may get the face, the pose and the colour grade too, because all of that is in the embedding.
- It carries no geometry. There is no spatial correspondence between the reference and the output. For that you want ControlNet, and the two are routinely used together.
- No training is involved. Unlike a LoRA, nothing is fine-tuned. You are not teaching the model; you are conditioning one generation.
Why it exists alongside ControlNet and LoRA
| Tool | What it controls | Cost per new look |
|---|---|---|
| ControlNet | Structure — pose, depth, edges, composition | None; supply a control image |
| LoRA | A learned style or subject, deeply and consistently | Training run, dataset, time |
| IP-Adapter | Style or subject from one reference image | None; supply a reference |
The honest trade: a LoRA trained on forty images of your own paintings will reproduce your style more faithfully and more controllably than IP-Adapter ever will from one image. IP-Adapter’s advantage is that it costs you nothing and works on the image you found five minutes ago.
Getting the right files, which is where most people stall
The model repository is h94/IP-Adapter on Hugging Face, Apache-2.0, with around 1,450 likes. It contains several adapters, and picking the wrong one is the usual first failure.
The practical distinctions:
- Base model must match. Adapters for SD 1.5 do not work on SDXL and vice versa. The filenames say which.
_plusvariants use more image tokens and follow the reference more strongly. Start with plain, move topluswhen the effect is too weak._plus_face/ FaceID variants are specialised for identity, and need an additional face-analysis model (InsightFace). If you only want style, do not install these — they are the source of most dependency errors in this ecosystem.- The image encoder is a separate download. The adapter is useless without the matching CLIP vision model, and the
plusadapters generally expect the larger ViT-H encoder. A mismatch here produces either an error or silently terrible output.
In ComfyUI the adapters go in models/ipadapter/ and the vision encoder in models/clip_vision/.
The weight slider is the real control
Once it runs, nearly all of your tuning happens on one number.
- Low (around 0.3–0.5): the reference tints the result. Your prompt still drives the content. This is where style transfer usually wants to be.
- Middle (around 0.6–0.8): the reference asserts subject and composition. Your prompt becomes a modifier.
- High (0.9–1.0): you are approximately re-rendering the reference, and the prompt has little say.
Two further settings matter once you are past that. Start and end step confine the adapter’s influence to part of the denoising process — applying it only in later steps lets the prompt establish composition first and the reference supply surface and colour, which is often exactly what “style, not subject” means in practice. And on plus models, the weight type or embedding scaling options control whether the influence concentrates on style or on composition; this is the subject of the Latent Vision video above, by the author of the ComfyUI nodes.
A first workflow
- Load your checkpoint as normal.
- Load the CLIP vision encoder and the IP-Adapter model for your base.
- Load the reference image. Crop it to what you actually want. This is the highest-leverage step in the whole process — if you want the palette of a landscape and not its horizon, crop to a patch of sky and ground. The encoder sees the frame you give it.
- Apply the adapter to the model, weight 0.5.
- Write a prompt describing the content you want, not the style. The reference handles style.
- Generate, then move the weight in steps of 0.1 and regenerate before changing anything else.
Add ControlNet once that works, not before.
The maintenance situation, stated plainly
Both of the things you will install are stable to the point of being static. The original tencent-ailab/IP-Adapter repository (Apache-2.0, ~6,700 stars) was last pushed in June 2024. cubiq/ComfyUI_IPAdapter_plus (GPL-3.0, ~6,200 stars) was last pushed in April 2025.
For a tool this widely used, that is mostly a sign of completeness — the technique works and the implementation is finished. But it means two things for you: expect no support for architectures newer than the last release, and expect the newer unified style-and-subject approaches now appearing to eventually replace this workflow. Learn it anyway. The concept — conditioning on an image embedding alongside text — is the durable part, and it is the same idea underneath whatever supersedes it.
Sources
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models — arXiv:2308.06721
- h94/IP-Adapter model repository — Hugging Face
- tencent-ailab/IP-Adapter on GitHub
- cubiq/ComfyUI_IPAdapter_plus on GitHub
- Style and Composition with IPAdapter and ComfyUI — Latent Vision (YouTube)
- ComfyUI documentation