ControlNet Review (2026): Structural Control for Stable Diffusion

ControlNet works by duplicating the encoding layers of a pretrained diffusion model and connecting the copy to the original network through what the paper calls "zero convolution" layers — convolution layers whose weights start at zero, so training begins from the base model's exact original behavior and only gradually learns the new conditioning signal. The original model's weights stay locked (frozen) during this process, which is why adding ControlNet does not degrade the base Stable Diffusion model's image quality.

Practically, this means you feed the pipeline two things instead of one: a text prompt (as usual) and a control image — a pose skeleton, a depth map, an edge outline, or a rough scribble. The output follows the composition and structure of the control image while still incorporating the prompt's style and subject description.

ControlNet is not a finished application. The paper describes an architecture, and the reference GitHub repository provides code and pretrained model weights, but there is no ControlNet installer you double-click. To actually use it, you install it as an extension or built-in feature inside a Stable Diffusion interface you already run — see How to Install and Use ControlNet below.

The lead author, Lvmin Zhang, is a researcher affiliated with Stanford University (with earlier study at the Chinese University of Hong Kong and Soochow University) who publishes under the GitHub handle lllyasviel. The same person also created Fooocus, a simplified Stable Diffusion interface built around fewer clicks to a first image — a connection worth knowing if you use both projects, since design decisions in one sometimes carry over to the other.

ControlNet's code and first pretrained weights were published on GitHub in February 2023, alongside the arXiv preprint. A more refined "ControlNet 1.1" release followed on April 14, 2023, expanding the set of supported conditioning types and improving robustness across them. The paper was formally published at ICCV 2023 and received the conference's Marr Prize, confirmed on the Computer Vision Foundation's official awards page.

ControlNet was built around, and first demonstrated on, the original Stable Diffusion 1.5 architecture. The open-source community — and later Stability AI itself — subsequently trained additional ControlNet checkpoints for newer base models, including Stable Diffusion XL and Stable Diffusion 3.5, extending the same conditioning technique to each new model generation. See Stable Diffusion Review for background on those base models.

ControlNet 1.1 also includes additional specialized models beyond this core set, such as M-LSD (straight-line detection for architecture and interiors), Shuffle (content-preserving recomposition), Inpaint, and Tile (detail enhancement for upscaling workflows). Each conditioning type is a separate model file — you load the one matching your control image, not a single universal model.

ControlNet setup differs by UI because it is not a standalone installer. The two most common paths are outlined below.

This is the single most important fact to check before using ControlNet in a commercial product: the license is not one number. The reference implementation's code is Apache License 2.0, an unrestricted permissive license. But the model checkpoint you download is a separate file with its own license, and that license generally reflects the license family of the Stable Diffusion base model the checkpoint was trained against.

The original ControlNet checkpoints for Stable Diffusion 1.5, published under the lllyasviel account on Hugging Face, are listed as "openrail" — the same CreativeML OpenRAIL-M license family used by SD 1.5 itself, which permits commercial use subject to use-based content restrictions rather than a revenue cap.

Community-trained SDXL ControlNet checkpoints, such as the widely used models published under the xinsir Hugging Face account, are commonly released under the plain Apache License 2.0 — check the specific model card, since different community authors can choose different terms for their own checkpoints.

Stability AI's own official ControlNets for Stable Diffusion 3.5 Large — covering Blur, Canny, and Depth conditioning, released November 26, 2024 — carry the Stability AI Community License, the same revenue-gated license used by SD 3.5 itself: free for individuals and organizations under $1,000,000 in aggregate annual revenue with registration, and an Enterprise license required above that threshold.

ControlNet adds a duplicated set of encoding layers on top of whichever Stable Diffusion model you are running, so it increases VRAM use beyond the base model's own requirement rather than replacing it.

Community reports and UI documentation generally describe roughly 1–2 GB of additional VRAM for a single ControlNet model on Stable Diffusion 1.5, and a larger increase — commonly cited around 2–4 GB per model — on SDXL, since SDXL's encoder layers are larger to duplicate. Using multiple ControlNet models simultaneously (for example, depth plus pose plus Canny at once) stacks this overhead further. Lighter alternatives such as T2I-Adapter (see the alternatives section below) are designed to add substantially less VRAM overhead than ControlNet, at the cost of somewhat less precise control in some community comparisons.

Because these figures come from community benchmarking rather than a controlled test PromptQuorum ran itself, treat them as planning ranges: confirm current VRAM behavior in your specific UI's documentation or issue tracker before assuming a borderline GPU will handle a given combination of base model and ControlNet.

ControlNet has no separate price — it is free and open-weights, distributed as code and model checkpoints rather than as a product with its own subscription or license fee. There is no PromptQuorum-tracked paid tier, membership, or hosted ControlNet-as-a-service offering from the original research team.

Your actual cost is the Stable Diffusion UI you already run (AUTOMATIC1111, ComfyUI, and InvokeAI are all themselves free and open source) plus your own GPU hardware and electricity. The only recurring cost that could apply is whatever hosted or cloud GPU service you might choose to run that UI on, which is unrelated to ControlNet itself.

ControlNet, T2I-Adapter, and IP-Adapter are not mutually exclusive — many workflows combine ControlNet (for pose or composition) with IP-Adapter (for style or character consistency) in the same generation. Which combination makes sense depends on what you are trying to fix: structure (ControlNet, T2I-Adapter) versus appearance (IP-Adapter).

For the underlying Stable Diffusion model choice these techniques attach to, see Stable Diffusion Review. For a broader comparison of local versus cloud image generation overall, see Local AI Image Generation vs. Cloud.

On the UI side, ComfyUI has native ControlNet nodes with no separate extension needed; AUTOMATIC1111 needs the community sd-webui-controlnet extension; InvokeAI integrates it as built-in Control Layers on its canvas; and Fooocus — built by the same researcher who created ControlNet — includes its own simplified, ControlNet-based structural-guidance features rather than exposing raw ControlNet model selection.

ControlNet earns its place as the standard way to add precise structural control to Stable Diffusion — not as a product in its own right, but as a well-documented research technique with a large ecosystem of checkpoints built around it since its ICCV 2023 debut. Readers who already run AUTOMATIC1111, ComfyUI, or InvokeAI can add pose, depth, edge, or scribble conditioning without switching platforms, and the underlying code carries an unrestricted Apache-2.0 license. The trade-off is real: it is not an app, it adds a genuine setup step and meaningful VRAM overhead on top of the base model, and licensing is not a single answer — it depends on which checkpoint you download, from "openrail" on the oldest SD 1.5 models to the revenue-gated Stability AI Community License on Stability's own SD 3.5 ControlNets. Readers who want lighter VRAM overhead for similar structural control should also evaluate T2I-Adapter; readers who have not yet set up a Stable Diffusion UI should start there first, since ControlNet has nothing to attach to on its own. For anyone who already generates images locally and needs more control than prompt wording provides, ControlNet remains the most thoroughly supported way to get it.