feat: hot-reload ControlNet - swap without rebuilding the context#1768
Merged
Conversation
Adds three new C API entry points on stable-diffusion.h so a long-lived process
(server / GUI / integration) can swap the ControlNet weights without paying the
cost of tearing down the ctx and reloading the base diffusion / VAE / text
encoders (multi-GB, tens of seconds):
SD_API bool sd_ctx_load_control_net(sd_ctx_t*, const char* path);
SD_API bool sd_ctx_unload_control_net(sd_ctx_t*);
SD_API bool sd_ctx_has_control_net(const sd_ctx_t*);
Implementation:
- StableDiffusionGGML gains load_control_net_from_file(path) and
unload_control_net() helpers. They reuse the existing ControlNet construction
path (backend pair, conv2d-direct setting, register_runner_params), so hot-
loaded ControlNets are indistinguishable at runtime from init-time ones.
- ModelManager gains unregister_param_tensors(desc) which releases both the
compute and params backend buffers for a runner's tensors and removes them
from tensor_states_by_name_ / tensor_states_, so a subsequent register with
the same tensor names does not collide.
- diffusion_conv_direct + control_net_params_mem_size are promoted from
local-scope in the initial-load path to StableDiffusionGGML members so that
the reload path can honor the ctx's original conv-direct setting and keep the
memory bookkeeping consistent.
- The three C API functions are thin wrappers over those helpers, matching the
existing sd_ctx_* pattern in stable-diffusion.h.
Doc caveats in the header note that this API is intended for use when no
generation is in flight — thread safety is the caller's responsibility, same as
the rest of the sd_ctx API surface.
Smoke test (create ctx w/o ControlNet, load canny, swap to depth, unload,
reload canny, free) exits cleanly with correct has_control_net transitions.
Existing --control-net init-time path is unchanged (regression-tested with
sd-cli).
Motivation is to unblock two follow-ups:
1. Servers can offer a ControlNet dropdown without a full model reload.
2. The same pattern generalizes to other module families (LoRA, VAE, upscaler);
the private helpers were factored with that extension in mind.
fszontagh
changed the title
Owner
|
Thank you for your contribution. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds three C API entry points so a long-lived process can swap the ControlNet weights on an existing
sd_ctx_twithout tearing it down:sd_ctx_load_control_netacts as both first-load and replace. Reuses the same construction path as--control-net, so hot-loaded ControlNets are indistinguishable at runtime.Why
Loading the diffusion model and text encoders takes tens of seconds. Switching ControlNet (canny to depth, etc.) shouldn't require rebuilding the ctx.
How
StableDiffusionGGMLgainsload_control_net_from_file(path)andunload_control_net()sharing the existing construction code.ModelManagergainsunregister_param_tensors(desc)so re-registering the same tensor names does not collide.diffusion_conv_directandcontrol_net_params_mem_sizepromoted from locals to members so the reload path honors the ctx's original settings.The helpers are shaped so the same pattern generalizes to other module families (VAE, LoRA, upscaler) later; only ControlNet is exposed here.
Caveats
sd_ctx_*.ControlNetctor.Verification
sd-cli --control-net ...init path unchanged.