Git commit
$git rev-parse HEAD
9ef6e73
Operating System & Version
Ubuntu 22.04
GGML backends
CUDA
Command-line arguments used
/app/sd-cli -v --seed 42 --offload-to-cpu --vae-tiling --diffusion-fa --sampling-method euler --steps 4 --cfg-scale 1.0 --backend "diffusion=cuda0&cuda1,te=cuda0&cuda1" -p "a lovely cat" --diffusion-model /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf --vae /root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors --llm /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf -H 2048 -W 2048 --output output.png
Steps to reproduce
- Use stable-diffusion.cpp version master-766-9ef6e73.
- Run a generation command using the Flux 2 Klein 9b model with multi-GPU backends enabled (--backend "diffusion=cuda0&cuda1,te=cuda0&cuda1").
- Set a high resolution (-H 2048 -W 2048 in my example).
- Observe that the graph-cut layer split logic assigns all 201 tensors of the diffusion model to CUDA0 instead of partitioning them across both GPUs.
- Note the resulting error: CUDA error: out of memory or CUDA error: an illegal memory access was encountered.
What you expected to happen
The graph-cut mechanism should correctly partition the model tensors across the available CUDA devices, similar to how it functioned in master-765-bb84971.
In the working version (master-765), the logs show a successful split:
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA0 <- blocks [0, 6) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA1 <- blocks [6, 24)
In the regression version (master-766), the logs show the entire diffusion model being dumped onto the first device:
[INFO ] layer_split_partition.cpp:238 - flux graph-cut layer split: CUDA0 <- segments [0, 34), 201 tensors, 9516.0 MB
What actually happened
The graph-cut layer split logic fails to distribute the workload. Instead of partitioning the model, it assigns the entire diffusion model to a single segment on CUDA0.
Log from the regression:
[INFO ] layer_split_partition.cpp:238 - flux graph-cut layer split: CUDA0 <- segments [0, 34), 201 tensors, 9516.0 MB
Because all 201 tensors are dumped onto CUDA0, the system encounters a CUDA error: out of memory or an illegal memory access when attempting to process the high-resolution generation, as the single GPU cannot handle the amount of memory needed.
Logs / error messages / stack trace
Version 765:
/app/sd-cli -v --seed 42 --offload-to-cpu --stream-layers --vae-tiling --diffusion-fa --sampling-method euler --steps 4 --cfg-scale 1.0 --backend "diffusion=cuda0&cuda1,te=cuda0&cuda1" -p "a lovely cat" --diffusion-model /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf --vae /root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors --llm /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf --output output.png -H 1536 -W 1536
[DEBUG] main.cpp:625 - version: stable-diffusion.cpp version master-765-bb84971, commit bb84971
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 31968 MiB):
Device 0: NVIDIA RTX A4000, compute capability 8.6, VMM: yes, VRAM: 15984 MiB
Device 1: NVIDIA RTX A4000, compute capability 8.6, VMM: yes, VRAM: 15984 MiB
[DEBUG] main.cpp:626 - System Info:
SSE3 = 1 | SSSE3 = 1 | AVX = 1 | F16C = 1 | OPENMP = 1 | REPACK = 1 |
[DEBUG] main.cpp:627 - SDCliParams {
mode: img_gen,
output_path: "output.png",
image_path: "",
metadata_format: "text",
verbose: true,
color: false,
canny_preprocess: false,
convert_name: false,
preview_method: none,
preview_interval: 1,
preview_path: "preview.png",
preview_fps: 16,
taesd_preview: false,
preview_noisy: false,
imatrix_out: "",
metadata_raw: false,
metadata_brief: false,
metadata_all: false
}
[DEBUG] main.cpp:628 - SDContextParams {
n_threads: 16,
model_path: "",
clip_l_path: "",
clip_g_path: "",
clip_vision_path: "",
t5xxl_path: "",
llm_path: "/root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf",
llm_vision_path: "",
diffusion_model_path: "/root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf",
high_noise_diffusion_model_path: "",
uncond_diffusion_model_path: "",
embeddings_connectors_path: "",
vae_path: "/root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors",
vae_format: "auto",
audio_vae_path: "",
taesd_path: "",
esrgan_path: "",
control_net_path: "",
embedding_dir: "",
embeddings: {
}
wtype: NONE,
tensor_type_rules: "",
lora_model_dir: ".",
hires_upscalers_dir: "",
photo_maker_path: "",
rng_type: cuda,
sampler_rng_type: NONE,
offload_params_to_cpu: true,
max_vram: "0",
stream_layers: true,
eager_load: false,
backend: "diffusion=cuda0&cuda1,te=cuda0&cuda1",
params_backend: "",
split_mode: "",
model_args: "",
auto_fit: false,
enable_mmap: false,
control_net_cpu: false,
clip_on_cpu: false,
vae_on_cpu: false,
flash_attn: false,
diffusion_flash_attn: true,
diffusion_conv_direct: false,
vae_conv_direct: false,
prediction: NONE,
lora_apply_mode: auto,
force_sdxl_vae_conv_scale: false
}
[DEBUG] main.cpp:629 - SDGenerationParams {
loras: "{
}",
high_noise_loras: "{
}",
prompt: "a lovely cat",
negative_prompt: "",
clip_skip: -1,
width: 1536,
height: 1536,
batch_count: 1,
qwen_image_layers: 3,
init_image_path: "",
end_image_path: "",
mask_image_path: "",
control_image_path: "",
ref_image_paths: [],
control_video_path: "",
auto_resize_ref_image: true,
increase_ref_index: false,
pm_id_images_dir: "",
pm_id_embed_path: "",
pm_style_strength: 20,
skip_layers: [7, 8, 9],
sample_params: (txt_cfg: 1.00, img_cfg: 1.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: euler, sample_steps: 4, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
high_noise_skip_layers: [7, 8, 9],
high_noise_sample_params: (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: NONE, sample_steps: 20, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
custom_sigmas: [],
cache_mode: "",
cache_option: "",
cache: disabled (threshold=inf, start=0.15, end=0.95),
moe_boundary: 0.875,
video_frames: 1,
fps: 16,
vace_strength: 1,
strength: 0.75,
control_strength: 0.9,
seed: 42,
upscale_repeats: 1,
upscale_tile_size: 128,
hires: { enabled: false, upscaler: "Latent", model_path: "", scale: 2, target_width: 0, target_height: 0, steps: 0, denoising_strength: 0.7, custom_sigmas: [], upscale_tile_size: 128 },
vae_tiling_params: { 1, 0, 0, 0, 0.5, 0, 0, "" },
}
[DEBUG] model_loader.cpp:228 - using 16 threads for model loading
[INFO ] stable-diffusion.cpp:630 - loading diffusion model from '/root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf'
[INFO ] model_loader.cpp:236 - load /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf using gguf format
[DEBUG] model_loader.cpp:285 - init from '/root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf'
[INFO ] stable-diffusion.cpp:692 - loading llm from '/root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf'
[INFO ] model_loader.cpp:236 - load /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf using gguf format
[DEBUG] model_loader.cpp:285 - init from '/root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf'
[INFO ] stable-diffusion.cpp:706 - loading vae from '/root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors'
[INFO ] model_loader.cpp:239 - load /root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors using safetensors format
[DEBUG] model_loader.cpp:313 - init from '/root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors', prefix = 'vae.'
[INFO ] stable-diffusion.cpp:757 - Version: Flux.2 klein
[DEBUG] ggml_extend_backend.cpp:391 - Initializing backend: CUDA0
[DEBUG] ggml_extend_backend.cpp:391 - Initializing backend: CPU
[INFO ] stable-diffusion.cpp:808 - Weight type stat: f32: 395 | q8_0: 112 | q4_K: 217 | q6_K: 36 | i32: 1 | bf16: 89
[INFO ] stable-diffusion.cpp:809 - Conditioner weight type stat: f32: 145 | q4_K: 217 | q6_K: 36
[INFO ] stable-diffusion.cpp:810 - Diffusion model weight type stat: q8_0: 112 | bf16: 89
[INFO ] stable-diffusion.cpp:811 - VAE weight type stat: f32: 250 | i32: 1
[DEBUG] stable-diffusion.cpp:813 - ggml tensor size = 400 bytes
[DEBUG] qwen2_tokenizer.cpp:14 - merges size 151387
[DEBUG] qwen2_tokenizer.cpp:39 - vocab size: 151674
[DEBUG] llm.hpp:259 - llm: num_layers = 36, vocab_size = 151936, hidden_size = 4096, intermediate_size = 12288
[DEBUG] flux.hpp:182 - flux: depth = 8, depth_single_blocks = 24, guidance_embed = false, context_in_dim = 12288, hidden_size = 4096, num_heads = 32
[DEBUG] ggml_extend_backend.cpp:391 - Initializing backend: CUDA1
[INFO ] layer_split_partition.cpp:210 - Conditioner model layer split: CUDA0 <- blocks [0, 7) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Conditioner model layer split: CUDA1 <- blocks [7, 36)
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA0 <- blocks [0, 6) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA1 <- blocks [6, 24)
[WARN ] ggml_extend.hpp:3040 - flux: --stream-layers is not supported with multiple runtime backends; ignoring
[INFO ] stable-diffusion.cpp:1244 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 96
[INFO ] stable-diffusion.cpp:1362 - Using flash attention in the diffusion model
[DEBUG] stable-diffusion.cpp:1370 - validating model metadata
[DEBUG] stable-diffusion.cpp:1427 - model metadata validated; weights will be prepared lazily
[INFO ] stable-diffusion.cpp:1468 - total params memory size = 15977.60MB (VRAM 0.00MB, RAM 15977.60MB): text_encoders 6342.49MB(RAM), diffusion_model 9516.04MB(RAM), vae 119.08MB(RAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1583 - running in Flux FLOW mode
[INFO ] stable-diffusion.cpp:4966 - generate_image 1536x1536
[INFO ] denoiser.hpp:1083 - get_sigmas with Flux2 scheduler
[DEBUG] denoiser.hpp:789 - Flux2 scheduler: image_seq_len=9216, steps=4, mu=2.017
[INFO ] stable-diffusion.cpp:3844 - sampling using Euler method
[DEBUG] conditioner.hpp:1785 - parse '<|im_start|>user
a lovely cat<|im_end|>
<|im_start|>assistant
<think>
</think>
' to [['<|im_start|>user
', 1], ['a lovely cat', 1], ['<|im_end|>
<|im_start|>assistant
<think>
</think>
', 1], ]
[DEBUG] bpe_tokenizer.cpp:208 - split prompt "<|im_start|>user
" to tokens ["<|im_start|>", "user", "Ċ", ]
[DEBUG] bpe_tokenizer.cpp:208 - split prompt "a lovely cat" to tokens ["a", "Ġlovely", "Ġcat", ]
[DEBUG] bpe_tokenizer.cpp:208 - split prompt "<|im_end|>
<|im_start|>assistant
<think>
</think>
" to tokens ["<|im_end|>", "Ċ", "<|im_start|>", "assistant", "Ċ", "<think>", "ĊĊ", "</think>", "ĊĊ", ]
[DEBUG] model_loader.cpp:1006 - loading 298/398 tensors from /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf
|##################################################| 298/298 - 3.02GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 1.06s (read: 0.31s, memcpy: 0.00s, convert: 0.02s, copy_to_backend: 0.00s)
[DEBUG] model_manager.cpp:259 - model manager prepared params backend buffer (5316.84 MB, 298 tensors, RAM)
[DEBUG] model_manager.cpp:355 - model manager staged compute params (3165.76 MB, 78 tensors) to CUDA0, taking 0.32s
[DEBUG] model_manager.cpp:355 - model manager staged compute params (2151.08 MB, 220 tensors) to CUDA1, taking 0.38s
[DEBUG] model_manager.cpp:772 - model manager releasing compute params (3165.76 MB, 78 tensors) from CUDA0
[DEBUG] model_manager.cpp:772 - model manager releasing compute params (2151.08 MB, 220 tensors) from CUDA1
[DEBUG] conditioner.hpp:2281 - computing condition graph completed, taking 8968 ms
[INFO ] stable-diffusion.cpp:4664 - get_learned_condition completed, taking 8.97s
[INFO ] stable-diffusion.cpp:5013 - generating image: 1/1 - seed 42
[DEBUG] model_loader.cpp:1006 - loading 201/201 tensors from /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf
|##################################################| 201/201 - 9.15GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 1.02s (read: 0.91s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[DEBUG] model_manager.cpp:259 - model manager prepared params backend buffer (9516.04 MB, 201 tensors, RAM)
[DEBUG] model_manager.cpp:355 - model manager staged compute params (4654.02 MB, 105 tensors) to CUDA0, taking 0.42s
[DEBUG] model_manager.cpp:355 - model manager staged compute params (4862.02 MB, 96 tensors) to CUDA1, taking 0.84s
|==================================================| 4/4 - 6.87s/it
[DEBUG] model_manager.cpp:772 - model manager releasing compute params (4654.02 MB, 105 tensors) from CUDA0
[DEBUG] model_manager.cpp:772 - model manager releasing compute params (4862.02 MB, 96 tensors) from CUDA1
[INFO ] stable-diffusion.cpp:5045 - sampling completed, taking 43.32s
[INFO ] stable-diffusion.cpp:5057 - generating 1 latent images completed, taking 43.32s
[INFO ] stable-diffusion.cpp:4689 - decoding 1 latents
[DEBUG] vae.hpp:189 - VAE Tile size: 32x32
[DEBUG] ggml_extend.hpp:882 - num tiles : 5, 5
[DEBUG] ggml_extend.hpp:883 - optimal overlap : 0.500000, 0.500000 (targeting 0.500000)
[DEBUG] ggml_extend.hpp:884 - processing 25 tiles
[DEBUG] model_loader.cpp:1006 - loading 140/251 tensors from /root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors
|##################################################| 140/140 - 517.53MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.21s (read: 0.01s, memcpy: 0.00s, convert: 0.01s, copy_to_backend: 0.00s)
[DEBUG] model_manager.cpp:259 - model manager prepared params backend buffer ( 53.36 MB, 140 tensors, RAM)
[DEBUG] model_manager.cpp:355 - model manager staged compute params ( 53.36 MB, 140 tensors) to CUDA0, taking 0.02s
[DEBUG] ggml_extend.hpp:2150 - vae compute buffer size: 1248.50 MB(VRAM)
|==================================================| 25/25 - 3.85it/s
[DEBUG] vae.hpp:219 - computing vae decode graph completed, taking 7.76s
[INFO ] stable-diffusion.cpp:4737 - latent 1 decoded, taking 7.89s
[INFO ] stable-diffusion.cpp:4741 - decode_first_stage completed, taking 7.89s
[INFO ] stable-diffusion.cpp:5195 - generate_image completed in 60.69s
[DEBUG] model_manager.cpp:772 - model manager releasing compute params ( 53.36 MB, 140 tensors) from CUDA0
[DEBUG] model_manager.cpp:818 - model manager releasing params backend buffer (5316.84 MB, 298 tensors, RAM)
[DEBUG] model_manager.cpp:818 - model manager releasing params backend buffer (9516.04 MB, 201 tensors, RAM)
[DEBUG] model_manager.cpp:818 - model manager releasing params backend buffer ( 53.36 MB, 140 tensors, RAM)
[INFO ] main.cpp:490 - save result image 0 to 'output.png' (success)
[INFO ] main.cpp:562 - 1/1 images saved
Version 766:
/app/sd-cli -v --seed 42 --offload-to-cpu --stream-layers --vae-tiling --diffusion-fa --sampling-method euler --steps 4 --cfg-scale 1.0 --backend "diffusion=cuda0&cuda1,te=cuda0&cuda1" -p "a lovely cat" --diffusion-model /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf --vae /root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors --llm /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf --output output.png -H 1536 -W 1536
[DEBUG] main.cpp:625 - version: stable-diffusion.cpp version master-766-9ef6e73, commit 9ef6e73
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 31968 MiB):
Device 0: NVIDIA RTX A4000, compute capability 8.6, VMM: yes, VRAM: 15984 MiB
Device 1: NVIDIA RTX A4000, compute capability 8.6, VMM: yes, VRAM: 15984 MiB
[DEBUG] main.cpp:626 - System Info:
SSE3 = 1 | SSSE3 = 1 | AVX = 1 | F16C = 1 | OPENMP = 1 | REPACK = 1 |
[DEBUG] main.cpp:627 - SDCliParams {
mode: img_gen,
output_path: "output.png",
image_path: "",
metadata_format: "text",
verbose: true,
color: false,
canny_preprocess: false,
convert_name: false,
preview_method: none,
preview_interval: 1,
preview_path: "preview.png",
preview_fps: 16,
taesd_preview: false,
preview_noisy: false,
imatrix_out: "",
metadata_raw: false,
metadata_brief: false,
metadata_all: false
}
[DEBUG] main.cpp:628 - SDContextParams {
n_threads: 16,
model_path: "",
clip_l_path: "",
clip_g_path: "",
clip_vision_path: "",
t5xxl_path: "",
llm_path: "/root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf",
llm_vision_path: "",
diffusion_model_path: "/root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf",
high_noise_diffusion_model_path: "",
uncond_diffusion_model_path: "",
embeddings_connectors_path: "",
vae_path: "/root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors",
vae_format: "auto",
audio_vae_path: "",
taesd_path: "",
esrgan_path: "",
control_net_path: "",
embedding_dir: "",
embeddings: {
}
wtype: NONE,
tensor_type_rules: "",
lora_model_dir: ".",
hires_upscalers_dir: "",
photo_maker_path: "",
rng_type: cuda,
sampler_rng_type: NONE,
offload_params_to_cpu: true,
max_vram: "0",
stream_layers: true,
eager_load: false,
backend: "diffusion=cuda0&cuda1,te=cuda0&cuda1",
params_backend: "",
split_mode: "",
model_args: "",
auto_fit: false,
enable_mmap: false,
control_net_cpu: false,
clip_on_cpu: false,
vae_on_cpu: false,
flash_attn: false,
diffusion_flash_attn: true,
diffusion_conv_direct: false,
vae_conv_direct: false,
prediction: NONE,
lora_apply_mode: auto,
force_sdxl_vae_conv_scale: false
}
[DEBUG] main.cpp:629 - SDGenerationParams {
loras: "{
}",
high_noise_loras: "{
}",
prompt: "a lovely cat",
negative_prompt: "",
clip_skip: -1,
width: 1536,
height: 1536,
batch_count: 1,
qwen_image_layers: 3,
init_image_path: "",
end_image_path: "",
mask_image_path: "",
control_image_path: "",
ref_image_paths: [],
control_video_path: "",
auto_resize_ref_image: true,
increase_ref_index: false,
pm_id_images_dir: "",
pm_id_embed_path: "",
pm_style_strength: 20,
skip_layers: [7, 8, 9],
sample_params: (txt_cfg: 1.00, img_cfg: 1.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: euler, sample_steps: 4, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
high_noise_skip_layers: [7, 8, 9],
high_noise_sample_params: (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: NONE, sample_steps: 20, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
custom_sigmas: [],
cache_mode: "",
cache_option: "",
cache: disabled (threshold=inf, start=0.15, end=0.95),
moe_boundary: 0.875,
video_frames: 1,
fps: 16,
vace_strength: 1,
strength: 0.75,
control_strength: 0.9,
seed: 42,
upscale_repeats: 1,
upscale_tile_size: 128,
hires: { enabled: false, upscaler: "Latent", model_path: "", scale: 2, target_width: 0, target_height: 0, steps: 0, denoising_strength: 0.7, custom_sigmas: [], upscale_tile_size: 128 },
vae_tiling_params: { 1, 0, 0, 0, 0.5, 0, 0, "" },
}
[DEBUG] model_loader.cpp:228 - using 16 threads for model loading
[INFO ] stable-diffusion.cpp:627 - loading diffusion model from '/root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf'
[INFO ] model_loader.cpp:236 - load /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf using gguf format
[DEBUG] model_loader.cpp:285 - init from '/root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf'
[INFO ] stable-diffusion.cpp:689 - loading llm from '/root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf'
[INFO ] model_loader.cpp:236 - load /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf using gguf format
[DEBUG] model_loader.cpp:285 - init from '/root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf'
[INFO ] stable-diffusion.cpp:703 - loading vae from '/root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors'
[INFO ] model_loader.cpp:239 - load /root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors using safetensors format
[DEBUG] model_loader.cpp:313 - init from '/root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors', prefix = 'vae.'
[INFO ] stable-diffusion.cpp:754 - Version: Flux.2 klein
[DEBUG] ggml_extend_backend.cpp:391 - Initializing backend: CUDA0
[DEBUG] ggml_extend_backend.cpp:391 - Initializing backend: CPU
[INFO ] stable-diffusion.cpp:809 - Weight type stat: f32: 395 | q8_0: 112 | q4_K: 217 | q6_K: 36 | i32: 1 | bf16: 89
[INFO ] stable-diffusion.cpp:810 - Conditioner weight type stat: f32: 145 | q4_K: 217 | q6_K: 36
[INFO ] stable-diffusion.cpp:811 - Diffusion model weight type stat: q8_0: 112 | bf16: 89
[INFO ] stable-diffusion.cpp:812 - VAE weight type stat: f32: 250 | i32: 1
[DEBUG] stable-diffusion.cpp:814 - ggml tensor size = 400 bytes
[DEBUG] qwen2_tokenizer.cpp:14 - merges size 151387
[DEBUG] qwen2_tokenizer.cpp:39 - vocab size: 151674
[DEBUG] llm.hpp:259 - llm: num_layers = 36, vocab_size = 151936, hidden_size = 4096, intermediate_size = 12288
[DEBUG] flux.hpp:182 - flux: depth = 8, depth_single_blocks = 24, guidance_embed = false, context_in_dim = 12288, hidden_size = 4096, num_heads = 32
[DEBUG] ggml_extend_backend.cpp:391 - Initializing backend: CUDA1
[INFO ] stable-diffusion.cpp:482 - Conditioner model graph-cut layer split: deferring 398 tensors across 2 runtime backends until first graph
[WARN ] ggml_extend.hpp:3204 - flux: --stream-layers is not supported with multiple runtime backends; ignoring
[INFO ] stable-diffusion.cpp:482 - Diffusion model graph-cut layer split: deferring 201 tensors across 2 runtime backends until first graph
[INFO ] stable-diffusion.cpp:1245 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 96
[INFO ] stable-diffusion.cpp:1363 - Using flash attention in the diffusion model
[DEBUG] stable-diffusion.cpp:1371 - validating model metadata
[DEBUG] stable-diffusion.cpp:1428 - model metadata validated; weights will be prepared lazily
[INFO ] stable-diffusion.cpp:1469 - total params memory size = 15977.60MB (VRAM 0.00MB, RAM 15977.60MB): text_encoders 6342.49MB(RAM), diffusion_model 9516.04MB(RAM), vae 119.08MB(RAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1584 - running in Flux FLOW mode
[INFO ] stable-diffusion.cpp:4967 - generate_image 1536x1536
[INFO ] denoiser.hpp:1083 - get_sigmas with Flux2 scheduler
[DEBUG] denoiser.hpp:789 - Flux2 scheduler: image_seq_len=9216, steps=4, mu=2.017
[INFO ] stable-diffusion.cpp:3845 - sampling using Euler method
[DEBUG] conditioner.hpp:1903 - parse '<|im_start|>user
a lovely cat<|im_end|>
<|im_start|>assistant
<think>
</think>
' to [['<|im_start|>user
', 1], ['a lovely cat', 1], ['<|im_end|>
<|im_start|>assistant
<think>
</think>
', 1], ]
[DEBUG] bpe_tokenizer.cpp:208 - split prompt "<|im_start|>user
" to tokens ["<|im_start|>", "user", "Ċ", ]
[DEBUG] bpe_tokenizer.cpp:208 - split prompt "a lovely cat" to tokens ["a", "Ġlovely", "Ġcat", ]
[DEBUG] bpe_tokenizer.cpp:208 - split prompt "<|im_end|>
<|im_start|>assistant
<think>
</think>
" to tokens ["<|im_end|>", "Ċ", "<|im_start|>", "assistant", "Ċ", "<think>", "ĊĊ", "</think>", "ĊĊ", ]
[INFO ] ggml_graph_cut.cpp:938 - qwen3 build cached graph cut plan done (taking 1 ms)
[INFO ] layer_split_partition.cpp:238 - qwen3 graph-cut layer split: CUDA0 <- segments [0, 1), 1 tensors, 2374.0 MB
[INFO ] layer_split_partition.cpp:238 - qwen3 graph-cut layer split: CUDA1 <- segments [1, 28), 297 tensors, 2942.8 MB
[DEBUG] model_loader.cpp:1006 - loading 298/398 tensors from /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf
|##################################################| 298/298 - 1.81GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 1.77s (read: 0.80s, memcpy: 0.00s, convert: 0.03s, copy_to_backend: 0.00s)
[DEBUG] model_manager.cpp:261 - model manager prepared params backend buffer (5316.84 MB, 298 tensors, RAM)
[DEBUG] model_manager.cpp:357 - model manager staged compute params (2374.00 MB, 1 tensors) to CUDA0, taking 0.23s
[DEBUG] model_manager.cpp:357 - model manager staged compute params (2942.84 MB, 297 tensors) to CUDA1, taking 0.99s
[DEBUG] model_manager.cpp:774 - model manager releasing compute params (2374.00 MB, 1 tensors) from CUDA0
[DEBUG] model_manager.cpp:774 - model manager releasing compute params (2942.84 MB, 297 tensors) from CUDA1
[DEBUG] conditioner.hpp:2399 - computing condition graph completed, taking 8825 ms
[INFO ] stable-diffusion.cpp:4665 - get_learned_condition completed, taking 8.83s
[INFO ] stable-diffusion.cpp:5014 - generating image: 1/1 - seed 42
[INFO ] ggml_graph_cut.cpp:938 - flux build cached graph cut plan done (taking 7 ms)
[INFO ] layer_split_partition.cpp:238 - flux graph-cut layer split: CUDA0 <- segments [0, 5), 54 tensors, 2283.0 MB
[INFO ] layer_split_partition.cpp:238 - flux graph-cut layer split: CUDA1 <- segments [5, 34), 147 tensors, 7233.0 MB
[DEBUG] model_loader.cpp:1006 - loading 201/201 tensors from /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf
|##################################################| 201/201 - 10.57GB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.88s (read: 0.82s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[DEBUG] model_manager.cpp:261 - model manager prepared params backend buffer (9516.04 MB, 201 tensors, RAM)
[DEBUG] model_manager.cpp:357 - model manager staged compute params (2283.01 MB, 54 tensors) to CUDA0, taking 0.21s
[DEBUG] model_manager.cpp:357 - model manager staged compute params (7233.03 MB, 147 tensors) to CUDA1, taking 2.23s
[ERROR] ggml_extend.hpp:72 - CUDA error: out of memory
[ERROR] ggml_extend.hpp:72 - current device: 0, in function ggml_cuda_mul_mat_q at /src/stable-diffusion.cpp/ggml/src/ggml-cuda/mmq.cu:145
[ERROR] ggml_extend.hpp:72 - cudaGetLastError()
/src/stable-diffusion.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:104: CUDA error
/app/sd-cli(+0x1420256)[0x56fdd811c256]
/app/sd-cli(+0x142084a)[0x56fdd811c84a]
/app/sd-cli(+0x1420a25)[0x56fdd811ca25]
/app/sd-cli(+0x66b235)[0x56fdd7367235]
/app/sd-cli(+0x6ab76e)[0x56fdd73a776e]
/app/sd-cli(+0x685399)[0x56fdd7381399]
/app/sd-cli(+0x14429e1)[0x56fdd813e9e1]
/app/sd-cli(+0x144383b)[0x56fdd813f83b]
/app/sd-cli(+0x40a8ac)[0x56fdd71068ac]
/app/sd-cli(+0x26509f)[0x56fdd6f6109f]
/app/sd-cli(+0x422079)[0x56fdd711e079]
/app/sd-cli(+0x292a5c)[0x56fdd6f8ea5c]
/app/sd-cli(+0x4248fc)[0x56fdd71208fc]
/app/sd-cli(+0x427e0c)[0x56fdd7123e0c]
/app/sd-cli(+0x391213)[0x56fdd708d213]
/app/sd-cli(+0x269fe2)[0x56fdd6f65fe2]
/app/sd-cli(+0x7d593)[0x56fdd6d79593]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7f7e7f7991ca]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7f7e7f79928b]
/app/sd-cli(+0xf4d95)[0x56fdd6df0d95]
Additional context / environment details
No response
Git commit
$git rev-parse HEAD
9ef6e73
Operating System & Version
Ubuntu 22.04
GGML backends
CUDA
Command-line arguments used
/app/sd-cli -v --seed 42 --offload-to-cpu --vae-tiling --diffusion-fa --sampling-method euler --steps 4 --cfg-scale 1.0 --backend "diffusion=cuda0&cuda1,te=cuda0&cuda1" -p "a lovely cat" --diffusion-model /root/.cache/sd/models/diffusion_models/flux-2-klein-9b-Q8_0.gguf --vae /root/.cache/sd/models/vae/full_encoder_small_decoder.safetensors --llm /root/.cache/sd/models/text_encoders/Qwen_Qwen3-8B-Q4_K_M.gguf -H 2048 -W 2048 --output output.png
Steps to reproduce
What you expected to happen
The graph-cut mechanism should correctly partition the model tensors across the available CUDA devices, similar to how it functioned in master-765-bb84971.
In the working version (master-765), the logs show a successful split:
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA0 <- blocks [0, 6) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA1 <- blocks [6, 24)
In the regression version (master-766), the logs show the entire diffusion model being dumped onto the first device:
[INFO ] layer_split_partition.cpp:238 - flux graph-cut layer split: CUDA0 <- segments [0, 34), 201 tensors, 9516.0 MB
What actually happened
The graph-cut layer split logic fails to distribute the workload. Instead of partitioning the model, it assigns the entire diffusion model to a single segment on CUDA0.
Log from the regression:
[INFO ] layer_split_partition.cpp:238 - flux graph-cut layer split: CUDA0 <- segments [0, 34), 201 tensors, 9516.0 MB
Because all 201 tensors are dumped onto CUDA0, the system encounters a CUDA error: out of memory or an illegal memory access when attempting to process the high-resolution generation, as the single GPU cannot handle the amount of memory needed.
Logs / error messages / stack trace
Version 765:
Version 766:
Additional context / environment details
No response