Patch 1

[HtetSec-0ps]

"features": {
"ghcr.io/devcontainers/features/github-cli:1": {}
}

[HtetSec-0ps]

internal/featuredetection/feature_detection.go

[HtetSec-0ps]

Question / Problem Description
Hi everyone,

I am loving Marker for its incredible OCR and layout detection capabilities! However, I’ve run into a slight architectural bottleneck when trying to run the entire pipeline on a single local GPU setup alongside a local OpenAI-compatible LLM service (like LM Studio or Ollama for text correction).

During the process, Marker loads Surya, Texify, and other vision models into the VRAM. By the time it transitions to the LLM correction phase, these vision models are still occupying the VRAM entirely. This forces my local LLM to be completely offloaded to the CPU, making the text refinement phase extremely slow or leading to common OOM issues.

Since many single-GPU users face similar memory constraints, I wanted to humbly ask:

Is there already an existing workaround or a hidden configuration that allows Marker to dynamically unload/garbage-collect its vision models from the CUDA cache before initiating the LLM service API call? Or perhaps a way to run the LLM correction standalone? If so, I would deeply appreciate any guidance on how to achieve this!

If this isn't supported yet, could we consider adding a feature for this in future releases?
For example, either by explicitly invoking torch.cuda.empty_cache() before the LLM phase, or introducing a standalone post-processing command (e.g., allowing us to run the LLM refiner on already generated text.md and blocks.json).

I believe this would be incredibly helpful for the open-source community and users with budget/VRAM-limited hardware.

Thank you so much for your time and for maintaining this amazing project!