pdf2md-local

Offline PDF → Markdown on your own machine — text, tables, and LaTeX formulas — with one command:

pdf2md file paper.pdf -o out/            # academic paper (MinerU, formulas)
pdf2md file report.pdf -o out/ -p doc    # general doc (PyMuPDF4LLM, fast)
pdf2md dir papers/ -o out/ -p paper      # batch a directory
pdf2md doctor                            # health check

Four best-in-class open-source engines — PyMuPDF4LLM, Docling, MinerU, Nougat — behind a single CLI. Because they pin mutually-incompatible transformers versions, each runs in its own conda environment and the CLI dispatches to the right one automatically. See docs/01_architecture.md.

MinerU is the default and recommended engine — with no flags, pdf2md uses it (the paper pipeline: formulas + tables + layout). The others are situational: PyMuPDF4LLM (-p doc) for speed on plain documents, Docling (-t docling) for scanned tables/OCR, Nougat (-t nougat) as an English-only formula fallback.

This packages a proven, version-pinned stack (magic-pdf 1.3.12). The installer reproduces every fix documented in docs/02_pitfalls.md, so a fresh machine just works.

Prerequisites

Requirement	Notes
OS	Linux or Windows
conda	Miniconda / Miniforge on PATH (conda command works)
GPU	NVIDIA GPU + driver for the paper/MinerU pipeline (the doc pipeline is CPU-only). Use --device cpu for no GPU — MinerU still runs, just slowly
Disk	~30 GB for MinerU models (one-time download)
git	to clone this repo

Install

git clone <this-repo-url> pdf2md && cd pdf2md

Linux / macOS (from a shell with conda base active):

./install.sh                    # default: CUDA, PyPI
./install.sh --device cpu       # no GPU

Windows (from an Anaconda PowerShell Prompt, base active):

.\install.ps1
.\install.ps1 --torch-index-url https://download.pytorch.org/whl/cu124   # CUDA wheels

The installer:

1. creates 3 conda envs (pdf2md, pdf2md_mineru, pdf2md_nougat) with pinned deps,
2. patches the MinerU env (installer/apply_patches.py),
3. downloads MinerU models + writes ~/magic-pdf.json,
4. writes the env-map config,
5. installs the pdf2md command (pip install -e .),
6. installs the optional Claude Code skill,
7. runs pdf2md doctor.

Useful flags (forwarded to installer/setup_all.py --help):

Flag	Purpose
--device cuda|cpu	MinerU inference device
--pip-index URL	PyPI mirror (e.g. Tsinghua)
--torch-index-url URL	CUDA torch wheels (recommended on Windows GPU)
--hf-mirror	download models via hf-mirror.com
--skip-models / --skip-envs / --skip-skill	partial / re-runs
--force	recreate envs from scratch

China networks (faster)

./install.sh --pip-index https://pypi.tuna.tsinghua.edu.cn/simple --hf-mirror

Verify

pdf2md doctor                              # envs + config + magic-pdf.json
python tests/smoke_test.py some.pdf        # end-to-end on the fast pipeline

Troubleshooting

The three most likely first-install snags (the rest work out of the box):

1. magic-pdf[full]==1.3.12 won't resolve / pip errors

Most common on a fresh machine or Windows. Checklist:

• The MinerU env must be Python 3.12 (the installer enforces this).
• magic-pdf 1.3.x hosts a few wheels off-PyPI; the installer already passes --extra-index-url https://wheels.myhloli.com. If your network blocks it, add a PyPI mirror too: --pip-index https://pypi.tuna.tsinghua.edu.cn/simple.
• Re-run just the envs without redoing everything: ./install.sh --skip-models --skip-cli --skip-skill (add --force to rebuild).

2. Model download 404 / stalls (PDF-Extract-Kit-1.0, hantian/layoutreader)

The model repo names are pinned in installer/download_models.py. If HuggingFace returns 404 or times out:

• Use a mirror: ./install.sh --hf-mirror (routes via hf-mirror.com).
• If a repo id has since moved, cross-check the current name against the MinerU docs and edit the constants at the top of download_models.py.
• The download is ~30 GB and resumable — re-run ./install.sh --skip-envs to continue; already-fetched files are skipped.
• Verify afterwards: pdf2md doctor must show magic-pdf.json present.

3. Windows GPU: MinerU runs on CPU (slow) instead of the GPU

PyPI's torch is CPU-only on Windows, so a plain install silently lacks CUDA. Install the CUDA wheels explicitly (match your driver — cu124 or cu121):

.\install.ps1 --torch-index-url https://download.pytorch.org/whl/cu124

Confirm CUDA is visible inside the MinerU env:

& "$env:USERPROFILE\miniconda3\envs\pdf2md_mineru\python.exe" -c "import torch; print(torch.cuda.is_available())"

Expect True. If False, reinstall torch from the CUDA index (above) or set --device cpu to accept slow CPU inference.

More engine-specific failure modes (MinerU formula errors, Nougat's broken deps, Chinese-PDF PUA characters) are catalogued in docs/02_pitfalls.md.

Environments

Env	Python	Engine(s)	Key pin
pdf2md	3.12	PyMuPDF4LLM + Docling	—
pdf2md_mineru	3.12	MinerU (magic-pdf 1.3.12)	transformers==4.49.0
pdf2md_nougat	3.10	Nougat	transformers==4.37.2

Repo layout

src/pdf2md/            dispatcher CLI (stdlib) + config + postprocess
installer/             setup_all.py, create_envs, apply_patches, download_models
  requirements/        pinned deps per env (reference)
  templates/           magic-pdf.json template
install.sh / .ps1      bootstrap for each OS
skill/pdf2md/          Claude Code skill (optional)
docs/                  architecture, pitfalls, usage
tests/                 smoke test

Documentation

• Architecture — why three environments
• Pitfalls — the failures already solved
• Usage — decision tree, batch, OCR, post-processing

Licensing & attribution

This project's own code (the dispatcher, installer, docs) is MIT (see LICENSE). It installs, but does not redistribute, third-party engines, each under its own license — PyMuPDF4LLM (AGPL/commercial), Docling (MIT), MinerU/magic-pdf (AGPL-3.0), Nougat (MIT), and their models. The patches in installer/apply_patches.py are applied to your local install; no third-party source is vendored here. Review each engine's license before any commercial or redistributive use.