httk-workflow

httk-workflow is the filesystem-native workflow engine for httk₂.

It provides the httk.workflow Python API in three layers — the language-neutral filesystem protocol (httk.workflow.protocol), the execution and authoring surface (httk.workflow — Runner, Attempt), and orchestration and management (Workspace, TaskManager, harvest, and named submodules) — and registers httk workflow, the canonical command tree for everything below. The httk-taskmanager and httk-v1-taskmanager executables remain installed as aliases of that tree; the second one prepares and executes legacy ht_steps/ht_run task templates through the same httk₂ workspace. Jobs communicate through atomically published filesystem state, so interrupted managers and calculations can be recovered without cleanup hooks.

From nothing to a finished VASP relaxation, without writing a runner:

httk workflow workspace init workflow-workspace --extension transactional-data-v1
httk workflow job new workflow-workspace --template vasp-relax --from POSCAR --tag silicon
export HTTK_VASP_COMMAND="$PWD/examples/mock_vasp.py"   # or: srun -n 32 vasp_std
httk workflow manager run workflow-workspace --until-idle
httk workflow harvest workflow-workspace

docs/quickstart.md explains each command, and examples/quickstart.sh runs the whole sequence — with the mock VASP above standing in for VASP on a machine that has none.

Install

python -m pip install httk-workflow

One optional extra exists. httk-workflow[cwl] adds the CWL parser needed to run httk workflow import cwl; executing what was imported needs nothing extra, so the extra belongs only on the machine that does the importing. Importing Python Workflow Definition documents needs no extra at all.

Running tests

The everyday regression gate is the normal profile: make test (or PYTHONPATH=src python -m pytest -q). Pytest-xdist runs it in parallel and the default marker selection omits only full-depth extended parameter cases. Profiled tests keep one test body and reduce their input scale in normal mode; they still exercise every property with representative inputs.

Run make test-extended at phase ends and in CI to select every parameter case at its current full depth. The underlying knob is HTTK_TEST_PROFILE=normal|extended; an explicit extended invocation is HTTK_TEST_PROFILE=extended PYTHONPATH=src python -m pytest -q -m "". make ci uses the same extended profile with fast-fail enabled. Tests whose process timing must remain comparable use xdist load groups, so they stay serial within their group while the rest of the suite runs in parallel.

What it does

• Runs workflows without a graph. A step decides at run time which children to spawn and which step runs next, so a two-step relaxation and a partitioned child campaign are the same engine — runners in Python or in Bash, with a normative parity table between the two.
• Recovers instead of cleaning up. One atomically renamed state marker per job is the source of truth, so an interrupted manager, node, or calculation is resumed from what is on disk. The protocol is specified in docs/workflow_filesystem_api.md.
• Ships complete VASP runners, so an ordinary relaxation or single point needs no runner written at all — see docs/vasp_runners.md.
• Runs workflows written elsewhere. Python Workflow Definition and CWL documents become ordinary jobs; see docs/importing_workflows.md.
• Reaches other machines. Versioned remote adapters send work to a cluster, start managers there, and fetch results back through crash-recoverable detached transfer.
• Manages projects and identity: XDG configuration, signed project manifests, and workspace policy — see docs/workflow_cli.md.
• Hands results to a data layer. harvest yields one record per stopped job; httk-workflow itself has no database dependency.
• Keeps httk v1 workflows running. ht_steps/ht_run task directories execute unchanged on this engine — see docs/v1_compatibility.md and the migration guide.

httk workflow project init . --name example
httk workflow project manifest create
httk workflow workspace status .