Retry a command whose handler throws instead of terminal-failing it (cmd-queue 0.4.1)

[pditommaso]

Stacked on #100 — base is revert/stream-1.5.0-cmdqueue-0.4.0. Merge #100 first.

Recovers the fix originally released as 0.5.1 (6c7b171, #87), which #100's revert to 0.4.0 dropped. Same hunk, re-cut on the 0.4.0 tree, as @jordeu requested in their review of #100.

The bug this restores the fix for

CommandServiceImpl.processCommandWithHandler's catch treated any escaping exception as a terminal command outcome:

} catch (Exception e) {
    log.error("Command processing failed: id={}", msg.commandId(), e);
    store.save(state.failed(e.getMessage()));   // CommandStatus.FAILED — terminal
    return true;                                 // consume() then xacks + xdels
}

return true makes RedisMessageStream.consume:155 run xack + xdel, so the entry leaves both the stream and the PEL. No redelivery, ever.

That conflates "the handler threw" with "the command failed", and it fails asymmetrically because the two live in different stores — command state in Redis, the domain work in Postgres:

1. Micronaut closes the HikariCP pool while the queue is still draining.
2. The handler's execute()/checkStatus() throws a JDBC error.
3. The catch's Redis write succeeds — Redis is still up.
4. The entry is acked and deleted.

So the code records a permanent verdict using the store that still works, about a failure caused by the store that doesn't — while the domain entity was never transitioned. Queue empty, command FAILED, entity dangling, nothing left to advance it, polling clients hanging forever (seqeralabs/sched#712).

The fix

Log and return false. The entry stays unacked in the PEL for XAUTOCLAIM to hand to a live consumer, and since the catch never persists started(), status stays SUBMITTED so the next delivery re-enters execute(). A genuine failure is signalled by returning a FAILED CommandResult, which the terminal branch above already handles.

Independent of the async model #100 removed. This works on the synchronous 1.5.0 stream precisely because consume() only xack/xdels when the consumer returns true — no heartbeat-lease machinery involved.

Unchanged: a returned FAILED CommandResult is still terminal, and an unknown command type is still failed and acked at CommandServiceImpl:208 (no handler exists to retry). Both are outside this catch and covered by existing tests.

⚠️ Version collision with #101 — needs a decision

Both this PR and #101 are stacked on #100 and both currently claim 0.4.1. And sched's fix/ordered-shutdown-drain (f8f09293f, sched#888) already pins 0.4.1 expecting the drain.

Proposal: 0.4.1 = this fix (smaller, more urgent, one logical change), 0.4.2 = #101's drain, and sched#888 bumps its pin to 0.4.2. Happy to invert it instead — but the two cannot both be 0.4.1.

Worth noting they are complementary, not alternatives: the drain narrows the window in which a handler is cut short, but step 2 is deadline-bounded (CommandServiceImpl:128) and on expiry it closes the queue and returns false anyway (:138-144). Any handler still running then throws — into this catch. Drain narrows the window; this makes what's left recoverable. Neither is sufficient alone.

Also note there is no safety net behind this: sched#772 re-accepted the regression assuming ReconcileCron would re-drive stranded tasks, but 6d36bad0b ("re-drive PENDING tasks stranded by a lost command") is not reachable from sched's origin/master, and there is no ReconcileCron on master at all.

Not included

#94's api-scope fix — CommandQueue publicly extends AbstractMessageStream and exposes MessageConsumer, but #100 restores implementation project(':lib-data-stream-redis'), breaking downstream subclassers in Gradle composite builds. Out of scope here (this PR is #87 only); it needs its own hunk somewhere in the 0.4.x line.

Release marker

No [release] in the title, matching #100. But unlike #100, 0.4.1 is a new version — it will not reach the Maven repo, and sched cannot consume it, until a merge commit carries [release]. Add the marker when you actually want it published.

Verification

• New spec should retry a handler that throws instead of failing it terminally (the one from [release] Retry command on handler exception instead of terminal-failing it #87) — confirmed to fail against the 0.4.0 catch and pass with the fix, so it genuinely guards the behaviour.
• :lib-cmd-queue-redis:test — 16 tests, 0 failures, on a forced --rerun.

🤖 Generated with Claude Code