Fix L3 worker child-exit hangs

[ndleslx]

Summary

• plumb forked child PIDs from Python Worker startup into C++ WorkerThread registration
• detect and reap child process exits while waiting for TASK_DONE/CONTROL_DONE, raising a clear RuntimeError instead of spinning forever
• make Worker.close tolerate children already reaped by the liveness check

Motivation

Fixes the L3 Worker hang from #980 where a chip child could exit before publishing TASK_DONE, leaving the parent polling the mailbox indefinitely and the child as .

Verification

• rm -rf build && pip install -v .
• task-submit --device 4 --max-time 0 --run "cd /data/liuxu/pypto/runtime && PTO_ISA_ROOT=/data/liuxu/pto-isa PTO2_RING_HEAP=536870912 PTO2_RING_TASK_WINDOW=131072 PTO2_RING_DEP_POOL=131072 python examples/workers/l3/child_memory/main.py -p a2a3 -d 4"
  • golden + reused-worker multi-run checks PASSED
• synthetic sub-worker os._exit(0) repro now raises: WorkerThread::dispatch_process: child process pid=... exited before mailbox completion (exit_status=0)
• task-submit --device 4 --max-time 0 --run "cd /data/liuxu/pypto-serving && PTO2_RING_HEAP=2147483648 PTO2_RING_TASK_WINDOW=262144 PTO2_RING_DEP_POOL=262144 python examples/model/qwen3_14b/npu_generate.py --model-dir /data/linyifan/models/Qwen3-14B --prompt 'Huawei is' --platform a2a3 --device-id 4 --max-seq-len 512 --max-new-tokens 5"
  • text: a the the the the

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 32cb8592-b0eb-423a-bae4-42248f93f655

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces child process PID tracking and liveness checks to prevent the parent process from spin-polling indefinitely if a child worker dies. The review feedback highlights several critical improvements: replacing the Python 3.9-incompatible strict=True in zip() with manual length checks, improving the robustness of check_child_alive in C++ (such as handling EINTR and correcting child_reaped_ logic), and resetting the mailbox state to IDLE when exceptions are thrown during polling to prevent deadlocks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Since this project targets Python 3.9+, using strict=True in zip() is not supported and will raise a TypeError at runtime. To maintain compatibility while preventing silent truncation on sequences expected to be of equal length, we should explicitly verify their lengths are identical and raise a ValueError on mismatch before zipping.

Suggested change

	for shm, pid in zip(self._chip_shms, self._chip_pids, strict=True):
	dw.add_next_level_worker(_mailbox_addr(shm), pid)
	# Register Worker children as NEXT_LEVEL (L4+)
	for shm in self._next_level_shms:
	dw.add_next_level_worker(_mailbox_addr(shm))
	for shm, pid in zip(self._next_level_shms, self._next_level_pids, strict=True):
	dw.add_next_level_worker(_mailbox_addr(shm), pid)
	for shm in self._sub_shms:
	dw.add_sub_worker(_mailbox_addr(shm))
	for shm, pid in zip(self._sub_shms, self._sub_pids, strict=True):
	dw.add_sub_worker(_mailbox_addr(shm), pid)
	if len(self._chip_shms) != len(self._chip_pids):
	raise ValueError("Sequences must be of equal length")
	for shm, pid in zip(self._chip_shms, self._chip_pids):
	dw.add_next_level_worker(_mailbox_addr(shm), pid)
	# Register Worker children as NEXT_LEVEL (L4+)
	if len(self._next_level_shms) != len(self._next_level_pids):
	raise ValueError("Sequences must be of equal length")
	for shm, pid in zip(self._next_level_shms, self._next_level_pids):
	dw.add_next_level_worker(_mailbox_addr(shm), pid)
	if len(self._sub_shms) != len(self._sub_pids):
	raise ValueError("Sequences must be of equal length")
	for shm, pid in zip(self._sub_shms, self._sub_pids):
	dw.add_sub_worker(_mailbox_addr(shm), pid)

References

1. Before using zip on sequences that are expected to be of equal length, verify their lengths are identical and raise an error on mismatch to prevent silent truncation.

[gemini-code-assist]

There are a few robustness and correctness issues in check_child_alive:

1. If child_reaped_ is already true, the child process is dead. Any subsequent call should fail immediately with an error instead of returning silently, which would otherwise cause the parent to spin-poll or timeout on the dead mailbox.
2. waitpid can be interrupted by a signal and return -1 with errno == EINTR. We should retry waitpid in a loop when EINTR occurs to prevent transient failures from being treated as fatal errors.
3. child_reaped_ is set to true unconditionally before checking if waitpid actually reaped the child. It should only be set to true if the child was successfully reaped (i.e., r == child_pid_ or errno == ECHILD).

void WorkerThread::check_child_alive(const char *op_name) {
    if (child_pid_ <= 0) return;
    if (child_reaped_) {
        throw std::runtime_error(
            std::string(op_name) + ": child process pid=" + std::to_string(child_pid_) + 
            " is dead (already reaped)"
        );
    }

    int status = 0;
    pid_t r;
    do {
        r = waitpid(static_cast<pid_t>(child_pid_), &status, WNOHANG);
    } while (r < 0 && errno == EINTR);

    if (r == 0) return;

    if (r == static_cast<pid_t>(child_pid_)) {
        child_reaped_ = true;
        throw std::runtime_error(std::string(op_name) + ": " + child_status_message(child_pid_, status));
    }
    if (r < 0 && errno == ECHILD) {
        child_reaped_ = true;
        throw std::runtime_error(
            std::string(op_name) + ": child process pid=" + std::to_string(child_pid_) + 
            " is no longer waitable before mailbox completion"
        );
    }
    if (r < 0) {
        throw std::runtime_error(
            std::string(op_name) + ": waitpid(pid=" + std::to_string(child_pid_) + 
            ") failed: " + std::strerror(errno)
        );
    }
}

[gemini-code-assist]

When performing IPC or control commands that can fail in a child process, we must ensure the communication state (e.g., mailbox state) is reset to a safe or idle state before throwing an exception or unwinding to prevent deadlocks in subsequent operations. Additionally, we can avoid potential signed overflow and modulo overhead by using a simple counter that resets to 0.

    try {
        int poll_count = 0;
        while (read_mailbox_state() != MailboxState::TASK_DONE) {
            if (++poll_count >= 200) {
                poll_count = 0;
                check_child_alive("WorkerThread::dispatch_process");
            }
            std::this_thread::sleep_for(std::chrono::microseconds(50));
        }
    } catch (...) {
        write_mailbox_state(MailboxState::IDLE);
        throw;
    }

References

1. When performing IPC or control commands that can fail in a child process, always check the returned error code. Ensure the communication state (e.g., mailbox state) is reset to a safe or idle state before throwing an exception or unwinding to prevent deadlocks in subsequent operations.

[gemini-code-assist]

When performing IPC or control commands that can fail in a child process, we must ensure the communication state (e.g., mailbox state) is reset to a safe or idle state before throwing an exception or unwinding to prevent deadlocks in subsequent operations. Additionally, we can avoid potential signed overflow and modulo overhead by using a simple counter that resets to 0.

    try {
        int poll_count = 0;
        while (read_mailbox_state() != MailboxState::CONTROL_DONE) {
            if (++poll_count >= 200) {
                poll_count = 0;
                check_child_alive(op_name);
            }
            std::this_thread::sleep_for(std::chrono::microseconds(50));
        }
    } catch (...) {
        write_mailbox_state(MailboxState::IDLE);
        throw;
    }

References

1. When performing IPC or control commands that can fail in a child process, always check the returned error code. Ensure the communication state (e.g., mailbox state) is reset to a safe or idle state before throwing an exception or unwinding to prevent deadlocks in subsequent operations.