Skip to content

Process-isolated container remains "Running" for exactly TcpTimedWaitDelay after all processes exit, if a loopback TCP connection was closed #644

Description

@mika-fischer

Describe the bug

A process-isolated Windows container whose workload opened and closed a loopback (127.0.0.1) TCP connection is not reported as stopped when its last process exits. Instead, the compute system remains in Running state for exactly the host's effective TcpTimedWaitDelay (default 120 seconds), measured from the moment the last loopback connection was closed. During this window no container processes exist (verified on the host with System Informer / Process Explorer), docker ps still shows the container as Running, docker events emits the die event only at the end of the window, and docker exec into the container hangs and eventually fails with container ... is not running.

The closed connection's TIME_WAIT TCBs are kernel-owned (not attributable to any process) and appear to pin the container's network compartment, which blocks silo/compute-system teardown until they expire. The delay scales linearly with TcpTimedWaitDelay (120 → ~120 s, 30 → ~30 s), which confirms the mechanism. Hyper-V-isolated containers are unaffected. The container engine version is irrelevant (reproduced identically on Docker Engine 25.0.4 and 29.5.2); the wait happens below the engine, in HCS.

Practical impact: any container workload that talks to a localhost helper daemon — compiler caches like sccache, local package proxies, etc. — silently adds up to 2 minutes to every container run. In our CI, every Windows build job carried an invisible, exactly-120-second gap at the end of the build step. Nothing in user space is observable during the window, which makes this extremely hard to diagnose.

To Reproduce

Steps to reproduce the behavior:

  1. On a Windows 11 25H2 host (build 10.0.26200.8875) with Docker and process isolation, run a container whose entrypoint opens and closes a single loopback TCP connection and then exits:
Measure-Command { docker run --rm --isolation=process mcr.microsoft.com/windows/servercore:ltsc2022 powershell -Command '$l=[Net.Sockets.TcpListener]::new([Net.IPAddress]::Loopback,0); $l.Start(); $c=[Net.Sockets.TcpClient]::new(); $c.Connect("127.0.0.1",$l.LocalEndpoint.Port); $s=$l.AcceptTcpClient(); $c.Close(); $s.Close(); $l.Stop(); "done"' }
  1. Run the identical control container with no TCP activity:
Measure-Command { docker run --rm --isolation=process mcr.microsoft.com/windows/servercore:ltsc2022 powershell -Command '"done"' }
  1. Compare wall-clock times. Optionally observe during the gap: docker ps (container Running), process list on the host (no container processes), docker events (die delayed), docker exec (hangs, then container is not running).

Measured results across two hosts on the same OS build:

Host Engine TcpTimedWaitDelay Isolation Repro (1 TCP conn) Control (no TCP)
A Docker 25.0.4 not set (default 120 s) process 128.7 s 10.7 s
B Docker 29.5.2 30 (0x1e) process 32.9 s 4.2 s
B Docker 29.5.2 30 hyperv 5.1 s 4.9 s

(StrictTimeWaitSeqCheck is not set on either host.)

Expected behavior

The container is reported as stopped promptly after its last process exits. TIME_WAIT drain of already-closed sockets should not gate compute-system teardown — for comparison, the equivalent Linux container reports its exit immediately (a lingering network namespace does not delay the container's exit status).

Configuration:

  • Edition: Windows 11 Pro 25H2, build 10.0.26200.8875 (two independent hosts)
  • Base Image being used: mcr.microsoft.com/windows/servercore:ltsc2022 (cross-version process isolation on a 26200 host; also reproduced with a custom image derived from it)
  • Container engine: docker (Docker Engine, Windows containers)
  • Container Engine version: 25.0.4 and 29.5.2 (identical behavior on both)

Additional context

  • The delay is anchored at the close of the last loopback TCP connection, not at process exit — if the connection closes earlier in the container's life, the visible tail delay shrinks accordingly.
  • Workaround: setting HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay to 30 (the minimum) on the host shortens the delay to ~30 s, but cannot eliminate it.
  • Real-world trigger: sccache (a compiler cache) runs a localhost TCP server inside the container; its final sccache --stop-server call closes the last connection right at the end of the CI script, so every CI build job paid the full 120 s.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriageNew and needs attention

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions