Skip to content

zulip: Reset session on ConnectionError to fix stale connections. - #922

Open
HeetRanpura wants to merge 1 commit into
zulip:mainfrom
HeetRanpura:fix-stale-connection-session-reset
Open

zulip: Reset session on ConnectionError to fix stale connections.#922
HeetRanpura wants to merge 1 commit into
zulip:mainfrom
HeetRanpura:fix-stale-connection-session-reset

Conversation

@HeetRanpura

Copy link
Copy Markdown
Collaborator

Overview

When a network middlebox silently drops an idle TCP connection,
requests.Session keeps that dead socket in its connection pool. The next
API call tries to reuse it and raises a ConnectionError. Previously the
retry loop in do_api_query caught this error but never closed the broken
session, so every retry attempt reused the same dead connection and failed.

Root Cause

ensure_session() was called once before the while True loop, so after a
ConnectionError the session was never rebuilt on retry. All 10 retries
attempted to send requests over the same stale TCP connection.

Fix

Two minimal changes to do_api_query:

  1. Move ensure_session() inside the while True loop so it runs on every
  2. iteration.
    1. On ConnectionError, close and null out self.session before retrying.
  3. The next iteration calls ensure_session(), sees None, and creates a
  4. fresh connection.
if self.session is not None:
    self.session.close()
self.session = None

The existing error_retry machinery is reused completely unchanged. No new
parameters, timeouts, or retry counters were added.

Why This Approach

PR #854 added a parallel retry system with new timeouts and keep-alive
headers - too broad. PR #913 was on the right track (session reset) but
had linting failures and stalled. This PR takes the same minimal approach
with full lint compliance and proper test coverage.

Test Results

============================= test session starts ==============================
platform darwin -- Python 3.9.6, pytest-8.4.2

zulip/tests/test_do_api_query.py::TestDoApiQuery::test_session_not_reset_when_never_connected PASSED [ 50%]
zulip/tests/test_do_api_query.py::TestDoApiQuery::test_session_reset_on_connection_error PASSED [100%]

========================= 2 passed in 2.14s ============================

Full project suite (437 tests, 0 failures):

========================= 437 passed, 5 warnings in 4.57s ========================

Linting (ruff + mypy):

mypy | Success: no issues found in 72 source files
ruff | All checks passed.

Gitlint:

gitlint --msg-filename /dev/stdin   # exit 0, no violations

Tests Added

zulip/tests/test_do_api_query.py:

When a network middlebox silently drops an idle TCP connection, the next
request raises a ConnectionError. Previously the retry logic would reuse
the same dead session, failing all retry attempts because the connection
was never refreshed.

Now we close and null out the session on ConnectionError so
ensure_session() creates a fresh one on the next retry.

Fixes zulip#761.
@HeetRanpura
HeetRanpura force-pushed the fix-stale-connection-session-reset branch from c595a57 to 2287d04 Compare July 31, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve error handling when a request fails due to the persistent connection being expired

2 participants