Skip to content

Fix SuppressStdoutStderr permanently silencing the process when contexts overlap - #3290

Open
choksi2212 wants to merge 2 commits into
unit8co:masterfrom
choksi2212:fix/suppress-stdout-thread-safety-3276
Open

choksi2212 wants to merge 2 commits into
unit8co:masterfrom
choksi2212:fix/suppress-stdout-thread-safety-3276

Conversation

@choksi2212

Copy link
Copy Markdown
Contributor

SuppressStdoutStderr used to duplicate fd 1 and 2 in __init__ and restore them in __exit__, per instance, with no synchronization. If a second context was created while a first one was active (two threads fitting Prophet models side by side, or just overlapping with blocks), the second one saved the devnull descriptors as if they were the real ones, and whoever exited last pointed fd 1 and 2 back at devnull for the rest of the process lifetime — the scenario from #3276, where after two overlapping fits everything printed afterwards was gone. Creating the context in interpreters without usable standard streams (pythonw on Windows, GUI-embedded hosts) also raised OSError: Bad file descriptor straight from __init__, leaking the descriptors opened just before the failure.

The redirection is now shared state: a module-level lock plus a depth counter, where the outermost context captures and restores the real descriptors, contexts entered while a redirection is already active just join it, and all descriptor work happens in __enter__ with a graceful no-op fallback when the standard streams cannot be saved. I left Prophet's suppression path as-is — it becomes safe through the context manager itself, and switching it over to cmdstanpy logger levels felt like a separate design call.

How I checked it: on master, the new threaded test fails exactly like the issue describes (assert b'' == b'survivor' — the write after both contexts lands nowhere) and the fallback test dies with OSError: [Errno 9] Bad file descriptor out of __init__. With the fix both pass, along with the round-trip and nested tests and the rest of test_logging.py (8 passed). The overlap choreography is event-driven rather than sleep-based, so the interleaving is deterministic. ruff check and ruff format (0.7.2, same as pre-commit) are clean.

Closes #3276.

The context manager duplicated and restored the process-level stdout and
stderr descriptors per instance, with no synchronization. A second context
created while a first one was active saved the devnull descriptors as if
they were the real ones, and whoever exited last dup2-ed devnull back onto
fd 1 and 2, permanently silencing the process (unit8co#3276). Constructing it in
interpreters without real standard streams (pythonw, GUI hosts) also raised
OSError from __init__ and leaked the descriptors opened before the failure.

The redirection is now shared: a module-level lock plus a depth counter, the
outermost context captures and restores the real descriptors, contexts
entered while a redirection is active just join it, and all fd work moved
into __enter__ with a graceful no-op fallback when the standard streams
cannot be saved.

Signed-off-by: choksi2212 <choksi2212@users.noreply.github.com>
Signed-off-by: choksi2212 <choksi2212@users.noreply.github.com>
@codecov

codecov Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.13%. Comparing base (555c8d1) to head (47435a9).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3290   +/-   ##
=======================================
  Coverage   97.13%   97.13%           
=======================================
  Files         175      175           
  Lines       19408    19447   +39     
=======================================
+ Hits        18851    18890   +39     
  Misses        557      557           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] SuppressStdoutStderr in Prophet.fit() is not thread-safe and permanently redirects process FD 1/2 to /dev/null under concurrent execution

1 participant