Skip to content

fix: keep paho network thread alive when MQTT-info refresh fails on disconnect - #82

Merged
stackia merged 2 commits into
mainfrom
cursor/mqtt-disconnect-thread-safety-27ca
Sep 4, 2026
Merged

stackia merged 2 commits into
mainfrom
cursor/mqtt-disconnect-thread-safety-27ca

Conversation

@stackia

@stackia stackia commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Fixes #67. Root cause of stackia/ha-deye-dehumidifier#105 as well.

Root cause

_mqtt_on_disconnect runs on paho's network thread and did
asyncio.run_coroutine_threadsafe(self._set_mqtt_info(), self._loop).result().

Two things are wrong with that:

  1. Any exception kills the network thread. paho's _do_on_disconnect re-raises callback exceptions (suppress_exceptions defaults to False), and the exception propagates out of loop_forever() / _thread_main(). The trigger in the issue is exactly the common case: the disconnect and the cloud outage share a cause (router reboot), so get_deye_platform_mqtt_info() fails with a DNS timeout -> DeyeCloudApiCannotConnectError -> thread gone. No reconnect, no keepalive, ever again.
  2. is_connected() keeps returning True afterwards. In paho's _loop_rc_handle (socket read/write error path) on_disconnect is called before _state = MQTT_CS_CONNECTION_LOST, so when the callback raises the state is never updated. That is why the integration's MqttDisconnectMonitor never fires, entities stay "available", and _publish_command_bytes calls paho.publish() on a dead client instead of queueing to _pending_commands — commands vanish silently. Only reloading the integration (new client) recovers.

Blocking the paho thread on the event loop is also a latent deadlock: disconnect() runs on the loop and calls loop_stop(), which join()s the very thread that is waiting on the loop.

Fix

  • The MQTT-info refresh is scheduled on the event loop and the callback returns immediately. paho carries on with its own reconnect/backoff using the previous credentials; if they were rotated, the refused CONNACK re-enters on_disconnect after the refresh has landed.
  • The refresh is bounded by MQTT_INFO_REFRESH_TIMEOUT (30 s), swallows and logs failures, is deduplicated while in flight, and is cancelled by disconnect(). A closed event loop is handled without leaking an un-awaited coroutine.
  • Second commit: _mqtt_on_message caught only JSONDecodeError/KeyError; any other payload error would kill the thread the same way. It now decodes once and logs/drops on any exception.

Verification

Reproduced end to end with a local MQTT broker stub, a real paho loop_start() thread, and a cloud API mock that raises DeyeCloudApiCannotConnectError during the disconnect (same script, main vs this branch):

### BEFORE FIX (origin/main)
[1] connected=True connects=1 subscribed=['/pid/did/status/hex']
[2] after drop: is_connected=True paho_thread_alive=False uncaught_thread_exceptions=['DeyeCloudApiCannotConnectError']
[3] network back: is_connected=True connects=1 subscribes=['/pid/did/status/hex']
[4] poll query delivered to broker: NO (dropped)
RESULT: STUCK - needs integration reload

### AFTER FIX
WARNING libdeye.mqtt_client: MQTT disconnected (Unspecified error) and refreshing the MQTT connection info failed (Timeout while contacting DNS servers); reconnecting with the previous credentials
[1] connected=True connects=1 subscribed=['/pid/did/status/hex']
[2] after drop: is_connected=False paho_thread_alive=True uncaught_thread_exceptions=[]
[3] network back: is_connected=True connects=2 subscribes=['/pid/did/status/hex', '/pid/did/status/hex']
[4] poll query delivered to broker: [('ep/pid/did/command/hex', b'\x00\x01')]
RESULT: RECOVERED
  • pre-commit, mypy, pylint: clean. pytest: 172 passed (7 new tests, run with -W error::RuntimeWarning to catch un-awaited coroutines).
  • ha-deye-dehumidifier test suite passes against this branch (28 passed).

Note on ha-deye-dehumidifier#105

The HA-side attribution ("coordinator returns cache so it cannot detect the dead link") describes a symptom, not the cause. Returning cached data is by design for a cloud_push integration; once the network thread survives, paho's keepalive/reconnect handles link loss and is_connected() is truthful again, so MqttDisconnectMonitor covers prolonged outages. No integration change is needed beyond bumping to the libdeye release containing this fix.

Open in Web Open in Cursor 

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MQTT 断线时云 API 或 DNS 异常会导致 Paho 网络线程永久退出

2 participants