crash: native abort when the node restarts after returning from the background

@jvsena42

Refs:

What happened?

Bitkit Android crashed with a native SIGABRT when the Lightning node restarted after the app returned from the background to the foreground:

A Tokio 1.x context was found, but it is being shutdown.

Seen by @jvsena42 on master (14:48:36 UTC, 16 Oct) during the device check of #1532 (#1532 (comment)), outside that PR's change. It looks like work still being scheduled on the node's Tokio runtime while that runtime shuts down during the restart, the same family as the teardown crash fixed in synonymdev/ldk-node#111.

Expected behavior

Restarting the node on resume never crashes the app: work started by the old runtime is cancelled or finished before it shuts down, and nothing is scheduled on a runtime that is shutting down.

Steps to Reproduce

Open Bitkit Android with a running node.
Send it to the background long enough for the node to stop, then bring it back so the node restarts.
Occasionally the app aborts with the message above.

Logs / Screenshots / Recordings

See the device check comment linked above.

Bitkit Version

3.0.0 RC2 (master at 16 Oct)

Device / OS

Android

Reproducibility

Rarely (seen once)

Production data: this crash predates 3.0.0

Android vitals (Play Console, last 60 days) show the same panic, A Tokio 1.x context was found, but it is being shutdown., in released builds:

The app's top crash cluster, native SIGABRT in split_config.arm64_v8a.apk: 114 events, 30 users, versions 181 (2.2.0), 187 (2.4.0), 188 (2.4.1), 190 (2.5.0) and 191 (2.5.1). The 2.4.1 sample (Pixel 7, Android 16) carries this message; that build's stack is unsymbolicated.
Two symbolicated 2.5.1 (191) issues with this message: 1 user and 2 events (20 hours ago), and 1 user and 1 event (3 days ago), both on Pixel 7, Android 16, foreground.

The 2.5.1 stack locates the panic in ldk-node: a periodic task spawned by Node::start (RuntimeControl::spawn_tracked_task) polls tokio::time::Interval::tick while the runtime shuts down, from a blocking task in RuntimeControl::try_block_on:

#11 core::panicking::panic_display<tokio>
#12 <tokio::time::sleep::Sleep as Future>::poll
#13 tokio::time::interval::Interval::poll_tick
#14 tokio::time::interval::Interval::tick (ldk_node)
#15 ldk_node::Node::start::{closure}
#16 ldk_node::runtime::RuntimeControl::spawn_tracked_task
#27 ... multi_thread::worker::block_in_place → ldk_node::runtime::RuntimeControl::try_block_on

So this is a long-standing ldk-node shutdown race, not a regression from the 3.0.0 changes (rc.74 → rc.76).

Seen again on RC3 (3.0.0 build 194, 6e4bc01f5, ldk-node 0.7.0-rc.76), devDebug regtest on an Android 16 emulator, 19 Oct.

Verdict: this crash existed before 3.0.0 and is not an rc.76 regression. Play Vitals shows the same panic and the same Interval::tick inside try_block_on/block_in_place stack in 2.4.1 (188) and 2.5.1 (191): #1536 (comment). The emulator runs below only compare how often rc.74 and rc.76 hit it under the same recipe. Neither reproduced it, so they show no rise with rc.76.

Symbolicated stack

Abort message: 'A Tokio 1.x context was found, but it is being shutdown.', tid 17443 (Thread-894), process uptime 4088 s. Every app frame is in libldk_node.so (BuildId 52659fe1, a local rc.76 build; see below). There is no JNI frame: the abort happened on a worker thread of the old node's own Tokio runtime.

#01-#11  std::process::abort <- panic_abort <- core::panicking::panic_display
#12      tokio::runtime::time::entry::TimerEntry::poll_elapsed      tokio-1.53.1/src/runtime/time/entry.rs:539  (assert!(!driver.is_shutdown()))
         <- tokio::time::Sleep::poll                                 tokio-1.53.1/src/time/sleep.rs:465
#13-#14  tokio::time::Interval::poll_tick / tick                     tokio-1.53.1/src/time/interval.rs:459, 442
#15-#16  ldk_node::chain::ChainSource::start_tx_based_sync_loop      src/chain/mod.rs:938 (the tokio::select! over the sync intervals)
#17      ldk_node::Node::spawn_chain_sync_task_with_receiver         src/lib.rs:812
         <- ldk_node::runtime::RuntimeControl::spawn_tracked_task
#19-#23  tokio task harness / multi_thread worker Context::run_task
#24-#27  tokio::runtime::scheduler::multi_thread::worker::run
#29-#33  BlockingTask<block_in_place<RuntimeControl::try_block_on<VssStore (KVStoreSync) ...>>>
#35-#36  tokio blocking-pool thread start

The crashing APK was a local devDebug build whose ldk-node came from a locally built rc.76 in ~/.m2 (BuildId 52659fe1), not the published rc.76 AAR (ee23b4f6). The stack above is symbolicated exactly against that local build's unstripped libldk_node.so with llvm-symbolizer. ldk-node's published AAR and its native-debug-symbols match each other, and so do the 3.0.1 (195) release artifacts. #1574 makes mavenLocal() opt-in so local QA builds use the published libraries.

What the stack shows:

The chain-sync loop is a tracked background task.
It was polled by a worker core that had been handed to a blocking-pool thread. The handoff happened when a VssStore sync call went through RuntimeControl::try_block_on → block_in_place (src/runtime.rs:299-306).
The loop polled its Interval after the old runtime's time driver had shut down, which is what makes the timer assert fire.

Timeline (emulator logcat + app log, CEST)

10:00:01.166  LightningService: Stopping node…                    (app backgrounded, network already down)
10:00:06.360  ERROR ldk_node::runtime:356  Stopping background tasks timed out.
10:00:11.363  ERROR ldk_node::runtime:362  Detaching background tasks after cancellation timed out.
10:00:41.377  ERROR ldk_node::runtime:397  Stopping event handling timed out: deadline has elapsed
10:00:42.388  WARN  LightningService.kt:538  Node handle release still pending after 1s
10:01:05.865  ActivityManager: freezing 15604 to.bitkit.dev
11:06:15.849  ActivityManager: sync unfroze 15604 to.bitkit.dev      (Shop link brought Bitkit back)
11:06:16.833  LightningService.kt:250  Building node…
11:06:16.968  F libc: Fatal signal 6 (SIGABRT) in tid 17443 (Thread-894)

The stop never logged Stopping background syncing on-chain wallet., so the chain-sync loop was the tracked task that got detached.
Thread 17443 is the old runtime's worker: it logged that node's fee-rate updates at 10:00:00.
The abort comes about 1.1 s after the process thaws and before the new node starts. No Kotlin call is involved.
Thawing the frozen process let the detached task of the old runtime run again. The runtime had already started shutting down through node.destroy() → Runtime::drop → shutdown_background() (src/runtime.rs:88-97).
So any resume works as a trigger (here a Shop checkout link). The checkout itself plays no part.

Not the rc.76 Electrum reconnect

git diff v0.7.0-rc.74 v0.7.0-rc.76 changes only src/chain/electrum.rs (plus version files).
src/runtime.rs, src/chain/mod.rs, src/lib.rs and src/io/vss_store.rs are identical between the two tags.
The new reconnect_main_client closure (spawn_blocking) is synchronous and never creates or polls a timer.
None of the crashing frames are in electrum.rs.

This matches the first sighting on master with rc.74 (16 Oct, #1532 (comment)).

Repro on emulators

Recipe:

Fresh regtest wallet on a new API 35 google_apis emulator.
Node running, then an iptables DROP on VSS/RGS (34.65.252.32:443) for 30 s while the app is in the foreground. Electrum stays up.
Home → the node stop times out.
SIGSTOP the process 8 s after Node stopped, standing in for the freezer. The API 35 image didn't freeze the app on its own.
After 150-240 s: restore the network, SIGCONT, and bring the app back from the launcher.

This reproduces the stop on 5654 exactly. All three lines appeared on every run:

Stopping background tasks timed out
Detaching background tasks after cancellation timed out
Stopping event handling timed out

Node handle release still pending followed each time. While VSS is blackholed, the chain-sync loop is stuck in a VSS write (//node_metrics) through block_in_place until the VSS client's 180 s retry budget runs out (src/io/vss_store.rs:693-697). That's why it ignores the stop.

Build	ldk-node	Runs	Stop timed out (all 3 lines)	Freeze/thaw + resume	Abort
RC2 193 (`19449fb0d`)	rc.74	5	5	3 (SIGSTOP 150-240 s)	0
RC3 194 (`6e4bc01f5`)	rc.76	3	3	3 (SIGSTOP 200 s)	0

Two of the RC2 runs had no freeze: one used airplane mode on an API 36.1 image with real freezing, and one used the VSS drop. In those, the stuck loop finished by itself after the retry budget ran out and logged Stopping background syncing on-chain wallet..

On 5654 the process stayed frozen for 65 minutes, and the abort came 1.1 s after the thaw. A freeze of a few minutes wasn't enough to hit it, which fits the low rate in production. 0/3 against 0/3 says nothing about the rate either way. The code path and the Vitals data are what show the crash predates 3.0.0.

iOS

Same exposure in the code, but iOS reaches it less often on resume.

bitkit-ios master (7b3f093b) pins rc.76 (Package.resolved), so it has the same runtime.rs, VSS block_in_place and chain-sync loop.
iOS doesn't stop the node when the app goes to the background. AppScene.handleScenePhaseChange (Bitkit/AppScene.swift:1250-1290) only pauses Paykit, locks the PIN and notifies the hardware wallet on .background. On .active it calls nodeRestarter.retryOnForeground, which restarts only a node in errorStarting (Bitkit/Utilities/NodeRestarter.swift:34-41). A suspended app therefore resumes its one live runtime, with no timed-out stop and no runtime that has already shut down.
iOS does stop and drop a node in these places:
- reset network graph (Bitkit/ViewModels/WalletViewModel.swift:1171)
- wipe (:1885)
- restart on recovery or network restore (Bitkit/AppScene.swift:2006, NodeRestarter.swift:61)
LightningService.stop() (Bitkit/Services/LightningService.swift:300-320) sets node = nil straight after node.stop(). The Runtime drop and shutdown_background() then run there. A stop that times out with work still stuck in block_in_place would leave the same detached task behind on iOS too.

The mechanism lives in ldk-node and doesn't depend on the app version:

a tracked task that ignores the stop because it's stuck in block_in_place
detached on timeout
the runtime shut down underneath it

The fix belongs in ldk-node. A task left detached after a timed-out stop must never be polled once its runtime's time driver has shut down. Today the timer asserts (tokio-1.53.1/src/runtime/time/entry.rs:539) and panic = "abort" turns that into the SIGABRT.

Additional production evidence from the Oct 26, 2026 Play Console review:

Version: 3.0.1 (195).
Cluster: 025a57b3db54dc722bb0cbed27ed8e12.
Filter: Sep 28–Oct 26, 2026, user-perceived crashes, version 195.
1 crash / 1 affected user; last occurrence shown as 3 days ago.
Samsung Galaxy S25+, Android 16.
Abort message: A Tokio 1.x context was found, but it is being shutdown.
Native library BuildId: ee23b4f6171a3eb8317e265d94b01b3a41eed556.

This adds current-release evidence to the already tracked shutdown failure. The new sample's native frames are unsymbolicated; it does not independently establish the exact task/lock sequence discussed above.

Across all versions the same cluster shows 18 events / 7 users in this 28-day window. Those totals include older releases and must not be described as this week's or version 195's totals.

Additional external device evidence from João's October 28 check of #1686: the same A Tokio 1.x context was found, but it is being shutdown. SIGABRT was observed while backgrounding with VSS unreachable, Electrum available and a SETTINGS backup pending.

Reported controlled comparison on a Pixel_9 emulator with the cached-apps freezer disabled:

App revision	Pending backup	Idle wallet
master `ea89ed1`	1/4 crashed	0/3
#1686 `0fd5e424`	2/4 crashed	0/3

The reviewer reports byte-identical libldk_node.so (BuildId prefix 7224179a…), the abort 2–4 ms after a failed //node_metrics write, and matching 39-frame traces apart from one offset. This is attributed reviewer evidence: I inspected the complete posted comparison, but not its raw APKs, tombstones or full logs. These small samples do not establish a rate difference or independently prove an exact task/lock cause.

Tracking here instead of opening a duplicate or treating the failure as introduced by #1686's profile-restore change. The existing shutdown/VSS investigation and native symbolication requirements remain applicable; no unchanged CI rerun or recovery/spending gate relaxation follows from this report.

Additional production evidence from the 2 November 2026 Play Console review. 3.0.1 (195) now has 8 user-perceived crash events across four clusters matching this shutdown panic. These are rolling-window counts, not weekly totals, and affected users must not be summed across clusters.

Window: 5 October–2 November 2026; source updated Sunday, 8:00 AM (timezone not displayed). The current production release is still 3.0.1 (195), released 21 October at 7:47 AM as displayed.

Play cluster	Events / users on build 195	Last occurred, as displayed
025a57b3db54dc722bb0cbed27ed8e12	4 / 3	5 days ago
132d3db9abd2116915993151b660a45f	2 / 1	29 hours ago
8f3c8074acb8cc4a8fe63dee1d6f1683	1 / 1	2 days ago
dcb96458b28cdbbf61d6bf5259b90bdd	1 / 1	34 hours ago

The last three cluster IDs were absent from the 26 October review. All three inspected samples are Google Pixel 7 / Android 16 / build 195, foreground, with the explicit message:

A Tokio 1.x context was found, but it is being shutdown.
BuildId: ee23b4f6171a3eb8317e265d94b01b3a41eed556

Their symbolicated frames include Sleep::poll → Interval::poll_tick / tick → ldk-node tracked tasks. Cluster 8f3c… includes ChainSource::start_tx_based_sync_loop, Node::spawn_chain_sync_task_with_receiver, and RuntimeControl::try_block_on / block_in_place. Clusters 132d… and dcb9… include Node::start and RuntimeControl::spawn_tracked_task.

These signatures match the existing shutdown investigation, so no duplicate issues were opened. This does not establish a new root cause or a crash-rate increase: production crash rates remain unavailable, and the reporting windows overlap.

Production recurrence in the 9 November 2026 Play Console review, on 3.0.1 (195). Three existing symbolicated clusters each gained one event since the 2 November review:

Cluster	Events / users, 2 Nov → 9 Nov	Latest occurrence shown on 9 Nov
132d3db9abd2116915993151b660a45f	2 / 1 → 3 / 1	2 days ago
8f3c8074acb8cc4a8fe63dee1d6f1683	1 / 1 → 2 / 1	4 days ago
dcb96458b28cdbbf61d6bf5259b90bdd	1 / 1 → 2 / 1	6 days ago

The fourth mapped cluster, 025a57b3db54dc722bb0cbed27ed8e12, remains at 4 events / 3 users on build 195. Together the four clusters show 11 events on 195, compared with 8 in the previous review. Do not sum affected users across clusters.

All three inspected detail views still identify Google Pixel 7 / Android 16 / foreground / build 195, with A Tokio 1.x context was found, but it is being shutdown. and native BuildId ee23b4f6171a3eb8317e265d94b01b3a41eed556. The available 8f3c… and dcb9… samples are dated 4 and 6 days ago; the 132d… sample is 8 days old even though its aggregate last occurrence is 2 days ago. The former retains ChainSource::start_tx_based_sync_loop and tracked-task frames; the latter retains Node::start / RuntimeControl::spawn_tracked_task. No distinct failure signature or new root cause is established.

Current source window: 12 October–9 November, user-perceived crashes/ANRs, build 195; data updated Sunday 8 November, 9:00 AM as displayed, timezone unstated. These overlapping rolling-window counts are not weekly totals or crash rates.

3.1.0 (196) was released today (Play displays 9 November, 8:31 AM). Its release dashboard reports stability data unavailable, and build 196 is not yet offered in the vitals version filter. This evidence does not establish recurrence or resolution on 3.1.0.

Proposed handling, given the Breez/Spark migration

ldk-node is still in the app through the migration. feat/move-to-spark (#1774) keeps ldk-node-android 0.7.0-rc.79 alongside breez-sdk-spark, Move to Spark pays over the LDK channels and then closes them, and ldk-node is still the on-chain wallet until a BDK replacement lands. So every user still runs the crashing code path. A proper fix in ldk-node's runtime and VSS code is a lot of work for code we plan to remove, so I propose a cheap app-side mitigation instead.

1. End the process after a timed-out stop (now)

When the node stop times out in the background (Stopping background tasks timed out / Node handle release still pending), end the process instead of keeping it alive. The detached task on the old runtime can then never be thawed and polled, and the next launch starts clean. The change is small and lives in LightningService.kt. It doesn't depend on the migration.

2. Stop starting ldk-node once it has nothing left to resolve (after on-chain moves to BDK)

"No channels" is not enough. A closed channel can still need the node:

coop close not yet buried (reorg)
force close: CSV-delayed to_self output and HTLC claims/timeouts are swept by the node
a revoked state broadcast before the funding output is spent for good needs the justice tx

Gate on all of these after a full sync:

listChannels() is empty
listBalances().lightningBalances is empty (no ChannelMonitor with anything claimable or awaiting confirmations)
listBalances().pendingBalancesFromChannelClosures is empty (nothing left in the sweeper)

Save that as a flag and skip starting the node while it is set. Clear the flag on anything that could bring LDK state back, such as a restore from VSS.

Caveats:

The node has to run at least once after the last close resolves, so users who rarely open the app keep running it longer.
This only works once on-chain is off ldk-node. Close and sweep outputs land in ldk-node's internal wallet, so the BDK wallet must see them (same seed and derivation, or a one-time sweep) before the node stops starting.

3. Keep this issue open as known

Close it when ldk-node is removed. Revisit if the BDK move slips, or if 3.1.0 (196) vitals show the crash rate going up.

Navigation Menu

Repository navigation

crash: native abort when the node restarts after returning from the background #1536

What happened?

Expected behavior

Steps to Reproduce

Logs / Screenshots / Recordings

Bitkit Version

Device / OS

Reproducibility

Production data: this crash predates 3.0.0

Symbolicated stack

Timeline (emulator logcat + app log, CEST)

Not the rc.76 Electrum reconnect

Repro on emulators

iOS

Proposed handling, given the Breez/Spark migration

1. End the process after a timed-out stop (now)

2. Stop starting ldk-node once it has nothing left to resolve (after on-chain moves to BDK)

3. Keep this issue open as known

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

crash: native abort when the node restarts after returning from the background #1536

Description

What happened?

Expected behavior

Steps to Reproduce

Logs / Screenshots / Recordings

Bitkit Version

Device / OS

Reproducibility

Activity

ovitrif commented on Oct 19, 2026

Production data: this crash predates 3.0.0

ovitrif commented on Oct 19, 2026

Symbolicated stack

Timeline (emulator logcat + app log, CEST)

Not the rc.76 Electrum reconnect

Repro on emulators

iOS

piotr-iohk commented on Oct 26, 2026

ben-kaufman commented on Oct 28, 2026

piotr-iohk commented on Nov 2, 2026

piotr-iohk commented on Nov 9, 2026

jvsena42 commented on Nov 10, 2026

Proposed handling, given the Breez/Spark migration

1. End the process after a timed-out stop (now)

2. Stop starting ldk-node once it has nothing left to resolve (after on-chain moves to BDK)

3. Keep this issue open as known

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions