Date: Fri, 26 Jun 2026 14:16:43 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: =?UTF-8?B?W0J1ZyAyOTYyOTFdIGN1c2UoMyk6IGN1c2Vfc2VydmVyX2ZyZWUo?= =?UTF-8?B?KSBidXN5LXdhaXRzIHVua2lsbGFibHkgKHBhdXNlKCJXIikpIHdoZW4gYSBj?= =?UTF-8?B?bGllbnQgcmVmICAgaXMgbm90IHJlbGVhc2VkIG9uIHZpcnR1YWxfb3NzIHRl?= =?UTF-8?B?YXJkb3duIOKAlCBwcm9jZXNzIHN0dWNrIGluIEQsIGN1c2Uua28gcGlubmVk?= =?UTF-8?B?LCByZWJvb3QgcmVxdWlyZWQ=?= Message-ID: <bug-296291-227@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296291 Bug ID: 296291 Summary: cuse(3): cuse_server_free() busy-waits unkillably (pause("W")) when a client ref is not released on virtual_oss teardown — process stuck in D, cuse.ko pinned, reboot required Product: Base System Version: 15.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: delleceste@gmail.com Every time virtual_oss(8) is stopped, its exit path busy-waits in the kernel for its cuse clients to drop their references — a routine multi-second stall. INTERMITTENTLY a client reference is never released, so the wait never ends: the exiting process is left wedged forever in an uninterruptible, SIGKILL-immune state, cuse.ko becomes pinned (cannot be unloaded), and no new virtual_oss can recreate the devices. Only a reboot recovers the machine. The proximate severity is that the kernel wait is unbounded and uninterruptible (a plain pause("W")), so any reference leak becomes permanent. ENVIRONMENT FreeBSD 15.1-RELEASE releng/15.1-n283562-96841ea08dcf GENERIC amd64 Base components: sys/fs/cuse/cuse.c (cuse.ko) and usr.sbin/virtual_oss. Setup: a loopback audio chain. virtual_oss -i 8 -C 2 -c 2 -b 32 -s 200ms -f /dev/null -a 0 \ -d dsp.play -L dsp.loop MPD opens /dev/dsp.play (cuse client); BruteFIR opens /dev/dsp.loop (cuse client) and writes the USB DAC. OBSERVED $ ps -o pid,ppid,stat,wchan,mwchan,command -p 5992 PID PPID STAT WCHAN MWCHAN COMMAND 5992 5988 D<E W W virtual_oss ... -d dsp.play -L dsp.loop - D = uninterruptible kernel sleep, < = realtime prio (-i 8), E = exiting. - kill -TERM and kill -9 have NO effect (thread is past signal delivery). - cuse.ko is pinned; "kldunload cuse" then also hangs in D. - A fresh virtual_oss dies with "Could not create CUSE DSP device". - No userland process holds /dev/dsp.play or /dev/dsp.loop open at this point (procstat -f sweep) — the leaked reference is purely kernel-side. - Only a reboot clears it; every virtual_oss stop/restart is a reboot risk. ROOT CAUSE (kernel) MWCHAN "W" pinpoints cuse_server_free(), the cdevpriv destructor for the daemon's /dev/cuse fd, run when the daemon exits (sys/fs/cuse/cuse.c:765): static void cuse_server_free(void *arg) { struct cuse_server *pcs = arg; while (cuse_server_do_close(pcs) != 1) /* returns pcs->refs */ pause("W", hz); /* <-- MWCHAN "W" */ cuse_server_unref(pcs); } The loop busy-waits (1s ticks) until pcs->refs == 1, i.e. until every client handle is released. pcs->refs is ++'d per client open (cuse.c:1506, cuse_client_open) and dropped in the client cdevpriv destructor cuse_client_free() -> cuse_server_unref() (cuse.c:1480). If a client ref is never dropped, this loop never terminates and is not interruptible (plain pause(), no signal/timeout escape) -> the exiting process is stuck in D forever. A mitigation exists but is escaped: cuse_server_do_close() sets pcs->is_closing and wakes all clients (cuse.c:752-755), and cuse_client_receive_command_locked() returns CUSE_ERR_OTHER when is_closing is set (cuse.c:631). The comment above the loop ("the client cdevpriv destructor, which cannot destroy itself") points at the self-referential case, exactly the loopback (-L/-l) device, where virtual_oss is both server and client of the same cuse server. EMPIRICAL EVIDENCE (instrumented cuse.ko + DTrace) A diagnostic cuse.ko logging pcs->refs inside the loop, over many start/stop cycles of the real chain (MPD + BruteFIR clients): 1. EVERY teardown enters the loop with refs > 1 (it is the normal stop path): cuse: server pid 11027 exit stuck: refs=3 (want 1) after 1 s cuse: server pid 11036 exit stuck: refs=2 (want 1) after 1 s refs=3 = server + BruteFIR(dsp.loop) + MPD(dsp.play); refs=2 = server + 1. 2. DTrace lifecycle (fbt:cuse:cuse_client_open/free, cuse_server_free, by execname/pid): musicpd[2278] OPEN brutefir[9705] OPEN virtual_oss[9697] SERVER_FREE begin (teardown wait) ... loop spins ... brutefir[9705] FREE (cuse_client_free -> cuse_server_unref) musicpd[2278] FREE (refs reaches 1, server exits) 3. USUAL outcome: clients close within ~1-6 s and the server RECOVERS — a routine multi-second busy-wait on every stop (e.g. pid 11036 logged "after 1 s" then "after 6 s" before recovering). 4. SEVERE/INTERMITTENT outcome: a client ref never drops, the loop spins forever, virtual_oss is stuck D<E (SIGKILL-immune), cuse.ko pinned, reboot-only (pid 5992). So is_closing DOES release clients in the common case; the permanent hang is an unlucky-timing race where a client close cannot be serviced/completed. This is consistent with virtual_oss's teardown: usr.sbin/virtual_oss/.../main.c destroy_threads() (main.c:2498) pthread_cancel()s the cuse worker threads without pthread_join(), the workers block in cuse_wait_and_process() (an ioctl(2), NOT a POSIX cancellation point, main.c:1990), and only the CTL device is cuse_dev_destroy()'d on exit (main.c:2630) — the DSP/WAV/loopback devices (created at main.c:1918/1935) are never destroyed. On unlucky timing the close command for a client thus has no server thread left to process it, and the unbounded uninterruptible pause("W") turns that transient leak into a reboot-only hang. REPRODUCE 1. virtual_oss -i 8 -C 2 -c 2 -b 32 -s 200ms -f /dev/null -a 0 \ -d dsp.play -L dsp.loop 2. Open clients on the devices (BruteFIR reading /dev/dsp.loop S32_LE 2ch, and/or MPD playing to /dev/dsp.play). 3. Stop virtual_oss (kill, killall, or stop consumers then virtual_oss). 4. Intermittently it never exits: ps shows "D<E W"; kill -9 no effect; cuse.ko cannot be unloaded; a fresh virtual_oss cannot recreate the devices. Frequency rises with rapid start/stop churn and a client attached at stop time. IMPACT Unkillable process; pinned kernel module; reboot is the only recovery. Any virtual_oss-based audio routing can be bricked until reboot by a normal stop/restart (e.g. a sample-rate change). SUGGESTED FIX Kernel (sys/fs/cuse/cuse.c): cuse_server_free() must not busy-wait unboundedly/uninterruptibly. Bound the wait (timeout + forced reclamation), or restructure teardown so a closing server forcibly invalidates client refs that can no longer be serviced (the is_closing wakeup does not cover the stuck case, notably the loopback/self-client). virtual_oss (usr.sbin/virtual_oss/.../main.c): on exit, cuse_dev_destroy() ALL created devices (DSP, WAV, loopback), not just CTL, before tearing down threads; and pthread_join() the workers instead of pthread_cancel()+free(). A candidate userland patch doing this (store each cuse_dev handle in the profile; destroy all devices before exit) has been written and compile-tested — it makes the client refs drop before process exit so cuse_server_free() finds refs==1 and never busy-waits. That fixes the trigger; the kernel pause("W") hardening is still wanted so a future ref leak cannot become an unrecoverable hang. -- You are receiving this mail because: You are the assignee for the bug.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-296291-227>