Date: Fri, 24 Jul 2026 19:55:07 +0000 From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 297030] Logging serial console to a file can cause an echo storm that crashes bhyve guest. Message-ID: <bug-297030-27103@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297030 Bug ID: 297030 Summary: Logging serial console to a file can cause an echo storm that crashes bhyve guest. Product: Base System Version: 15.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: sears@cs.berkeley.edu Created attachment 273153 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=273153&action=edit LLM-generated investigation, repro scripts and draft bugfixes. Running FreeBSD 15.1-RELEASE-p1 or Rocky Linux 9 inside a FreeBSD 15.1-RELEASE-p1 bhyve VM is intermittently unreliable when the serial console is attached to a host null modem device, and a program (like cat or logging infrastructure) is reading from the .1B console. In both cases, a serial port echo storm causes the guest to become unresponsive. In FreeBSD guests: sometimes getty goes into an infinite loop, and the guest CPU interrupt % approaches 200% on my 2 core VM. In Rocky Linux, grub or early boot hangs. I could not reproduce this with Ubuntu. If FreeBSD boots the issue can sometimes be reproduced from the host side by doing a 'cat ...1B > /dev/null' and then attaching to the serial console .1A and sending an \r. This path seems dependent on the set of keystrokes that were already echoed back to the VM and isn't particularly reliable. When it works: It causes getty to respond with an ASCII BEL, which is echoed back by the host (via .1B), which causes a BEL, leading to each side 'beeping' at each the other around 60K times per second on my machine. This is probably a duplicate of Bug #229188 which references reports of the problem dating as far back as 2009. I don't see how to reopen it via the web UI and no one is watching it. The reporter of that says it only reproduced when the VM console was attached to a tty device (as opposed to stdin/stdout of the terminal that started the VM). Reproducing the problem: The attached LLM generated tarball has a repro directory with a README.md and some scripts to automatically reproduce the issue with Rocky Linux guests, where it is much easier to hit. You only need the 01-... script with Rocky, but be sure to run the cat in the README.md before the VM gets past early boot. The rocky repro sometimes hits an (unrelated?) assertion failure in emulate_inout at Linux's inout.c:228 That driver seems to have something to do with bhyve. There's an INOUT_ASSERTION_FAIL.md in the tarball with more info. vm_set_register(vcpu, VM_REG_GUEST_RAX, eax) returned non-zero. Root cause: nmdm.c (the null modem emulator) in the kernel source defaults to 'cooked mode' (including 'echo on'), and byhve does not disable 'echo on'. Changing the default behavior to disable host->guest echo is clearly the right call: Current mainstream operating systems (Linux, BSDs, etc) use the serial port for debugging / emergency access, and do not expect the terminal emulator on the other end to route displayed characters back into their keyboards. For very old guests (where the guest VM *is* the terminal emulator, talking over serial to a mainframe or something), this will cause keystrokes to not echo unless whatever attaches to the host-side tty also enables echo. This is pretty easy to diagnose and should not lead to live-locking echo storms. Proposed fix: There are two attached draft patches with independent fixes. One changes the kernel default behavior. The other has bhyve explicitly disable echo. Either one should fix the problem on its own, both probably need a bit of cleanup (or replacement with a better approach). The kernel change to the null modem driver would lead to less surprising default behavior (physical null modem cables cannot echo!) but also break backward compatibility. This is the cleanest fix, but has the biggest blast radius. I have not tested it on a live system, or carefully read the set of flags it proposes. Let me know if you'd like me to test a kernel patch. The bhyve userspace fix is more complicated than I'd like: It has to hold an fd open indefinitely to prevent nmdm from resetting the tty back to cooked mode (nmdm does this when the open count reaches zero). This is a little hacky, but passed quick testing on a live system. The string-based path validation of the device name probably needs more scrutiny. I also have a script (not shared) that works around this by running cat > log.txt in race with a tool that sets the tty mode. This works, but the tool has to poll until cat starts up so that it can set the mode then exit without causing the open count to drop to zero. This is incredibly hacky, but I haven't had any issues with it in the last few days. I used an LLM to help track this down. It asserts the bug has not been fixed in 16.0. I've attached its patches and its investigation in case you want more details. It includes detailed reproduction steps, line-for-line tracing of the null modem setup path, and historical background on serial line modes. Links to guest images that reproduce the problem: https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base-9.8-20260525.0.x86_64.qcow2 https://download.freebsd.org/releases/VM-IMAGES/15.1-RELEASE/amd64/Latest/FreeBSD-15.1-RELEASE-amd64-BASIC-CLOUDINIT-ufs.raw.xz -- You are receiving this mail because: You are the assignee for the bug.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-297030-27103>