From nobody Fri Jul 24 19:55:07 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4h6JcW708Zz6mfFq for ; Fri, 24 Jul 2026 19:55:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4h6JcW6VT5z3SyR for ; Fri, 24 Jul 2026 19:55:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1784922907; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=71dvIg5FiJg2yLBy5UP22Wez9xQtQWUXkcuKUu7wuMQ=; b=nqhdhWR3BIh5zylCKxHN/dL4RrKjKpm/ISW23PhD7SJMgJco8YHUtnFFGXKKTV33iYkrTa h/MNF5QUJZcmxb1cZNRJ7RfvF5+JmYIwU4n/C5f8Lmj/0AI+4SMXLoGB8hPg0b7seTCBVx H7JxwBCrEQJWM6dyfyiKjJOGZ4jz7F1DzYLo36qBOwSvyhCdzgCup6yghw6o1YiIPUOB// 75vITR6jnA/vTnM3s73xtZNHJE8gQX+CFkAkxghQDdgfwPBFsNKgdKi5v82IFePXkgw6SD 0WpQ/5NmkUZ+fwNqY0PGQc+xqe8eRpsmxFEKBu+J/oqtqQ8RQVPiZLP7BVti3w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1784922907; a=rsa-sha256; cv=none; b=ULpn5TILB4BODy/EZ219aSL9hA+T6W6OkTGwaWZnTIKz/8WC7Rr7rhYVVb5OG55p/B1czA I0ulmCSo/o1HitcCLrT1aLBgd8DGvmXLsoweLQePJgbLrYra43L4ez7+kaJyMqcJaNlqXM iBgfeQuU4qe1KbWHhbPVtVMHjkm3eJSr9r2MwjZROBaLsRl7SiHFRQ63I+xt/QLziaP1lk ex+B4qkxmUBbN+V08ya5yE5NZux6vfWA++KZONpRmYNwpX9eKtEAE7KcA7pCBupqhbomel 0osumRvqPPP/NE2IRn3+v6B+G1sgVL/SHUGIPqzJdSaar8bcdxiMr1qf7uTmMQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1784922907; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=71dvIg5FiJg2yLBy5UP22Wez9xQtQWUXkcuKUu7wuMQ=; b=muUDWUJqUARFT+9ls1v9jF5KgaBMY3IgcoZAMVVyq8rs4X6J2kcEvRsYev+NRNXsASIJTl +/nwiV+LVoJbkfVy+4FObPXlcFLecDSZ7+TsRuyV3Rt7wvy9r1yo3fWPwjq7uj2vq9nwZv SzUgrz/qrfCetairiBJaZMxjp8MS4lUyzYJ2XoSXsjJ9YTg5sHGM2WDpkwW0lHo1ELcEld /T0WYkK/Ry/Eeiux3dbjW3yxAQYcfWYuOf6bq3Ls9zBvN53L3N2Ti9P+lINdN+rUqvCcIW OzhDSsP8Qj1UJFY+Rcde3iyVUpTz9/Q8n7qB8Dy9TGYqC5S8KidD7jL0EbdVVg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4h6JcW5B07z15Nw for ; Fri, 24 Jul 2026 19:55:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 66OJt7rQ071404 for ; Fri, 24 Jul 2026 19:55:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 66OJt7oT071403 for virtualization@FreeBSD.org; Fri, 24 Jul 2026 19:55:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 297030] Logging serial console to a file can cause an echo storm that crashes bhyve guest. Date: Fri, 24 Jul 2026 19:55:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 15.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: sears@cs.berkeley.edu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D297030 Bug ID: 297030 Summary: Logging serial console to a file can cause an echo storm that crashes bhyve guest. Product: Base System Version: 15.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: sears@cs.berkeley.edu Created attachment 273153 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D273153&action= =3Dedit LLM-generated investigation, repro scripts and draft bugfixes. Running FreeBSD 15.1-RELEASE-p1 or Rocky Linux 9 inside a FreeBSD 15.1-RELEASE-p1 bhyve VM is intermittently unreliable when the serial conso= le is attached to a host null modem device, and a program (like cat or logging infrastructure) is reading from the .1B console. In both cases, a serial p= ort echo storm causes the guest to become unresponsive. In FreeBSD guests: sometimes getty goes into an infinite loop, and the guest CPU interrupt % approaches 200% on my 2 core VM. In Rocky Linux, grub or early boot hangs. I could not reproduce this with Ubuntu. If FreeBSD boots the issue can sometimes be reproduced from the host side by doing a 'cat ...1B > /dev/null' and then attaching to the serial console .= 1A and sending an \r. This path seems dependent on the set of keystrokes that were already echoed back to the VM and isn't particularly reliable. When it works: It causes getty to respond with an ASCII BEL, which is echoed back by the host (via .1B), which causes a BEL, leading to each side 'beeping' at e= ach the other around 60K times per second on my machine. This is probably a duplicate of Bug #229188 which references reports of the problem dating as far back as 2009. I don't see how to reopen it via the w= eb UI and no one is watching it. The reporter of that says it only reproduced when the VM console was attached to a tty device (as opposed to stdin/stdou= t of the terminal that started the VM). Reproducing the problem: The attached LLM generated tarball has a repro directory with a README.md a= nd some scripts to automatically reproduce the issue with Rocky Linux guests, where it is much easier to hit. You only need the 01-... script with Rocky, but be sure to run the cat in the README.md before the VM gets past early b= oot. The rocky repro sometimes hits an (unrelated?) assertion failure in emulate_inout at Linux's inout.c:228 That driver seems to have something t= o do with bhyve. There's an INOUT_ASSERTION_FAIL.md in the tarball with more in= fo.=20 vm_set_register(vcpu, VM_REG_GUEST_RAX, eax) returned non-zero. Root cause: nmdm.c (the null modem emulator) in the kernel source defaults to 'cooked m= ode' (including 'echo on'), and byhve does not disable 'echo on'. Changing the default behavior to disable host->guest echo is clearly the ri= ght call:=20=20 Current mainstream operating systems (Linux, BSDs, etc) use the serial port= for debugging / emergency access, and do not expect the terminal emulator on the other end to route displayed characters back into their keyboards. For very old guests (where the guest VM *is* the terminal emulator, talking over ser= ial to a mainframe or something), this will cause keystrokes to not echo unless whatever attaches to the host-side tty also enables echo. This is pretty e= asy to diagnose and should not lead to live-locking echo storms. Proposed fix: There are two attached draft patches with independent fixes. One changes t= he kernel default behavior. The other has bhyve explicitly disable echo. Eit= her one should fix the problem on its own, both probably need a bit of cleanup = (or replacement with a better approach). The kernel change to the null modem driver would lead to less surprising default behavior (physical null modem cables cannot echo!) but also break backward compatibility. This is the cleanest fix, but has the biggest blast radius. I have not tested it on a live system, or carefully read the set of flags it proposes. Let me know if you'd like me to test a kernel patch. The bhyve userspace fix is more complicated than I'd like: It has to hold = an fd open indefinitely to prevent nmdm from resetting the tty back to cooked = mode (nmdm does this when the open count reaches zero). This is a little hacky,= but passed quick testing on a live system. The string-based path validation of= the device name probably needs more scrutiny.=20 I also have a script (not shared) that works around this by running cat > log.txt in race with a tool that sets the tty mode. This works, but the to= ol has to poll until cat starts up so that it can set the mode then exit witho= ut causing the open count to drop to zero. This is incredibly hacky, but I haven't had any issues with it in the last few days. I used an LLM to help track this down. It asserts the bug has not been fix= ed in 16.0. I've attached its patches and its investigation in case you want = more details. It includes detailed reproduction steps, line-for-line tracing of= the null modem setup path, and historical background on serial line modes. Links to guest images that reproduce the problem: https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Ba= se-9.8-20260525.0.x86_64.qcow2 https://download.freebsd.org/releases/VM-IMAGES/15.1-RELEASE/amd64/Latest/F= reeBSD-15.1-RELEASE-amd64-BASIC-CLOUDINIT-ufs.raw.xz --=20 You are receiving this mail because: You are the assignee for the bug.=