From owner-freebsd-virtualization@freebsd.org Sat Dec 22 07:30:18 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7987F1348B77 for ; Sat, 22 Dec 2018 07:30:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1363274A68 for ; Sat, 22 Dec 2018 07:30:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id CACD61348B76; Sat, 22 Dec 2018 07:30:17 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 901011348B75 for ; Sat, 22 Dec 2018 07:30:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F14074A61 for ; Sat, 22 Dec 2018 07:30:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 761171E475 for ; Sat, 22 Dec 2018 07:30:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBM7UGGl017515 for ; Sat, 22 Dec 2018 07:30:16 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBM7UGdC017514 for virtualization@FreeBSD.org; Sat, 22 Dec 2018 07:30:16 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 231117] I/O lockups inside bhyve vms Date: Sat, 22 Dec 2018 07:30:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kkc@duckpond.dk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Dec 2018 07:30:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231117 --- Comment #12 from Kristian K. Christensen --- (In reply to Kristian K. Christensen from comment #11) So, I can reproduce the issue on a test vm, regretably I can't get ps outpu= t or run procstat, they both hang. On the hypervisor, I get the following from procstat -kk on the bhyve pid: machiavelli# procstat -kk 35743 PID TID COMM TDNAME KSTACK=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 35743 100908 bhyve mevent mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 kqueue_kevent+0= x290 kern_kevent+0xb5 sys_kevent+0xa4 amd64_syscall+0xa38 fast_syscall_common+0x= 101=20 35743 101987 bhyve vtnet-2:0 tx mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101988 bhyve blk-3:0-0 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101989 bhyve blk-3:0-1 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101990 bhyve blk-3:0-2 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101991 bhyve blk-3:0-3 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101992 bhyve blk-3:0-4 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101993 bhyve blk-3:0-5 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101994 bhyve blk-3:0-6 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101995 bhyve blk-3:0-7 mi_switch+0xe6 sleepq_catch_signals+0x40c sleepq_wait_sig+0xf _sleep+0x231 umtxq_sleep+0x1= 43 do_wait+0x427 __umtx_op_wait_uint_private+0x53 amd64_syscall+0xa38 fast_syscall_common+0x101=20 35743 101996 bhyve vcpu 0 mi_switch+0xe6 sleepq_timedwait+0x2f msleep_spin_sbt+0x161 vm_run+0x7d7 vmmdev_ioctl+0x85f devfs_ioctl_f+0x128 kern_ioctl+0x26d sys_ioctl+0x15c amd64_syscall+0xa38 fast_syscall_common+0x101=20 The system isn't completely dead, but anything at all that requires io hangs indefinitely. I can type on the console, provided it was logged in beforeha= nd, but trying to tab complete or run anything that's not in memory, hangs. I'm going to try to run procstat and ps more frequently in cron, hoping to = keep them in memory. Any other suggestions as to how to provide you with any use= ful input would be welcomed. --=20 You are receiving this mail because: You are the assignee for the bug.=