From owner-freebsd-bugs@freebsd.org Mon Apr 29 05:18:53 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B6A51580CBF for ; Mon, 29 Apr 2019 05:18:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 91C998C063 for ; Mon, 29 Apr 2019 05:18:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 529631580CBA; Mon, 29 Apr 2019 05:18:52 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CA6A1580CB5 for ; Mon, 29 Apr 2019 05:18:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 815E98C061 for ; Mon, 29 Apr 2019 05:18:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B2ABE168FB for ; Mon, 29 Apr 2019 05:18:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x3T5IosY019633 for ; Mon, 29 Apr 2019 05:18:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x3T5IotV019632 for bugs@FreeBSD.org; Mon, 29 Apr 2019 05:18:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 237636] bhyve guest ZFS filesystem freezes in zcw->zcw_cv state Date: Mon, 29 Apr 2019 05:18:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd-bugs@morgandavis.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Apr 2019 05:18:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D237636 Bug ID: 237636 Summary: bhyve guest ZFS filesystem freezes in zcw->zcw_cv state Product: Base System Version: 12.0-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: freebsd-bugs@morgandavis.net I experience random freezes during file I/O that locks up the entire virtual machine (except networking). Environment is FreeBSD 12.0-RELEASE bhyve guest running with ZFS on a FreeBSD 12.0-RELEASE host also on ZFS. During some specific disk operations, such as a sqlite database vacuum or g= it 'gc' operation, the VM will freeze up and cease performing disk operations.= =20 The VM can still respond to pings and accept network connections, but any d= isk operation will just cause complete blocking. The only remedy is to kill -9= the bhyve host process and restart the guest. Trying Control-T during one of these episodes always shows the kernel stuck= in zcw->zcw_cv. For example, this "git gc" operation: Enumerating objects: 12282, done. Counting objects: 100% (12282/12282), done. Delta compression using up to 4 threads Compressing objects: 100% (12259/12259), done. <--- system completely stops responding here, control-T entered --> load: 0.47 cmd: git 90093 [zcw->zcw_cv] 31.31r 9.77u 0.09s 0% 123612k This does not always happen and is not reproducible at will. However, the freezes are occurring mostly with sqlite and occasionally with git. Host system has 16GB RAM and guest has 5GB allocated. The host does nothing but run this one guest bhyve instance. Guest's /boot/loader.conf disables ZFS prefetching: vfs.zfs.prefetch_disable=3D"1" And after much searching for help on the freezing, I found this article whi= ch suggested reducing ZFS arc max. https://www.reddit.com/r/freebsd/comments/1u358s/freebsd_random_freeze_wher= e_to_start/ So I added this to loader.conf, too: vfs.zfs.arc_max=3D"1G" This occurs on FreeBSD 11 and now 12 as well. I don't recall having this i= ssue under FreeBSD 10. --=20 You are receiving this mail because: You are the assignee for the bug.=