From owner-freebsd-bugs@freebsd.org Mon Sep 3 11:17:12 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A4CCFE60B9 for ; Mon, 3 Sep 2018 11:17:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B55677FF6C for ; Mon, 3 Sep 2018 11:17:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 760D4FE60B8; Mon, 3 Sep 2018 11:17:11 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64957FE60B7 for ; Mon, 3 Sep 2018 11:17:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A17007FF6B for ; Mon, 3 Sep 2018 11:17:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id D05311EBC4 for ; Mon, 3 Sep 2018 11:17:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w83BH9WB096568 for ; Mon, 3 Sep 2018 11:17:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w83BH935096567 for bugs@FreeBSD.org; Mon, 3 Sep 2018 11:17:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 231117] I/O lockups inside bhyve vms Date: Mon, 03 Sep 2018 11:17:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: niels@kristensen.io X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Sep 2018 11:17:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231117 Bug ID: 231117 Summary: I/O lockups inside bhyve vms Product: Base System Version: 11.2-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: niels@kristensen.io We're running bhyve VMs on ~10-20 hypervisors. Since upgrading everything to 11.2-RELEASE last week, we've seen i/o lockups inside different VMs on multiple of these hypervisors. The VMs also run Fre= eBSD 11.2, and both the hypervisors and the VMs run ZFS for backup and quota reasons. We havn't had any issues on 11.1, but when we ran 11.0, we had some problems with the ARC inside the VMs. The VM would be unable to make any disk i/o as= we experience now as well, but in 11.0 the problem could be "resolved" by launching a memory hungry bin from tmpfs, causing ARC to release some memor= y, and then we got working i/o again. On 11.0 the affected VMs were generally i/o intensive VMs. Now we see random lockups on different workloads. The zvol timestamp on the hypervisors are not updated, so clearly writes fa= il to go through. Does anyone have some advise for debugging this issue further? --=20 You are receiving this mail because: You are the assignee for the bug.=