From owner-svn-src-all@freebsd.org Wed Oct 10 22:59:16 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38DD510BF56E; Wed, 10 Oct 2018 22:59:16 +0000 (UTC) (envelope-from allanjude@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E27BE75F96; Wed, 10 Oct 2018 22:59:15 +0000 (UTC) (envelope-from allanjude@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D8E5B27666; Wed, 10 Oct 2018 22:59:15 +0000 (UTC) (envelope-from allanjude@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w9AMxFLk035796; Wed, 10 Oct 2018 22:59:15 GMT (envelope-from allanjude@FreeBSD.org) Received: (from allanjude@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w9AMxFf4035795; Wed, 10 Oct 2018 22:59:15 GMT (envelope-from allanjude@FreeBSD.org) Message-Id: <201810102259.w9AMxFf4035795@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: allanjude set sender to allanjude@FreeBSD.org using -f From: Allan Jude Date: Wed, 10 Oct 2018 22:59:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r339299 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Group: head X-SVN-Commit-Author: allanjude X-SVN-Commit-Paths: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs X-SVN-Commit-Revision: 339299 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Oct 2018 22:59:16 -0000 Author: allanjude Date: Wed Oct 10 22:59:15 2018 New Revision: 339299 URL: https://svnweb.freebsd.org/changeset/base/339299 Log: Pull in a follow-on commit to resolve a deadlock in ZFS sequential resilver (r334844) MFV/ZoL: Fix deadlock in IO pipeline commit a76f3d0437e5e974f0f748f8735af3539443b388 Author: Brian Behlendorf Date: Fri Mar 16 16:46:06 2018 -0700 Fix deadlock in IO pipeline In vdev_queue_aggregate() the zio_execute() bypass should not be called under the vdev queue lock. This can result in a deadlock as shown in the stack traces below. Drop the vdev queue lock then walk the parents of the aggregate IO to determine the list of component IOs to be bypassed. This can be done safely without holding the io_lock since the new aggregate IO has not yet been returned and its parents cannot change. --- THREAD 1 --- arc_read() zio_nowait() zio_vdev_io_start() vdev_queue_io() <--- mutex_enter(vq->vq_lock) vdev_queue_io_to_issue() vdev_queue_aggregate() zio_execute() vdev_queue_io_to_issue() vdev_queue_aggregate() zio_execute() zio_vdev_io_assess() zio_wait_for_children() <- mutex_enter(zio->io_lock) --- THREAD 2 --- (inverse order) arc_read() zio_change_priority() <- mutex_enter(zio->zio_lock) vdev_queue_change_io_priority() <- mutex_enter(vq->vq_lock) Reviewed-by: Tom Caputi Reviewed-by: Don Brady Signed-off-by: Brian Behlendorf Reported by: ZFS Leadership Meeting Reviewed by: mav Approved by: re (kib) Obtained from: ZFS-on-Linux MFC after: 2 weeks Sponsored by: Klara Systems Differential Revision: https://reviews.freebsd.org/D17495 Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c ============================================================================== --- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Wed Oct 10 22:55:31 2018 (r339298) +++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c Wed Oct 10 22:59:15 2018 (r339299) @@ -670,6 +670,7 @@ static zio_t * vdev_queue_aggregate(vdev_queue_t *vq, zio_t *zio) { zio_t *first, *last, *aio, *dio, *mandatory, *nio; + zio_link_t *zl = NULL; uint64_t maxgap = 0; uint64_t size; boolean_t stretch; @@ -813,9 +814,18 @@ vdev_queue_aggregate(vdev_queue_t *vq, zio_t *zio) zio_add_child(dio, aio); vdev_queue_io_remove(vq, dio); + } while (dio != last); + + /* + * We need to drop the vdev queue's lock to avoid a deadlock that we + * could encounter since this I/O will complete immediately. + */ + mutex_exit(&vq->vq_lock); + while ((dio = zio_walk_parents(aio, &zl)) != NULL) { zio_vdev_io_bypass(dio); zio_execute(dio); - } while (dio != last); + } + mutex_enter(&vq->vq_lock); return (aio); }