From owner-svn-src-head@freebsd.org Wed Dec 20 20:32:52 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3111EA309E; Wed, 20 Dec 2017 20:32:52 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk0-x232.google.com (mail-qk0-x232.google.com [IPv6:2607:f8b0:400d:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91D0F158C; Wed, 20 Dec 2017 20:32:52 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk0-x232.google.com with SMTP id i190so7938227qka.5; Wed, 20 Dec 2017 12:32:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=o6+kd9MFKMS/QX8kv3g+JrcJCDZlL6V6JIqXiFem99A=; b=P0rkdsBh1B0YExBOnx0lrmUHrVQLKO3NmS7Snyw5vom2T73T+1EP69qM906SGEgX+S jhCgrv8cX61ukG9iIqcbFo2WR4g/W1TDsGW03X/PfUMzimFAq/UEHw6oAtGtycSl8sls pj+Yg0XX4zCEJZLgUcKw+U8kWbxcl+SsMi0azigROMA5Bo5xXSYrzFD7ZTQQHubsybdq gELsQsA/Ynd/gR1grQi2goilCsO1QgJawngBa+JBHRx0zXCxmSdVfXIX/PA+xm28vhdM 2lpYhZdCws3wcxdrpGQN1tforQ6yV+qMPT1a+XSpNNmYZHYBRBuDYlcSt4eHyFthFgjd bqVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=o6+kd9MFKMS/QX8kv3g+JrcJCDZlL6V6JIqXiFem99A=; b=Ak9108t+PZ2jO7QFjARZmYPKuUNj874WOnQyu86VFLemQIYBadwxwB37zMcWklX8KB 6q6AWjhXMbJX4TTl1AlOAsuKDixx1kn9HMG6VsRWECr69EV7vEfo2sbrqdlFr/o9oV5w 34eeWYpSG+cCM5gRKW5Xh3wPk5Uo4GLEJpScw8g0l0Y3+TITFUS5ewkhcjIy659AAzzd 3WYBT9iXnb7XpxDiUA4klSDFjZQANTM/RdUFak7mrLIOz+u8yGwHIxGSNszUPv2zehb9 kYeV7faYDwL02V+jnVR1HaTthOJKgxMTG+OcO+WZoIWLMFk6xv8vySkBXvm8I7mnDVqM wkmA== X-Gm-Message-State: AKGB3mIoQYi1oyV5MHMobakyfgV9IC9qkE3XGwLAc8NQ0meGhRs3iWhi AJgRj9Z1PrnAmFJS7zG4TbJ82g== X-Google-Smtp-Source: ACJfBosFWhyrnZmzzw/0922jp6zNPY3Bw4OqrwWaeD4hi6w5VnBsoUYbT8jdN65HjMribV7E5SHdAw== X-Received: by 10.55.115.130 with SMTP id o124mr11816700qkc.260.1513801971313; Wed, 20 Dec 2017 12:32:51 -0800 (PST) Received: from raichu (toroon0560w-lp140-01-69-159-38-22.dsl.bell.ca. [69.159.38.22]) by smtp.gmail.com with ESMTPSA id q13sm12092014qta.22.2017.12.20.12.32.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Dec 2017 12:32:50 -0800 (PST) Sender: Mark Johnston Date: Wed, 20 Dec 2017 15:32:48 -0500 From: Mark Johnston To: Warner Losh Cc: src-committers , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Subject: Re: svn commit: r326731 - head/sys/ufs/ffs Message-ID: <20171220203248.GB11032@raichu> References: <201712091544.vB9FiVUI096790@repo.freebsd.org> <20171210031450.GB15275@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171210031450.GB15275@raichu> User-Agent: Mutt/1.9.1 (2017-09-22) X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Dec 2017 20:32:52 -0000 On Sat, Dec 09, 2017 at 10:14:50PM -0500, Mark Johnston wrote: > On Sat, Dec 09, 2017 at 07:36:59PM -0700, Warner Losh wrote: > > On Sat, Dec 9, 2017 at 11:03 AM, Andriy Gapon wrote: > > > > > On 09/12/2017 17:44, Mark Johnston wrote: > > > > Some GEOMs do not appear to handle BIO_ORDERED correctly, meaning that > > > the > > > > barrier write may not work as intended. > > > > > > There's a few places we send down a BIO_ORDERED BIO_FLUSH command > > (see softdep_synchronize for one). Will those matter? > > Some classes have separate handling for BIO_FLUSH, so it depends. I > think gmirror's handling is buggy independent of BIO_ORDERED: > g_mirror_start() sends BIO_FLUSH commands directly to the mirrors, > while reads and writes are queued for handling by the gmirror worker > thread. So as far as I can tell, a BIO_WRITE which arrives at the > gmirror provider before a BIO_FLUSH might be sent to the mirrors > after that BIO_FLUSH. I would expect BIO_FLUSH to implicitly have > something like release semantics, i.e., a BIO_FLUSH shouldn't be > reordered with a BIO_WRITE that preceded it. But I might be > misunderstanding. > > > As I've noted elsewhere: I'd really like to kill BIO_ORDERED since it has > > too many icky effects (and BIO_FLUSH + BIO_ORDERED isn't guaranteed to do, > > well, anything since it can turn into a NOP in a number of places. Plus > > many of the implementations of BIO_ORDERED assume the drive is like SCSI > > and you just set the right tag to make it 'ordered'. For ATA we issue a non > > NCQ command, which is a full drain of outstanding commands, send this > > command, then start them again which really shuts down the parallelism we > > implemented NCQ for :(. We do similar for NVME which is even worse. There > > we have multiple submission queues in the hardware. To simulated it, we do > > a similar drain, but that's going to get in the way as we move to NUMA and > > systems where we try to do the I/O entirely on one CPU (both submission and > > completion) and ordered I/O is guaranteed lock contention. > > Independent of this, it doesn't really look like we have any way of > handling write errors when dependencies are enforced using BIO_ORDERED. > In the case of the babarrierwrite() consumer in FFS, what happens if the > inode block write fails due to a transient error, but the following CG > update succeeds? Looking at this some more, I think iosched is ok since bioq_disksort() does in fact handle BIO_ORDERED. I'm concerned though that there are places in the tree where we should be using BIO_ORDERED but aren't. gmirror metadata writes for instance are performed synchronously but should not be reordered with preceding BIO_WRITEs. Anyway, I've posted a review for some gmirror I/O ordering fixes here if anyone is interested: https://reviews.freebsd.org/D13559