From owner-freebsd-geom@FreeBSD.ORG Sun Sep 23 07:10:17 2012 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 314B7106566C; Sun, 23 Sep 2012 07:10:17 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 028218FC15; Sun, 23 Sep 2012 07:10:16 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id q8N7AFUE017029 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 23 Sep 2012 00:10:15 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id q8N7AFda017025; Sun, 23 Sep 2012 00:10:15 -0700 (PDT) (envelope-from jmg) Date: Sun, 23 Sep 2012 00:10:15 -0700 From: John-Mark Gurney To: Pawel Jakub Dawidek Message-ID: <20120923070842.GJ19036@funkthat.com> Mail-Followup-To: Pawel Jakub Dawidek , freebsd-geom@freebsd.org References: <20120919040430.GF19036@funkthat.com> <20120922162025.GE1454@garage.freebsd.pl> <20120923044828.GI19036@funkthat.com> <20120923064307.GK1454@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120923064307.GK1454@garage.freebsd.pl> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sun, 23 Sep 2012 00:10:16 -0700 (PDT) Cc: freebsd-geom@freebsd.org Subject: Re: geli and BIO_FLUSH and/or BIO_ORDERED issue? X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Sep 2012 07:10:17 -0000 Pawel Jakub Dawidek wrote this message on Sun, Sep 23, 2012 at 08:43 +0200: > On Sat, Sep 22, 2012 at 09:48:28PM -0700, John-Mark Gurney wrote: > > Pawel Jakub Dawidek wrote this message on Sat, Sep 22, 2012 at 18:20 +0200: > > > On Tue, Sep 18, 2012 at 09:04:30PM -0700, John-Mark Gurney wrote: > > > > I was looking at geli and I'm not sure if it's implementing BIO_FLUSH > > > > and/or BIO_ORDERED properly... > > > > > > > > >From my understanding is the BIO_ORDERED is suppose to wait for the > > > > previous _WRITES to complete before returning so that you can ensure > > > > that data is on disk, i.e. _ORDERED is set on a BIO_FLUSH... > > > > > > > > BIO_ORDERED is handled by diskq_* code such that when you add an _ORDERED > > > > command, all commands are put after it, but there doesn't appear to > > > > be any code to ensure that an _ORDERED command waits for prevoius > > > > pending commands to complete.. > > > > > > > > This is extra obvious in eli in that a _FLUSH is immediately dispatched, > > > > even when there may be _WRITEs that haven't been finished encrypting and > > > > sent down to the disk to get _FLUSHed... > > > > > > > > Any comments about this? > > > > > > Hmm, BIO_ORDERED was introduced pretty recently and GEOM classes were > > > not updated to honour it, but it also seems to be to complex to handle > > > in GEOM classes. I wonder if we could hold off new writes and wait for > > > the in-progress writes in GEOM if we spot BIO_ORDERED request without > > > the need to implement this logic in GEOM classes. > > > > Yeh. When I was looking at it, it definately seems like it should be > > something that we provide a generic method of handling (as part of > > bioq_*), since all the geom classes need to handle it... > > No, in most cases this is not a problem, because most of GEOM classes > just pass all I/O requests without any reordering, so it is enough if > the very last layer (eg. disk driver) handles BIO_ORDERED properly. > > I thought what you meant with GELI was that it can reorder writes, for > which it needs more time with BIO_FLUSH requests that it handles > immediately. I did mean that GELI can reorder writes, since when it schedules the writes on a queue, there is nothing that ensures that each thread on an SMP system (I have 6 on mine) will complete the requets in the order they were queued.. So, even if GELI simply added the _FLUSH command to the queue, we'd still need to have a method for GELI to serialize the writes either going into the queue (by holding off the _ORDERED command till all outstanding _WRITES are back, but this will increase latency) or out of the queue (by giving each bio an ordered id, recording which id's have the _ORDERED flag set, and only submiting the _ORDERED command once all the previous _WRITES have been submitted)... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."