Date: Sun, 23 Sep 2012 00:10:15 -0700 From: John-Mark Gurney <jmg@funkthat.com> To: Pawel Jakub Dawidek <pjd@freebsd.org> Cc: freebsd-geom@freebsd.org Subject: Re: geli and BIO_FLUSH and/or BIO_ORDERED issue? Message-ID: <20120923070842.GJ19036@funkthat.com> In-Reply-To: <20120923064307.GK1454@garage.freebsd.pl> References: <20120919040430.GF19036@funkthat.com> <20120922162025.GE1454@garage.freebsd.pl> <20120923044828.GI19036@funkthat.com> <20120923064307.GK1454@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
Pawel Jakub Dawidek wrote this message on Sun, Sep 23, 2012 at 08:43 +0200:
> On Sat, Sep 22, 2012 at 09:48:28PM -0700, John-Mark Gurney wrote:
> > Pawel Jakub Dawidek wrote this message on Sat, Sep 22, 2012 at 18:20 +0200:
> > > On Tue, Sep 18, 2012 at 09:04:30PM -0700, John-Mark Gurney wrote:
> > > > I was looking at geli and I'm not sure if it's implementing BIO_FLUSH
> > > > and/or BIO_ORDERED properly...
> > > >
> > > > >From my understanding is the BIO_ORDERED is suppose to wait for the
> > > > previous _WRITES to complete before returning so that you can ensure
> > > > that data is on disk, i.e. _ORDERED is set on a BIO_FLUSH...
> > > >
> > > > BIO_ORDERED is handled by diskq_* code such that when you add an _ORDERED
> > > > command, all commands are put after it, but there doesn't appear to
> > > > be any code to ensure that an _ORDERED command waits for prevoius
> > > > pending commands to complete..
> > > >
> > > > This is extra obvious in eli in that a _FLUSH is immediately dispatched,
> > > > even when there may be _WRITEs that haven't been finished encrypting and
> > > > sent down to the disk to get _FLUSHed...
> > > >
> > > > Any comments about this?
> > >
> > > Hmm, BIO_ORDERED was introduced pretty recently and GEOM classes were
> > > not updated to honour it, but it also seems to be to complex to handle
> > > in GEOM classes. I wonder if we could hold off new writes and wait for
> > > the in-progress writes in GEOM if we spot BIO_ORDERED request without
> > > the need to implement this logic in GEOM classes.
> >
> > Yeh. When I was looking at it, it definately seems like it should be
> > something that we provide a generic method of handling (as part of
> > bioq_*), since all the geom classes need to handle it...
>
> No, in most cases this is not a problem, because most of GEOM classes
> just pass all I/O requests without any reordering, so it is enough if
> the very last layer (eg. disk driver) handles BIO_ORDERED properly.
>
> I thought what you meant with GELI was that it can reorder writes, for
> which it needs more time with BIO_FLUSH requests that it handles
> immediately.
I did mean that GELI can reorder writes, since when it schedules the
writes on a queue, there is nothing that ensures that each thread on
an SMP system (I have 6 on mine) will complete the requets in the
order they were queued..
So, even if GELI simply added the _FLUSH command to the queue, we'd
still need to have a method for GELI to serialize the writes either
going into the queue (by holding off the _ORDERED command till all
outstanding _WRITES are back, but this will increase latency) or out
of the queue (by giving each bio an ordered id, recording which id's
have the _ORDERED flag set, and only submiting the _ORDERED command
once all the previous _WRITES have been submitted)...
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120923070842.GJ19036>