Date: Wed, 16 Jan 2013 00:57:03 -0000 From: "Steven Hartland" <killing@multiplay.co.uk> To: "Artem Belevich" <art@freebsd.org>, "Nicolas Rachinsky" <fbsd-mas-0@ml.turing-complete.org> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: slowdown of zfs (tx->tx) Message-ID: <00F86FD0E85D4EEEA1A01E115497F022@multiplay.co.uk> References: <CAFqOu6jrng=v8eVyhqV-PBqJM_dYy%2BU7X4%2B=ahBeoxvK4mxcSA@mail.gmail.com> <20130110193949.GA10023@mid.pc5.i.0x5.de> <20130111073417.GA95100@mid.pc5.i.0x5.de> <CAFqOu6gWpMsWN0pTBiv10WfwyGWMfO9GzMLWTtcVxHixr-_i3Q@mail.gmail.com> <20130114094010.GA75529@mid.pc5.i.0x5.de> <CAFqOu6hxfGt_M6Jo9qWeifDz9YnNc_Bd9H-GEe4RYtutaPvH5w@mail.gmail.com> <20130114195148.GA20540@mid.pc5.i.0x5.de> <CAFqOu6jwJ4qhbOovN_NhzusdQJvrbvUC3g93sziR=Uw99SGenw@mail.gmail.com> <20130114214652.GA76779@mid.pc5.i.0x5.de> <CAFqOu6jKX-Ks6C1RK5GwZ51ZVUSnGSe7S99_EfK%2BfwLPjAFFYw@mail.gmail.com> <20130115224556.GA41774@mid.pc5.i.0x5.de> <CAFqOu6jJnWdbikPmE1-UML5i_x7meF%2BiyY=9WBRyv2j7AeOaSg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message -----
From: "Artem Belevich" <art@freebsd.org>
To: "Nicolas Rachinsky" <fbsd-mas-0@ml.turing-complete.org>
Cc: "freebsd-fs" <freebsd-fs@freebsd.org>
Sent: Wednesday, January 16, 2013 12:16 AM
Subject: Re: slowdown of zfs (tx->tx)
> On Tue, Jan 15, 2013 at 2:45 PM, Nicolas Rachinsky
> <fbsd-mas-0@ml.turing-complete.org> wrote:
>> 147 0 100098 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 space_map_load_wait+0x20
>> metaslab_activate+0x73 metaslab_alloc+0x7b2 zio_dva_allocate+0x9a zio_execute+0xc3 taskqueue_run_locked+0x85
>> taskqueue_thread_loop+0x4e fork_exit+0x11f fork_trampoline+0xe
>
> It appears that lots of threads are stuck in
> metaslab_activate->space_map_load_wait path. This sounds like CR#
> 6876962 in Solaris: "degraded write performance with threads held up
> by space_map_load_wait(). This bug is fixed in patch 147440-05, -06 or
> -07, which is current and contains the fix." Alas, I could not find
> specifics on how the issue got fixed and whether the same fix is
> present in illumos and FreeBSD.
>
> You may want to update your system to very recent FreeBSD as quite a
> few fixes were recently imported from illumos. Hopefully it will deal
> with the issue. I'm out of ideas otherwise. Sorry.
That would tend to indicate its blocking on write. If this is the case
yet the rsync is copying from this box, with little else doing writes
it could be atime which is causing the issue.
A test for this would be to use the following to disable atime and see
if that helps:
zfs set atime=off [filesystem]
Also out of interest does the pool have many snapshots?
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?00F86FD0E85D4EEEA1A01E115497F022>