From owner-freebsd-fs@FreeBSD.ORG Wed Jan 16 00:56:52 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 49CFD7C9; Wed, 16 Jan 2013 00:56:52 +0000 (UTC) (envelope-from prvs=1728d5906c=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 5EB15A16; Wed, 16 Jan 2013 00:56:51 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50001720881.msg; Wed, 16 Jan 2013 00:56:43 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 16 Jan 2013 00:56:43 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1728d5906c=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <00F86FD0E85D4EEEA1A01E115497F022@multiplay.co.uk> From: "Steven Hartland" To: "Artem Belevich" , "Nicolas Rachinsky" References: <20130110193949.GA10023@mid.pc5.i.0x5.de> <20130111073417.GA95100@mid.pc5.i.0x5.de> <20130114094010.GA75529@mid.pc5.i.0x5.de> <20130114195148.GA20540@mid.pc5.i.0x5.de> <20130114214652.GA76779@mid.pc5.i.0x5.de> <20130115224556.GA41774@mid.pc5.i.0x5.de> Subject: Re: slowdown of zfs (tx->tx) Date: Wed, 16 Jan 2013 00:57:03 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2013 00:56:52 -0000 ----- Original Message ----- From: "Artem Belevich" To: "Nicolas Rachinsky" Cc: "freebsd-fs" Sent: Wednesday, January 16, 2013 12:16 AM Subject: Re: slowdown of zfs (tx->tx) > On Tue, Jan 15, 2013 at 2:45 PM, Nicolas Rachinsky > wrote: >> 147 0 100098 kernel zio_write_issue_ mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 space_map_load_wait+0x20 >> metaslab_activate+0x73 metaslab_alloc+0x7b2 zio_dva_allocate+0x9a zio_execute+0xc3 taskqueue_run_locked+0x85 >> taskqueue_thread_loop+0x4e fork_exit+0x11f fork_trampoline+0xe > > It appears that lots of threads are stuck in > metaslab_activate->space_map_load_wait path. This sounds like CR# > 6876962 in Solaris: "degraded write performance with threads held up > by space_map_load_wait(). This bug is fixed in patch 147440-05, -06 or > -07, which is current and contains the fix." Alas, I could not find > specifics on how the issue got fixed and whether the same fix is > present in illumos and FreeBSD. > > You may want to update your system to very recent FreeBSD as quite a > few fixes were recently imported from illumos. Hopefully it will deal > with the issue. I'm out of ideas otherwise. Sorry. That would tend to indicate its blocking on write. If this is the case yet the rsync is copying from this box, with little else doing writes it could be atime which is causing the issue. A test for this would be to use the following to disable atime and see if that helps: zfs set atime=off [filesystem] Also out of interest does the pool have many snapshots? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.