From owner-freebsd-fs@FreeBSD.ORG Wed Jan 16 00:16:47 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CC3AE599 for ; Wed, 16 Jan 2013 00:16:47 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-vc0-f180.google.com (mail-vc0-f180.google.com [209.85.220.180]) by mx1.freebsd.org (Postfix) with ESMTP id 8CF557C4 for ; Wed, 16 Jan 2013 00:16:47 +0000 (UTC) Received: by mail-vc0-f180.google.com with SMTP id p16so762001vcq.39 for ; Tue, 15 Jan 2013 16:16:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=RZ4UGXlPyiFkjowbzCuy+4v/0dm3vdNPzEKHocW0Nhc=; b=lf1S93PQXn8v7iWzMVhoRF/N8O2yUWBseY9wFk9fN5Zgtmf72aFFP02lv/rfrNUYuY zzVIR55EAaGxe0XMFqpcLbfy4uQcfx5bMa4zxI2tkW+quHkLE6fZNqCh9d3wbQjC4ZY0 eRbe2wL4vGQFQwocjUjW1yE0OmRUeuDKVdJgWTyhKloHJ2C6P7scU2RHlo3/WD8A/buF V2uL12pS/nmVW/g+fjEr/F19be4OBihNm9KyEylErzdz5FoBWqVslhqqwoEpDb37S/zT 0g+RzyrVpDUtsPCDknCo86OPo9KhD2FY4VP/X2ka411KbQrBbgiuOYsIdf02PZQBgiOP dc9w== MIME-Version: 1.0 Received: by 10.220.153.201 with SMTP id l9mr106550322vcw.33.1358295401115; Tue, 15 Jan 2013 16:16:41 -0800 (PST) Sender: artemb@gmail.com Received: by 10.220.122.196 with HTTP; Tue, 15 Jan 2013 16:16:40 -0800 (PST) In-Reply-To: <20130115224556.GA41774@mid.pc5.i.0x5.de> References: <20130110193949.GA10023@mid.pc5.i.0x5.de> <20130111073417.GA95100@mid.pc5.i.0x5.de> <20130114094010.GA75529@mid.pc5.i.0x5.de> <20130114195148.GA20540@mid.pc5.i.0x5.de> <20130114214652.GA76779@mid.pc5.i.0x5.de> <20130115224556.GA41774@mid.pc5.i.0x5.de> Date: Tue, 15 Jan 2013 16:16:40 -0800 X-Google-Sender-Auth: kDVbLxBGd15xXKwDJfHspV66yps Message-ID: Subject: Re: slowdown of zfs (tx->tx) From: Artem Belevich To: Nicolas Rachinsky Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2013 00:16:47 -0000 On Tue, Jan 15, 2013 at 2:45 PM, Nicolas Rachinsky wrote: > 147 0 100098 kernel zio_write_issue_ mi_switch+0x176 sleep= q_wait+0x42 _cv_wait+0x129 space_map_load_wait+0x20 metaslab_activate+0x73 = metaslab_alloc+0x7b2 zio_dva_allocate+0x9a zio_execute+0xc3 taskqueue_run_l= ocked+0x85 taskqueue_thread_loop+0x4e fork_exit+0x11f fork_trampoline+0xe It appears that lots of threads are stuck in metaslab_activate->space_map_load_wait path. This sounds like CR# 6876962 in Solaris: "degraded write performance with threads held up by space_map_load_wait(). This bug is fixed in patch 147440-05, -06 or -07, which is current and contains the fix." Alas, I could not find specifics on how the issue got fixed and whether the same fix is present in illumos and FreeBSD. You may want to update your system to very recent FreeBSD as quite a few fixes were recently imported from illumos. Hopefully it will deal with the issue. I'm out of ideas otherwise. Sorry. --Artem