From owner-freebsd-fs@FreeBSD.ORG Tue Jan 8 20:47:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 16ACC527 for ; Tue, 8 Jan 2013 20:47:40 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-vb0-f52.google.com (mail-vb0-f52.google.com [209.85.212.52]) by mx1.freebsd.org (Postfix) with ESMTP id A127EA37 for ; Tue, 8 Jan 2013 20:47:39 +0000 (UTC) Received: by mail-vb0-f52.google.com with SMTP id ez10so872122vbb.11 for ; Tue, 08 Jan 2013 12:47:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=0wKGk8NiR/gzbHt2wwZCLsU4tiSUL57mDpUY9wLUw4s=; b=FpQzB5BIQgFK7BrIdNULqINLMQQvZcQ9YawjUPAqSe8EZPVrxKACoL4qBanTGqtj+l D1MYBH2YeCedcqvjLEjGDzzSf+f8PeC4xHd8T1iSNYpyu4eKXQJxkCyRlUtOo56pkh+m 0ICAJOexTpq2Bzt0RJv+MJpIL2+thEsBVZJMNtOeVHepZfzKYj+10CMvpWaKJRXtz8Gy x3/wf4g3PNLSJfQ9xFa0MhIRHIP75hjGe/4B3DxA7cJuyXSsrwdxok8g8OCEwI+LKKeZ uF3/xysJwVWsQOSoKzRafu8cysArzzl8b63l3qZOuHMHD3TRm88b1IjGZaMz4e5ul/vb iAkA== MIME-Version: 1.0 Received: by 10.220.151.83 with SMTP id b19mr86687753vcw.25.1357678058734; Tue, 08 Jan 2013 12:47:38 -0800 (PST) Sender: artemb@gmail.com Received: by 10.220.122.196 with HTTP; Tue, 8 Jan 2013 12:47:38 -0800 (PST) In-Reply-To: <20130108174225.GA17260@mid.pc5.i.0x5.de> References: <20130108174225.GA17260@mid.pc5.i.0x5.de> Date: Tue, 8 Jan 2013 12:47:38 -0800 X-Google-Sender-Auth: Y5qqrkapwiEmXHcuD5d-LMC04q8 Message-ID: Subject: Re: slowdown of zfs (tx->tx) From: Artem Belevich To: Nicolas Rachinsky Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 20:47:40 -0000 On Tue, Jan 8, 2013 at 9:42 AM, Nicolas Rachinsky wrote: > NAME STATE READ WRITE CKSUM > pool1 DEGRADED 0 0 0 > raidz2-0 DEGRADED 0 0 0 > ada5 ONLINE 0 0 0 > ada8 ONLINE 0 0 0 > ada2 ONLINE 0 0 0 > ada3 ONLINE 0 0 0 > 11846390416703086268 UNAVAIL 0 0 0 was /dev/dsk/ada1 > ada6 ONLINE 0 0 0 > ada0 ONLINE 0 0 1 > ada7 ONLINE 0 0 0 > ada4 ONLINE 0 0 3 You seem to have some checksum errors which does suggest hardware troubles. For starters, check smart info for all drives and see if they have any relocated sectors. Use gstat during your workload to see if any of the drives takes much longer than others to handle its job. > There is almost no disk activity during this time. What kind of disk activity *is* there? Sleeping on 'tx->tx...' usually means that ZFS is trying to commit data to disk. Normally it happens once every few seconds (10 is default if I remember correctly). It may happen more often if you do a lot of synchronous writes. I believe there was an iostat-like dtrace script that would show synchronous write rate, but I can't seem to find it. > sync is disabled for the whole pool. If that's the case (assyming you're talking about sync=disabled zfs property), then synchronous writes are probably not the cause of slowdown. My guess would be either failing HDD or something funky with cabling or sata controller. --Artem