From owner-freebsd-fs@FreeBSD.ORG Fri Jan 11 20:39:50 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 145878BA for ; Fri, 11 Jan 2013 20:39:50 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-vb0-f50.google.com (mail-vb0-f50.google.com [209.85.212.50]) by mx1.freebsd.org (Postfix) with ESMTP id B0CC3F5D for ; Fri, 11 Jan 2013 20:39:49 +0000 (UTC) Received: by mail-vb0-f50.google.com with SMTP id ft2so1828565vbb.23 for ; Fri, 11 Jan 2013 12:39:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=YTp8OpFe66CMNyTOIh1OhoVCn4e+9sdZQQUnBNOX2JM=; b=U7s/M7ffnmiBTFmeK5B8jK2MFgOvvAjIXtAVRDKhvw4/tmiope8TWrb/lkc3C6gJaU Pl6s5WEtpeXBuW36dwC/ITJN6AhDvE08k8jgEdHry5pcpsKyR3G2xtflC/kv/yGt9DcN Uff+dzzrGeUVErNpPbONP5KEJhCHXvp+gpo7mGvrxH11hQUiKmWKJdB9aKbXwjzlmeY2 z1+mKvWy0hbyf1BzruzKE4vtGlAM1zeeQeNd03GZJLLFWJDjLn/cW2nZAIpTwttebAQc bHQStJKYhk6JbNQvqn1X+OpQcTpNU8VEWXRRbJE6RePY3m97ZqpiVPq7Xe3h4Kj+kHhw q9UA== MIME-Version: 1.0 Received: by 10.220.153.201 with SMTP id l9mr94404963vcw.33.1357936783342; Fri, 11 Jan 2013 12:39:43 -0800 (PST) Sender: artemb@gmail.com Received: by 10.220.122.196 with HTTP; Fri, 11 Jan 2013 12:39:43 -0800 (PST) In-Reply-To: <20130111073417.GA95100@mid.pc5.i.0x5.de> References: <20130108174225.GA17260@mid.pc5.i.0x5.de> <20130109162613.GA34276@mid.pc5.i.0x5.de> <20130110193949.GA10023@mid.pc5.i.0x5.de> <20130111073417.GA95100@mid.pc5.i.0x5.de> Date: Fri, 11 Jan 2013 12:39:43 -0800 X-Google-Sender-Auth: Os9MM8m1yqNVQHWmrImVZq9nbp8 Message-ID: Subject: Re: slowdown of zfs (tx->tx) From: Artem Belevich To: Nicolas Rachinsky Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2013 20:39:50 -0000 On Thu, Jan 10, 2013 at 11:34 PM, Nicolas Rachinsky wrote: > * Nicolas Rachinsky [2013-01-10 20:39 +0100]: >> after replacing one of the controllers, all problems seem to have >> disappeared. Thank you very much for your advice! > > Now the problem is back. > > After changing the controller, there were no more timeouts logged. > > No UDMA_CRC_Error_Count changed. > Is there anything special about ada8? It does seem to have noticeably higher service time compared to other disks. Cound you do gstat with 1-second interval. Some of the 5-second samples show that ada8 is the bottleneck -- it has its request queue full (L(q)=10) when all other drives were done with their jobs. And that's a 5-sec average. Its write service time also seems to be a lot higher than for other drives. Does the drive have its write cache disabled by any chance? That could explain why it takes so much longer to service writes. Can you remove ada8 and see if your performance go back to normal? --Artem