From owner-freebsd-fs@FreeBSD.ORG Wed Aug 31 09:03:58 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CFA301065670 for ; Wed, 31 Aug 2011 09:03:58 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 944008FC19 for ; Wed, 31 Aug 2011 09:03:58 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:6407:f3f9:7d93:d34c]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id DDBFD4AC31; Wed, 31 Aug 2011 13:03:56 +0400 (MSD) Date: Wed, 31 Aug 2011 13:03:54 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <177519198.20110831130354@serebryakov.spb.ru> To: Daniel Kalchev In-Reply-To: <4E5DF560.1050507@digsys.bg> References: <1945418039.20110830231024@serebryakov.spb.ru> <317753422.20110830231815@serebryakov.spb.ru> <20110831004251.GA89979@icarus.home.lan> <147623060.20110831123623@serebryakov.spb.ru> <4E5DF560.1050507@digsys.bg> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Very inconsistent (read) speed on UFS2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Aug 2011 09:03:58 -0000 Hello, Daniel. You wrote 31 =E0=E2=E3=F3=F1=F2=E0 2011 =E3., 12:48:32: > On 31.08.11 11:36, Lev Serebryakov wrote: >> device r/s w/s kr/s kw/s wait svc_t %b >> ada1 340.9 292.9 43138.8 146.5 0 1.2 42 >> ada2 340.9 293.9 43138.8 147.0 0 1.9 63 >> ada3 340.9 292.9 43044.7 146.5 0 1.5 57 >> ada4 341.9 292.9 43232.9 146.5 0 1.3 42 >> ada5 341.9 292.0 43138.8 146.0 2 1.3 40 >> > Very interesting, this writes. You need to find out what is causing these. Yep. I've been very surprised by them. > Just some random thoughts: > This flapping may have something to do with the drives' internal caches. > What are the drives? WD20EARS, it is WD Green 2Tb, advanced format. Yes, I know, that they are not best performers at all, when here are seeks. It is why I don't expect good performance in random or multi-threaded (multi-client) access patterns here. And, yes, I know about advanced format. Stripe size is 128Kb, and GEOM is built from raw drives, so all stripes are aligned. FS is created on raw GEOM, without any partitioning again, and block size is 32Kb, so everything should be aligned here too. Really, if all reading speeds were, say, 120MiB/s, but every time and consistent, I don't start this thread. In case I would blame HDDs and my parsimony, but not software :) > SATA drives, unlike SAS have simplex communication with the host, that > is, the drive cannot simultaneously read and write data and commands=20 > (from/to host). There might be some, perhaps locking contention in=20 > there? It is not contention for bandwidth obviously. Yep... > In any case, you cannot measure read performance as long as it > intermixes with writes, especially as you noted that your RAID5 code has > some non-obvious write characteristics/optimizations. I understand. Now I should understand how to pin down these writes. --=20 // Black Lion AKA Lev Serebryakov