From owner-freebsd-fs@FreeBSD.ORG Wed Aug 31 08:48:42 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6FC71106564A for ; Wed, 31 Aug 2011 08:48:42 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id F09158FC14 for ; Wed, 31 Aug 2011 08:48:41 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p7V8mXtE067659 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Wed, 31 Aug 2011 11:48:38 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4E5DF560.1050507@digsys.bg> Date: Wed, 31 Aug 2011 11:48:32 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110822 Thunderbird/6.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <1945418039.20110830231024@serebryakov.spb.ru> <317753422.20110830231815@serebryakov.spb.ru> <20110831004251.GA89979@icarus.home.lan> <147623060.20110831123623@serebryakov.spb.ru> In-Reply-To: <147623060.20110831123623@serebryakov.spb.ru> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Very inconsistent (read) speed on UFS2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Aug 2011 08:48:42 -0000 On 31.08.11 11:36, Lev Serebryakov wrote: > device r/s w/s kr/s kw/s wait svc_t %b > ada1 340.9 292.9 43138.8 146.5 0 1.2 42 > ada2 340.9 293.9 43138.8 147.0 0 1.9 63 > ada3 340.9 292.9 43044.7 146.5 0 1.5 57 > ada4 341.9 292.9 43232.9 146.5 0 1.3 42 > ada5 341.9 292.0 43138.8 146.0 2 1.3 40 > Very interesting, this writes. You need to find out what is causing these. Just some random thoughts: This flapping may have something to do with the drives' internal caches. What are the drives? SATA drives, unlike SAS have simplex communication with the host, that is, the drive cannot simultaneously read and write data and commands (from/to host). There might be some, perhaps locking contention in there? It is not contention for bandwidth obviously. Most consumer drives have rather low IOPS performance. This is especially pronounced when there are both reads and writes. Your IOPS rate here is relatively high for such drive, although the busy percentage is low -- but then, it may not be accurate. In any case, you cannot measure read performance as long as it intermixes with writes, especially as you noted that your RAID5 code has some non-obvious write characteristics/optimizations. Daniel