From owner-freebsd-fs@FreeBSD.ORG Wed Aug 31 08:36:28 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57544106564A for ; Wed, 31 Aug 2011 08:36:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 06D688FC13 for ; Wed, 31 Aug 2011 08:36:28 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:6407:f3f9:7d93:d34c]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id AC6074AC31; Wed, 31 Aug 2011 12:36:26 +0400 (MSD) Date: Wed, 31 Aug 2011 12:36:23 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <147623060.20110831123623@serebryakov.spb.ru> To: Jeremy Chadwick In-Reply-To: <20110831004251.GA89979@icarus.home.lan> References: <1945418039.20110830231024@serebryakov.spb.ru> <317753422.20110830231815@serebryakov.spb.ru> <20110831004251.GA89979@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Very inconsistent (read) speed on UFS2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Aug 2011 08:36:28 -0000 Hello, Jeremy. You wrote 31 =E0=E2=E3=F3=F1=F2=E0 2011 =E3., 4:42:51: > Furthermore, why are these benchmarks not providing speed data > per-device (e.g. gstat or iostat -x data)? There is a possibility that > one of your drives could be performing at less-than-ideal rates (yes, > intermittently) and therefore impacts (intermittently) your overall I/O > throughput. Ok. I've run my benchamrk when `iostat -x -d -c 999999' is running. Results are like this: device r/s w/s kr/s kw/s wait svc_t %b ada1 340.9 292.9 43138.8 146.5 0 1.2 42 ada2 340.9 293.9 43138.8 147.0 0 1.9 63 ada3 340.9 292.9 43044.7 146.5 0 1.5 57 ada4 341.9 292.9 43232.9 146.5 0 1.3 42 ada5 341.9 292.0 43138.8 146.0 2 1.3 40 Yes, NUMBERS are different from sample to sample and oscillate from 16MB/s to 80Mb/s, but they VERY consistent among disks in question. Slow read? All disks work slowly. Fast read? All disks work fast. I don't like this low-level speed oscillation too. I understand, that something higher on stack cause it. And want to understand -- WHAT. What additionally surprise me: 1) benchmark induce some writing. atime modification? No, I've turned this one off, but it doesn't help. I afraid, that this read-write interleaving could be cause of "problems", but I don't understand, WHY here is some writing (1 writing per 2 reads in average) when read-only benchmark runs. It doesn't write any logs, etc. Yes, writing speed is very low, every write transaction is about 2Kb, but WHY they are here?! If I stop benchmark, here will be less than 1 write transaction per second. 2) without `-x' it shows, that typical read transaction size is about 50Kb. It is very strange, as geom_raid5 shows (I have diagnostics in it), that almost all file access is aligned and is 128Kb-sized... P.S. Several samples for example of consistency in ONE sample and inconsistency BETWEEN samples. Random pick from output, no editing, they were in exact this order: extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ada1 165.3 87.0 10515.9 43.5 2 5.0 50 ada2 165.3 87.0 10547.2 43.5 2 7.7 61 ada3 167.2 87.0 10703.7 43.5 1 6.1 55 ada4 165.3 87.0 10484.6 43.5 3 4.9 44 ada5 160.4 87.0 10265.5 43.5 5 5.1 48 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ada1 884.1 350.9 56583.1 175.4 0 1.0 49 ada2 886.1 350.9 56677.2 175.4 0 1.3 58 ada3 882.2 349.9 56489.0 175.0 2 1.7 63 ada4 885.1 350.9 56614.5 175.4 0 1.4 64 ada5 887.1 350.9 56739.9 175.4 0 1.5 63 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ada1 640.6 261.5 41001.3 130.8 0 0.9 40 ada2 639.7 261.5 40969.9 130.8 0 0.9 35 ada3 637.7 262.5 40844.5 131.3 0 1.5 46 ada4 640.6 260.6 41001.3 130.3 1 1.3 65 ada5 638.7 261.5 40875.9 130.8 0 1.3 46 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ada1 243.7 102.8 15660.2 51.4 2 1.9 36 ada2 240.8 102.8 15503.6 51.4 3 1.9 43 ada3 242.7 103.7 15566.2 51.9 0 1.9 30 ada4 244.7 103.7 15785.5 51.9 2 2.4 56 ada5 243.7 102.8 15566.2 51.4 2 1.8 30 --=20 // Black Lion AKA Lev Serebryakov