From owner-freebsd-questions@freebsd.org Wed Sep 23 20:42:55 2015 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44786A07A5D for ; Wed, 23 Sep 2015 20:42:55 +0000 (UTC) (envelope-from healer@rpi.edu) Received: from smtp9.server.rpi.edu (gateway.canit.rpi.edu [128.113.2.229]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "canit.localdomain", Issuer "canit.localdomain" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0609A1961 for ; Wed, 23 Sep 2015 20:42:54 +0000 (UTC) (envelope-from healer@rpi.edu) Received: from smtp-auth2.server.rpi.edu (route.canit.rpi.edu [128.113.2.232]) by smtp9.server.rpi.edu (8.14.3/8.14.3/Debian-9.4) with ESMTP id t8NKd64E015218 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Wed, 23 Sep 2015 16:39:06 -0400 Received: from smtp-auth2.server.rpi.edu (localhost [127.0.0.1]) by smtp-auth2.server.rpi.edu (Postfix) with ESMTP id 11EDB18112 for ; Wed, 23 Sep 2015 16:39:06 -0400 (EDT) Received: from [129.161.61.54] (biotech-lower-wl-295.dynamic2.rpi.edu [129.161.61.54]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: healer) by smtp-auth2.server.rpi.edu (Postfix) with ESMTPSA id F2AF918105 for ; Wed, 23 Sep 2015 16:39:02 -0400 (EDT) Subject: Re: zfs performance degradation To: freebsd-questions@freebsd.org References: <56019211.2050307@dim.lv> From: Bob Healey Message-ID: <56030DE8.20009@rpi.edu> Date: Wed, 23 Sep 2015 16:39:04 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <56019211.2050307@dim.lv> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP X-Bayes-Prob: 0.0001 (Score 0, tokens from: outgoing, @@RPTN) X-Spam-Score: 0.00 () [Hold at 7.10] X-CanIt-Incident-Id: 02PkID6TX X-CanIt-Geo: ip=129.161.61.54; country=US; region=Connecticut; city=Hartford; latitude=41.7424; longitude=-72.6905; http://maps.google.com/maps?q=41.7424,-72.6905&z=6 X-CanItPRO-Stream: outgoing X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.229 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Sep 2015 20:42:55 -0000 That sounds similar to the issues I posted about 10 days ago, but my issues seem to be more time than capacity based and go away on reboot. I haven't had time to follow up on my thread due to more pressing issues at work involving Microsoft products, but its still on my TODO list. Bob Healey Systems Administrator Biocomputation and Bioinformatics Constellation and Molecularium healer@rpi.edu (518) 276-4407 On 9/22/2015 1:38 PM, Dmitrijs wrote: > Goog afternoon, > > I've encountered strange ZFS behavior - serious performance > degradation over few days. Right after setup on fresh ZFS (2 hdd in a > mirror) I made a test on a file 30Gb size with dd like > dd if=test.mkv of=/dev/null bs=64k > and got 150+Mbs speed. > > Today I got only 90Mbs, tested with different blocksizes, many times, > speed seems to be stable +-5% > > nas4free: divx# dd if=test.mkv of=/dev/null bs=64k > 484486+1 records in > 484486+1 records out > 31751303111 bytes transferred in 349.423294 secs (90867734 bytes/sec) > > > > Computer\system details: > > nas4free: /mnt# uname -a > FreeBSD nas4free.local 10.2-RELEASE-p2 FreeBSD 10.2-RELEASE-p2 #0 > r287260M: Fri Aug 28 18:38:18 CEST 2015 > root@dev.nas4free.org:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64 > > RAM 4Gb > I've got brand new 2x HGST HDN724040ALE640, 4Тб, 7200rpm (ada0, ada1) > for pool data4. > Another pool, data2, performs slightly better even on older\cheaper WD > Green 5400 HDDs, up to 99Mbs. > > > nas4free: /mnt# zpool status > pool: data2 > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > data2 ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ada2 ONLINE 0 0 0 > ada3 ONLINE 0 0 0 > > errors: No known data errors > > pool: data4 > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > data4 ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ada0 ONLINE 0 0 0 > ada1 ONLINE 0 0 0 > > errors: No known data errors > > > While dd is running, gstat is showing like: > > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 0 366 366 46648 1.1 0 0 0.0 39.6| ada0 > 1 432 432 54841 1.0 0 0 0.0 45.1| ada1 > > > > so iops are very high, while %busy is quite low. It averages about > 50%, rare peaks till 85-90% > > Even top shows no significant load: > > last pid: 61983; load averages: 0.44, 0.34, 0.37 up 11+07:51:31 16:44:56 > 40 processes: 1 running, 39 sleeping > CPU: 0.3% user, 0.0% nice, 6.4% system, 1.1% interrupt, 92.1% idle > Mem: 21M Active, 397M Inact, 2101M Wired, 56M Cache, 94M Buf, 1044M Free > ARC: 1024M Total, 232M MFU, 692M MRU, 160K Anon, 9201K Header, 91M Other > Swap: 4096M Total, 4096M Free > Not displaying idle processes. > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 61981 root 1 30 0 12364K 2084K zio->i 3 0:09 18.80% dd > 61966 root 1 22 0 58392K 7144K select 3 0:24 3.86% proftpd > > > > Zpool list: > > nas4free: /mnt# zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > data2 1.81T 578G 1.25T - 11% 31% 1.00x ONLINE - > data4 3.62T 2.85T 797G - 36% 78% 1.00x ONLINE - > > > Could it happen because of pool being 78% full? So I cannot fill puls > full? > Can anyone please advice how could I fix the situation - or is it normal? > I've googled a lot about vmaxnodes, vminnodes but advices are mostly > controversial and doesn't help. > I can provide additional system output on request. > > best regards, > Dmitry > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org"