From owner-freebsd-current@freebsd.org Wed Mar 22 21:25:40 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87FF1D1153E for ; Wed, 22 Mar 2017 21:25:40 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E31911AC6 for ; Wed, 22 Mar 2017 21:25:39 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from thor.intern.walstatt.dynvpn.de ([85.179.167.156]) by mail.gmx.com (mrgmx003 [212.227.17.190]) with ESMTPSA (Nemesis) id 0MhiTL-1cUTNj0I8R-00MwSx; Wed, 22 Mar 2017 22:25:36 +0100 Date: Wed, 22 Mar 2017 22:25:24 +0100 From: "O. Hartmann" To: Michael Gmelin Cc: "O. Hartmann" , FreeBSD CURRENT Subject: Re: CURRENT: slow like crap! ZFS scrubbing and ports update > 25 min Message-ID: <20170322222524.2db39c65@thor.intern.walstatt.dynvpn.de> In-Reply-To: <70346774-2E34-49CA-8B62-497BD346CBC8@grem.de> References: <20170322210225.511da375@thor.intern.walstatt.dynvpn.de> <70346774-2E34-49CA-8B62-497BD346CBC8@grem.de> Organization: WALSTATT User-Agent: OutScare 3.1415926 X-Operating-System: ImNotAnOperatingSystem 3.141592527 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/k0xYkZg4PbQTRQ0eQ5S1Va3"; protocol="application/pgp-signature" X-Provags-ID: V03:K0:tL656RmJBQelSvl4FP6NhdjpGNU/OcpGtHQNul2fqnEOpR+wC8j mosN+kaidkeEHAkcbzVge5w4ri3T2+SW8D5/3WnHLo5SiWi406JJI59Z3k87Adajobkb98o DI1gMTixb6kaXtNhycpoQfAUmnXoAL4LcnaK3AxYesQjoQLriTrTl+Pthl0IrO8ACU4w4WP JLb9L+1TEqj9eWqqfd/1A== X-UI-Out-Filterresults: notjunk:1;V01:K0:d3lsdyv/gNQ=:FMaAAm+1VbOnm48zmKVs31 41/5sP5FQgFBIqCkFllKp/qoNMTmQln2b6SMyl9AJM3L9mXXyPqCma6ZBtIqOxJxcsvLH0Y5u epmCclP/ngK5szf+Voi1jHyNnDmAJ01FMF5Kx/Z9ir1es7nBbk+Ihnhoz7nz1RC0IlRieHGlj 3jvZCfmdWG47AW6TIcTHE7jm8u1jornH+5BjTNdFrv7Uelcb9tfcqXohpNpnZxt0tugNWhRU6 Fo3ZSzGhme6yDm0iLrgQzrNbXWnA5a5MDE8U49HQB1zsI1fphQb4yz2t0gXc89dd7Z+SkYpIC LWdKk/uOXcakEDY1EAlR/jLEf5RsGyc/FMs/NlB3rOeESQmEzdX/ArkMdCpDZhfEibpMF5yKw Er69XYZ1mEoqjvjJzl33QCIE30nbEPdiLKGieXli+KGqdFRpTJYUJipI6y06lgxZ8iRWd+/ux coAW31CGWk6X/1P5UULZ2dUkRurbBdAYYWXtZA0k7uZjxhWFLE66b1usqvVV6NCqqCFnKA2/M oPr7T9qMaG4DMiZ7wyLyR22di1f4/ga589JMQ+mhwFY/o6elJzAmvZEvyZ0T5xeSeVSko72D8 kvYTcgvQN2trMvI3GKS1OgrV3xKWClul2+c6DIddR9z9N0EZa5Sug5itsH/mt1l4cOITkk/Zp S0Op4CWbZH/EeEN1La5OWBNsLyywlKXPwSHXzuI+b71vNSf+8odvOTNXT0Enbu7CR5BcLCnU3 TFf2Ht8bE7esUS95Z8pAsajYvdvaqDVu+RboRA7Xo1oim7L3LAsq+RP5yqw= X-Mailman-Approved-At: Thu, 23 Mar 2017 10:58:42 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 21:25:40 -0000 --Sig_/k0xYkZg4PbQTRQ0eQ5S1Va3 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Am Wed, 22 Mar 2017 21:10:51 +0100 Michael Gmelin schrieb: > > On 22 Mar 2017, at 21:02, O. Hartmann wrote: > >=20 > > CURRENT (FreeBSD 12.0-CURRENT #82 r315720: Wed Mar 22 18:49:28 CET 2017= amd64) is > > annoyingly slow! While scrubbing is working on my 12 GB ZFS volume, > > updating /usr/ports takes >25 min(!). That is an absolute record now. > >=20 > > I do an almost daily update of world and ports tree and have periodic = scrubbing ZFS > > volumes every 35 days, as it is defined in /etc/defaults. Prts tree has= n't grown much, > > the content of the ZFS volume hasn't changed much (~ 100 GB, its fill i= s about 4 TB > > now) and this is now for ~ 2 years constant.=20 > >=20 > > I've experienced before that while scrubbing the ZFS volume, some opera= tions, even the > > update of /usr/ports which resides on that ZFS RAIDZ volume, takes a bi= t longer than > > usual - but never that long like now! > >=20 > > Another box is quite unusable while it is scrubbing and it has been usa= ble times > > before. The change is dramatic ... > > =20 >=20 > What do "zpool list", "gstat" and "zpool status" show? >=20 >=20 >=20 zpool list: NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTRO= OT TANK00 10.9T 5.45T 5.42T - 7% 50% 1.58x ONLINE - Deduplication is off right now, I had one ZFS filesystem with dedup enabled gstat: not shown here, but the drives comprise the volume (4x 3 TB) show 10= 0% busy each, but one drive is always a bit off (by 10% lower) and this drive is walking = through all four drives ada2, ada3, ada4 and ada5. Nothing unusual in that situation. B= ut the throughput is incredible low, for example ada4: L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 2 174 174 1307 11.4 0 0 0.0 99.4| ada4 kBps (kilo Bits per second I presume) are peaking at ~ 4800 - 5000. On anot= her bos, this is ~ 20x higher! Most time, kBps r and w stay at ~ 500 -600. zpool status: pool: TANK00 state: ONLINE scan: scrub in progress since Wed Mar 22 19:12:41 2017 167G scanned out of 5.62T at 15.9M/s, 99h46m to go 0 repaired, 2.91% done config: NAME STATE READ WRITE CKSUM TANK00 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/tank00 ONLINE 0 0 0 gpt/tank01 ONLINE 0 0 0 gpt/tank02 ONLINE 0 0 0 gpt/tank03 ONLINE 0 0 0 logs gpt/zil00 ONLINE 0 0 0 cache gpt/l2arc00 ONLINE 0 0 0 errors: No known data errors Here are the incredible low numbers more clear. Usually, the hardware is ca= pable of doing ~ 280 - 300 MBytes/s. It is stuck for hours at 15 - 16 MBytes/s. --Sig_/k0xYkZg4PbQTRQ0eQ5S1Va3 Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iLUEARMKAB0WIQQZVZMzAtwC2T/86TrS528fyFhYlAUCWNLrxAAKCRDS528fyFhY lDgaAf4901djZ+7HsfgS9kDrGcWXKwMO14DRCRFHla8MHLv1eTp7XEkYzTX3W1Yn 5Cc89j2XT+HqAivLkfOTs8KuqD3KAgCde4j6MFK8ypzuZgBZh5ILsuNDEyv/NXcx GTvBUJLU4vlDIRWrytlPdSMxI748DQaLb8xsSurUxTdHDLrGuMg8 =V+Mn -----END PGP SIGNATURE----- --Sig_/k0xYkZg4PbQTRQ0eQ5S1Va3--