From owner-freebsd-questions@freebsd.org Fri Jan 22 13:08:25 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 271D8A8CAEC for ; Fri, 22 Jan 2016 13:08:25 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay01.ispgateway.de (smtprelay01.ispgateway.de [80.67.31.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DCE3D1F3F for ; Fri, 22 Jan 2016 13:08:24 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.136.252] (helo=fabiankeil.de) by smtprelay01.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1aMbRk-0004PW-5m; Fri, 22 Jan 2016 14:08:12 +0100 Date: Fri, 22 Jan 2016 14:05:49 +0100 From: Fabian Keil To: Mason Loring Bliss Cc: freebsd-questions@freebsd.org Subject: Re: ZFS performance help sought Message-ID: <20160122140549.053fea0f@fabiankeil.de> In-Reply-To: <20160122000523.GL4538@blisses.org> References: <20160121205139.GG4538@blisses.org> <20160121232810.GJ4538@blisses.org> <20160121235545.GK4538@blisses.org> <20160122000523.GL4538@blisses.org> Reply-To: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/JL2.QyCqkFPVF6wmbAVcHVL"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 13:08:25 -0000 --Sig_/JL2.QyCqkFPVF6wmbAVcHVL Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Mason Loring Bliss wrote: > On Thu, Jan 21, 2016 at 06:55:45PM -0500, Mason Loring Bliss wrote: >=20 > > Here's what I set: > >=20 > > vfs.zfs.arc_max=3D"4096M" > > vfs.zfs.arc_min=3D"1024M" > > vfs.zfs.txg.timeout=3D"3" > > vfs.zfs.write_limit_override=3D"512M" > >=20 > > I'm not seeing any obvious way to verify the write_limit_override setti= ng - > > it appears not to show up in sysctl output. > >=20 > > I'll wait for the current big transfer to finish and then I'll try it w= ith > > prefetch disabled too. =20 >=20 > Disabling prefetch doesn't do a thing here - the system is still painfully > overloaded, doing something that was simply unproblematic under Linux. >=20 > I'd be grateful for further debugging or tuning tips. Is it possible this= has > nothing to do with ZFS and that I need to play with FreeBSD's scheduling > somehow? >=20 > Again, FreeBSD 10.2, ZFS tuned at noted above, and with prefetch disabled. > Eight gigs of RAM, pools less than 1TB. Doing a send/receive between pool= s on > different disks is bringing the system to its knees, where the literal sa= me > hardware under Linux/ZoL doesn't break a sweat. >=20 > What else can I try? It's conceivable that you are seeing the result of lock contention that is slowing down arc_get_data_buf(), for details see: https://www.fabiankeil.de/gehacktes/electrobsd/zfs-arc-tuning/ You could try the referenced DTrace script to check if that's the case. Even if it's a different issue, you may be able to work around it by throttling the send/receive throughput with mbuffer or a similar tool. Fabian --Sig_/JL2.QyCqkFPVF6wmbAVcHVL Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlaiKS8ACgkQBYqIVf93VJ25xgCfU/7/ZXZiZeSgvPonL+2ob4XH 1C0AoLK5e5x1CTi3JWsx4217yzHiFQ3v =Dmk9 -----END PGP SIGNATURE----- --Sig_/JL2.QyCqkFPVF6wmbAVcHVL--