From owner-freebsd-hackers@FreeBSD.ORG Thu Nov 13 13:10:41 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4C2E7455 for ; Thu, 13 Nov 2014 13:10:41 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 12F6F91D for ; Thu, 13 Nov 2014 13:10:40 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArcEAEysZFSDaFve/2dsb2JhbABbg2JZBIMCyWiHUwKBNQEBAQEBfYQCAQEBAwEjVgUUAhgCAg0ZAlkGE4g4CZ0KnF+WGAEBAQEBAQEDAQEBAQEBAQEBGYEtjx4VATMHgneBVAWFIgOGZpAWg0SDT5FkhBofMIEHQYEDAQEB X-IronPort-AV: E=Sophos;i="5.07,377,1413259200"; d="scan'208";a="167822535" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 13 Nov 2014 08:09:31 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id C6180B40D6; Thu, 13 Nov 2014 08:09:31 -0500 (EST) Date: Thu, 13 Nov 2014 08:09:31 -0500 (EST) From: Rick Macklem To: Mark Schouten Message-ID: <54043890.12566825.1415884171801.JavaMail.root@uoguelph.ca> In-Reply-To: <2826701214-10966@kerio.tuxis.nl> Subject: Re: Slow nfsd write performance, tweaks needed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 13:10:41 -0000 Mark Schouten wrote: > Hi, >=20 >=20 > I am in the process of switching from a ZFS On Linux-based NFS-server > to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is > way superiour over ZoL, and the box serves as storage for a > virtualizationplatform, so stability is welcome. :) >=20 >=20 > The box is stable, but performs terribly. Surely, I'm doing something > wrong, but I would like some tips and tricks to speed things up. >=20 >=20 >=20 >=20 > Here's my setup: > CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is > enabled) > RAM: 64GB > NIC: 2x igb in lagg0 (loadbalancing) Oops again. I don't think igb had the 64K TSO problem (I get the Intel nics confused;-), but I would definitely suggest trying without lagg and then disabling jumbo packets, if possible. Please let us know if you fix the problem, rick > Disks: >=20 > export1 =C2=A0 =C2=A0 1.81T =C2=A0 914G =C2=A0 942G =C2=A0 =C2=A049% =C2= =A01.00x =C2=A0ONLINE =C2=A0- > =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 mirror =C2=A0 =C2=A09.94G =C2=A0 173M =C2=A09.77G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da4p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da5p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > cache =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 = =C2=A0- > =C2=A0 da4p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0= =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 da5p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0= =C2=A0 =C2=A0 =C2=A0 - >=20 >=20 > da0-3 are 1TB WDs > da4-5 are 240GB=C2=A0Samsung SSD 840s >=20 >=20 > Here's (related) info from rc.conf. >=20 > nfs_server_enable=3D"YES" > nfs_server_flags=3D"-u -t -n 128" > rpcbind_enable=3D"YES" > mountd_enable=3D"YES" > rpc_lockd_enable=3D"YES" > rpc_statd_enable=3D"YES" >=20 >=20 >=20 >=20 > I have compression enabled on all the ZFS-filesystems, and > jumboframes are enabled on the nics. >=20 >=20 > As soon as one of the (Linux) clients start to do some IO, NFS > responsetimes go up bigtime (yesterday up to 13 seconds), while the > hardware is pretty much idle, I must be doing something very wrong. > I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is > appreciated. >=20 >=20 > Regards, >=20 >=20 > -- > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ > Mark Schouten | Tuxis Internet Engineering > KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/ > T: 0318 200208 | info@tuxis.nl