From owner-freebsd-hackers@FreeBSD.ORG Thu Nov 13 13:00:17 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EBECCFA1 for ; Thu, 13 Nov 2014 13:00:16 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id B2E6B792 for ; Thu, 13 Nov 2014 13:00:16 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArcEAKiqZFSDaFve/2dsb2JhbABbg2JZBIMCyWiHUwKBNQEBAQEBfYQCAQEBAwEjVgUUAhgCAg0ZAlkGE4g4CZ0EnF+WGgEBAQEBAQEDAQEBAQEBAQEBGYEtjx4VATMHgneBVAWFIgOGZpAWmHeEGh8wgQdBgQMBAQE X-IronPort-AV: E=Sophos;i="5.07,377,1413259200"; d="scan'208";a="166405315" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 13 Nov 2014 08:00:15 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A72CBB40AE; Thu, 13 Nov 2014 08:00:15 -0500 (EST) Date: Thu, 13 Nov 2014 08:00:15 -0500 (EST) From: Rick Macklem To: Mark Schouten Message-ID: <1841043385.12560911.1415883615679.JavaMail.root@uoguelph.ca> In-Reply-To: <2826701214-10966@kerio.tuxis.nl> Subject: Re: Slow nfsd write performance, tweaks needed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 13:00:17 -0000 Mark Schouten wrote: > Hi, >=20 >=20 > I am in the process of switching from a ZFS On Linux-based NFS-server > to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is > way superiour over ZoL, and the box serves as storage for a > virtualizationplatform, so stability is welcome. :) >=20 >=20 > The box is stable, but performs terribly. Surely, I'm doing something > wrong, but I would like some tips and tricks to speed things up. >=20 >=20 >=20 >=20 > Here's my setup: > CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is > enabled) > RAM: 64GB > NIC: 2x igb in lagg0 (loadbalancing) Oops, I didn't see this before my last post. igb had problems with the 64K TSO issue and I'd try to get rid of lagg as well. (You might be much better off just using a single net interface without lagg.) Again, good luck with it, rick > Disks: >=20 > export1 =C2=A0 =C2=A0 1.81T =C2=A0 914G =C2=A0 942G =C2=A0 =C2=A049% =C2= =A01.00x =C2=A0ONLINE =C2=A0- > =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 mirror =C2=A0 =C2=A09.94G =C2=A0 173M =C2=A09.77G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da4p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da5p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > cache =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 = =C2=A0- > =C2=A0 da4p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0= =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 da5p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0= =C2=A0 =C2=A0 =C2=A0 - >=20 >=20 > da0-3 are 1TB WDs > da4-5 are 240GB=C2=A0Samsung SSD 840s >=20 >=20 > Here's (related) info from rc.conf. >=20 > nfs_server_enable=3D"YES" > nfs_server_flags=3D"-u -t -n 128" > rpcbind_enable=3D"YES" > mountd_enable=3D"YES" > rpc_lockd_enable=3D"YES" > rpc_statd_enable=3D"YES" >=20 >=20 >=20 >=20 > I have compression enabled on all the ZFS-filesystems, and > jumboframes are enabled on the nics. >=20 >=20 > As soon as one of the (Linux) clients start to do some IO, NFS > responsetimes go up bigtime (yesterday up to 13 seconds), while the > hardware is pretty much idle, I must be doing something very wrong. > I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is > appreciated. >=20 >=20 > Regards, >=20 >=20 > -- > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ > Mark Schouten | Tuxis Internet Engineering > KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/ > T: 0318 200208 | info@tuxis.nl