Date: Thu, 13 Nov 2014 08:09:31 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Mark Schouten <mark@tuxis.nl> Cc: freebsd-hackers@freebsd.org Subject: Re: Slow nfsd write performance, tweaks needed Message-ID: <54043890.12566825.1415884171801.JavaMail.root@uoguelph.ca> In-Reply-To: <2826701214-10966@kerio.tuxis.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
Mark Schouten wrote: > Hi, >=20 >=20 > I am in the process of switching from a ZFS On Linux-based NFS-server > to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is > way superiour over ZoL, and the box serves as storage for a > virtualizationplatform, so stability is welcome. :) >=20 >=20 > The box is stable, but performs terribly. Surely, I'm doing something > wrong, but I would like some tips and tricks to speed things up. >=20 >=20 >=20 >=20 > Here's my setup: > CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is > enabled) > RAM: 64GB > NIC: 2x igb in lagg0 (loadbalancing) Oops again. I don't think igb had the 64K TSO problem (I get the Intel nics confused;-), but I would definitely suggest trying without lagg and then disabling jumbo packets, if possible. Please let us know if you fix the problem, rick > Disks: >=20 > export1 =C2=A0 =C2=A0 1.81T =C2=A0 914G =C2=A0 942G =C2=A0 =C2=A049% =C2= =A01.00x =C2=A0ONLINE =C2=A0- > =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2= =A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 mirror =C2=A0 =C2=A09.94G =C2=A0 173M =C2=A09.77G =C2=A0 =C2=A0 = =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da4p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 =C2=A0 da5p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - > cache =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 = =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 = =C2=A0- > =C2=A0 da4p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0= =C2=A0 =C2=A0 =C2=A0 - > =C2=A0 da5p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0= =C2=A0 =C2=A0 =C2=A0 - >=20 >=20 > da0-3 are 1TB WDs > da4-5 are 240GB=C2=A0Samsung SSD 840s >=20 >=20 > Here's (related) info from rc.conf. >=20 > nfs_server_enable=3D"YES" > nfs_server_flags=3D"-u -t -n 128" > rpcbind_enable=3D"YES" > mountd_enable=3D"YES" > rpc_lockd_enable=3D"YES" > rpc_statd_enable=3D"YES" >=20 >=20 >=20 >=20 > I have compression enabled on all the ZFS-filesystems, and > jumboframes are enabled on the nics. >=20 >=20 > As soon as one of the (Linux) clients start to do some IO, NFS > responsetimes go up bigtime (yesterday up to 13 seconds), while the > hardware is pretty much idle, I must be doing something very wrong. > I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is > appreciated. >=20 >=20 > Regards, >=20 >=20 > -- > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ > Mark Schouten | Tuxis Internet Engineering > KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/ > T: 0318 200208 | info@tuxis.nl
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54043890.12566825.1415884171801.JavaMail.root>