Date: Thu, 13 Nov 2014 07:57:48 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Mark Schouten <mark@tuxis.nl> Cc: freebsd-hackers@freebsd.org Subject: Re: Slow nfsd write performance, tweaks needed Message-ID: <1690681228.12559777.1415883468777.JavaMail.root@uoguelph.ca> In-Reply-To: <2826701214-10966@kerio.tuxis.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
Mark Schouten wrote:
> Hi,
>=20
>=20
> I am in the process of switching from a ZFS On Linux-based NFS-server
> to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is
> way superiour over ZoL, and the box serves as storage for a
> virtualizationplatform, so stability is welcome. :)
>=20
>=20
> The box is stable, but performs terribly. Surely, I'm doing something
> wrong, but I would like some tips and tricks to speed things up.
>=20
>=20
>=20
>=20
> Here's my setup:
> CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is
> enabled)
> RAM: 64GB
> NIC: 2x igb in lagg0 (loadbalancing)
> Disks:
>=20
> export1 =C2=A0 =C2=A0 1.81T =C2=A0 914G =C2=A0 942G =C2=A0 =C2=A049% =C2=
=A01.00x =C2=A0ONLINE =C2=A0-
> =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 -
> =C2=A0 =C2=A0 da0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=
=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
> =C2=A0 =C2=A0 da1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=
=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
> =C2=A0 mirror =C2=A0 =C2=A0 928G =C2=A0 457G =C2=A0 471G =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 -
> =C2=A0 =C2=A0 da2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=
=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
> =C2=A0 =C2=A0 da3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=
=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
> =C2=A0 mirror =C2=A0 =C2=A09.94G =C2=A0 173M =C2=A09.77G =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 -
> =C2=A0 =C2=A0 da4p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 =
=C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
> =C2=A0 =C2=A0 da5p1 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 =
=C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
> cache =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 =C2=A0- =C2=A0 =
=C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =
=C2=A0-
> =C2=A0 da4p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0=
=C2=A0 =C2=A0 =C2=A0 -
> =C2=A0 da5p2 =C2=A0 =C2=A0 =C2=A0223G =C2=A0 223G =C2=A0 =C2=A0 8M =C2=A0=
=C2=A0 =C2=A0 =C2=A0 -
>=20
>=20
> da0-3 are 1TB WDs
> da4-5 are 240GB=C2=A0Samsung SSD 840s
>=20
>=20
> Here's (related) info from rc.conf.
>=20
> nfs_server_enable=3D"YES"
> nfs_server_flags=3D"-u -t -n 128"
> rpcbind_enable=3D"YES"
> mountd_enable=3D"YES"
> rpc_lockd_enable=3D"YES"
> rpc_statd_enable=3D"YES"
>=20
>=20
>=20
>=20
> I have compression enabled on all the ZFS-filesystems, and
> jumboframes are enabled on the nics.
>=20
>=20
> As soon as one of the (Linux) clients start to do some IO, NFS
> responsetimes go up bigtime (yesterday up to 13 seconds), while the
> hardware is pretty much idle, I must be doing something very wrong.
> I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is
> appreciated.
>=20
When an NFS server demonstrates terrible performance, it is usually
a problem with the underlying network fabric.
Possibilities include:
- A problem with TSO enabled network interface doing I/O of near
64K. (This should be mostly fixed in recent code, but you didn't
mention what version of FreeBSD you are using, so...)
Workarounds: disable TSO or make sure all your clients are using
rsize=3D32768,wsize=3D32768
- A problem with your network interface device driver.
Workaround: Try a different kind of network interface.
(You didn't give hardware specs for the server, so we don't
know what kind of NIC you are using. FreeBSD doesn't have
the marketshare of Linux, so the drivers vary greatly depending
on the hardware.)
- I have no idea what effect jumboframes might have. If possible,
I suggest you try disabling them.
I have no expertise on the ZFS side, so maybe someone else will
make some suggestions there.
Good luck with it, rick
ps: There is no "ideal" mailing list for NFS stuff. Most would
use either freebsd-stable@ or freebsd-current@, depending
on whether they are using a stable release or -head.
Others might use freebsd-fs@.
>=20
> Regards,
>=20
>=20
> --
> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
> Mark Schouten | Tuxis Internet Engineering
> KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/
> T: 0318 200208 | info@tuxis.nl
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1690681228.12559777.1415883468777.JavaMail.root>