Date: Mon, 08 Dec 2014 08:36:21 +0000 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" <loic.blot@unix-experience.fr> To: "Rick Macklem" <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: High Kernel Load with nfsv4 Message-ID: <ad6a65b88d79848ea80e14e6c4221e5d@mail.unix-experience.fr> In-Reply-To: <581583623.5730217.1417788866930.JavaMail.root@uoguelph.ca> References: <581583623.5730217.1417788866930.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick,=0AI stopped the jails this week-end and started it this morning,= i'll give you some stats this week.=0A=0AHere is my nfsstat -m output (w= ith your rsize/wsize tweaks)=0A=0Anfsv4,tcp,resvport,hard,cto,sec=3Dsys,a= cdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negna= metimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=3D32768,readahead=3D= 1,wcommitsize=3D773136,timeout=3D120,retrans=3D2147483647=0A=0AOn server = side my disks are on a raid controller which show a 512b volume and write= performances are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs= =3D4096 count=3D100000000 =3D> 450MBps)=0A=0ARegards,=0A=0ALo=C3=AFc Blot= ,=0AUNIX Systems, Network and Security Engineer=0Ahttp://www.unix-experie= nce.fr=0A=0A5 d=C3=A9cembre 2014 15:14 "Rick Macklem" <rmacklem@uoguelph.= ca> a =C3=A9crit: =0A> Loic Blot wrote:=0A> =0A>> Hi,=0A>> i'm trying to = create a virtualisation environment based on jails.=0A>> Those jails are = stored under a big ZFS pool on a FreeBSD 9.3 which=0A>> export a NFSv4 vo= lume. This NFSv4 volume was mounted on a big=0A>> hypervisor (2 Xeon E5v3= + 128GB memory and 8 ports (but only 1 was=0A>> used at this time).=0A>>= =0A>> The problem is simple, my hypervisors runs 6 jails (used 1% cpu an= d=0A>> 10GB RAM approximatively and less than 1MB bandwidth) and works=0A= >> fine at start but the system slows down and after 2-3 days become=0A>>= unusable. When i look at top command i see 80-100% on system and=0A>> co= mmands are very very slow. Many process are tagged with nfs_cl*.=0A> =0A>= To be honest, I would expect the slowness to be because of slow response= =0A> from the NFSv4 server, but if you do:=0A> # ps axHl=0A> on a client = when it is slow and post that, it would give us some more=0A> information= on where the client side processes are sitting.=0A> If you also do somet= hing like:=0A> # nfsstat -c -w 1=0A> and let it run for a while, that sho= uld show you how many RPCs are=0A> being done and which ones.=0A> =0A> # = nfsstat -m=0A> will show you what your mount is actually using.=0A> The o= nly mount option I can suggest trying is "rsize=3D32768,wsize=3D32768",= =0A> since some network environments have difficulties with 64K.=0A> =0A>= There are a few things you can try on the NFSv4 server side, if it appea= rs=0A> that the clients are generating a large RPC load.=0A> - disabling = the DRC cache for TCP by setting vfs.nfsd.cachetcp=3D0=0A> - If the serve= r is seeing a large write RPC load, then "sync=3Ddisabled"=0A> might help= , although it does run a risk of data loss when the server=0A> crashes.= =0A> Then there are a couple of other ZFS related things (I'm not a ZFS g= uy,=0A> but these have shown up on the mailing lists).=0A> - make sure yo= ur volumes are 4K aligned and ashift=3D12 (in case a drive=0A> that uses = 4K sectors is pretending to be 512byte sectored)=0A> - never run over 70-= 80% full if write performance is an issue=0A> - use a zil on an SSD with = good write performance=0A> =0A> The only NFSv4 thing I can tell you is th= at it is known that ZFS's=0A> algorithm for determining sequential vs ran= dom I/O fails for NFSv4=0A> during writing and this can be a performance = hit. The only workaround=0A> is to use NFSv3 mounts, since file handle af= finity apparently fixes=0A> the problem and this is only done for NFSv3.= =0A> =0A> rick=0A> =0A>> I saw that there are TSO issues with igb then i'= m trying to disable=0A>> it with sysctl but the situation wasn't solved.= =0A>> =0A>> Someone has got ideas ? I can give you more informations if y= ou=0A>> need.=0A>> =0A>> Thanks in advance.=0A>> Regards,=0A>> =0A>> Lo= =C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer=0A>> http:= //www.unix-experience.fr=0A>> ___________________________________________= ____=0A>> freebsd-fs@freebsd.org mailing list=0A>> http://lists.freebsd.o= rg/mailman/listinfo/freebsd-fs=0A>> To unsubscribe, send any mail to "fre= ebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ad6a65b88d79848ea80e14e6c4221e5d>