Date: Wed, 9 Mar 2022 14:39:39 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Yoshihiro Ota <ota@j.email.ne.jp>, freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Message-ID: <YT2PR01MB9730D7B51D325258AAA29828DD0A9@YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <20220309034601.ea3135e31aec3ffb2623f145@j.email.ne.jp> References: <20220309034601.ea3135e31aec3ffb2623f145@j.email.ne.jp>
next in thread | previous in thread | raw e-mail | index | archive | help
Yoshihiro Ota <ota@j.email.ne.jp> wrote:=0A= > Hi,=0A= >=0A= > I'm on stable/13 with latest code base.=0A= > I started testing pre-13.1 branch.=0A= >=0A= > I noticed major performance degrades with NFS when all CPUs are fully =0A= > utilized.=0A= >=0A= > This happends with stable/13 but not releng/13.0 nor releng/12.3.=0A= NFS performance is sensitive to RPC response time.=0A= Since this only happens when the COUs are busy, I'd suspect:=0A= - Kernel thread scheduling changes=0A= or=0A= - Timing of receive socket upcalls (which wake up the nfsd kernel threads).= =0A= =0A= I suspect bisecting to the actual commit that causes this is the only way= =0A= to find it.=0A= If you know of a working stable/13 that is more recent than 13.0, it would= =0A= help. If not, you start at this commit (which did make socket upcall change= s):=0A= commit 55cc0a478506ee1c2db7b2f9aadb9855e5490af3=0A= which was done on May 21, 2021.=0A= =0A= Maybe others can suggest commits related to thread scheduling (which I=0A= know nothing about).=0A= =0A= If you don't have the time/resources to bisect, I doubt this will get resol= ved.=0A= =0A= Good luck with it, rick=0A= =0A= I had NFS server with above versions and rsynced nfs mount to ufs mount on = NFS clients.=0A= My NFS server has 4 cores.=0A= When I had load average of 3 with make buildworld -j3, NFS server was fine.= =0A= After adding another 1 load, NFS server throughput came down to about 10% o= f before.=0A= After taking back to 3 load avg, performance recovered and down again after= getting over 4.=0A= Disk was fully avaiable for rsync; buildworld was done on another disk.=0A= =0A= =0A= Someone told me his smbfs was also slow and he suspected TCP/IP regression = instead of NFS, by the way.=0A= =0A= Hiro=0A= =0A= =0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YT2PR01MB9730D7B51D325258AAA29828DD0A9>