Date: Tue, 23 May 2006 13:58:20 +0200 (CEST) From: "Ralf S. Engelschall" <rse@FreeBSD.org> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/97686: massive NFS server performance break under "options QUOTA" Message-ID: <200605231158.k4NBwKbd031603@en1.home.engelschall.com> Resent-Message-ID: <200605231200.k4NC0dWt035403@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 97686
>Category: kern
>Synopsis: massive NFS server performance break under "options QUOTA"
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue May 23 12:00:39 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator: Ralf S. Engelschall
>Release: FreeBSD 6.1-STABLE i386
>Organization:
FreeBSD
>Environment:
System: FreeBSD sv0 6.1-STABLE FreeBSD 6.1-STABLE #0: Tue May 23 12:05:59 CEST 2006 root@sv0:/usr/obj/usr/src/sys/SV0 i386
>Description:
After an upgrade we experienced massive serious NFS server problems
on a NFS server under FreeBSD 6.1-STABLE (as of 2006-05-23). It
still worked fine (exactly same kernel configuration) under FreeBSD
5.4-STABLE (as of about December 2005).
The effect is that the machine becomes totally unusable (nearly a
system freeze) after about 10 minutes of just moderate NFS traffic
of a single NFS client. We figured out that once the "options QUOTA"
was removed from the kernel configuration the machine performance
was 100% ok and we do not experience any performance breaks again.
Even disabling SMP or deconfiguring all Netgraph configurations (for
interface bundling and Netflow statistics) caused no effect. Only the
"options QUOTA" caused the problem and its removal finally fixed the
problem.
There seems to be a strange interaction between NFS and UFS Quotas...
>How-To-Repeat:
- build a FreeBSD 6.1 kernel with "options QUOTA" and "options NFSSERVER".
- configure nfs_server_enable="YES" and nfs_server_flags="-u -n 8" in rc.conf.
- cause some usual NFS traffic by unpacking a large distribution
tarball (lots of small files), removing lots of files or just doing a
"cvs update" in a large source tree.
- in parallel on the NFS server watch a steadily increasing CPU
utilitzation from nfsd(8) and about 45% of "CPU time in system
mode" (= "sy") in "iostat 1" outputs (numbers on a 2 CPU HP DL380
box). Also the system loads increases steadily up to 15-20.
- after maximally 10 minutes the box becomes more and more totally
unusable. Even typing in the SSH session becomes a problem. Once
the NFS traffic is stopped, the machines becomes usable within
30-60 seconds again.
>Fix:
workaround by removing "options QUOTA" from the kernel configuration
the machine works fine and no performance problems occur at all. With
the above "How-To-Repeat" procedure the NFS performance is fully ok
and the NFS server itself also never reaches any high loads. Also
"iostat 1" shows never a "CPU time in system mode" of more than 5%.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200605231158.k4NBwKbd031603>