From owner-freebsd-bugs@FreeBSD.ORG Wed Feb 4 12:40:02 2009 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5274E1065672 for ; Wed, 4 Feb 2009 12:40:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 262688FC0C for ; Wed, 4 Feb 2009 12:40:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n14Ce227046401 for ; Wed, 4 Feb 2009 12:40:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n14Ce1ZX046400; Wed, 4 Feb 2009 12:40:02 GMT (envelope-from gnats) Resent-Date: Wed, 4 Feb 2009 12:40:02 GMT Resent-Message-Id: <200902041240.n14Ce1ZX046400@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Martin Birgmeier Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FF701065670 for ; Wed, 4 Feb 2009 12:30:51 +0000 (UTC) (envelope-from martin@email.aon.at) Received: from email.aon.at (warsl404pip4.highway.telekom.at [195.3.96.117]) by mx1.freebsd.org (Postfix) with ESMTP id 7762A8FC1D for ; Wed, 4 Feb 2009 12:30:49 +0000 (UTC) (envelope-from martin@email.aon.at) Received: (qmail 30617 invoked from network); 4 Feb 2009 12:30:48 -0000 Received: from 80-121-4-38.adsl.highway.telekom.at (HELO gandalf.xyzzy) ([80.121.4.38]) (envelope-sender ) by smarthub91.highway.telekom.at (qmail-ldap-1.03) with SMTP for ; 4 Feb 2009 12:30:47 -0000 Received: from gandalf.xyzzy (localhost.xyzzy [127.0.0.1]) by gandalf.xyzzy (8.14.3/8.14.3) with ESMTP id n14CUM5v003973 for ; Wed, 4 Feb 2009 13:30:23 +0100 (CET) (envelope-from martin@gandalf.xyzzy) Received: (from martin@localhost) by gandalf.xyzzy (8.14.3/8.14.3/Submit) id n14CUL0L003970; Wed, 4 Feb 2009 13:30:21 +0100 (CET) (envelope-from martin) Message-Id: <200902041230.n14CUL0L003970@gandalf.xyzzy> Date: Wed, 4 Feb 2009 13:30:21 +0100 (CET) From: Martin Birgmeier To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: kern/131360: [nfs] poor scaling behavior of the NFS server under load X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Martin Birgmeier List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2009 12:40:02 -0000 >Number: 131360 >Category: kern >Synopsis: [nfs] poor scaling behavior of the NFS server under load >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Feb 04 12:40:01 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Martin Birgmeier >Release: FreeBSD 7.1-RELEASE i386 >Organization: MBi at home >Environment: System: FreeBSD gandalf.xyzzy 7.1-RELEASE FreeBSD 7.1-RELEASE #0: Sun Jan 4 10:53:24 CET 2009 root@atpcdvvc.xyzzy:/usr/VOL/OBJ/FreeBSD/RELENG_7_1_0_RELEASE/src/sys/XYZZY i386 >Description: [Please note that the email address given is not valid due to security reasons. Just reply to the gnats entry, and I'll follow it via the web interface.] Between FreeBSD 6.3 and 7.1, the behavior of the NFS server changed for the worse. Under 6.3, the load generated by the nfsd's would never exceed their number (I am using nfs_server_flags="-u -t -n 8"). With 7.1, when the client generates a lot of requests, it seems that the load on the NFS server can grow nearly without bounds, rendering the server all but unusable. I have been testing this with two scenarios: - running a FreeBSD buildworld -j4 with src on NFS and obj local, so there are only NFS reads, and - compiling openoffice.org-3-devel with WRKDIRPREFIX set to an NFS-mounted directory, and MAXMODULES=3, MAXPROCESSES=3, resulting in both reads and writes. In both cases, the NFS server reaches absurdly high load values: with 'buildword -j4' on the client I currently read from 'systat -vm 1' on the server: - a load just passing the 15 mark - nearly 100% 'sys' - there seems to be nearly no disk i/o activity - very poor general network throughput: an xterm from the server to the client is nearly unresponsive (in fact, I again have to suspend the build on the client right now in order to be able to write this bug report in an xterm running on the server). - resuming the build immediately sends the load skyrocketing again. I am quite sure that this is a server issue, as for a while I had the client running 7.0 and server 6.3, and in this scenario this did not happen. BTW, the NFS mounts on the client are done via amd - this probably influences which mount type is used. >How-To-Repeat: Generate a high NFS load on a client, and watch the server load skyrocket. >Fix: Unknown. >Release-Note: >Audit-Trail: >Unformatted: