From owner-freebsd-fs@FreeBSD.ORG Thu Feb 26 16:10:02 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 836F210656D9 for ; Thu, 26 Feb 2009 16:10:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 714DE8FC21 for ; Thu, 26 Feb 2009 16:10:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n1QGA2oD073770 for ; Thu, 26 Feb 2009 16:10:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n1QGA2gv073769; Thu, 26 Feb 2009 16:10:02 GMT (envelope-from gnats) Date: Thu, 26 Feb 2009 16:10:02 GMT Message-Id: <200902261610.n1QGA2gv073769@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Peter Keel Cc: Subject: kern/131360: [nfs] poor scaling behavior of the NFS server under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Peter Keel List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Feb 2009 16:10:02 -0000 The following reply was made to PR kern/131360; it has been noted by GNATS. From: Peter Keel To: bug-followup@FreeBSD.org, martin@email.aon.at Cc: Subject: kern/131360: [nfs] poor scaling behavior of the NFS server under load Date: Thu, 26 Feb 2009 16:50:18 +0100 We experience exactly the same problem. With FreeBSD 7.0 everything was alright, but with the upgrade to 7.1 performance became abyssmal. Normally it seems to work for some time; but after a few hours, the load climbs and the 4 nfsd-processes each use 100% CPU (on a 4way-system). The system only serves as nfs-root (ro) for about 20 systems, so there's not too much nfs-traffic going on. I wasn't able to capture it when every nfsd was using 100% CPU (because, well, some thousands of users depend on it); this is what it looks when it's half-working (hundreds of timeouts on the clients): 37907 root 1 4 0 4604K 1112K - 2 46:29 33.59% nfsd 37908 root 1 4 0 4604K 1112K - 2 15:29 17.38% nfsd 37909 root 1 4 0 4604K 1112K - 3 5:59 5.66% nfsd 37910 root 1 4 0 4604K 1112K - 2 2:38 0.00% nfsd We're suspecting the ULE scheduler as responsible for the mess. Right now we're testing a kernel with the 4BSD scheduler; we'll know more next week. But now it looks like this: 1040 root 1 4 0 4604K 1068K - 3 5:49 6.49% nfsd 1039 root 1 4 0 4604K 1068K - 3 2:20 1.86% nfsd 1042 root 1 4 0 4604K 1068K - 1 0:48 0.05% nfsd 1041 root 1 4 0 4604K 1068K - 2 0:15 0.00% nfsd Promising. Regards Peter -- "Those who give up essential liberties for temporary safety deserve neither liberty nor safety." -- Benjamin Franklin "It's also true that those who would give up privacy for security are likely to end up with neither." -- Bruce Schneier