From owner-freebsd-current@FreeBSD.ORG Wed Nov 2 14:58:50 2005 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 256BA16A41F; Wed, 2 Nov 2005 14:58:50 +0000 (GMT) (envelope-from gavin.atkinson@ury.york.ac.uk) Received: from mail-gw1.york.ac.uk (mail-gw1.york.ac.uk [144.32.128.246]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3610F43D70; Wed, 2 Nov 2005 14:58:44 +0000 (GMT) (envelope-from gavin.atkinson@ury.york.ac.uk) Received: from buffy.york.ac.uk (buffy-128.york.ac.uk [144.32.128.160]) by mail-gw1.york.ac.uk (8.12.10/8.12.10) with ESMTP id jA2Ewba2006984; Wed, 2 Nov 2005 14:58:37 GMT Received: from buffy.york.ac.uk (localhost [127.0.0.1]) by buffy.york.ac.uk (8.13.4/8.13.4) with ESMTP id jA2Ewa2S052113; Wed, 2 Nov 2005 14:58:36 GMT (envelope-from gavin.atkinson@ury.york.ac.uk) Received: (from ga9@localhost) by buffy.york.ac.uk (8.13.4/8.13.4/Submit) id jA2Ewa1o052112; Wed, 2 Nov 2005 14:58:36 GMT (envelope-from gavin.atkinson@ury.york.ac.uk) X-Authentication-Warning: buffy.york.ac.uk: ga9 set sender to gavin.atkinson@ury.york.ac.uk using -f From: Gavin Atkinson To: freebsd-current@freebsd.org Content-Type: text/plain Content-Transfer-Encoding: 7bit Date: Wed, 02 Nov 2005 14:58:36 +0000 Message-Id: <1130943516.51544.34.camel@buffy.york.ac.uk> Mime-Version: 1.0 X-Mailer: Evolution 2.2.3 FreeBSD GNOME Team Port X-York-MailScanner: Found to be clean X-York-MailScanner-From: gavin.atkinson@ury.york.ac.uk Cc: rwatson@freebsd.org Subject: Poor NFS server performance in 6.0 with SMP and mpsafenet=1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Nov 2005 14:58:50 -0000 Hi, I'm seeing incredibly poor performance when serving files from an SMP FreeBSD 6.0RC1 server to a Solaris 10 client. I've done some experimenting and have discovered that either removing SMP from the kernel, or setting debug.mpsafenet=0 in loader.conf massively improves the speed. Switching preemption off seems to also help. A quick summary of my findings: switching SMP off and setting mpsafenet=0 gives more than 700% boost in performance. PREEMPTION is a pessimisation in the SMP case, mpsafenet=1 is a pessimisation in all cases. Given these are three of the major new features in 6.0, this doesn't seem ideal :) The following are average times, in seconds, to serve a 512meg file over NFS to the Solaris host. Each experiment was performed immediately after a reboot of the FreeBSD host, and after one untimed transfer (in order to warm the FreeBSD vfs cache). No SMP, mpsafenet=1 59.4 No SMP, mpsafenet=0 49.4 No SMP, mpsafenet=1, no PREEMPTION 53.1 No SMP, mpsafenet=0, no PREEMPTION 51.9 SMP, mpsafenet=1 351.7 SMP, mpsafenet=0 74.5 SMP, mpsafenet=1, no PREEMPTION 264.9 SMP, mpsafenet=0, no PREEMPTION 53.7 Obviously this is not ideal, but at least I have a usable workaround. I'm just worried that the default SMP configuration performs so poorly (although I appreciate that GENERIC does not include SMP, anyone with a multiprocessor machine presumably would want to use it...) The FreeBSD machine is a twin 866MHz P3, running 6.-RC1 with ACPI enabled. The network cards are 64bit/66MHz em(4) cards using checksum offloading etc. Apart from the changes mentioned above, the kernel is GENERIC. I'll get a dmesg online somewhere. I'll also recreate the experiment, serving to a FreeBSD machine as opposed to Solaris, just to compare. I'm happy to do any more investigating to get to the bottom of this, although the machines in question will be going live at the end of the month. Gavin