From owner-freebsd-current@FreeBSD.ORG Fri Feb 3 22:47:41 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2E6F016A420 for ; Fri, 3 Feb 2006 22:47:41 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id B7EE943D46 for ; Fri, 3 Feb 2006 22:47:38 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.13.4/8.13.4) with ESMTP id k13MlbC1025430 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 3 Feb 2006 17:47:37 -0500 (EST) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id k13MlW9U077914; Fri, 3 Feb 2006 17:47:32 -0500 (EST) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17379.56708.421007.613310@grasshopper.cs.duke.edu> Date: Fri, 3 Feb 2006 17:47:32 -0500 (EST) To: freebsd-current@freebsd.org X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Subject: machdep.cpu_idle_hlt and SMP perf? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Feb 2006 22:47:41 -0000 Why dooes machdep.cpu_idle_hlt=1 drop my 10GbE network rx performance by a considerable amount (7.5Gbs -> 5.5Gbs)? I've (blindly) tried leaving machdep.cpu_idle_hlt=1 enabled and playing with the vast array of kern.sched.ipiwakeup.* sysctls, but receive performance remains limited to ~5.5Gb/sec or less. This is an 'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+' running FreeBSD-current as of about one week ago. The interrupt load is about 22,000 device interrupts/sec (ithreaded). Interestingly, the more I decrease the interrupt load by increasing the interrupt coalescing timer, the worse the machdep.cpu_idle_hlt=1 case does. Is this just a case of the wakeup IPI taking a long time or blocking on some lock? Drew PS: Here is what I mean: rome% ssh venice-my netperf224 -Hrome-my -tTCP_SENDFILE -F /boot/vmlinuz-2.6.9-11.EL -- -S 131072 TCP SENDFILE TEST to rome-my Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 131072 65536 65536 10.00 5460.73 rome% sudo sysctl machdep.cpu_idle_hlt=0 machdep.cpu_idle_hlt: 1 -> 0 rome% ssh venice-my netperf224 -Hrome-my -tTCP_SENDFILE -F /boot/vmlinuz-2.6.9-11.EL -- -S 131072 TCP SENDFILE TEST to rome-my Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 131072 65536 65536 10.00 7842.41