From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 12 15:08:25 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5C5BF16A4CE for ; Wed, 12 Nov 2003 15:08:25 -0800 (PST) Received: from mail.speakeasy.net (mail8.speakeasy.net [216.254.0.208]) by mx1.FreeBSD.org (Postfix) with ESMTP id 74F6D43FD7 for ; Wed, 12 Nov 2003 15:08:24 -0800 (PST) (envelope-from jhb@FreeBSD.org) Received: (qmail 12253 invoked from network); 12 Nov 2003 23:08:23 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender )encrypted SMTP for ; 12 Nov 2003 23:08:23 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.9/8.12.9) with ESMTP id hACN80ce028196; Wed, 12 Nov 2003 18:08:00 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20031112103358.S11644@bowser.eecs.harvard.edu> Date: Wed, 12 Nov 2003 18:07:58 -0500 (EST) From: John Baldwin To: Daniel Ellard X-Spam-Checker-Version: SpamAssassin 2.55 (1.174.2.19-2003-05-19-exp) cc: freebsd-hackers@freebsd.org Subject: RE: Confused about HyperThreading and Performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2003 23:08:25 -0000 On 12-Nov-2003 Daniel Ellard wrote: > > Can someone point me at some non-marketing documentation about > hyperthreading on the latest Intel chips? I'm seeing some strange > performance measurements and I would like to figure out what they > mean. > > My systems have the following CPU: > > CPU: Intel(R) Xeon(TM) CPU 1.80GHz (1799.81-MHz 686-class CPU) Origin = "GenuineIntel" Id = > 0xf27 Stepping = 7 > > Features=0xbfebfbff H,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > They are running FreeBSD 4.8-RELEASE-p13. > > I'm running an application that pumps a lot of data quickly over a > gigabit NIC (an Alteon AceNIC 1000baseT Gigabit Ethernet, using the ti > driver). The network uses normal ethernet frames (MTU=1500) instead > of jumbo frames, so there are lots of interrupts. In this case, the > system is mostly reading data from the network, in case that makes a > difference. The application is multi-threaded, using pthreads. > > When I run a kernel using the default configuration (no SMP, no APIC), > the application takes an average of 8.45 seconds (wall-clock time) to > run, and this number is consistent from run to run. When I run a > kernel built with either SMP or SMP+APIC, it takes an average of 13.25 > seconds, and this number is also quite consistent. (It's not the > Alteon; using the Intel Pro/1000 XT server adapter with the em driver > gives a similar difference, although the intel NIC is slower.) > > Interestingly, the CPU utilization for the default kernel is about > 75-80%, while for the APIC kernel it is over 95%. My guess would be > that something is spinning on a lock in the APIC kernel, but that's > just a hunch. > > My questions: > > 1. Any idea why the performance plummets from 8.45 seconds to 13.25? > > 2. Is the situation any better in FreeBSD 5.1 (or 4.9)? I know in > 4.9 APIC is turned on by default, but if this is the result > than I won't be upgrading to 4.9 any time soon! APIC is not on by default for 4.9, it will be for 5.2. 4.9 does have HTT on by default when you build an SMP kernel though. Did you include 'options HTT' in your 4.8 SMP kernel? If not, you aren't actually using your second CPU. Also, as someone else mentioned, setting 'machdep.cpu_idle_hlt=1' can be useful on some HTT systems. However, p4's have a problem with their interrupt routing that can leave the second CPU halted for a long time if you do that. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/