From owner-freebsd-smp@FreeBSD.ORG Tue May 31 07:38:53 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 649E016A420 for ; Tue, 31 May 2005 07:38:53 +0000 (GMT) (envelope-from briar@typos.ch) Received: from ptrt.dravanet.hu (ptrt.dravanet.hu [81.183.253.7]) by mx1.FreeBSD.org (Postfix) with SMTP id C96D543D54 for ; Tue, 31 May 2005 07:38:52 +0000 (GMT) (envelope-from briar@typos.ch) Received: from [37.59.180.70] (port=4376 helo=[celebrate]) by ptrt.dravanet.hu with esmtp id 89025118074archeologist11326 for freebsd-smp@freebsd.org; Tue, 31 May 2005 09:40:33 +0200 Mime-Version: 1.0 (Apple Message framework v728) Content-Transfer-Encoding: 7bit Message-Id: <1220732380.11643140502@ptrt.dravanet.hu> Content-Type: text/plain; charset=US-ASCII; format=flowed To: freebsd-smp@freebsd.org From: Margie Date: Tue, 31 May 2005 09:40:32 +0200 X-Mailer: Apple Mail (2.728) Subject: Generic ED drugs directly from manufacturer. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 07:38:53 -0000 Looking for a specific medication? Let us know what you need! http://zryeoz.hwks3ihae9z730h.euphone74c8.com Thinking is only a process of talking to yourself. To find a fault is easy; to do better may be difficult. I have found power in the mysteries of thought. From owner-freebsd-smp@FreeBSD.ORG Tue May 31 17:37:41 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 48F4516A41C for ; Tue, 31 May 2005 17:37:41 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0178543D4C for ; Tue, 31 May 2005 17:37:40 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from smtp2.corp.netapp.com (10.57.159.114) by mx2.netapp.com with ESMTP; 31 May 2005 10:37:40 -0700 X-IronPort-AV: i="3.93,153,1115017200"; d="scan'208,217"; a="222079502:sNHT23215572" Received: from svlexc03.hq.netapp.com (svlexc03.corp.netapp.com [10.57.156.149]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id j4VHbeio019475 for ; Tue, 31 May 2005 10:37:40 -0700 (PDT) Received: from lavender.hq.netapp.com ([10.56.11.75]) by svlexc03.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 31 May 2005 10:37:40 -0700 Received: from magenta.hq.netapp.com ([10.56.11.84]) by lavender.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 31 May 2005 10:37:40 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Tue, 31 May 2005 10:37:40 -0700 Message-ID: <637A278D8D0DBC438EA5E75C6E1818B904410DAA@magenta.hq.netapp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 5.4 network performance Thread-Index: AcVmB3E19Wf8FdVvR/KzZtdkjNmYnQ== From: "Singh, Vijay" To: X-OriginalArrivalTime: 31 May 2005 17:37:40.0868 (UTC) FILETIME=[71B8FC40:01C56607] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 17:37:41 -0000 Hello. I am trying to benchmark 5.4 performance for a company project. I've got: =20 FreeBSD 5.4-RELEASE #0: Fri May 27 20:52:58 PDT 2005 admin@netpc13.lab.netapp.com:/usr/obj/usr/src/sys/SMP Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2395.92-MHz 686-class CPU) Origin =3D "GenuineIntel" Id =3D 0xf27 Stepping =3D 7 =20 Features=3D0xbfebfbff Hyperthreading: 2 logical CPUs real memory =3D 1073676288 (1023 MB) avail memory =3D 1041121280 (992 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 em0: port 0xc800-0xc83f mem 0xfe8c0000-0xfe8dffff irq 18 at device 2.0 on pci1 em1: port 0xd000-0xd03f mem 0xfe8e0000-0xfe8fffff irq 19 at device 2.1 on pci1 The UP version of the builds is able to deliver close to line rate on these 2 interfaces. However the SMP build (with WITNESS and INVARIANTS disabled) gives me half the line rate on them. I am using netperf. =20 /opt/netperf/netperf -H x.x.x.x -f m -l 120 =20 Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec =20 65536 16384 16384 120.00 429.14 65536 16384 16384 120.00 501.74 =20 Is there something I can do to make the system scale? I cannot move to -CURRENT, but I can try pulling patches. =20 br vijay From owner-freebsd-smp@FreeBSD.ORG Tue May 31 19:11:56 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 180B716A41C for ; Tue, 31 May 2005 19:11:56 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8C98843D53 for ; Tue, 31 May 2005 19:11:54 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.2.R) with ESMTP id md50001450056.msg for ; Tue, 31 May 2005 20:06:44 +0100 Message-ID: <004e01c56614$8098e510$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Singh, Vijay" , References: <637A278D8D0DBC438EA5E75C6E1818B904410DAA@magenta.hq.netapp.com> Date: Tue, 31 May 2005 20:11:05 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Tue, 31 May 2005 20:06:44 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-smp@freebsd.org X-MDAV-Processed: multiplay.co.uk, Tue, 31 May 2005 20:06:45 +0100 Cc: Subject: Re: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 19:11:56 -0000 Ensure that HT is disabled and see if that makes any difference. Steve ----- Original Message ----- From: "Singh, Vijay" Is there something I can do to make the system scale? I cannot move to -CURRENT, but I can try pulling patches. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-smp@FreeBSD.ORG Tue May 31 20:48:04 2005 Return-Path: X-Original-To: freebsd-smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4ED4F16A41C for ; Tue, 31 May 2005 20:48:04 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: from mail26.sea5.speakeasy.net (mail26.sea5.speakeasy.net [69.17.117.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id DA85B43D54 for ; Tue, 31 May 2005 20:48:03 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 32148 invoked from network); 31 May 2005 20:48:03 -0000 Received: from server.baldwin.cx ([216.27.160.63]) (envelope-sender ) by mail26.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 31 May 2005 20:48:03 -0000 Received: from [10.50.41.231] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.13.1/8.13.1) with ESMTP id j4VKlujR064394; Tue, 31 May 2005 16:47:56 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-smp@FreeBSD.org Date: Tue, 31 May 2005 16:44:25 -0400 User-Agent: KMail/1.8 References: <637A278D8D0DBC438EA5E75C6E1818B904410DAA@magenta.hq.netapp.com> In-Reply-To: <637A278D8D0DBC438EA5E75C6E1818B904410DAA@magenta.hq.netapp.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200505311644.26315.jhb@FreeBSD.org> X-Spam-Status: No, score=-102.8 required=4.2 tests=ALL_TRUSTED, USER_IN_WHITELIST autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on server.baldwin.cx Cc: "Singh, Vijay" Subject: Re: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 20:48:04 -0000 On Tuesday 31 May 2005 01:37 pm, Singh, Vijay wrote: > Hello. I am trying to benchmark 5.4 performance for a company project. > I've got: > > FreeBSD 5.4-RELEASE #0: Fri May 27 20:52:58 PDT 2005 > admin@netpc13.lab.netapp.com:/usr/obj/usr/src/sys/SMP > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2395.92-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 > > Features=0xbfebfbff ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,S > SE2,SS,HTT,TM,PBE> > Hyperthreading: 2 logical CPUs > real memory = 1073676288 (1023 MB) > avail memory = 1041121280 (992 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > > em0: port > 0xc800-0xc83f mem 0xfe8c0000-0xfe8dffff irq 18 at device 2.0 on pci1 > em1: port > 0xd000-0xd03f mem 0xfe8e0000-0xfe8fffff irq 19 at device 2.1 on pci1 > > The UP version of the builds is able to deliver close to line rate on > these 2 interfaces. However the SMP build (with WITNESS and INVARIANTS > disabled) gives me half the line rate on them. I am using netperf. > > /opt/netperf/netperf -H x.x.x.x -f m -l 120 > > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 65536 16384 16384 120.00 429.14 > 65536 16384 16384 120.00 501.74 > > Is there something I can do to make the system scale? I cannot move to > -CURRENT, but I can try pulling patches. Does UP with 'device apic' also show poor performance? Try disabling USB support in the kernel as some Intel server motherboards have a "feature" that our APIC code trips over that can hurt performance by adding lots of stray interrupts on one of the USB controllers. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-smp@FreeBSD.ORG Tue May 31 21:19:17 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3286216A440; Tue, 31 May 2005 21:19:17 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id D6E8043D1F; Tue, 31 May 2005 21:19:16 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from smtp2.corp.netapp.com (10.57.159.114) by mx2.netapp.com with ESMTP; 31 May 2005 14:19:17 -0700 X-IronPort-AV: i="3.93,154,1115017200"; d="scan'208"; a="222101109:sNHT20244380" Received: from svlexc02.hq.netapp.com (svlexc02.corp.netapp.com [10.57.157.136]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id j4VLJGxV011263; Tue, 31 May 2005 14:19:16 -0700 (PDT) Received: from burgundy.hq.netapp.com ([10.56.10.66]) by svlexc02.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 31 May 2005 14:19:16 -0700 Received: from magenta.hq.netapp.com ([10.56.11.84]) by burgundy.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 31 May 2005 14:19:16 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 31 May 2005 14:19:15 -0700 Message-ID: <637A278D8D0DBC438EA5E75C6E1818B904410F31@magenta.hq.netapp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 5.4 network performance Thread-Index: AcVmIgtbjpjTcJyRQkasq2ZRJOcgVgAA+DRw From: "Singh, Vijay" To: "John Baldwin" , X-OriginalArrivalTime: 31 May 2005 21:19:16.0063 (UTC) FILETIME=[6646BEF0:01C56626] Cc: Subject: RE: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 21:19:17 -0000 > On Tuesday 31 May 2005 01:37 pm, Singh, Vijay wrote: > > Hello. I am trying to benchmark 5.4 performance for a=20 > company project. > > I've got: > > > > FreeBSD 5.4-RELEASE #0: Fri May 27 20:52:58 PDT 2005 > > admin@netpc13.lab.netapp.com:/usr/obj/usr/src/sys/SMP > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2395.92-MHz 686-class CPU) > > Origin =3D "GenuineIntel" Id =3D 0xf27 Stepping =3D 7 > > > >=20 > = Features=3D0xbfebfbff > GE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,S > > SE2,SS,HTT,TM,PBE> > > Hyperthreading: 2 logical CPUs > > real memory =3D 1073676288 (1023 MB) > > avail memory =3D 1041121280 (992 MB) > > ACPI APIC Table: > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0=20 > (BSP): APIC=20 > > ID: 0 > > cpu1 (AP): APIC ID: 1 > > cpu2 (AP): APIC ID: 2 > > cpu3 (AP): APIC ID: 3 > > > > em0: port=20 > > 0xc800-0xc83f mem 0xfe8c0000-0xfe8dffff irq 18 at device 2.0 on pci1 > > em1: port=20 > > 0xd000-0xd03f mem 0xfe8e0000-0xfe8fffff irq 19 at device 2.1 on pci1 > > > > The UP version of the builds is able to deliver close to=20 > line rate on=20 > > these 2 interfaces. However the SMP build (with WITNESS and=20 > INVARIANTS > > disabled) gives me half the line rate on them. I am using netperf. > > > > /opt/netperf/netperf -H x.x.x.x -f m -l 120 > > > > Socket Socket Message Elapsed > > Size Size Size Time Throughput > > bytes bytes bytes secs. 10^6bits/sec > > > > 65536 16384 16384 120.00 429.14 > > 65536 16384 16384 120.00 501.74 > > > > Is there something I can do to make the system scale? I=20 > cannot move to=20 > > -CURRENT, but I can try pulling patches. >=20 > Does UP with 'device apic' also show poor performance? Try=20 > disabling USB support in the kernel as some Intel server=20 > motherboards have a "feature" that our APIC code trips over=20 > that can hurt performance by adding lots of stray interrupts=20 > on one of the USB controllers. >=20 Thanks. I have the following in GENERIC, do I need to comment out all? Is the problem you mention responsible for the "Interrupt storm detected" messages? Also I see some ACPI messages (below). Could that also be a problem? # USB support device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface #device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices device ugen # Generic device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse device urio # Diamond Rio 500 MP3 player device uscanner # Scanners # USB Ethernet, requires mii device aue # ADMtek USB Ethernet device axe # ASIX Electronics USB Ethernet device cdce # Generic USB over Ethernet device cue # CATC USB Ethernet device kue # Kawasaki LSI USB Ethernet device rue # RealTek RTL8150 USB Ethernet ACPI-0698: *** Warning: Type override - [DEB_] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0698: *** Warning: Type override - [MLIB] had invalid type (Integer) for Scope operator, changed to (Scope) [...] ACPI-0698: *** Warning: Type override - [KBC_] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0698: *** Warning: Type override - [OEM_] had invalid type (Integer) for Scope operator, changed to (Scope) Br vijay From owner-freebsd-smp@FreeBSD.ORG Tue May 31 21:23:10 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7DE9216A41C for ; Tue, 31 May 2005 21:23:10 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from mx1.netapp.com (mx1.netapp.com [216.240.18.38]) by mx1.FreeBSD.org (Postfix) with ESMTP id 43FEC43D4C for ; Tue, 31 May 2005 21:23:10 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from smtp2.corp.netapp.com (10.57.159.114) by mx1.netapp.com with ESMTP; 31 May 2005 14:23:10 -0700 X-IronPort-AV: i="3.93,154,1115017200"; d="scan'208"; a="175941989:sNHT17072248" Received: from svlexc03.hq.netapp.com (svlexc03.corp.netapp.com [10.57.156.149]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id j4VLN9W0012116; Tue, 31 May 2005 14:23:09 -0700 (PDT) Received: from lavender.hq.netapp.com ([10.56.11.75]) by svlexc03.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 31 May 2005 14:23:09 -0700 Received: from magenta.hq.netapp.com ([10.56.11.84]) by lavender.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 31 May 2005 14:23:09 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 31 May 2005 14:23:08 -0700 Message-ID: <637A278D8D0DBC438EA5E75C6E1818B904410F37@magenta.hq.netapp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 5.4 network performance Thread-Index: AcVmFKMvMQa5IkMuQDiwRxQ1ZQyPzwAEgzng From: "Singh, Vijay" To: "Steven Hartland" , X-OriginalArrivalTime: 31 May 2005 21:23:09.0485 (UTC) FILETIME=[F1681DD0:01C56626] Cc: Subject: RE: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 21:23:10 -0000 Hello. Disabling HTT by sysctl -w machdep.halt_logical_cpus=3D1 improves performance by a little bit, but still not close the the UP performance. I am trying now to disable USB. -vijay=20 > -----Original Message----- > From: Steven Hartland [mailto:killing@multiplay.co.uk]=20 > Sent: Tuesday, May 31, 2005 12:11 PM > To: Singh, Vijay; freebsd-smp@freebsd.org > Subject: Re: 5.4 network performance >=20 > Ensure that HT is disabled and see if that makes any difference. >=20 > Steve > ----- Original Message ----- > From: "Singh, Vijay" > =20 > Is there something I can do to make the system scale? I=20 > cannot move to -CURRENT, but I can try pulling patches. >=20 >=20 > = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > This e.mail is private and confidential between Multiplay=20 > (UK) Ltd. and the person or entity to whom it is addressed.=20 > In the event of misdirection, the recipient is prohibited=20 > from using, copying, printing or otherwise disseminating it=20 > or any information contained in it.=20 >=20 > In the event of misdirection, illegible or incomplete=20 > transmission please telephone (023) 8024 3137 or return the=20 > E.mail to postmaster@multiplay.co.uk. >=20 > _______________________________________________ > freebsd-smp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-smp > To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" >=20 From owner-freebsd-smp@FreeBSD.ORG Tue May 31 21:50:02 2005 Return-Path: X-Original-To: freebsd-smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9A05F16A41C for ; Tue, 31 May 2005 21:50:02 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: from mail21.sea5.speakeasy.net (mail21.sea5.speakeasy.net [69.17.117.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5023E43D53 for ; Tue, 31 May 2005 21:50:02 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 28206 invoked from network); 31 May 2005 21:50:02 -0000 Received: from server.baldwin.cx ([216.27.160.63]) (envelope-sender ) by mail21.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 31 May 2005 21:50:01 -0000 Received: from [10.50.41.231] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.13.1/8.13.1) with ESMTP id j4VLnpX3064952; Tue, 31 May 2005 17:49:52 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-smp@FreeBSD.org Date: Tue, 31 May 2005 17:46:29 -0400 User-Agent: KMail/1.8 References: <637A278D8D0DBC438EA5E75C6E1818B904410F31@magenta.hq.netapp.com> In-Reply-To: <637A278D8D0DBC438EA5E75C6E1818B904410F31@magenta.hq.netapp.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200505311746.30830.jhb@FreeBSD.org> X-Spam-Status: No, score=-102.8 required=4.2 tests=ALL_TRUSTED, USER_IN_WHITELIST autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on server.baldwin.cx Cc: "Singh, Vijay" Subject: Re: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 21:50:02 -0000 On Tuesday 31 May 2005 05:19 pm, Singh, Vijay wrote: > > On Tuesday 31 May 2005 01:37 pm, Singh, Vijay wrote: > > > Hello. I am trying to benchmark 5.4 performance for a > > > > company project. > > > > > I've got: > > > > > > FreeBSD 5.4-RELEASE #0: Fri May 27 20:52:58 PDT 2005 > > > admin@netpc13.lab.netapp.com:/usr/obj/usr/src/sys/SMP > > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > > CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2395.92-MHz 686-class CPU) > > > Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 > > > > Features=0xbfebfbff > > > > GE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,S > > > SE2,SS,HTT,TM,PBE> > > > Hyperthreading: 2 logical CPUs > > > real memory = 1073676288 (1023 MB) > > > avail memory = 1041121280 (992 MB) > > > ACPI APIC Table: > > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 > > > > (BSP): APIC > > > > > ID: 0 > > > cpu1 (AP): APIC ID: 1 > > > cpu2 (AP): APIC ID: 2 > > > cpu3 (AP): APIC ID: 3 > > > > > > em0: port > > > 0xc800-0xc83f mem 0xfe8c0000-0xfe8dffff irq 18 at device 2.0 on pci1 > > > em1: port > > > 0xd000-0xd03f mem 0xfe8e0000-0xfe8fffff irq 19 at device 2.1 on pci1 > > > > > > The UP version of the builds is able to deliver close to > > > > line rate on > > > > > these 2 interfaces. However the SMP build (with WITNESS and > > > > INVARIANTS > > > > > disabled) gives me half the line rate on them. I am using netperf. > > > > > > /opt/netperf/netperf -H x.x.x.x -f m -l 120 > > > > > > Socket Socket Message Elapsed > > > Size Size Size Time Throughput > > > bytes bytes bytes secs. 10^6bits/sec > > > > > > 65536 16384 16384 120.00 429.14 > > > 65536 16384 16384 120.00 501.74 > > > > > > Is there something I can do to make the system scale? I > > > > cannot move to > > > > > -CURRENT, but I can try pulling patches. > > > > Does UP with 'device apic' also show poor performance? Try > > disabling USB support in the kernel as some Intel server > > motherboards have a "feature" that our APIC code trips over > > that can hurt performance by adding lots of stray interrupts > > on one of the USB controllers. > > Thanks. I have the following in GENERIC, do I need to comment out all? > Is the problem you mention responsible for the "Interrupt storm > detected" messages? Also I see some ACPI messages (below). Could that > also be a problem? > > # USB support > device uhci # UHCI PCI->USB interface > device ohci # OHCI PCI->USB interface > #device ehci # EHCI PCI->USB interface (USB 2.0) > device usb # USB Bus (required) > #device udbp # USB Double Bulk Pipe devices > device ugen # Generic > device uhid # "Human Interface Devices" > device ukbd # Keyboard > device ulpt # Printer > device umass # Disks/Mass storage - Requires scbus > and da > device ums # Mouse > device urio # Diamond Rio 500 MP3 player > device uscanner # Scanners > # USB Ethernet, requires mii > device aue # ADMtek USB Ethernet > device axe # ASIX Electronics USB Ethernet > device cdce # Generic USB over Ethernet > device cue # CATC USB Ethernet > device kue # Kawasaki LSI USB Ethernet > device rue # RealTek RTL8150 USB Ethernet Yes, comment all of that out. Disabling USB should solve your interrupt storm, yes. > ACPI-0698: *** Warning: Type override - [DEB_] had invalid type > (Integer) for Scope operator, changed to (Scope) > ACPI-0698: *** Warning: Type override - [MLIB] had invalid type > (Integer) for Scope operator, changed to (Scope) > [...] > ACPI-0698: *** Warning: Type override - [KBC_] had invalid type > (Integer) for Scope operator, changed to (Scope) > ACPI-0698: *** Warning: Type override - [OEM_] had invalid type > (Integer) for Scope operator, changed to (Scope) I don't think you need to worry about these. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-smp@FreeBSD.ORG Tue May 31 22:54:38 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C69516A41C; Tue, 31 May 2005 22:54:38 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id E9A8A43D1F; Tue, 31 May 2005 22:54:37 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from smtp1.corp.netapp.com (10.57.156.124) by mx2.netapp.com with ESMTP; 31 May 2005 15:54:37 -0700 X-IronPort-AV: i="3.93,155,1115017200"; d="scan'208"; a="222108641:sNHT19230936" Received: from svlexc03.hq.netapp.com (svlexc03.corp.netapp.com [10.57.156.149]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id j4VMsbSL000122; Tue, 31 May 2005 15:54:37 -0700 (PDT) Received: from lavender.hq.netapp.com ([10.56.11.75]) by svlexc03.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 31 May 2005 15:54:37 -0700 Received: from magenta.hq.netapp.com ([10.56.11.84]) by lavender.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 31 May 2005 15:54:37 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 31 May 2005 15:54:36 -0700 Message-ID: <637A278D8D0DBC438EA5E75C6E1818B904410FC3@magenta.hq.netapp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 5.4 network performance Thread-Index: AcVmKrNWuL/7aXAaSTuUrhOOydnsXQACMSqQ From: "Singh, Vijay" To: "John Baldwin" , X-OriginalArrivalTime: 31 May 2005 22:54:37.0338 (UTC) FILETIME=[B86C07A0:01C56633] Cc: Subject: RE: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 22:54:38 -0000 Hi, sorry for the trouble, but even after commenting the USB lines from GENERIC, and a "make buildkernel" I see the following in dmesg: usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 4 ports with 4 removable, self powered Is there something else I need to do to disable USB support in the kernel? Br vijay > -----Original Message----- > From: John Baldwin [mailto:jhb@FreeBSD.org]=20 > Sent: Tuesday, May 31, 2005 2:46 PM > To: freebsd-smp@FreeBSD.org > Cc: Singh, Vijay > Subject: Re: 5.4 network performance >=20 > On Tuesday 31 May 2005 05:19 pm, Singh, Vijay wrote: > > > On Tuesday 31 May 2005 01:37 pm, Singh, Vijay wrote: > > > > Hello. I am trying to benchmark 5.4 performance for a > > > > > > company project. > > > > > > > I've got: > > > > > > > > FreeBSD 5.4-RELEASE #0: Fri May 27 20:52:58 PDT 2005 > > > > admin@netpc13.lab.netapp.com:/usr/obj/usr/src/sys/SMP > > > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > > > CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2395.92-MHz 686-class CPU) > > > > Origin =3D "GenuineIntel" Id =3D 0xf27 Stepping =3D 7 > > > > > >=20 > Features=3D0xbfebfbff > > ,P > > > > > > > GE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,S > > > > SE2,SS,HTT,TM,PBE> > > > > Hyperthreading: 2 logical CPUs > > > > real memory =3D 1073676288 (1023 MB) avail memory =3D=20 > 1041121280 (992=20 > > > > MB) > > > > ACPI APIC Table: > > > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 > > > > > > (BSP): APIC > > > > > > > ID: 0 > > > > cpu1 (AP): APIC ID: 1 > > > > cpu2 (AP): APIC ID: 2 > > > > cpu3 (AP): APIC ID: 3 > > > > > > > > em0: 1.7.35> port=20 > > > > 0xc800-0xc83f mem 0xfe8c0000-0xfe8dffff irq 18 at device 2.0 on=20 > > > > pci1 > > > > em1: 1.7.35> port=20 > > > > 0xd000-0xd03f mem 0xfe8e0000-0xfe8fffff irq 19 at device 2.1 on=20 > > > > pci1 > > > > > > > > The UP version of the builds is able to deliver close to > > > > > > line rate on > > > > > > > these 2 interfaces. However the SMP build (with WITNESS and > > > > > > INVARIANTS > > > > > > > disabled) gives me half the line rate on them. I am=20 > using netperf. > > > > > > > > /opt/netperf/netperf -H x.x.x.x -f m -l 120 > > > > > > > > Socket Socket Message Elapsed > > > > Size Size Size Time Throughput > > > > bytes bytes bytes secs. 10^6bits/sec > > > > > > > > 65536 16384 16384 120.00 429.14 > > > > 65536 16384 16384 120.00 501.74 > > > > > > > > Is there something I can do to make the system scale? I > > > > > > cannot move to > > > > > > > -CURRENT, but I can try pulling patches. > > > > > > Does UP with 'device apic' also show poor performance? Try=20 > > > disabling USB support in the kernel as some Intel server=20 > > > motherboards have a "feature" that our APIC code trips=20 > over that can=20 > > > hurt performance by adding lots of stray interrupts on one of the=20 > > > USB controllers. > > > > Thanks. I have the following in GENERIC, do I need to=20 > comment out all? > > Is the problem you mention responsible for the "Interrupt storm=20 > > detected" messages? Also I see some ACPI messages (below).=20 > Could that=20 > > also be a problem? > > > > # USB support > > device uhci # UHCI PCI->USB interface > > device ohci # OHCI PCI->USB interface > > #device ehci # EHCI PCI->USB interface (USB 2.0) > > device usb # USB Bus (required) > > #device udbp # USB Double Bulk Pipe devices > > device ugen # Generic > > device uhid # "Human Interface Devices" > > device ukbd # Keyboard > > device ulpt # Printer > > device umass # Disks/Mass storage -=20 > Requires scbus > > and da > > device ums # Mouse > > device urio # Diamond Rio 500 MP3 player > > device uscanner # Scanners > > # USB Ethernet, requires mii > > device aue # ADMtek USB Ethernet > > device axe # ASIX Electronics USB Ethernet > > device cdce # Generic USB over Ethernet > > device cue # CATC USB Ethernet > > device kue # Kawasaki LSI USB Ethernet > > device rue # RealTek RTL8150 USB Ethernet >=20 > Yes, comment all of that out. Disabling USB should solve=20 > your interrupt storm, yes. >=20 > > ACPI-0698: *** Warning: Type override - [DEB_] had invalid type > > (Integer) for Scope operator, changed to (Scope) > > ACPI-0698: *** Warning: Type override - [MLIB] had invalid type > > (Integer) for Scope operator, changed to (Scope) > > [...] > > ACPI-0698: *** Warning: Type override - [KBC_] had invalid type > > (Integer) for Scope operator, changed to (Scope) > > ACPI-0698: *** Warning: Type override - [OEM_] had invalid type > > (Integer) for Scope operator, changed to (Scope) >=20 > I don't think you need to worry about these. >=20 > -- > John Baldwin <>< =20 > http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to=20 > Serve" =3D http://www.FreeBSD.org >=20 From owner-freebsd-smp@FreeBSD.ORG Tue May 31 23:07:07 2005 Return-Path: X-Original-To: freebsd-smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2484F16A422; Tue, 31 May 2005 23:07:07 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 709C643D1F; Tue, 31 May 2005 23:07:06 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.2.R) with ESMTP id md50001450536.msg; Wed, 01 Jun 2005 00:01:52 +0100 Message-ID: <008101c56635$5a7a1fe0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Singh, Vijay" , "John Baldwin" , References: <637A278D8D0DBC438EA5E75C6E1818B904410FC3@magenta.hq.netapp.com> Date: Wed, 1 Jun 2005 00:06:14 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Wed, 01 Jun 2005 00:01:52 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Wed, 01 Jun 2005 00:01:53 +0100 Cc: Subject: Re: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 23:07:07 -0000 Ensure the modules not built as well: makeoptions MODULES_OVERRIDE="acpi" should do it but ENSURE you have all drivers in the kernel before you do. Alternatively just delete the module from /boot/kernel Steve ----- Original Message ----- From: "Singh, Vijay" Is there something else I need to do to disable USB support in the kernel? ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-smp@FreeBSD.ORG Tue May 31 23:08:49 2005 Return-Path: X-Original-To: freebsd-smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 11B9516A41C; Tue, 31 May 2005 23:08:49 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.232]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA0B043D1F; Tue, 31 May 2005 23:08:48 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 9123F515A0; Tue, 31 May 2005 16:08:47 -0700 (PDT) Date: Tue, 31 May 2005 16:08:47 -0700 From: Kris Kennaway To: "Singh, Vijay" Message-ID: <20050531230847.GA39097@xor.obsecurity.org> References: <637A278D8D0DBC438EA5E75C6E1818B904410FC3@magenta.hq.netapp.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8t9RHnE3ZwKMSgU+" Content-Disposition: inline In-Reply-To: <637A278D8D0DBC438EA5E75C6E1818B904410FC3@magenta.hq.netapp.com> User-Agent: Mutt/1.4.2.1i Cc: freebsd-smp@FreeBSD.org, John Baldwin Subject: Re: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 23:08:49 -0000 --8t9RHnE3ZwKMSgU+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 31, 2005 at 03:54:36PM -0700, Singh, Vijay wrote: > Hi, sorry for the trouble, but even after commenting the USB lines from > GENERIC, and a "make buildkernel" I see the following in dmesg: >=20 > usb0: OHCI version 1.0, legacy support > usb0: SMM does not respond, resetting > usb0: on ohci0 > usb0: USB revision 1.0 > uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub0: 4 ports with 4 removable, self powered >=20 > Is there something else I need to do to disable USB support in the > kernel? 'make installkernel'? :) Kris --8t9RHnE3ZwKMSgU+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCnO5+Wry0BWjoQKURAqRjAKDEJYKYbtIImyPnSJI3hiY19U1gvgCeIyZ8 XPy7maIds3Y96CLzogh/qsI= =cYK5 -----END PGP SIGNATURE----- --8t9RHnE3ZwKMSgU+-- From owner-freebsd-smp@FreeBSD.ORG Wed Jun 1 01:20:07 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 69A5A16A41F for ; Wed, 1 Jun 2005 01:20:07 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.197]) by mx1.FreeBSD.org (Postfix) with ESMTP id CBB1843D49 for ; Wed, 1 Jun 2005 01:20:06 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by rproxy.gmail.com with SMTP id i8so2030920rne for ; Tue, 31 May 2005 18:20:06 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=A2QrOnaTeFq8x0FX2bLBUDWEOuZDhNszR0oQAj/Sq9kUwQliMmc/I2HcUQajRvBR/j/aXIBfN54o6Yck8IZDCPCTVKiTxgFAxgn56xsACXdD+C0IMF9r/rygyY1Z7aocqwEWzRjSyMYpmSfsD21mwxV35OXyZZ5nRLioL7jB5CE= Received: by 10.38.89.40 with SMTP id m40mr2076242rnb; Tue, 31 May 2005 18:20:06 -0700 (PDT) Received: by 10.38.209.31 with HTTP; Tue, 31 May 2005 18:20:06 -0700 (PDT) Message-ID: <84dead72050531182067293818@mail.gmail.com> Date: Wed, 1 Jun 2005 06:50:06 +0530 From: Joseph Koshy To: "Singh, Vijay" In-Reply-To: <20050531230847.GA39097@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <637A278D8D0DBC438EA5E75C6E1818B904410FC3@magenta.hq.netapp.com> <20050531230847.GA39097@xor.obsecurity.org> Cc: freebsd-smp@freebsd.org Subject: Re: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Joseph Koshy List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2005 01:20:07 -0000 > GENERIC, and a "make buildkernel" I see the following in=20 > dmesg: ... > Is there something else I need to do to disable USB support in the > kernel? Also check the "usbd_enable" line in "/etc/rc.conf". =20 --=20 FreeBSD Volunteer, http://people.freebsd.org/~jkoshy From owner-freebsd-smp@FreeBSD.ORG Wed Jun 1 01:34:09 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4D5E216A41C; Wed, 1 Jun 2005 01:34:08 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0BF3643D49; Wed, 1 Jun 2005 01:34:07 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from smtp2.corp.netapp.com (10.57.159.114) by mx2.netapp.com with ESMTP; 31 May 2005 18:34:07 -0700 X-IronPort-AV: i="3.93,155,1115017200"; d="scan'208"; a="222119522:sNHT17021804" Received: from svlexc03.hq.netapp.com (svlexc03.corp.netapp.com [10.57.156.149]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id j511Y6MO027975; Tue, 31 May 2005 18:34:07 -0700 (PDT) Received: from lavender.hq.netapp.com ([10.56.11.75]) by svlexc03.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 31 May 2005 18:34:06 -0700 Received: from magenta.hq.netapp.com ([10.56.11.84]) by lavender.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 31 May 2005 18:34:06 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 31 May 2005 18:34:06 -0700 Message-ID: <637A278D8D0DBC438EA5E75C6E1818B90441109D@magenta.hq.netapp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 5.4 network performance Thread-Index: AcVmNbSxb43GfCcjQm2JEpm2rgWHAwAFEV3g From: "Singh, Vijay" To: "Kris Kennaway" X-OriginalArrivalTime: 01 Jun 2005 01:34:06.0560 (UTC) FILETIME=[001F6E00:01C5664A] Cc: freebsd-smp@FreeBSD.org, John Baldwin Subject: RE: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2005 01:34:09 -0000 Yep, I did that. :-)=20 > -----Original Message----- > From: Kris Kennaway [mailto:kris@obsecurity.org]=20 > Sent: Tuesday, May 31, 2005 4:09 PM > To: Singh, Vijay > Cc: John Baldwin; freebsd-smp@FreeBSD.org > Subject: Re: 5.4 network performance >=20 > On Tue, May 31, 2005 at 03:54:36PM -0700, Singh, Vijay wrote: > > Hi, sorry for the trouble, but even after commenting the USB lines=20 > > from GENERIC, and a "make buildkernel" I see the following in dmesg: > >=20 > > usb0: OHCI version 1.0, legacy support > > usb0: SMM does not respond, resetting > > usb0: on ohci0 > > usb0: USB revision 1.0 > > uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > > uhub0: 4 ports with 4 removable, self powered > >=20 > > Is there something else I need to do to disable USB support in the=20 > > kernel? >=20 > 'make installkernel'? :) >=20 > Kris >=20 From owner-freebsd-smp@FreeBSD.ORG Wed Jun 1 15:13:49 2005 Return-Path: X-Original-To: freebsd-smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B1DE716A424 for ; Wed, 1 Jun 2005 15:13:49 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: from mail21.sea5.speakeasy.net (mail21.sea5.speakeasy.net [69.17.117.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id 74EF843D48 for ; Wed, 1 Jun 2005 15:13:49 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 20602 invoked from network); 1 Jun 2005 15:13:49 -0000 Received: from server.baldwin.cx ([216.27.160.63]) (envelope-sender ) by mail21.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 1 Jun 2005 15:13:48 -0000 Received: from [10.50.41.231] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.13.1/8.13.1) with ESMTP id j51FDbTN074139; Wed, 1 Jun 2005 11:13:41 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-smp@FreeBSD.org Date: Wed, 1 Jun 2005 09:27:17 -0400 User-Agent: KMail/1.8 References: <637A278D8D0DBC438EA5E75C6E1818B904410FC3@magenta.hq.netapp.com> In-Reply-To: <637A278D8D0DBC438EA5E75C6E1818B904410FC3@magenta.hq.netapp.com> MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200506010927.17911.jhb@FreeBSD.org> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-102.8 required=4.2 tests=ALL_TRUSTED, USER_IN_WHITELIST autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on server.baldwin.cx Cc: "Singh, Vijay" Subject: Re: 5.4 network performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2005 15:13:49 -0000 On Tuesday 31 May 2005 06:54 pm, Singh, Vijay wrote: > Hi, sorry for the trouble, but even after commenting the USB lines from > GENERIC, and a "make buildkernel" I see the following in dmesg: > > usb0: OHCI version 1.0, legacy support > usb0: SMM does not respond, resetting > usb0: on ohci0 > usb0: USB revision 1.0 > uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub0: 4 ports with 4 removable, self powered > > Is there something else I need to do to disable USB support in the > kernel? Turn off 'usbd_enable=YES' in your /etc/rc.conf so that it doesn't try to autoload the USB kernel modules during boot as well. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-smp@FreeBSD.ORG Thu Jun 2 01:58:47 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76ABF16A41C for ; Thu, 2 Jun 2005 01:58:47 +0000 (GMT) (envelope-from oceanare@pacific.net.sg) Received: from sarajevo.pacific.net.sg (sarajevo.pacific.net.sg [203.120.90.134]) by mx1.FreeBSD.org (Postfix) with SMTP id AA6BE43D4C for ; Thu, 2 Jun 2005 01:58:46 +0000 (GMT) (envelope-from oceanare@pacific.net.sg) Received: (qmail 6483 invoked from network); 2 Jun 2005 01:58:44 -0000 Received: from unknown (HELO maxwell2.pacific.net.sg) (203.120.90.192) by sarajevo with SMTP; 2 Jun 2005 01:58:44 -0000 Received: from [192.168.0.107] ([210.24.246.101]) by maxwell2.pacific.net.sg with ESMTP id <20050602015843.GJHV1130.maxwell2.pacific.net.sg@[192.168.0.107]> for ; Thu, 2 Jun 2005 09:58:43 +0800 Message-ID: <429E67CB.6090901@pacific.net.sg> Date: Thu, 02 Jun 2005 09:58:35 +0800 From: Erich Dollansky Organization: oceanare pte ltd User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050514) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-smp@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Assignmet of CPUs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2005 01:58:47 -0000 Hi, it is a general problem of systems with more than one CPU to assign the proper CPU to a task. The most obvious problem is the cache. If a task is migrating all the time, the cache has to be reloaded again and again. The next problem is the location of the data in RAM on NUMA machines. It is better to leave a thread on a CPU if the RAM connected to it also has the data stored. This could block a task from running if to many running tasks have their data stored on the same CPU. The last, but hardly used parameter, is the CPU temperature. I noticed that FreeBSD tends to use always the same CPU to start a task. This makes one CPU real hot while the other stays cool. Taking the CPU temperature into account for starting at least new threads would also have an advantage of systems with an less then ideal cooling system. Spreading tasks all over the system with the coolest CPU being the one to be started next will make systems a bit cooler. Erich From owner-freebsd-smp@FreeBSD.ORG Thu Jun 2 13:16:55 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 423F216A41C for ; Thu, 2 Jun 2005 13:16:55 +0000 (GMT) (envelope-from hsoftdev17@gmail.com) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id DED4243D54 for ; Thu, 2 Jun 2005 13:16:54 +0000 (GMT) (envelope-from hsoftdev17@gmail.com) Received: by wproxy.gmail.com with SMTP id 69so290715wri for ; Thu, 02 Jun 2005 06:16:54 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=qApvclx11LfRuEdIU97zLoRzouoQctSRvQzdxvAQyEiVvJjsRxtPBidi36Nyu67qpSy++ZsWGZ9ojcTX3E2mCRpDysF8QJ8AINBR6zAZFhzcIXKVdaFm7/6fGPfQrOiQiNnySUz6DoVYxCuCn0WdGL+yrdka2UrByYkThFUxTlE= Received: by 10.54.8.26 with SMTP id 26mr269927wrh; Thu, 02 Jun 2005 06:16:07 -0700 (PDT) Received: by 10.54.133.7 with HTTP; Thu, 2 Jun 2005 06:16:07 -0700 (PDT) Message-ID: <6845d25a0506020616293991e3@mail.gmail.com> Date: Thu, 2 Jun 2005 09:16:07 -0400 From: Dave Stephens To: freebsd-smp@freebsd.org In-Reply-To: <429E67CB.6090901@pacific.net.sg> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <429E67CB.6090901@pacific.net.sg> Subject: Re: Assignmet of CPUs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Dave Stephens List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2005 13:16:55 -0000 The main problem I see with using temperature is that the ability to detect CPU temp. is a feature limited to "more modern" machines. It has become very common even in desktops now a days, but it wouldn't be available in legacy hardware or custom hardware. Just a thought. On 6/1/05, Erich Dollansky wrote: > Hi, >=20 > it is a general problem of systems with more than one CPU to assign the > proper CPU to a task. >=20 > The most obvious problem is the cache. If a task is migrating all the > time, the cache has to be reloaded again and again. >=20 > The next problem is the location of the data in RAM on NUMA machines. It > is better to leave a thread on a CPU if the RAM connected to it also has > the data stored. This could block a task from running if to many running > tasks have their data stored on the same CPU. >=20 > The last, but hardly used parameter, is the CPU temperature. I noticed > that FreeBSD tends to use always the same CPU to start a task. This > makes one CPU real hot while the other stays cool. Taking the CPU > temperature into account for starting at least new threads would also > have an advantage of systems with an less then ideal cooling system. >=20 > Spreading tasks all over the system with the coolest CPU being the one > to be started next will make systems a bit cooler. >=20 > Erich > _______________________________________________ > freebsd-smp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-smp > To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" > From owner-freebsd-smp@FreeBSD.ORG Thu Jun 2 21:09:53 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2F93616A41C for ; Thu, 2 Jun 2005 21:09:53 +0000 (GMT) (envelope-from david@landgren.net) Received: from smtp10.wanadoo.fr (smtp10.wanadoo.fr [193.252.22.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id CECCD43D4C for ; Thu, 2 Jun 2005 21:09:52 +0000 (GMT) (envelope-from david@landgren.net) Received: from me-wanadoo.net (localhost [127.0.0.1]) by mwinf1002.wanadoo.fr (SMTP Server) with ESMTP id D0CA52400099 for ; Thu, 2 Jun 2005 23:09:50 +0200 (CEST) Received: from [82.120.199.250] (ASt-Lambert-152-1-62-250.w82-120.abo.wanadoo.fr [82.120.199.250]) by mwinf1002.wanadoo.fr (SMTP Server) with ESMTP id 0E60424000B3; Thu, 2 Jun 2005 23:09:49 +0200 (CEST) X-ME-UUID: 20050602210950595.0E60424000B3@mwinf1002.wanadoo.fr Message-ID: <429F759F.1000403@landgren.net> Date: Thu, 02 Jun 2005 23:09:51 +0200 From: David Landgren Organization: The Dusty Decadent Delights of Imperial Pompeii User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a6) Gecko/20050111 X-Accept-Language: en-us, en, fr-fr MIME-Version: 1.0 To: Dave Stephens References: <429E67CB.6090901@pacific.net.sg> <6845d25a0506020616293991e3@mail.gmail.com> In-Reply-To: <6845d25a0506020616293991e3@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-smp@freebsd.org Subject: Re: Assignmet of CPUs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2005 21:09:53 -0000 Dave Stephens wrote: > The main problem I see with using temperature is that the ability to > detect CPU temp. is a feature limited to "more modern" machines. It > has become very common even in desktops now a days, but it wouldn't be > available in legacy hardware or custom hardware. > > Just a thought. Well, round-robin startup would at least be better than always using one process to start all processes. That would go some of the way towards improving asymmetric heating. (Says me, whose knowledge of kernel scheduling could be written on the face of a chip and still leave room for footnotes...) David > > On 6/1/05, Erich Dollansky wrote: > [...] >>The last, but hardly used parameter, is the CPU temperature. I noticed >>that FreeBSD tends to use always the same CPU to start a task. This >>makes one CPU real hot while the other stays cool. Taking the CPU >>temperature into account for starting at least new threads would also >>have an advantage of systems with an less then ideal cooling system. >> >>Spreading tasks all over the system with the coolest CPU being the one >>to be started next will make systems a bit cooler. From owner-freebsd-smp@FreeBSD.ORG Fri Jun 3 00:16:07 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6B5E616A41C for ; Fri, 3 Jun 2005 00:16:07 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 30E6943D4C for ; Fri, 3 Jun 2005 00:16:07 +0000 (GMT) (envelope-from Vijay.Singh@netapp.com) Received: from smtp1.corp.netapp.com (10.57.156.124) by mx2.netapp.com with ESMTP; 02 Jun 2005 17:16:07 -0700 X-IronPort-AV: i="3.93,163,1115017200"; d="scan'208"; a="223414246:sNHT17376856" Received: from svlexc02.hq.netapp.com (svlexc02.corp.netapp.com [10.57.157.136]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id j530G6e6005573 for ; Thu, 2 Jun 2005 17:16:06 -0700 (PDT) Received: from lavender.hq.netapp.com ([10.56.11.75]) by svlexc02.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Thu, 2 Jun 2005 17:16:06 -0700 Received: from magenta.hq.netapp.com ([10.56.11.84]) by lavender.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.6713); Thu, 2 Jun 2005 17:16:06 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Thu, 2 Jun 2005 17:16:06 -0700 Message-ID: <637A278D8D0DBC438EA5E75C6E1818B90454D76D@magenta.hq.netapp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Benchmarking 6.0 Thread-Index: AcVn0W7X+lXhfP4SSGWKohQXtLACig== From: "Singh, Vijay" To: X-OriginalArrivalTime: 03 Jun 2005 00:16:06.0970 (UTC) FILETIME=[6FB229A0:01C567D1] Cc: Subject: Benchmarking 6.0 X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2005 00:16:07 -0000 Hello, I am trying to run network benchmarking using netperf. I have a snapshot of "systat -vmstat 1". Is it OK to see those large number of interrupts (~ 1998) for the lapics? 1 users Load 1.62 0.96 0.52 Jun 2 17:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 14096 2760 44296 2760 802184 count All 217936 5592 5941072 5780 pages =20 Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow 22014 total 1 27 40374 80521894792750 376 102916 wire 1: atkb 17368 act 6: fdc0 23.3%Sys 46.9%Intr 1.1%User 0.0%Nice 28.7%Idl 98232 inact 13: npx | | | | | | | | | | 792 cache 14: ata =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+++++++++++++++++++++++> = 801392 free 15: ata daefr 6083 18: em0 Namei Name-cache Dir-cache prcfr 6938 19: em1 Calls hits % hits % react 20: em2 pdwak 5 30: fxp zfod pdpgs 1998 lapic0: ti Disks ad0 ofod intrn 998 0: clk KB/t 0.00 %slo-z 108400 buf 1998 lapic1: ti tps 0 tfree 5 dirty 1997 lapic3: ti MB/s 0.00 69985 desir 1997elapic2: ti % busy 0 17496 numvnodes 1677 freevnodes Vijay Singh NetApp=20 From owner-freebsd-smp@FreeBSD.ORG Fri Jun 3 00:35:34 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BCB6C16A42A for ; Fri, 3 Jun 2005 00:35:34 +0000 (GMT) (envelope-from ketralnis@ketralnis.dyndns.org) Received: from ylpvm12.prodigy.net (ylpvm12-ext.prodigy.net [207.115.57.43]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2FA8E43D48 for ; Fri, 3 Jun 2005 00:35:33 +0000 (GMT) (envelope-from ketralnis@ketralnis.dyndns.org) Received: from pimout6-ext.prodigy.net (pimout6-int.prodigy.net [207.115.4.22]) by ylpvm12.prodigy.net (8.12.10 outbound/8.12.10) with ESMTP id j530ZV28025647 for ; Thu, 2 Jun 2005 20:35:31 -0400 X-ORBL: [64.173.11.131] Received: from ketralnis.dyndns.org (adsl-64-173-11-131.dsl.sntc01.pacbell.net [64.173.11.131]) by pimout6-ext.prodigy.net (8.12.10 milter /8.12.10) with ESMTP id j530ZWug422994 for ; Thu, 2 Jun 2005 20:35:32 -0400 Received: from [10.0.2.239] ([10.0.2.239]) by ketralnis.dyndns.org (8.13.1/8.13.1) with ESMTP id j530ZEre000515 for ; Thu, 2 Jun 2005 17:35:14 -0700 (PDT) (envelope-from ketralnis@ketralnis.dyndns.org) Mime-Version: 1.0 (Apple Message framework v622) In-Reply-To: <429F759F.1000403@landgren.net> References: <429E67CB.6090901@pacific.net.sg> <6845d25a0506020616293991e3@mail.gmail.com> <429F759F.1000403@landgren.net> Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <845448d08ea438ecc1d735af8eaf5f6f@ketralnis.dyndns.org> Content-Transfer-Encoding: 7bit From: David King Date: Thu, 2 Jun 2005 17:35:34 -0700 To: freebsd-smp@freebsd.org X-Mailer: Apple Mail (2.622) Subject: Re: Assignmet of CPUs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2005 00:35:34 -0000 > Well, round-robin startup would at least be better than always using > one process to start all processes. That would go some of the way > towards improving asymmetric heating. (Says me, whose knowledge of > kernel scheduling could be written on the face of a chip and still > leave room for footnotes...) In the spirit of small amounts of scheduling knowledge, are load averages kept on individual processors? Or can they be determined fast enough to not increase the process creation time significantly (or at least to be offset by the speed gained), in order to put a process or thread on the least encumbered processor? top(1) lists the assigned CPU, so it seems like it would be simple enough to determine on-the-fly, but if it's not kept somewhere, I wouldn't want to iterate every process to get its assigned CPU every time I create a new one. Is there a "scheduling-for-dummies" feasibly (even if not easily) read by non-kernel hackers? :) > David > >> On 6/1/05, Erich Dollansky wrote: > [...] > >>> The last, but hardly used parameter, is the CPU temperature. I >>> noticed >>> that FreeBSD tends to use always the same CPU to start a task. This >>> makes one CPU real hot while the other stays cool. Taking the CPU >>> temperature into account for starting at least new threads would also >>> have an advantage of systems with an less then ideal cooling system. >>> >>> Spreading tasks all over the system with the coolest CPU being the >>> one >>> to be started next will make systems a bit cooler. > > _______________________________________________ > freebsd-smp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-smp > To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" From owner-freebsd-smp@FreeBSD.ORG Fri Jun 3 07:36:12 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6B8EF16A41C for ; Fri, 3 Jun 2005 07:36:12 +0000 (GMT) (envelope-from david@landgren.net) Received: from sferics.mongueurs.net (sferics.mongueurs.net [81.80.147.197]) by mx1.FreeBSD.org (Postfix) with ESMTP id E03B543D1F for ; Fri, 3 Jun 2005 07:36:10 +0000 (GMT) (envelope-from david@landgren.net) Received: from [127.0.0.1] (exo.bpinet.com [81.80.147.206]) by sferics.mongueurs.net (Postfix) with ESMTP id 8035AAA3C; Fri, 3 Jun 2005 09:36:09 +0200 (CEST) Message-ID: <42A00867.7020505@landgren.net> Date: Fri, 03 Jun 2005 09:36:07 +0200 From: David Landgren Organization: The Lusty Decadent Delights of Imperial Pompeii User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: David King References: <429E67CB.6090901@pacific.net.sg> <6845d25a0506020616293991e3@mail.gmail.com> <429F759F.1000403@landgren.net> <845448d08ea438ecc1d735af8eaf5f6f@ketralnis.dyndns.org> In-Reply-To: <845448d08ea438ecc1d735af8eaf5f6f@ketralnis.dyndns.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-smp@freebsd.org Subject: Re: Assignmet of CPUs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2005 07:36:12 -0000 David King wrote: >> Well, round-robin startup would at least be better than always using >> one process to start all processes. That would go some of the way ^^^^^^^ I did of course mean to say processor here. David >> towards improving asymmetric heating. (Says me, whose knowledge of >> kernel scheduling could be written on the face of a chip and still >> leave room for footnotes...) > > > In the spirit of small amounts of scheduling knowledge, are load > averages kept on individual processors? Or can they be determined fast > enough to not increase the process creation time significantly (or at > least to be offset by the speed gained), in order to put a process or > thread on the least encumbered processor? top(1) lists the assigned CPU, > so it seems like it would be simple enough to determine on-the-fly, but > if it's not kept somewhere, I wouldn't want to iterate every process to > get its assigned CPU every time I create a new one. > > Is there a "scheduling-for-dummies" feasibly (even if not easily) read > by non-kernel hackers? :) > >> David From owner-freebsd-smp@FreeBSD.ORG Fri Jun 3 08:08:25 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4579116A41C for ; Fri, 3 Jun 2005 08:08:25 +0000 (GMT) (envelope-from oceanare@pacific.net.sg) Received: from sarajevo.pacific.net.sg (sarajevo.pacific.net.sg [203.120.90.134]) by mx1.FreeBSD.org (Postfix) with SMTP id 722AC43D1F for ; Fri, 3 Jun 2005 08:08:24 +0000 (GMT) (envelope-from oceanare@pacific.net.sg) Received: (qmail 28730 invoked from network); 3 Jun 2005 08:08:21 -0000 Received: from unknown (HELO maxwell6.pacific.net.sg) (203.120.90.212) by sarajevo with SMTP; 3 Jun 2005 08:08:21 -0000 Received: from [192.168.0.107] ([210.24.246.101]) by maxwell6.pacific.net.sg with ESMTP id <20050603080821.IPQC1233.maxwell6.pacific.net.sg@[192.168.0.107]>; Fri, 3 Jun 2005 16:08:21 +0800 Message-ID: <42A00FD7.3070205@pacific.net.sg> Date: Fri, 03 Jun 2005 16:07:51 +0800 From: Erich Dollansky Organization: oceanare pte ltd User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050514) X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Landgren References: <429E67CB.6090901@pacific.net.sg> <6845d25a0506020616293991e3@mail.gmail.com> <429F759F.1000403@landgren.net> <845448d08ea438ecc1d735af8eaf5f6f@ketralnis.dyndns.org> <42A00867.7020505@landgren.net> In-Reply-To: <42A00867.7020505@landgren.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-smp@freebsd.org, David King Subject: Re: Assignmet of CPUs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2005 08:08:25 -0000 Hi, David Landgren wrote: > David King wrote: > >>> Well, round-robin startup would at least be better than always using >>> one process to start all processes. That would go some of the way > > ^^^^^^^ > > I did of course mean to say processor here. > Round-robin has does not really work with processes which produce different loads as those processes would have to move to other CPUs later to be ale to run. This involves reloading of CPU caches. The whole problem is not this simple. NUMA adds here as the memory content is local to CPUs too. Moving a process to a neighbour is still not this bad compared to moving it to a different CPU node. Erich From owner-freebsd-smp@FreeBSD.ORG Fri Jun 3 09:12:23 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7336716A41C for ; Fri, 3 Jun 2005 09:12:23 +0000 (GMT) (envelope-from hatchet@arevalo.com) Received: from 200-161-170-106.dsl.telesp.net.br (200-161-170-106.dsl.telesp.net.br [200.161.170.106]) by mx1.FreeBSD.org (Postfix) with SMTP id 75CF343D5E for ; Fri, 3 Jun 2005 09:12:14 +0000 (GMT) (envelope-from hatchet@arevalo.com) Received: from [89.233.25.45] (port=4497 helo=[washer]) by 200-161-170-106.dsl.telesp.net.br with esmtp id 7446892714enumerate122259 for freebsd-smp@freebsd.org; Fri, 3 Jun 2005 07:11:58 -0200 Mime-Version: 1.0 (Apple Message framework v728) Content-Transfer-Encoding: 7bit Message-Id: <12054756221.3726397905@200-161-170-106.dsl.telesp.net.br> Content-Type: text/plain; charset=US-ASCII; format=flowed To: freebsd-smp@freebsd.org From: Greta Date: Fri, 3 Jun 2005 07:11:57 -0200 X-Mailer: Apple Mail (2.728) Subject: The Best Place To Find The Cheapest Medicine. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2005 09:12:23 -0000 Prices You Can Afford - Quality You Can Trust http://breathable.healingsphere.info/?Bernardinextvuycourtyardzvtbashes Self-respect is the cornerstone of all virtue. There are three ingredients to the good life; learning, earning, and yearning. We don't know a millionth of one percent about anything. Nixon, Ford, Carter, Reagan - a Mount Rushmore of incompetence. From owner-freebsd-smp@FreeBSD.ORG Fri Jun 3 14:42:05 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 72D1F16A41C for ; Fri, 3 Jun 2005 14:42:05 +0000 (GMT) (envelope-from laurels@guaranteedsuccess.com) Received: from p54837471.dip.t-dialin.net (p54837471.dip.t-dialin.net [84.131.116.113]) by mx1.FreeBSD.org (Postfix) with SMTP id A4ADA43D48 for ; Fri, 3 Jun 2005 14:42:04 +0000 (GMT) (envelope-from laurels@guaranteedsuccess.com) Received: from [155.46.124.223] (port=4114 helo=[pause]) by p54837471.dip.t-dialin.net with esmtp id 2572829060purchaser68825 for freebsd-smp@freebsd.org; Fri, 3 Jun 2005 16:42:13 +0200 Mime-Version: 1.0 (Apple Message framework v728) Content-Transfer-Encoding: 7bit Message-Id: <2307455641.39246121339@p54837471.dip.t-dialin.net> Content-Type: text/plain; charset=US-ASCII; format=flowed To: freebsd-smp@freebsd.org From: Moll Date: Fri, 3 Jun 2005 16:42:12 +0200 X-Mailer: Apple Mail (2.728) Subject: Web's cheapest pills - Internet special X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2005 14:42:05 -0000 Viagra shipped privately and discreetly to your door. No prescription necessary http://gbe.t8e4futm8ltjxub.upshootgm.com A good conscience is a continual feast. If there is no God, who pops up the next Kleenex? I can't complain, but sometimes I still do. From owner-freebsd-smp@FreeBSD.ORG Sat Jun 4 20:02:19 2005 Return-Path: X-Original-To: freebsd-smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4C63D16A41C; Sat, 4 Jun 2005 20:02:19 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1416743D49; Sat, 4 Jun 2005 20:02:18 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 773E846B06; Sat, 4 Jun 2005 16:02:18 -0400 (EDT) Date: Sat, 4 Jun 2005 21:02:47 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "Singh, Vijay" In-Reply-To: <637A278D8D0DBC438EA5E75C6E1818B90454D76D@magenta.hq.netapp.com> Message-ID: <20050604205958.A93312@fledge.watson.org> References: <637A278D8D0DBC438EA5E75C6E1818B90454D76D@magenta.hq.netapp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-smp@FreeBSD.org, jhb@FreeBSD.org Subject: Re: Benchmarking 6.0 X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jun 2005 20:02:19 -0000 On Thu, 2 Jun 2005, Singh, Vijay wrote: > Hello, I am trying to run network benchmarking using netperf. I have a > snapshot of "systat -vmstat 1". Is it OK to see those large number of > interrupts (~ 1998) for the lapics? That's a property of the increase in HZ to 1000 from 100, and that on 6.x, each lapic generates HZ interrupts/sec to the individual processors, rather than the programmable timer deliverable at HZ to a particular (or round-robin'd) CPU, then that CPU broadcasting the clock tick to the other CPUs using IPIs. The rate you see should be approximately HZ * #cpus, so about 2000 for a dual-processor system (which is indeed about 1998). While lowering HZ will lower the overhead associated with processing timer ticks, it will also reduce timer granularity, which can affect performance in other ways (for example, TCP may behave better with finer granularity timeouts). One of the things we don't show in vmstat -systat that we probably should is per-CPU IPI rates. I've CC'd John in the hopes of convincing him to add that to his todo list :-). Robert N M Watson