From owner-freebsd-smp@FreeBSD.ORG Sun Jun 22 23:13:29 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC25337B401 for ; Sun, 22 Jun 2003 23:13:28 -0700 (PDT) Received: from wally.statscout.com (wally.statscout.com [203.39.101.146]) by mx1.FreeBSD.org (Postfix) with ESMTP id AEB4543F3F for ; Sun, 22 Jun 2003 23:13:26 -0700 (PDT) (envelope-from pak@statscout.com) Received: from speedy.statscout.com (speedy.statscout.com [10.1.1.100]) by wally.statscout.com (8.11.6/8.11.3av) with ESMTP id h5N6DG145108 for ; Mon, 23 Jun 2003 16:13:16 +1000 (EST) (envelope-from pak@statscout.com) Received: from speedy.statscout.com (localhost.statscout.com [127.0.0.1]) by speedy.statscout.com (8.12.8p1/8.12.8) with ESMTP id h5N6DHVl001389 for ; Mon, 23 Jun 2003 16:13:17 +1000 (EST) (envelope-from pak@speedy.statscout.com) Received: from localhost (localhost [[UNIX: localhost]]) by speedy.statscout.com (8.12.8p1/8.12.8/Submit) id h5N6DGqf001388 for freebsd-smp@freebsd.org; Mon, 23 Jun 2003 16:13:16 +1000 (EST) From: Paul Koch Organization: Statscout To: freebsd-smp@freebsd.org Date: Mon, 23 Jun 2003 16:13:16 +1000 User-Agent: KMail/1.5 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200306231613.16662.paul.koch@statscout.com> Subject: FreeBSD 4.8 SMP performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jun 2003 06:13:29 -0000 Last week we did a fairly major installation of our network monitoring software at a customers' site on a couple of identical quad processor machines running FreeBSD-4.8. One as a production machine, the other as a warm spare. The machines were unfortunately "hand me downs" from the weenies server group: - Compaq server of some description - quad 700Mhz Intel Xeon 2M cache CPUs - 4 gigabytes of ram - usual arrangement of Compaq/Adaptec SCSI and raid controllers - 1 * raid 1 disk setup - 1 * raid 5 disk setup - single and dual port Intel NICs We originally intended to benchmark a 4.8 and 5.1 box side by side to see what the SMP performance gains were, but ran out of time due to SMP kernel and dual port Intel NIC problem. When running 5.1 with SMP, the dual port NICs would lock up with "fxp0 device timeout" messages. Thinking this was a 5.x related problem, we fell back to 4.8 SMP, eventually finding that the same problem also existed in it. A quick change of NIC fixed our immediate issue. Time ran short so we unfortunately never got back to 5.1 SMP or to investigate the dual port NIC problems. Having never run a SMP box before, we were surprised when running 'top' on the 5.1 machine to see a process state of "Giant". Thought this was suppose to be fine grain SMP :) Our product is a real-time network performance monitoring application, only available on FreeBSD of course. Statscout NPM is typically deployed as a "blanket monitoring" solution. That is, monitor every port on every switch/router/repeater each minute (ie. ping/snmp) for things like bytes, frames, errors, discards and status. This particular customer has are rather large Cisco network, consisting of over 5000 switches/routers and ~100,000 network interfaces. Our software was configured to poll some devices at 60 second intervals and others at 120 seconds, collecting 10 SNMP OIDs per interface per poll. That's approximately one million SNMP OIDs to collect. The customer provided a list of device names/IP addresses and then it took just under one hour to SNMP walk all 5000 devices and populate our configuration. Once running and with a bit of tuning to our software, the machine was cruising along collecting, processing and saving over 500,000 SNMP OIDs per minute (8333/sec), within ~20,000 SNMP requests, and ~4000 pings per minute, using less than 50% of available CPU. We were very impressed with how smoothly and responsive the SMP machine ran doing lots of simultaneous disk/network activity, while also servicing user requested real-time reports. It was a pity that the network wasn't big enough to push the machine to its limits ! We noticed several things on 4.8: - Some of our busy processes jumped around 'a lot' between the different processors. It would be nice if processor affinity was implemented to get the best out of those 2M CPU caches. - "systat -vm" would chew up ~10 percent on one CPU when running a SMP kernel. Don't know what was going on there. - The dual port Intel fxp lockup mentioned above. Paul. -- Paul Koch (CTO Statscout Pty Ltd) http://www.statscout.com Brisbane, Australia From owner-freebsd-smp@FreeBSD.ORG Sun Jun 22 23:19:54 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4D9C37B410 for ; Sun, 22 Jun 2003 23:19:54 -0700 (PDT) Received: from obsecurity.dyndns.org (adsl-64-169-104-32.dsl.lsan03.pacbell.net [64.169.104.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id A846343FAF for ; Sun, 22 Jun 2003 23:19:53 -0700 (PDT) (envelope-from kris@obsecurity.org) Received: from rot13.obsecurity.org (rot13.obsecurity.org [10.0.0.5]) by obsecurity.dyndns.org (Postfix) with ESMTP id CC6D966BE5; Sun, 22 Jun 2003 23:19:51 -0700 (PDT) Received: by rot13.obsecurity.org (Postfix, from userid 1000) id 9C1EC963; Sun, 22 Jun 2003 23:19:51 -0700 (PDT) Date: Sun, 22 Jun 2003 23:19:51 -0700 From: Kris Kennaway To: Paul Koch Message-ID: <20030623061951.GA18487@rot13.obsecurity.org> References: <200306231613.16662.paul.koch@statscout.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="82I3+IH0IqGh5yIs" Content-Disposition: inline In-Reply-To: <200306231613.16662.paul.koch@statscout.com> User-Agent: Mutt/1.4.1i cc: freebsd-smp@freebsd.org Subject: Re: FreeBSD 4.8 SMP performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jun 2003 06:19:55 -0000 --82I3+IH0IqGh5yIs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 23, 2003 at 04:13:16PM +1000, Paul Koch wrote: > Having never run a SMP box before, we were surprised when=20 > running 'top' on the 5.1 machine to see a process state of "Giant". > Thought this was suppose to be fine grain SMP :) Read the 5.1 release notes and early adopter's guide. SMP development in 5.x is ongoing. Kris --82I3+IH0IqGh5yIs Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iD8DBQE+9pwHWry0BWjoQKURAiKLAJ0ZXuQVPUa21SOMJ9yQlCxsxsPSzwCfftnf G3BSa8DBqQDrZsV/dO5xDDc= =Gmea -----END PGP SIGNATURE----- --82I3+IH0IqGh5yIs-- From owner-freebsd-smp@FreeBSD.ORG Mon Jun 23 06:55:50 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 89DD637B401 for ; Mon, 23 Jun 2003 06:55:50 -0700 (PDT) Received: from ztxmail05.ztx.compaq.com (ztxmail05.ztx.compaq.com [161.114.1.209]) by mx1.FreeBSD.org (Postfix) with ESMTP id B44CA43FFB for ; Mon, 23 Jun 2003 06:55:49 -0700 (PDT) (envelope-from john.cagle@hp.com) Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id 3E950E928; Mon, 23 Jun 2003 08:55:49 -0500 (CDT) Received: from cceexc19.americas.cpqcorp.net ([16.110.250.85]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Mon, 23 Jun 2003 08:55:47 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Mon, 23 Jun 2003 08:55:44 -0500 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FreeBSD 4.8 SMP performance Thread-Index: AcM5TpgmIr4syD0RQ5OYAMKvw1KNPAAP8fCg From: "Cagle, John (ISS-Houston)" To: "Paul Koch" X-OriginalArrivalTime: 23 Jun 2003 13:55:47.0900 (UTC) FILETIME=[261C47C0:01C3398F] cc: freebsd-smp@freebsd.org Subject: RE: FreeBSD 4.8 SMP performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jun 2003 13:55:50 -0000 Paul, There's a known issue with APIC IRQ routing when using Multi-Port NICs in PCI slots. The existing code doesn't know how to swizzle the IRQs through the PCI-to-PCI bridges that are on the NIC. If you want to use Multi-port NICs, you'll need to make sure you're not running in APIC Full Table mode. Double check that your OS Selection (in ROM Based Setup) is "Linux". Of course, that may not be the problem you're experiencing, but it's my best guess. Regards, John -------------------------------- John Cagle john.cagle@hp.com Principal Member Technical Staff Industry Standard Servers Hewlett-Packard Company -----Original Message----- From: Paul Koch [mailto:paul.koch@statscout.com]=20 Sent: Monday, June 23, 2003 1:13 AM To: freebsd-smp@freebsd.org Subject: FreeBSD 4.8 SMP performance Last week we did a fairly major installation of our network=20 monitoring software at a customers' site on a couple of=20 identical quad processor machines running FreeBSD-4.8. One as a production machine, the other as a warm spare. The machines were unfortunately "hand me downs" from the=20 weenies server group: - Compaq server of some description - quad 700Mhz Intel Xeon 2M cache CPUs - 4 gigabytes of ram - usual arrangement of Compaq/Adaptec SCSI and raid=20 controllers - 1 * raid 1 disk setup - 1 * raid 5 disk setup - single and dual port Intel NICs We originally intended to benchmark a 4.8 and 5.1 box side by side to see what the SMP performance gains were, but ran out of time due to SMP kernel and dual port Intel NIC problem. When running 5.1 with SMP, the dual port NICs would lock up with "fxp0 device timeout" messages. Thinking this was a 5.x=20 related problem, we fell back to 4.8 SMP, eventually finding=20 that the same problem also existed in it. A quick change of NIC fixed our immediate issue. Time ran short so we unfortunately never got back to 5.1 SMP or to investigate the dual port NIC problems. Having never run a SMP box before, we were surprised when=20 running 'top' on the 5.1 machine to see a process state of "Giant". Thought this was suppose to be fine grain SMP :) Our product is a real-time network performance monitoring=20 application, only available on FreeBSD of course. Statscout=20 NPM is typically deployed as a "blanket monitoring" solution.=20 That is, monitor every port on every switch/router/repeater each minute (ie. ping/snmp) for things like bytes, frames, errors,=20 discards and status. This particular customer has are rather large Cisco network,=20 consisting of over 5000 switches/routers and ~100,000 network interfaces. Our software was configured to poll some devices at 60 second intervals and others at 120 seconds, collecting 10=20 SNMP OIDs per interface per poll. That's approximately one=20 million SNMP OIDs to collect. The customer provided a list of device names/IP addresses and then it took just under one hour to SNMP walk all 5000 devices and populate our configuration. Once running and with a bit of=20 tuning to our software, the machine was cruising along collecting,=20 processing and saving over 500,000 SNMP OIDs per minute=20 (8333/sec), within ~20,000 SNMP requests, and ~4000 pings=20 per minute, using less than 50% of available CPU. We were very impressed with how smoothly and responsive the SMP machine ran doing lots of simultaneous disk/network activity, while also servicing user requested real-time reports. It was a pity that the network wasn't big enough to push the=20 machine to its limits ! We noticed several things on 4.8: - Some of our busy processes jumped around 'a lot' between the different processors. It would be nice if processor affinity=20 was implemented to get the best out of those 2M CPU caches. - "systat -vm" would chew up ~10 percent on one CPU when running a SMP kernel. Don't know what was going on there. - The dual port Intel fxp lockup mentioned above. Paul. --=20 Paul Koch (CTO Statscout Pty Ltd) http://www.statscout.com Brisbane, Australia _______________________________________________ freebsd-smp@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" From owner-freebsd-smp@FreeBSD.ORG Mon Jun 23 07:17:03 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B6FA937B405 for ; Mon, 23 Jun 2003 07:17:03 -0700 (PDT) Received: from smtp-relay2.barrysworld.com (smtp-relay2.barrysworld.com [213.221.172.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D47444008 for ; Mon, 23 Jun 2003 07:16:58 -0700 (PDT) (envelope-from killing@barrysworld.com) Received: from [213.221.181.50] (helo=barrysworld.com) by smtp-relay2.barrysworld.com with esmtp (Exim 4.12) id 19US7e-00074s-00 for freebsd-smp@freebsd.org; Mon, 23 Jun 2003 15:16:42 +0100 Received: from vader [212.135.219.179] by barrysworld.com with ESMTP (SMTPD32-7.15) id AC94C20014E; Mon, 23 Jun 2003 15:20:04 +0100 Message-ID: <023b01c33992$0fa5abf0$b3db87d4@vader> From: "Steven Hartland" To: References: <20030623155627.5d0a0ad3.db@traceroute.dk> Date: Mon, 23 Jun 2003 15:16:38 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 Subject: Current package system issues X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Steven Hartland List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jun 2003 14:17:04 -0000 After playing around with the current linux_base-debian it begins to show some holes in the current packaging system. 1. Files installed by ports install and package install can vary. 2. There doesnt seem to be anyway to supply alternative dependecies, in this case linux_base-7.1.x and linux_base-debian are in the main interchangable but so far I've had to alter the port install of nearly all ports that we use that have a dependency on "linux_base". 3. Current bsd.port.mk is redhat specific ( have a patch for that if people want it ) 4. Information duplication / difference between INDEX and port Makefile can lead to issues. Similar to 1 but not quite the same. Are there any package improvements in the works which would cover these issues? Steve From owner-freebsd-smp@FreeBSD.ORG Mon Jun 23 17:46:48 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6A99437B401 for ; Mon, 23 Jun 2003 17:46:48 -0700 (PDT) Received: from wally.statscout.com (wally.statscout.com [203.39.101.146]) by mx1.FreeBSD.org (Postfix) with ESMTP id B9E8D43F75 for ; Mon, 23 Jun 2003 17:46:45 -0700 (PDT) (envelope-from pak@statscout.com) Received: from speedy.statscout.com (speedy.statscout.com [10.1.1.100]) by wally.statscout.com (8.11.6/8.11.3av) with ESMTP id h5O0kb167989; Tue, 24 Jun 2003 10:46:37 +1000 (EST) (envelope-from pak@statscout.com) Received: from speedy.statscout.com (localhost.statscout.com [127.0.0.1]) h5O0kcok000664; Tue, 24 Jun 2003 10:46:38 +1000 (EST) (envelope-from pak@speedy.statscout.com) Received: from localhost (localhost [[UNIX: localhost]]) by speedy.statscout.com (8.12.8p1/8.12.8/Submit) id h5O0kbYe000663; Tue, 24 Jun 2003 10:46:37 +1000 (EST) From: Paul Koch Organization: Statscout To: "Cagle, John (ISS-Houston)" Date: Tue, 24 Jun 2003 10:46:37 +1000 User-Agent: KMail/1.5 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200306241046.37229.paul.koch@statscout.com> cc: freebsd-smp@freebsd.org Subject: Re: FreeBSD 4.8 SMP performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2003 00:46:48 -0000 Ah! thanks, I did a bit of reading on this and it sounds like we need to keep clear of these types of cards for the moment. We didn't plan on using the multiport NICs, it's just what came in the box. Is this only an issue when running SMP ? because the NICs appeared to work fine when initially running a non-SMP kernel. OS selection of "Linux": hmm, didn't know about that. More reading to be done here. I assume this does is some vendor specific trickery setting the hardware up. Paul. On Mon, 23 Jun 2003 11:55 pm, Cagle, John (ISS-Houston) wrote: > Paul, > > There's a known issue with APIC IRQ routing when using Multi-Port > NICs in PCI slots. The existing code doesn't know how to swizzle the > IRQs through the PCI-to-PCI bridges that are on the NIC. If you want > to use Multi-port NICs, you'll need to make sure you're not running > in APIC Full Table mode. Double check that your OS Selection (in ROM > Based Setup) is "Linux". > > Of course, that may not be the problem you're experiencing, but it's > my best guess. > > Regards, > John > -------------------------------- > John Cagle john.cagle@hp.com > Principal Member Technical Staff > Industry Standard Servers > Hewlett-Packard Company > > -----Original Message----- > From: Paul Koch [mailto:paul.koch@statscout.com] > Sent: Monday, June 23, 2003 1:13 AM > To: freebsd-smp@freebsd.org > Subject: FreeBSD 4.8 SMP performance > > > Last week we did a fairly major installation of our network > monitoring software at a customers' site on a couple of > identical quad processor machines running FreeBSD-4.8. > One as a production machine, the other as a warm spare. > > The machines were unfortunately "hand me downs" from the > weenies server group: > - Compaq server of some description > - quad 700Mhz Intel Xeon 2M cache CPUs > - 4 gigabytes of ram > - usual arrangement of Compaq/Adaptec SCSI and raid > controllers > - 1 * raid 1 disk setup > - 1 * raid 5 disk setup > - single and dual port Intel NICs > > We originally intended to benchmark a 4.8 and 5.1 box side > by side to see what the SMP performance gains were, but ran > out of time due to SMP kernel and dual port Intel NIC problem. When > running 5.1 with SMP, the dual port NICs would lock up with "fxp0 > device timeout" messages. Thinking this was a 5.x > related problem, we fell back to 4.8 SMP, eventually finding > that the same problem also existed in it. A quick change of NIC > fixed our immediate issue. Time ran short so we unfortunately never > got back to 5.1 SMP or to investigate the dual port NIC problems. > > Having never run a SMP box before, we were surprised when > running 'top' on the 5.1 machine to see a process state of "Giant". > Thought this was suppose to be fine grain SMP :) > > Our product is a real-time network performance monitoring > application, only available on FreeBSD of course. Statscout > NPM is typically deployed as a "blanket monitoring" solution. > That is, monitor every port on every switch/router/repeater each > minute (ie. ping/snmp) for things like bytes, frames, errors, > discards and status. > > This particular customer has are rather large Cisco network, > consisting of over 5000 switches/routers and ~100,000 network > interfaces. Our software was configured to poll some devices at 60 > second intervals and others at 120 seconds, collecting 10 > SNMP OIDs per interface per poll. That's approximately one > million SNMP OIDs to collect. > > The customer provided a list of device names/IP addresses and then it > took just under one hour to SNMP walk all 5000 devices and populate > our configuration. Once running and with a bit of > tuning to our software, the machine was cruising along collecting, > processing and saving over 500,000 SNMP OIDs per minute > (8333/sec), within ~20,000 SNMP requests, and ~4000 pings > per minute, using less than 50% of available CPU. We were very > impressed with how smoothly and responsive the SMP machine ran doing > lots of simultaneous disk/network activity, while also servicing user > requested real-time reports. > > It was a pity that the network wasn't big enough to push the > machine to its limits ! > > > We noticed several things on 4.8: > > - Some of our busy processes jumped around 'a lot' between > the different processors. It would be nice if processor affinity > was implemented to get the best out of those 2M CPU caches. > > - "systat -vm" would chew up ~10 percent on one CPU when > running a SMP kernel. Don't know what was going on there. > > - The dual port Intel fxp lockup mentioned above. > > Paul. From owner-freebsd-smp@FreeBSD.ORG Mon Jun 23 19:02:59 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 999AD37B401 for ; Mon, 23 Jun 2003 19:02:59 -0700 (PDT) Received: from ztxmail03.ztx.compaq.com (ztxmail03.ztx.compaq.com [161.114.1.207]) by mx1.FreeBSD.org (Postfix) with ESMTP id B2D3A43FA3 for ; Mon, 23 Jun 2003 19:02:58 -0700 (PDT) (envelope-from john.cagle@hp.com) Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id 4795FA02B; Mon, 23 Jun 2003 21:02:58 -0500 (CDT) Received: from cceexc19.americas.cpqcorp.net ([16.110.250.85]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Mon, 23 Jun 2003 21:02:57 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Mon, 23 Jun 2003 21:02:57 -0500 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FreeBSD 4.8 SMP performance Thread-Index: AcM56hfoNHGJ+on1ShOgO8vIx1ep7gABjE4A From: "Cagle, John (ISS-Houston)" To: "Paul Koch" X-OriginalArrivalTime: 24 Jun 2003 02:02:57.0721 (UTC) FILETIME=[BB82A290:01C339F4] cc: freebsd-smp@freebsd.org Subject: RE: FreeBSD 4.8 SMP performance X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2003 02:02:59 -0000 Actually, there's no reason to avoid using the multi-port NICs, as long as you don't use full table APIC mode (which doesn't work yet in FreeBSD). I know that the kernel maintainers are working on this, so it should be fixed RSN. Also, there's no real trickery involved in the OS selection menu. It simply tries to set the BIOS defaults appropriate for the Operating System you tell it you want to run. For instance, if you select "Linux" it will use APIC "Full Table Mapped" mode, which should work for FreeBSD. The reason it worked when you weren't running SMP is that it probably wasn't using the APICs. -----Original Message----- From: Paul Koch [mailto:paul.koch@statscout.com]=20 Sent: Monday, June 23, 2003 7:47 PM To: Cagle, John (ISS-Houston) Cc: freebsd-smp@freebsd.org Subject: Re: FreeBSD 4.8 SMP performance Ah! thanks, I did a bit of reading on this and it sounds like we need to keep clear of these types of cards for the moment. We didn't plan on using the multiport NICs, it's just what came in the box. Is this only an issue when running SMP ? because the NICs appeared to work fine when initially running a=20 non-SMP kernel. OS selection of "Linux": hmm, didn't know about that. More reading to be done here. I assume this does is some vendor specific trickery setting the hardware up. Paul. On Mon, 23 Jun 2003 11:55 pm, Cagle, John (ISS-Houston) wrote: > Paul, > > There's a known issue with APIC IRQ routing when using Multi-Port NICs > in PCI slots. The existing code doesn't know how to swizzle the IRQs=20 > through the PCI-to-PCI bridges that are on the NIC. If you want to=20 > use Multi-port NICs, you'll need to make sure you're not running in=20 > APIC Full Table mode. Double check that your OS Selection (in ROM=20 > Based Setup) is "Linux". > > Of course, that may not be the problem you're experiencing, but it's=20 > my best guess. > > Regards, > John > -------------------------------- > John Cagle john.cagle@hp.com > Principal Member Technical Staff > Industry Standard Servers > Hewlett-Packard Company > > -----Original Message----- > From: Paul Koch [mailto:paul.koch@statscout.com] > Sent: Monday, June 23, 2003 1:13 AM > To: freebsd-smp@freebsd.org > Subject: FreeBSD 4.8 SMP performance > > > Last week we did a fairly major installation of our network monitoring > software at a customers' site on a couple of identical quad processor=20 > machines running FreeBSD-4.8. One as a production machine, the other=20 > as a warm spare. > > The machines were unfortunately "hand me downs" from the weenies=20 > server group: > - Compaq server of some description > - quad 700Mhz Intel Xeon 2M cache CPUs > - 4 gigabytes of ram > - usual arrangement of Compaq/Adaptec SCSI and raid > controllers > - 1 * raid 1 disk setup > - 1 * raid 5 disk setup > - single and dual port Intel NICs > > We originally intended to benchmark a 4.8 and 5.1 box side > by side to see what the SMP performance gains were, but ran out of=20 > time due to SMP kernel and dual port Intel NIC problem. When running=20 > 5.1 with SMP, the dual port NICs would lock up with "fxp0 device=20 > timeout" messages. Thinking this was a 5.x related problem, we fell=20 > back to 4.8 SMP, eventually finding that the same problem also existed > in it. A quick change of NIC fixed our immediate issue. Time ran=20 > short so we unfortunately never got back to 5.1 SMP or to investigate=20 > the dual port NIC problems. > > Having never run a SMP box before, we were surprised when running=20 > 'top' on the 5.1 machine to see a process state of "Giant". Thought=20 > this was suppose to be fine grain SMP :) > > Our product is a real-time network performance monitoring application, > only available on FreeBSD of course. Statscout NPM is typically=20 > deployed as a "blanket monitoring" solution. That is, monitor every=20 > port on every switch/router/repeater each minute (ie. ping/snmp) for=20 > things like bytes, frames, errors, discards and status. > > This particular customer has are rather large Cisco network,=20 > consisting of over 5000 switches/routers and ~100,000 network=20 > interfaces. Our software was configured to poll some devices at 60=20 > second intervals and others at 120 seconds, collecting 10 SNMP OIDs=20 > per interface per poll. That's approximately one million SNMP OIDs to=20 > collect. > > The customer provided a list of device names/IP addresses and then it=20 > took just under one hour to SNMP walk all 5000 devices and populate=20 > our configuration. Once running and with a bit of tuning to our=20 > software, the machine was cruising along collecting, processing and=20 > saving over 500,000 SNMP OIDs per minute (8333/sec), within ~20,000=20 > SNMP requests, and ~4000 pings per minute, using less than 50% of=20 > available CPU. We were very impressed with how smoothly and responsive > the SMP machine ran doing lots of simultaneous disk/network activity,=20 > while also servicing user requested real-time reports. > > It was a pity that the network wasn't big enough to push the machine=20 > to its limits ! > > > We noticed several things on 4.8: > > - Some of our busy processes jumped around 'a lot' between > the different processors. It would be nice if processor affinity > was implemented to get the best out of those 2M CPU caches. > > - "systat -vm" would chew up ~10 percent on one CPU when > running a SMP kernel. Don't know what was going on there. > > - The dual port Intel fxp lockup mentioned above. > > Paul. From owner-freebsd-smp@FreeBSD.ORG Tue Jun 24 10:56:44 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 320CA37B401 for ; Tue, 24 Jun 2003 10:56:44 -0700 (PDT) Received: from chiller.chillinthemost.com (dsl093-024-089.hou1.dsl.speakeasy.net [66.93.24.89]) by mx1.FreeBSD.org (Postfix) with SMTP id A097E43FAF for ; Tue, 24 Jun 2003 10:56:43 -0700 (PDT) (envelope-from stefan@chillinthemost.com) Received: (qmail 72742 invoked by uid 80); 24 Jun 2003 17:55:14 -0000 Date: 24 Jun 2003 17:55:14 -0000 Message-ID: <20030624175514.72741.qmail@chiller.chillinthemost.com> From: "Stefan" To: freebsd-smp@freebsd.org X-Mailer: NeoMail 1.25 X-IPAddress: 158.81.251.201 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Subject: FreeBSD 5.1 SMP on Dell 2650 X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: stefan@chillinthemost.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2003 17:56:44 -0000 I have SMP running on a Dell PowerEdge 2650 with dual Intel Xeon 2800mhz. FreeBSD 5.1 detected two physical cpu's and two logical cpu's. So far I havn't noticed any problems. If anyone wants a copy of the dmesg.boot file let me know. Also, I'm not sure if I have the correct URL, I was going to send the info to Erich's list but I'm getting a 404. http://www.uruk.org/~erich/mps-hw.html Thanks, Stefan From owner-freebsd-smp@FreeBSD.ORG Tue Jun 24 18:23:47 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5DD6D37B401 for ; Tue, 24 Jun 2003 18:23:47 -0700 (PDT) Received: from hermes.fm.intel.com (fmr01.intel.com [192.55.52.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8672243FE5 for ; Tue, 24 Jun 2003 18:23:46 -0700 (PDT) (envelope-from h.castillo@intel.com) Received: from talaria.fm.intel.com (talaria.fm.intel.com [10.1.192.39]) 21:17:36 rfjohns1 Exp $) with ESMTP id h5P1J0422659 for ; Wed, 25 Jun 2003 01:19:00 GMT Received: from fmsmsxvs041.fm.intel.com (fmsmsxvs041.fm.intel.com [132.233.42.126]) 2003/05/22 21:18:01 rfjohns1 Exp $) with SMTP id h5P1OEl16222 for ; Wed, 25 Jun 2003 01:24:14 GMT Received: from fmsmsx331.amr.corp.intel.com ([132.233.42.135]) M2003062418201530250 for ; Tue, 24 Jun 2003 18:20:15 -0700 Received: from fmsmsx403.amr.corp.intel.com ([132.233.42.207]) by fmsmsx331.amr.corp.intel.com with Microsoft SMTPSVC(5.0.2195.5329); Tue, 24 Jun 2003 18:23:45 -0700 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Date: Tue, 24 Jun 2003 18:23:45 -0700 Message-ID: <331AD7BED1579543AD146F5A1A44D5250F111B@fmsmsx403.fm.intel.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Hyper threading support? Thread-Index: AcM6uGshN305IO1hRYGRgPP9SgYXGA== From: "Castillo Hernandez, Hector" To: X-OriginalArrivalTime: 25 Jun 2003 01:23:45.0611 (UTC) FILETIME=[6BF4E1B0:01C33AB8] Subject: Hyper threading support? X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jun 2003 01:23:47 -0000 Hello, do you know if the latest versions of FreeBSD(4.7), NetBSD and = OpenBSD(3.2) support intel's hyper threading technology? I mean that this OSs take advantage of logical processors thanks! Hector Castillo Intel Guadalajara, Mexico From owner-freebsd-smp@FreeBSD.ORG Tue Jun 24 18:29:27 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6BE0C37B401 for ; Tue, 24 Jun 2003 18:29:27 -0700 (PDT) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id DD0F443F3F for ; Tue, 24 Jun 2003 18:29:26 -0700 (PDT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (IDENT:brdavis@localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.12.9/8.12.3) with ESMTP id h5P1TNvN030703; Tue, 24 Jun 2003 18:29:23 -0700 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.12.9/8.12.3/Submit) id h5P1TNVo030702; Tue, 24 Jun 2003 18:29:23 -0700 Date: Tue, 24 Jun 2003 18:29:23 -0700 From: Brooks Davis To: "Castillo Hernandez, Hector" Message-ID: <20030625012923.GA28749@Odin.AC.HMC.Edu> References: <331AD7BED1579543AD146F5A1A44D5250F111B@fmsmsx403.fm.intel.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jRHKVT23PllUwdXP" Content-Disposition: inline In-Reply-To: <331AD7BED1579543AD146F5A1A44D5250F111B@fmsmsx403.fm.intel.com> User-Agent: Mutt/1.5.4i X-Virus-Scanned: by amavisd-milter (http://amavis.org/) on odin.ac.hmc.edu cc: freebsd-smp@freebsd.org Subject: Re: Hyper threading support? X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jun 2003 01:29:27 -0000 --jRHKVT23PllUwdXP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 24, 2003 at 06:23:45PM -0700, Castillo Hernandez, Hector wrote: > Hello, > do you know if the latest versions of FreeBSD(4.7), NetBSD and > OpenBSD(3.2) support intel's hyper threading technology? > I mean that this OSs take advantage of logical processors FreeBSD 4.8 and 5.1 can use logical CPUs as they there were physical CPUs. The schedular has not been optimized for logical CPUs at this point though I've seem posts about building the necessicary framework. I don't know about NetBSD or OpenBSD support. -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --jRHKVT23PllUwdXP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQE++PryXY6L6fI4GtQRAqRXAJ9icNPFctWt3xzOoSS1KmADD9U+FACgozei t9iBnPD0mp5dUlraJqIf/vw= =Nhtg -----END PGP SIGNATURE----- --jRHKVT23PllUwdXP-- From owner-freebsd-smp@FreeBSD.ORG Wed Jun 25 15:26:14 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76F6937B40B for ; Wed, 25 Jun 2003 15:26:14 -0700 (PDT) Received: from quartzo.cirp.usp.br (quartzo.cirp.usp.br [143.107.200.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id CCA2E43FDF for ; Wed, 25 Jun 2003 15:26:12 -0700 (PDT) (envelope-from aftaha@cirp.usp.br) Received: from cirp.usp.br (arenito.cirp.usp.br [143.107.200.101]) by quartzo.cirp.usp.br (8.11.6/8.11.6) with ESMTP id h5PMcUq22923 for ; Wed, 25 Jun 2003 19:38:31 -0300 Sender: aftaha@quartzo.cirp.usp.br Message-ID: <3EFA234E.7F82C7CD@cirp.usp.br> Date: Wed, 25 Jun 2003 19:33:50 -0300 From: Ali Faiez Taha X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.12 i386) X-Accept-Language: en, pt-BR, pt, es-ES, fr, ar, ar, de-DE, ru, el MIME-Version: 1.0 To: freebsd-smp@freebsd.org Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Intel Fortran on FreeBSD X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: aftaha@cirp.usp.br List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jun 2003 22:26:14 -0000 Hi... I'm trying to install the latest version of Intel Fortran in one machine with Dual Xeon 2.4 GHz (Motherboard : Intel SE7500CW2). The O.S. is FreeBSD 4.8-STABLE. According the ports (/usr/ports/lang/ifc) I need to download "l_fc_pu_7.1.016.tar" , but at moment that's impossible. I can't find this file. Where can I found this file for download ? On the Intel sites (described in pkg-desc and in the results of make install): impossible !!! The maintanner don't answer my e-mail ... At now I'm using the latest version of IFC for Linux (l_fc_p_7.1.008.tar). thanks... -- ------------------------------------------------ MSc. Eng. Ali Faiez Taha / Analista de Sistemas CIRP - USP - Ribeirão Preto - SP Fone:(16) 602-3622 Use Software Livre ! É livre ! É legal ! http://www.linorg.cirp.usp.br ------------------------------------------------ From owner-freebsd-smp@FreeBSD.ORG Thu Jun 26 01:00:28 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B567737B401 for ; Thu, 26 Jun 2003 01:00:28 -0700 (PDT) Received: from gidgate.gid.co.uk (gid.co.uk [194.32.164.225]) by mx1.FreeBSD.org (Postfix) with ESMTP id 59A2443FF3 for ; Thu, 26 Jun 2003 01:00:27 -0700 (PDT) (envelope-from rb@gid.co.uk) Received: (from rb@localhost) by gidgate.gid.co.uk (8.11.6/8.11.6) id h5Q80PK57577; Thu, 26 Jun 2003 09:00:25 +0100 (BST) (envelope-from rb) Message-Id: <4.3.2.7.2.20030626085756.02b90298@gid.co.uk> X-Sender: rbmail@gid.co.uk X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Thu, 26 Jun 2003 09:00:23 +0100 To: aftaha@cirp.usp.br, freebsd-smp@freebsd.org From: Bob Bishop In-Reply-To: <3EFA234E.7F82C7CD@cirp.usp.br> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Subject: Re: Intel Fortran on FreeBSD X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jun 2003 08:00:29 -0000 Hi, Here's the answer I got to the same question: >Date: Mon, 02 Jun 2003 18:28:15 +0900 (JST) >Message-Id: <20030602.182815.607958600.chat95@mbox.kyoto-inet.or.jp> >To: rb@gid.co.uk >Cc: maho@FreeBSD.org >Subject: Re: Intel Fortran port >From: Nakata Maho > >Hi, Bob > > > I see that the ifc port is up to version 7.1.016, but the latest > > non-commercial version on the Intel download site [etc] > >No. visit https://premier.intel.com/ (registration is required) At 23:33 25/6/03, Ali Faiez Taha wrote: >Hi... > >I'm trying to install the latest version of Intel Fortran in one machine >with >Dual Xeon 2.4 GHz (Motherboard : Intel SE7500CW2). The O.S. is FreeBSD >4.8-STABLE. > >According the ports (/usr/ports/lang/ifc) I need to download >"l_fc_pu_7.1.016.tar" , but at moment that's impossible. > >I can't find this file. Where can I found this file for download ? >On the Intel sites (described in pkg-desc and in the results of make >install): impossible !!! > >The maintanner don't answer my e-mail ... > >At now I'm using the latest version of IFC for Linux >(l_fc_p_7.1.008.tar). > >thanks... > >-- > ------------------------------------------------ > MSc. Eng. Ali Faiez Taha / Analista de Sistemas > CIRP - USP - Ribeir=E3o Preto - SP > Fone:(16) 602-3622 > Use Software Livre ! =C9 livre ! =C9 legal ! > http://www.linorg.cirp.usp.br > ------------------------------------------------ >_______________________________________________ >freebsd-smp@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-smp >To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" -- Bob Bishop +44 (0)118 977 4017 rb@gid.co.uk fax +44 (0)118 989 4254