Date: Mon, 23 Jun 2003 08:55:44 -0500 From: "Cagle, John (ISS-Houston)" <john.cagle@hp.com> To: "Paul Koch" <paul.koch@statscout.com> Cc: freebsd-smp@freebsd.org Subject: RE: FreeBSD 4.8 SMP performance Message-ID: <C50AB9511EE59B49B2A503CB7AE1ABD10476C694@cceexc19.americas.cpqcorp.net>
next in thread | raw e-mail | index | archive | help
Paul,
There's a known issue with APIC IRQ routing when using Multi-Port NICs
in PCI slots. The existing code doesn't know how to swizzle the IRQs
through the PCI-to-PCI bridges that are on the NIC. If you want to use
Multi-port NICs, you'll need to make sure you're not running in APIC
Full Table mode. Double check that your OS Selection (in ROM Based
Setup) is "Linux".
Of course, that may not be the problem you're experiencing, but it's my
best guess.
Regards,
John
--------------------------------
John Cagle john.cagle@hp.com
Principal Member Technical Staff
Industry Standard Servers
Hewlett-Packard Company
-----Original Message-----
From: Paul Koch [mailto:paul.koch@statscout.com]=20
Sent: Monday, June 23, 2003 1:13 AM
To: freebsd-smp@freebsd.org
Subject: FreeBSD 4.8 SMP performance
Last week we did a fairly major installation of our network=20
monitoring software at a customers' site on a couple of=20
identical quad processor machines running FreeBSD-4.8.
One as a production machine, the other as a warm spare.
The machines were unfortunately "hand me downs" from the=20
weenies server group:
- Compaq server of some description
- quad 700Mhz Intel Xeon 2M cache CPUs
- 4 gigabytes of ram
- usual arrangement of Compaq/Adaptec SCSI and raid=20
controllers
- 1 * raid 1 disk setup
- 1 * raid 5 disk setup
- single and dual port Intel NICs
We originally intended to benchmark a 4.8 and 5.1 box side
by side to see what the SMP performance gains were, but ran
out of time due to SMP kernel and dual port Intel NIC problem. When
running 5.1 with SMP, the dual port NICs would lock up with "fxp0 device
timeout" messages. Thinking this was a 5.x=20
related problem, we fell back to 4.8 SMP, eventually finding=20
that the same problem also existed in it. A quick change of NIC fixed
our immediate issue. Time ran short so we unfortunately never got back
to 5.1 SMP or to investigate the dual port NIC problems.
Having never run a SMP box before, we were surprised when=20
running 'top' on the 5.1 machine to see a process state of "Giant".
Thought this was suppose to be fine grain SMP :)
Our product is a real-time network performance monitoring=20
application, only available on FreeBSD of course. Statscout=20
NPM is typically deployed as a "blanket monitoring" solution.=20
That is, monitor every port on every switch/router/repeater each minute
(ie. ping/snmp) for things like bytes, frames, errors,=20
discards and status.
This particular customer has are rather large Cisco network,=20
consisting of over 5000 switches/routers and ~100,000 network
interfaces. Our software was configured to poll some devices at 60
second intervals and others at 120 seconds, collecting 10=20
SNMP OIDs per interface per poll. That's approximately one=20
million SNMP OIDs to collect.
The customer provided a list of device names/IP addresses and then it
took just under one hour to SNMP walk all 5000 devices and populate our
configuration. Once running and with a bit of=20
tuning to our software, the machine was cruising along collecting,=20
processing and saving over 500,000 SNMP OIDs per minute=20
(8333/sec), within ~20,000 SNMP requests, and ~4000 pings=20
per minute, using less than 50% of available CPU. We were very impressed
with how smoothly and responsive the SMP machine ran doing lots of
simultaneous disk/network activity, while also servicing user requested
real-time reports.
It was a pity that the network wasn't big enough to push the=20
machine to its limits !
We noticed several things on 4.8:
- Some of our busy processes jumped around 'a lot' between
the different processors. It would be nice if processor affinity=20
was implemented to get the best out of those 2M CPU caches.
- "systat -vm" would chew up ~10 percent on one CPU when
running a SMP kernel. Don't know what was going on there.
- The dual port Intel fxp lockup mentioned above.
Paul.
--=20
Paul Koch (CTO Statscout Pty Ltd)
http://www.statscout.com
Brisbane, Australia _______________________________________________
freebsd-smp@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-smp
To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C50AB9511EE59B49B2A503CB7AE1ABD10476C694>