From owner-freebsd-questions@FreeBSD.ORG Thu Jul 28 03:26:09 2005 Return-Path: X-Original-To: questions@freebsd.org Delivered-To: freebsd-questions@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 274DD16A41F for ; Thu, 28 Jul 2005 03:26:09 +0000 (GMT) (envelope-from dpk@dpk.net) Received: from shared10.hosting.flyingcroc.net (shared10.hosting.flyingcroc.net [207.246.149.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id D4D2443D48 for ; Thu, 28 Jul 2005 03:26:08 +0000 (GMT) (envelope-from dpk@dpk.net) Received: from shared10.hosting.flyingcroc.net (localhost [127.0.0.1]) by shared10.hosting.flyingcroc.net (8.12.9p2/8.12.10) with ESMTP id j6S3Q8rR039794 for ; Wed, 27 Jul 2005 20:26:08 -0700 (PDT) Received: from localhost (dpk@localhost) by shared10.hosting.flyingcroc.net (8.12.9p2/8.12.10/Submit) with ESMTP id j6S3Q8gl039791 for ; Wed, 27 Jul 2005 20:26:08 -0700 (PDT) X-Authentication-Warning: shared10.hosting.flyingcroc.net: dpk owned process doing -bs Date: Wed, 27 Jul 2005 20:26:07 -0700 (PDT) From: dpk X-X-Sender: dpk@shared10.hosting.flyingcroc.net To: questions@freebsd.org Message-ID: <20050727185853.D23753@shared10.hosting.flyingcroc.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: Subject: FreeBSD 5.4+SMP, severe network degredation X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jul 2005 03:26:09 -0000 We just received several SuperMicro servers, 3.0Ghz Xeon x 2, 4GB RAM. They're using the em driver and the ports are set to 1000Mbit (we also tried 100Mbit/full duplex on the card and on the switch). They're running FreeBSD 5.4. I ran a steady ping on a couple of them while they were running "GENERIC", and then rebooted them with a kernel built with the "PAE" kernel included with the installation, with "option SMP" added. The PAE-SMP-GENERIC kernel was built after cvsup'ing with "tag=RELENG_5_4" and the uname reports "5.4-RELEASE-p5". Here are the ping results: GENERIC: 117 packets transmitted, 117 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.451/0.554/0.856/0.059 ms PAE-SMP-GENERIC: 102 packets transmitted, 102 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.569/4.262/7.944/2.065 ms Fetching a 637MB ISO from a local server, also on 100/FDX: GENERIC: /dev/null 100% of 637 MB 10 MBps 00m00s real 0m58.071s user 0m1.954s sys 0m6.278s PAE-SMP-GENERIC: /dev/null 100% of 637 MB 5764 kBps 00m00s real 1m53.324s user 0m1.478s sys 0m5.624s Running GENERIC, systat shows about 7000 interrupts/second, and around 600 interrupts/second using PAE-SMP-GENERIC, while fetch was running. I've checked the errata and hardware notes, as well as gnats, and was not able to find anything that explains or matches this behavior. We've run SMP servers for years, using 4.5-4.11, but we've never seen the network performance cut in half (or pings go up 10x). Removing "option SMP" makes the problem go away, but at a very significant performance cost obviously. Could it be something from -p5? Is this explained/examined in a PR I've missed, and if so can I add some information?