From owner-freebsd-performance@FreeBSD.ORG Tue Aug 17 20:13:17 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3D1016A4CE; Tue, 17 Aug 2004 20:13:17 +0000 (GMT) Received: from gnosis.realityengine.ca (gnosis.realityengine.ca [69.55.224.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8789743D2F; Tue, 17 Aug 2004 20:13:17 +0000 (GMT) (envelope-from erik.rothwell@realityengine.ca) Received: from [192.168.5.3] (unknown [216.58.89.253]) by gnosis.realityengine.ca (Postfix) with ESMTP id 7D8EB9EF9; Tue, 17 Aug 2004 20:13:15 +0000 (UTC) User-Agent: Microsoft-Entourage/11.0.0.040405 Date: Tue, 17 Aug 2004 16:13:10 -0400 From: Erik Rothwell To: Robert Watson Message-ID: In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Network performance issues when writing to disk (5.2.1-RELEASE) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Aug 2004 20:13:17 -0000 On 8/12/04 11:17 AM, "Robert Watson" wrote: > Did you replace it with another if_dc card, or with a card using a > different interface driver? Well, I took the afflicted box out of production this weekend for some further testing. After swapping out one of the if_dc NICs with an Intel card using the if_fxp driver, the problem no longer occurs on that interface. I can only reproduce the problem using an interface with the if_dc driver. The difference in performance is night and day. > - What sort of kernel configuration are you using -- GENERIC, or a custom > kernel, and if so, is it an SMP box and is SMP enabled in the kernel? This is a single processor box. The kernel is compiled without SMP or APIC and without a number of unused drivers (SCSI, ISA NICs, PCMCIA, umass, &c). IPFIREWALL, IPDIVERT, QUOTA and DEVICE_POLLING are compiled in. (Device polling is not currently enabled via the sysctl however). > - When the system performance suddenly degrades, what is the apparent load > and condition of the system? In particular, at that point if you > measure the load average and CPU utilization, perhaps with "systat > -vmstat" or "top", are the CPU(s) maxed out? How much time is in user > vs. system vs. interrupt (vs. idle). if_dc: nc > /dev/null Load 0.79 0.25 0.13 CPU 41.5% sys, 28.5% interrupt, 5.5% user, 24.5% idle Interrupts 2501 clock, 128 rtc, 11684 dc0, 83 ata Disk ad0: 12.33 KB/t 4 tps 0.04 MB/s Disk ad1: 7.80 KB/t 80 tps 0.61 MB/s 8% busy if_dc: nc > /data/junk Load 0.11 0.26 0.17 CPU 9.0% sys, 2.0% interrupt, 0.5% user, 88.5% idle Interrupts 2504 clock, 128 rtc, 434 dc0, 54 ata Disk ad0: - Disk ad1: 11.75 KB/t 51 tps 0.58 MB/s 100% busy if_fxp: nc > /data/junk Load 0.42 0.38 0.23 CPU 61.0% sys, 31.2% interrupt, 2.5% user, 4.5% idle Interrupts 2501 clock, 128 rtc, 6036 fxp0, 145 ata Disk ad0: - Disk ad1: 128 KB/t 73 tps 9.07 MB/s 37% busy Plenty of free memory, little if any swapping at any time. Both disks are running in udma66. devinfo/dmesg doesn't indicate any conflict between disk controller and the network interfaces. There certainly appears to be some contention between the if_dc cards and the disk, however. Erik.