From owner-freebsd-performance@FreeBSD.ORG Wed Aug 11 00:10:25 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1524E16A4CE for ; Wed, 11 Aug 2004 00:10:25 +0000 (GMT) Received: from gnosis.realityengine.ca (gnosis.realityengine.ca [69.55.224.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0075843D3F for ; Wed, 11 Aug 2004 00:10:24 +0000 (GMT) (envelope-from erik.rothwell@realityengine.ca) Received: from [192.168.5.3] (unknown [216.58.89.253]) by gnosis.realityengine.ca (Postfix) with ESMTP id 17D7C7793 for ; Wed, 11 Aug 2004 00:10:24 +0000 (UTC) User-Agent: Microsoft-Entourage/11.0.0.040405 Date: Tue, 10 Aug 2004 20:10:22 -0400 From: Erik Rothwell To: Message-ID: Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Subject: Network performance issues when writing to disk (5.2.1-RELEASE) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2004 00:10:25 -0000 I have a workgroup server running FreeBSD 5.2.1 that is experiencing some serious network performance problems over very minimal load. The server has two miibus-based NICs: a WAN link via dc1 and a switched LAN serving ~7 clients attached via dc0. File transfers to the server (eg, scp, WebDAV, NFS, FTP) experience terrible throughput, starting off at about 2MB/s and dropping rapidly to ~200KB/s. At first, I suspected duplex mismatch or a misbehaving switch (unmanaged switch with all client interfaces set to autosense), but the problem can be replicated with the server connected directly to any of the clients with the media at both ends explicitly set. netstat -ni indicates no collisions but a few I/Oerrs during transfers. It appears from testing that the interface itself responds intermittently under this load (during transfers, the switch reports the port is intermittently inactive -- ifconfig on the server will also sometimes report "status: inactive") and packet loss approaches 75%. Occasionally, "dc0: TX underrun -- increasing TX threshold" would crop up in dmesg, but that's not usually noteworthy. I replaced the NIC, yet the problem curiously persists. After further testing, it seems the problem is isolated only to network file transfers that actually write to disk. For instance, using netcat to pipe random junk to /dev/null at the server, throughput is good and the NIC behaves itself: # nc -l -p 2332 -vv | dd if=/dev/stdin of=/dev/null bs=1k listening on [any] 2332 ... connect to [192.168.5.1] from [192.168.5.3] 51295 sent 0, rcvd 104857600 94731+21879 records in 94731+21879 records out 104857600 bytes transferred in 11.897846 secs (8813158 bytes/sec) When writing to a file, however: # nc -l -p 2332 -vv | dd if=/dev/stdin of=/data/junk bs=1k listening on [any] 2332 ... connect to [192.168.5.1] from [192.168.5.3] 51296 sent 0, rcvd 1130496 1046+115 records in 1046+115 records out 1130496 bytes transferred in 32.888345 secs (34374 bytes/sec) Conversely, using netcat to pipe a 1GB file on the server to one of the clients was no problem: # nc -l -p 2332 -vv > /tmp/junk ... 698018816 bytes transferred in 95.423028 secs (7314993 bytes/sec) Locally, disk performance isn't suffering unusually: # dd if=/data/junk of=/dev/null 2246100+0 records in 2246100+0 records out 1150003200 bytes transferred in 97.113807 secs (11841809 bytes/sec) # dd if=/dev/zero of=/data/junk bs=1k count=1m 1048576+0 records in 1048576+0 records out 1073741824 bytes transferred in 50.131779 secs (21418387 bytes/sec) After searching the lists and documentation, I'm still not sure what's causing the problem. Has anyone seen anything of this nature before? Any suggestions? Erik. From owner-freebsd-performance@FreeBSD.ORG Wed Aug 11 06:33:23 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 80F9416A4CE for ; Wed, 11 Aug 2004 06:33:23 +0000 (GMT) Received: from mail03.syd.optusnet.com.au (mail03.syd.optusnet.com.au [211.29.132.184]) by mx1.FreeBSD.org (Postfix) with ESMTP id B259043D31 for ; Wed, 11 Aug 2004 06:33:22 +0000 (GMT) (envelope-from david.burns@dugeem.net) Received: from [192.168.63.20] (c211-30-248-50.carlnfd2.nsw.optusnet.com.au [211.30.248.50])i7B6XJC0018929 for ; Wed, 11 Aug 2004 16:33:20 +1000 Message-ID: <4119BDAD.7030302@dugeem.net> Date: Wed, 11 Aug 2004 16:33:17 +1000 From: David Burns User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040616 X-Accept-Language: en, en-us MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: In-Reply-To: X-Enigmail-Version: 0.84.1.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Network performance issues when writing to disk (5.2.1-RELEASE) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2004 06:33:23 -0000 Erik Rothwell wrote: > > When writing to a file, however: > > # nc -l -p 2332 -vv | dd if=/dev/stdin of=/data/junk bs=1k > listening on [any] 2332 ... > connect to [192.168.5.1] from [192.168.5.3] 51296 > sent 0, rcvd 1130496 > 1046+115 records in > 1046+115 records out > 1130496 bytes transferred in 32.888345 secs (34374 bytes/sec) > You didn't mention what your hardware is... particularly disk IO controller... If ata you could verify your disk is in a DMA mode - with "atacontrol mode 0" etc One other possibility is PCI bus contention - particularly if the box has some ancient PCI implementation and/or slow memory. Regards, David From owner-freebsd-performance@FreeBSD.ORG Thu Aug 12 15:19:16 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 60A9316A4CE for ; Thu, 12 Aug 2004 15:19:16 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id E88A543D49 for ; Thu, 12 Aug 2004 15:19:15 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i7CFHal7056552; Thu, 12 Aug 2004 11:17:36 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i7CFHaB4056549; Thu, 12 Aug 2004 11:17:36 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 12 Aug 2004 11:17:36 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Erik Rothwell In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-performance@freebsd.org Subject: Re: Network performance issues when writing to disk (5.2.1-RELEASE) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Aug 2004 15:19:16 -0000 On Tue, 10 Aug 2004, Erik Rothwell wrote: > I have a workgroup server running FreeBSD 5.2.1 that is experiencing > some serious network performance problems over very minimal load. A couple of background questions: - What sort of kernel configuration are you using -- GENERIC, or a custom kernel, and if so, is it an SMP box and is SMP enabled in the kernel? - When the system performance suddenly degrades, what is the apparent load and condition of the system? In particular, at that point if you measure the load average and CPU utilization, perhaps with "systat -vmstat" or "top", are the CPU(s) maxed out? How much time is in user vs. system vs. interrupt (vs. idle). - Can you confirm that the system does or doesn't have enough memory, and is or isn't paging much when the degredation occurs? "systat -vmstat" will probably be useeful here also. > The server has two miibus-based NICs: a WAN link via dc1 and a switched > LAN serving ~7 clients attached via dc0. > > File transfers to the server (eg, scp, WebDAV, NFS, FTP) experience > terrible throughput, starting off at about 2MB/s and dropping rapidly to > ~200KB/s. It would be interesting to know what condition the processes in question are in. In particular, how much time they're spending blocked in network, how much in CPU, and how much I/O bound to/from disk. It might be useful to ktrace the process, although ktrace will change the system load by consuming some combination of memory, disk, and cpu, of course. > At first, I suspected duplex mismatch or a misbehaving switch (unmanaged > switch with all client interfaces set to autosense), but the problem can > be replicated with the server connected directly to any of the clients > with the media at both ends explicitly set. netstat -ni indicates no > collisions but a few I/Oerrs during transfers. > > It appears from testing that the interface itself responds > intermittently under this load (during transfers, the switch reports the > port is intermittently inactive -- ifconfig on the server will also > sometimes report "status: inactive") and packet loss approaches 75%. > Occasionally, "dc0: TX underrun -- increasing TX threshold" would crop > up in dmesg, but that's not usually noteworthy. This is the if_dc interface realizing that the packet flow is going faster and accomodating. However, the intermittent response thing sounds like a more serious problem. It might be useful to run a long-running ping session using "ping -c" over a couple of hours and monitor the connectivity of the box from another box and see if it comes and goes. A few years ago I ran into some weird problems with if_dc cards interacting poorly with several linksys switches. I found that they saw intermittent connectivity, and autonegotiation worked poorly between them and that particular line of switches. It could be that you're experiencing something similar. > I replaced the NIC, yet the problem curiously persists. Did you replace it with another if_dc card, or with a card using a different interface driver? Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research From owner-freebsd-performance@FreeBSD.ORG Fri Aug 13 06:14:28 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB18416A4CE for ; Fri, 13 Aug 2004 06:14:28 +0000 (GMT) Received: from hotmail.com (bay22-dav16.bay22.hotmail.com [64.4.16.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id D0B1343D1F for ; Fri, 13 Aug 2004 06:14:28 +0000 (GMT) (envelope-from forharryh@hotmail.com) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Thu, 12 Aug 2004 23:14:28 -0700 Received: from 66.81.120.73 by bay22-dav16.bay22.hotmail.com with DAV; Fri, 13 Aug 2004 06:14:28 +0000 X-Originating-IP: [66.81.120.73] X-Originating-Email: [forharryh@hotmail.com] X-Sender: forharryh@hotmail.com From: "HarryH" To: Date: Thu, 12 Aug 2004 23:14:29 -0700 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1437 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1441 Message-ID: X-OriginalArrivalTime: 13 Aug 2004 06:14:28.0820 (UTC) FILETIME=[CA599940:01C480FC] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: CUPS Printing setup problems. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Aug 2004 06:14:29 -0000 Hi, I have FreeBSD 4.10 installed and decided make it a print server using = CUPS instead of LPD. I followed all instructions and have CUPS up and = running. When I print a test page, CUPS acts as though all is fine and = declares the job "completed" yet there is not output to the printer and = nothing in the print queue. A problem (see below) appears to be where = Perl cannot find the Locale. I set LANG, LC_ALL and LC_CTYPE all equal = to en_US.ISO.8859-1 in the .cshrc script for startup but it does not = work. I added DefaultLanguage=3Den_US.ISO.8859-1 in the cupsd.conf file = - still no change. When I do a "env" to check the variables, all of = them look good and set correctly. =20 I have spent days all over the Web but have yet to find any answers that = work. It appears to me that Perl is trying to get the LANG and other = variables from some other place but where? I checked the locale = directories in /usr/share/locale and all seems to be there. One site Linux suggested that they fixed it by adding $ENV {'LC_ALL'} = =3D 'en_US'; to some Perl startup script, but I do not know Perl and = where its parts are. Any ideas as to what I need to do to fix this? =20 Any suggestions GREATLY appreciated! Thanks, Harry Heim =20 couldn't set locale correctly D [11/Mar/2004:02:38:33 +0500] [Job 55] perl: warning: Setting locale = failed. D [11/Mar/2004:02:38:33 +0500] [Job 55] perl: warning: Please check that = your locale settings: D [11/Mar/2004:02:38:33 +0500] [Job 55] LC_ALL =3D (unset), D [11/Mar/2004:02:38:33 +0500] [Job 55] LANG =3D "en" D [11/Mar/2004:02:38:33 +0500] [Job 55] are supported and installed on = your system. D [11/Mar/2004:02:38:33 +0500] [Job 55] perl: warning: Falling back to = the standard locale ("C"). From owner-freebsd-performance@FreeBSD.ORG Fri Aug 13 07:16:51 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2933716A4CE for ; Fri, 13 Aug 2004 07:16:51 +0000 (GMT) Received: from web14107.mail.yahoo.com (web14107.mail.yahoo.com [216.136.172.137]) by mx1.FreeBSD.org (Postfix) with SMTP id CD06643D31 for ; Fri, 13 Aug 2004 07:16:50 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Message-ID: <20040813071650.57007.qmail@web14107.mail.yahoo.com> Received: from [194.248.174.50] by web14107.mail.yahoo.com via HTTP; Fri, 13 Aug 2004 09:16:50 CEST Date: Fri, 13 Aug 2004 09:16:50 +0200 (CEST) From: =?iso-8859-1?q?Claus=20Guttesen?= To: HarryH , freebsd-performance@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: CUPS Printing setup problems. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Aug 2004 07:16:51 -0000 > I have FreeBSD 4.10 installed and decided make it a > print server using CUPS instead of LPD. I followed > all instructions and have CUPS up and running. When > I print a test page, CUPS acts as though all is fine > and declares the job "completed" yet there is not > output to the printer and nothing in the print > queue. Try removing /usr/bin/lp* which comes as a part of the base-system. CUPS-files installs in /usr/local/bin. Don't forget to activate NO_LPR= true in /etc/make.conf if you decide to keep CUPS. My personal experience is tha CUPS is a bit easier to manage than LPR. Claus Yahoo! Mail (http://dk.mail.yahoo.com) - Gratis: 6 MB lagerplads, spamfilter og virusscan