From owner-freebsd-performance@FreeBSD.ORG Mon Dec 26 21:24:28 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 285BA16A41F for ; Mon, 26 Dec 2005 21:24:28 +0000 (GMT) (envelope-from mihai@duras.ro) Received: from mail.duras.ro (mail.duras.ro [86.105.56.133]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7A3E743D49 for ; Mon, 26 Dec 2005 21:24:25 +0000 (GMT) (envelope-from mihai@duras.ro) Received: from localhost (localhost [127.0.0.1]) by mail.duras.ro (Postfix) with ESMTP id E47CB7CA32 for ; Mon, 26 Dec 2005 23:24:21 +0200 (EET) Received: from mail.duras.ro ([127.0.0.1]) by localhost (mail [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 02506-03 for ; Mon, 26 Dec 2005 23:24:20 +0200 (EET) Received: from [86.105.56.194] (ma.plimb.cu.barca.prin.padure.ro [86.105.56.194]) by mail.duras.ro (Postfix) with ESMTP id 34FEA7CBEC for ; Mon, 26 Dec 2005 23:24:20 +0200 (EET) Message-ID: <43B05F1E.3060908@duras.ro> Date: Mon, 26 Dec 2005 23:22:38 +0200 From: Mihai Tanasescu User-Agent: Mozilla Thunderbird 1.0.6 (Windows/20050716) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new-20030616-p10 (RedHat) at duras.ro Subject: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Dec 2005 21:24:28 -0000 Hello, I'm running the following setup: Freebsd Dual Xeon 3Ghz machine (SMP enabled) 3 x 100 Mbits/s links (used at 80% capacity) - 3 x Intel 100 fxp 1 x 1000 Gbit link to a cisco router (transfers downstream the other 3 links) - 1 x Intel em I'm getting something arround 100k pkt/sec input and 100k pkt/sec output as "systat -ip 1" shows. Kernel polling is enabled. I have tried options HZ=1000, options HZ=2500 to see if anything changes. The problem: If I ping this machine or anything that is routed through it I get response times of 10-15-30 ms and once in 30 seconds a packet is lost. If I disable kernel.polling.enable then I get response times of 1-2-3 ms but I also get a lot of interrupts and a kernel panic after about 20 min. I was wondering what else I could try to change or what settings would be the best choice in order to have polling enabled and to avoid packet loss for small packets that also require low delays. I'm sorry if this wasn't the appropriate list to post to, Thanks in advance, Mihai From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 01:15:27 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 16C5816A41F for ; Tue, 27 Dec 2005 01:15:27 +0000 (GMT) (envelope-from mikej@rogers.com) Received: from smtp108.rog.mail.re2.yahoo.com (smtp108.rog.mail.re2.yahoo.com [68.142.225.206]) by mx1.FreeBSD.org (Postfix) with SMTP id 527D643D4C for ; Tue, 27 Dec 2005 01:15:26 +0000 (GMT) (envelope-from mikej@rogers.com) Received: (qmail 73915 invoked from network); 27 Dec 2005 01:15:25 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=pxYrNZE52kJ1sa+9hJTBLa37mu8OwykCG+U5kPG1pH2ViOWZijLYsQnBSGju/UdHc1YZ97uaQ1dZPZT9tlP1jdwlSj3nbhacdB27S46c8/kXkc01+DQEZ51TY63QQMVagGRuaIyWndUtzZ0oJScaay9ftx4sMFYM+wEvMfqOIx0= ; Received: from unknown (HELO ?70.30.133.190?) (mikej@rogers.com@70.30.133.190 with plain) by smtp108.rog.mail.re2.yahoo.com with SMTP; 27 Dec 2005 01:15:25 -0000 Message-ID: <43B095C7.4090000@rogers.com> Date: Mon, 26 Dec 2005 20:15:51 -0500 From: Mike Jakubik User-Agent: Thunderbird 1.5 (Windows/20051201) MIME-Version: 1.0 To: Mihai Tanasescu References: <43B05F1E.3060908@duras.ro> In-Reply-To: <43B05F1E.3060908@duras.ro> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 01:15:27 -0000 Mihai Tanasescu wrote: > The problem: > If I ping this machine or anything that is routed through it I get > response times of 10-15-30 ms and once in 30 seconds a packet is lost. > > If I disable kernel.polling.enable then I get response times of 1-2-3 > ms but I also get a lot of interrupts and a kernel panic after about > 20 min. You should not be getting a panic, unless you have bad hardware. Can you provide more details of the panic? Also, provide the version of freebsd, any custom kernel configuration, and attach output of dmesg. From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 07:56:08 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CA45D16A41F for ; Tue, 27 Dec 2005 07:56:08 +0000 (GMT) (envelope-from mihai@duras.ro) Received: from mail.duras.ro (mail.duras.ro [86.105.56.133]) by mx1.FreeBSD.org (Postfix) with ESMTP id C84FA43D5C for ; Tue, 27 Dec 2005 07:56:07 +0000 (GMT) (envelope-from mihai@duras.ro) Received: from localhost (localhost [127.0.0.1]) by mail.duras.ro (Postfix) with ESMTP id 84BC97CBEB; Tue, 27 Dec 2005 09:56:06 +0200 (EET) Received: from mail.duras.ro ([127.0.0.1]) by localhost (mail [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 13118-03; Tue, 27 Dec 2005 09:56:04 +0200 (EET) Received: from [86.105.56.194] (ma.plimb.cu.barca.prin.padure.ro [86.105.56.194]) by mail.duras.ro (Postfix) with ESMTP id 6F12722850; Tue, 27 Dec 2005 09:56:04 +0200 (EET) Message-ID: <43B0F32E.7070905@duras.ro> Date: Tue, 27 Dec 2005 09:54:22 +0200 From: Mihai Tanasescu User-Agent: Mozilla Thunderbird 1.0.6 (Windows/20050716) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Jakubik References: <43B05F1E.3060908@duras.ro> <43B095C7.4090000@rogers.com> In-Reply-To: <43B095C7.4090000@rogers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new-20030616-p10 (RedHat) at duras.ro Cc: freebsd-performance@freebsd.org Subject: Re: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 07:56:08 -0000 Hello, Sorry for omitting the esential details about my system. You can find my dmesg at: http://pastebin.com/479977 I'm running: FreeBSD 6.0-RELEASE-p1 Here is my kernel configuration: http://pastebin.com/479981 I wasn't there when the panic happened, I just noticed the load average was increasing (with polling disabled); and it kept increasing until the machine become unresponsive. After 1-2 minutes I observed it had rebooted itself. Now I have polling enabled on it, I have kernel.idle.poll set and a latency of 1-5 ms inside the local network. But on the 3 links I have spoken of, altough they are not full I have between 1-5 ms and 20-30 ms (when the number of packets passive through the machine increases). Mike Jakubik wrote: > Mihai Tanasescu wrote: > >> The problem: >> If I ping this machine or anything that is routed through it I get >> response times of 10-15-30 ms and once in 30 seconds a packet is lost. >> >> If I disable kernel.polling.enable then I get response times of 1-2-3 >> ms but I also get a lot of interrupts and a kernel panic after about >> 20 min. > > > You should not be getting a panic, unless you have bad hardware. Can > you provide more details of the panic? Also, provide the version of > freebsd, any custom kernel configuration, and attach output of dmesg. > From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 12:39:25 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6EA5316A41F for ; Tue, 27 Dec 2005 12:39:25 +0000 (GMT) (envelope-from tec@mega.net.br) Received: from msrv.matik.com.br (msrv.matik.com.br [200.152.83.14]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2911D43D5C for ; Tue, 27 Dec 2005 12:39:22 +0000 (GMT) (envelope-from tec@mega.net.br) Received: from anb (anb.matik.com.br [200.152.83.34]) by msrv.matik.com.br (8.13.4/8.13.1) with ESMTP id jBRCdLde017787 for ; Tue, 27 Dec 2005 10:39:21 -0200 (BRST) (envelope-from tec@mega.net.br) From: TD Meganet To: freebsd-performance@freebsd.org Date: Tue, 27 Dec 2005 10:39:15 -0200 User-Agent: KMail/1.8.3 References: <43B05F1E.3060908@duras.ro> In-Reply-To: <43B05F1E.3060908@duras.ro> Organization: Wipnet Telecom Ltda. MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200512271039.15915.tec@mega.net.br> X-Filter-Version: 1.11a (msrv.matik.com.br) X-Virus-Scanned: ClamAV version 0.86.2, clamav-milter version 0.86 on msrv.matik.com.br X-Virus-Status: Clean Subject: Re: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 12:39:25 -0000 On Monday 26 December 2005 19:22, Mihai Tanasescu wrote: > > 3 x 100 Mbits/s links (used at 80% capacity) - 3 x Intel 100 fxp > > 1 x 1000 Gbit link to a cisco router (transfers downstream the other 3 > links) - 1 x Intel em > > > I'm getting something arround 100k pkt/sec input and 100k pkt/sec output > as "systat -ip 1" shows. > I guess fxp do not support polling but there is another value, you may try "ifconfig fxpN link0" anyway on SMP polling is not supported at all I believe I had strange problems with fxp interfaces when more than one installed, I got better results when setting media with ifconfig instead of autoselect but at the end I through them out and use 3Com now which do not give me any problem at all btw I am talking about releng_6 what I thought wired your dmesg is showing AMD Features ... Hans A mensagem foi scaneada pelo sistema de e-mail e pode ser considerada segura. Service fornecido pelo Datacenter Matik https://datacenter.matik.com.br From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 17:59:04 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B433716A41F for ; Tue, 27 Dec 2005 17:59:04 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6091543D5C for ; Tue, 27 Dec 2005 17:59:04 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id 5F447117D2; Tue, 27 Dec 2005 12:59:03 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id 36A631A0932; Tue, 27 Dec 2005 12:58:59 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17329.32994.838899.696704@canoe.dclg.ca> Date: Tue, 27 Dec 2005 12:58:58 -0500 From: Freebsd-lists@dclg.ca To: Mihai Tanasescu In-Reply-To: <43B05F1E.3060908@duras.ro> References: <43B05F1E.3060908@duras.ro> X-Mailer: VM 7.17 under 21.4 (patch 18) "Social Property" XEmacs Lucid X-Mailman-Approved-At: Tue, 27 Dec 2005 18:22:42 +0000 Cc: freebsd-performance@freebsd.org Subject: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 17:59:04 -0000 >>>>> "Mihai" == Mihai Tanasescu writes: Mihai> Hello, I'm running the following setup: Mihai> Freebsd Dual Xeon 3Ghz machine (SMP enabled) Mihai> 3 x 100 Mbits/s links (used at 80% capacity) - 3 x Intel 100 Mihai> fxp Mihai> 1 x 1000 Gbit link to a cisco router (transfers downstream the Mihai> other 3 links) - 1 x Intel em Mihai> I'm getting something arround 100k pkt/sec input and 100k Mihai> pkt/sec output as "systat -ip 1" shows. Mihai> Kernel polling is enabled. I have tried options HZ=1000, Mihai> options HZ=2500 to see if anything changes. Mihai> The problem: If I ping this machine or anything that is routed Mihai> through it I get response times of 10-15-30 ms and once in 30 Mihai> seconds a packet is lost. Mihai> If I disable kernel.polling.enable then I get response times of Mihai> 1-2-3 ms but I also get a lot of interrupts and a kernel panic Mihai> after about 20 min. We've done a bunch of different experiments on various hardware and various operating systems. 300 kpps of very small packets is about the forwarding limit of FreeBSD ... with any hardware we can find. So if your packets are non-trivial ... there is a lesser cost to packet size than packet number ... then 200kpps will likely show some loss due to FreeBSD's inability to forward more packets. Keep in mind that I've had two engineers spend months on this with some guidance from me. Now a stock linux on the same hardware can handle about 500 kpps, but there's a caveat. Linux hashes packet streams (key of sce ip, src port, dest ip, dest port) for both routed and terminated traffic. This hash has some advantages, but has a huge drawback. If you spray more than than the hash size worth of streams at the linux box (even if it's not routing), then it basically falls over. (not quite... packet performance goes from 500 kpps to less than 10kpps and everything is hosed until it stops. Profiling shows it spends 99% of it's time in the hash emptying and allocaing code) This means that a few megabit of fully random small packets will knock over an arbitrary linux box. Every time. Not even SCO boxes are that lame (hah... :) My current thinking is that the hardware model has to change to get better performance. I'm not really a hardware guy (hardware is evil), but as I understand it, we're approaching limits of PCI busses. One thing that might improve the performance of your box is to swap out the FXP's for EM's. Even at 100 megabit, the EM's are measurably more efficient at passing traffic. FXP's are no slouch (although, like em's, there's dozens of different kinds with differing performance characteristics), but EM's are better. The EM driver seems better at polling, too. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 18:42:33 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8226A16A41F for ; Tue, 27 Dec 2005 18:42:33 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 293A043D4C for ; Tue, 27 Dec 2005 18:42:33 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id 68C9A11335; Tue, 27 Dec 2005 13:42:32 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id 169BE1A08D7; Tue, 27 Dec 2005 13:42:28 -0500 (EST) From: David Gilbert Message-ID: <17329.35594.458303.75695@canoe.dclg.ca> Date: Tue, 27 Dec 2005 13:42:18 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit In-Reply-To: <43B05F1E.3060908@duras.ro> References: <43B05F1E.3060908@duras.ro> X-Mailer: VM 7.17 under 21.4 (patch 18) "Social Property" XEmacs Lucid To: Mihai Tanasescu Cc: freebsd-performance@freebsd.org Subject: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 18:42:33 -0000 >>>>> "Mihai" == Mihai Tanasescu writes: Mihai> Hello, I'm running the following setup: Mihai> Freebsd Dual Xeon 3Ghz machine (SMP enabled) Mihai> 3 x 100 Mbits/s links (used at 80% capacity) - 3 x Intel 100 Mihai> fxp Mihai> 1 x 1000 Gbit link to a cisco router (transfers downstream the Mihai> other 3 links) - 1 x Intel em Mihai> I'm getting something arround 100k pkt/sec input and 100k Mihai> pkt/sec output as "systat -ip 1" shows. Mihai> Kernel polling is enabled. I have tried options HZ=1000, Mihai> options HZ=2500 to see if anything changes. Mihai> The problem: If I ping this machine or anything that is routed Mihai> through it I get response times of 10-15-30 ms and once in 30 Mihai> seconds a packet is lost. Mihai> If I disable kernel.polling.enable then I get response times of Mihai> 1-2-3 ms but I also get a lot of interrupts and a kernel panic Mihai> after about 20 min. We've done a bunch of different experiments on various hardware and various operating systems. 300 kpps of very small packets is about the forwarding limit of FreeBSD ... with any hardware we can find. So if your packets are non-trivial ... there is a lesser cost to packet size than packet number ... then 200kpps will likely show some loss due to FreeBSD's inability to forward more packets. Keep in mind that I've had two engineers spend months on this with some guidance from me. Now a stock linux on the same hardware can handle about 500 kpps, but there's a caveat. Linux hashes packet streams (key of sce ip, src port, dest ip, dest port) for both routed and terminated traffic. This hash has some advantages, but has a huge drawback. If you spray more than than the hash size worth of streams at the linux box (even if it's not routing), then it basically falls over. (not quite... packet performance goes from 500 kpps to less than 10kpps and everything is hosed until it stops. Profiling shows it spends 99% of it's time in the hash emptying and allocaing code) This means that a few megabit of fully random small packets will knock over an arbitrary linux box. Every time. Not even SCO boxes are that lame (hah... :) My current thinking is that the hardware model has to change to get better performance. I'm not really a hardware guy (hardware is evil), but as I understand it, we're approaching limits of PCI busses. One thing that might improve the performance of your box is to swap out the FXP's for EM's. Even at 100 megabit, the EM's are measurably more efficient at passing traffic. FXP's are no slouch (although, like em's, there's dozens of different kinds with differing performance characteristics), but EM's are better. The EM driver seems better at polling, too. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 18:18:09 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 885FF16A420 for ; Tue, 27 Dec 2005 18:18:09 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id F098043D77 for ; Tue, 27 Dec 2005 18:18:08 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id 2CC15118C1; Tue, 27 Dec 2005 13:18:08 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id D1CEE1A06B5; Tue, 27 Dec 2005 13:18:02 -0500 (EST) Message-ID: <17329.34124.218821.310869@canoe.dclg.ca> Date: Tue, 27 Dec 2005 13:17:48 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit In-Reply-To: <43B05F1E.3060908@duras.ro> References: <43B05F1E.3060908@duras.ro> X-Mailer: VM 7.17 under 21.4 (patch 18) "Social Property" XEmacs Lucid From: Freebsd-lists@daveg.ca To: Mihai Tanasescu X-Mailman-Approved-At: Tue, 27 Dec 2005 18:48:10 +0000 Cc: freebsd-performance@freebsd.org Subject: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 18:18:09 -0000 >>>>> "Mihai" == Mihai Tanasescu writes: Mihai> Hello, I'm running the following setup: Mihai> Freebsd Dual Xeon 3Ghz machine (SMP enabled) Mihai> 3 x 100 Mbits/s links (used at 80% capacity) - 3 x Intel 100 Mihai> fxp Mihai> 1 x 1000 Gbit link to a cisco router (transfers downstream the Mihai> other 3 links) - 1 x Intel em Mihai> I'm getting something arround 100k pkt/sec input and 100k Mihai> pkt/sec output as "systat -ip 1" shows. Mihai> Kernel polling is enabled. I have tried options HZ=1000, Mihai> options HZ=2500 to see if anything changes. Mihai> The problem: If I ping this machine or anything that is routed Mihai> through it I get response times of 10-15-30 ms and once in 30 Mihai> seconds a packet is lost. Mihai> If I disable kernel.polling.enable then I get response times of Mihai> 1-2-3 ms but I also get a lot of interrupts and a kernel panic Mihai> after about 20 min. We've done a bunch of different experiments on various hardware and various operating systems. 300 kpps of very small packets is about the forwarding limit of FreeBSD ... with any hardware we can find. So if your packets are non-trivial ... there is a lesser cost to packet size than packet number ... then 200kpps will likely show some loss due to FreeBSD's inability to forward more packets. Keep in mind that I've had two engineers spend months on this with some guidance from me. Now a stock linux on the same hardware can handle about 500 kpps, but there's a caveat. Linux hashes packet streams (key of sce ip, src port, dest ip, dest port) for both routed and terminated traffic. This hash has some advantages, but has a huge drawback. If you spray more than than the hash size worth of streams at the linux box (even if it's not routing), then it basically falls over. (not quite... packet performance goes from 500 kpps to less than 10kpps and everything is hosed until it stops. Profiling shows it spends 99% of it's time in the hash emptying and allocaing code) This means that a few megabit of fully random small packets will knock over an arbitrary linux box. Every time. Not even SCO boxes are that lame (hah... :) My current thinking is that the hardware model has to change to get better performance. I'm not really a hardware guy (hardware is evil), but as I understand it, we're approaching limits of PCI busses. One thing that might improve the performance of your box is to swap out the FXP's for EM's. Even at 100 megabit, the EM's are measurably more efficient at passing traffic. FXP's are no slouch (although, like em's, there's dozens of different kinds with differing performance characteristics), but EM's are better. The EM driver seems better at polling, too. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 21:19:54 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 916C816A41F for ; Tue, 27 Dec 2005 21:19:54 +0000 (GMT) (envelope-from mikej@rogers.com) Received: from smtp109.rog.mail.re2.yahoo.com (smtp109.rog.mail.re2.yahoo.com [68.142.225.207]) by mx1.FreeBSD.org (Postfix) with SMTP id 6904D43D75 for ; Tue, 27 Dec 2005 21:19:47 +0000 (GMT) (envelope-from mikej@rogers.com) Received: (qmail 49942 invoked from network); 27 Dec 2005 21:19:46 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=hfkjbm9cTmZkh5jBYyhqtX5g1In0IUlz4UJG6F/VOMZcSWcnXyc4YuvlNqcP61U5WjLTevLPgMAIpQxhskORPgictN2dUv4y4fvYqdqd7AzFu2BfGHd2KduixjZBH3PRChi0M6ECxQh22qml/qoujLRSP04+JW7q8oaK1kMmPFI= ; Received: from unknown (HELO ?70.30.133.190?) (mikej@rogers.com@70.30.133.190 with plain) by smtp109.rog.mail.re2.yahoo.com with SMTP; 27 Dec 2005 21:19:46 -0000 Message-ID: <43B1B001.2060007@rogers.com> Date: Tue, 27 Dec 2005 16:20:01 -0500 From: Mike Jakubik User-Agent: Thunderbird 1.5 (Windows/20051201) MIME-Version: 1.0 To: David Gilbert References: <43B05F1E.3060908@duras.ro> <17329.35594.458303.75695@canoe.dclg.ca> In-Reply-To: <17329.35594.458303.75695@canoe.dclg.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Mihai Tanasescu Subject: Re: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 21:19:54 -0000 David Gilbert wrote: >>>>>> "Mihai" == Mihai Tanasescu writes: >>>>>> > > Mihai> Hello, I'm running the following setup: > > Mihai> Freebsd Dual Xeon 3Ghz machine (SMP enabled) > > Mihai> 3 x 100 Mbits/s links (used at 80% capacity) - 3 x Intel 100 > Mihai> fxp > > Mihai> 1 x 1000 Gbit link to a cisco router (transfers downstream the > Mihai> other 3 links) - 1 x Intel em > > Mihai> I'm getting something arround 100k pkt/sec input and 100k > Mihai> pkt/sec output as "systat -ip 1" shows. > > Mihai> Kernel polling is enabled. I have tried options HZ=1000, > Mihai> options HZ=2500 to see if anything changes. > > Mihai> The problem: If I ping this machine or anything that is routed > Mihai> through it I get response times of 10-15-30 ms and once in 30 > Mihai> seconds a packet is lost. > > Mihai> If I disable kernel.polling.enable then I get response times of > Mihai> 1-2-3 ms but I also get a lot of interrupts and a kernel panic > Mihai> after about 20 min. > > We've done a bunch of different experiments on various hardware and > various operating systems. 300 kpps of very small packets is about > the forwarding limit of FreeBSD ... with any hardware we can find. So > if your packets are non-trivial ... there is a lesser cost to packet > size than packet number ... then 200kpps will likely show some loss > due to FreeBSD's inability to forward more packets. > > Keep in mind that I've had two engineers spend months on this with > some guidance from me. > > Now a stock linux on the same hardware can handle about 500 kpps, but > there's a caveat. Linux hashes packet streams (key of sce ip, > src port, dest ip, dest port) for both routed and terminated traffic. > This hash has some advantages, but has a huge drawback. If you spray > more than than the hash size worth of streams at the linux box (even > if it's not routing), then it basically falls over. > > (not quite... packet performance goes from 500 kpps to less than > 10kpps and everything is hosed until it stops. Profiling shows it > spends 99% of it's time in the hash emptying and allocaing code) > > This means that a few megabit of fully random small packets will knock > over an arbitrary linux box. Every time. > > Not even SCO boxes are that lame (hah... :) > > My current thinking is that the hardware model has to change to get > better performance. I'm not really a hardware guy (hardware is evil), > but as I understand it, we're approaching limits of PCI busses. > Post the message enough times? Have you tried on a PCI Express bus? From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 21:27:45 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE41A16A41F for ; Tue, 27 Dec 2005 21:27:45 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7473143D5A for ; Tue, 27 Dec 2005 21:27:45 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id 9E4391113A; Tue, 27 Dec 2005 16:27:44 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id D579F1A0931; Tue, 27 Dec 2005 16:27:39 -0500 (EST) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17329.45515.347351.227564@canoe.dclg.ca> Date: Tue, 27 Dec 2005 16:27:39 -0500 To: Mike Jakubik In-Reply-To: <43B1B001.2060007@rogers.com> References: <43B05F1E.3060908@duras.ro> <17329.35594.458303.75695@canoe.dclg.ca> <43B1B001.2060007@rogers.com> X-Mailer: VM 7.17 under 21.4 (patch 18) "Social Property" XEmacs Lucid Cc: freebsd-performance@freebsd.org, Mihai Tanasescu , David Gilbert Subject: Re: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 21:27:45 -0000 >>>>> "Mike" == Mike Jakubik writes: [about packet-per-second performance] Mike> Post the message enough times? Have you tried on a PCI Express Mike> bus? Well... I kept getting the message that it wasn't from my subscribed address --- which in the past had ment that it just doesn't get posted (although that isn't what the message says). Anyways... I havn't tried PCI-Express. I havn't heard anything that would convince me that it's more efficient... but then again, I havn't really researched it. The lastest hardware available when we tested things was PCI-X (ie: 133 Mhz, 64 bit). Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Wed Dec 28 06:02:41 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F34F516A41F for ; Wed, 28 Dec 2005 06:02:40 +0000 (GMT) (envelope-from chrcoluk@gmail.com) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1995D43D70 for ; Wed, 28 Dec 2005 06:02:31 +0000 (GMT) (envelope-from chrcoluk@gmail.com) Received: by wproxy.gmail.com with SMTP id 71so1160667wra for ; Tue, 27 Dec 2005 22:02:31 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=jsz7SSz0Q697Znn2pJCaL8CfiBMsElK85/yhPnFP3T90njcFjmLq/xjE8XoRKWKhx1p5p2VZ2zNW0ftP99sdPxiK4khmCZ+5iDBVwE+97Bc4c+kyJ2tJcbfwXLlkbxLFHciWwQij7dW7fP6c8Gl6qJBJ9U0OBICDt4uiG08yS2A= Received: by 10.54.110.19 with SMTP id i19mr8044550wrc; Tue, 27 Dec 2005 22:02:31 -0800 (PST) Received: by 10.54.122.17 with HTTP; Tue, 27 Dec 2005 22:02:31 -0800 (PST) Message-ID: <3aaaa3a0512272202u715ccb4fo@mail.gmail.com> Date: Wed, 28 Dec 2005 06:02:31 +0000 From: Chris To: Michael Vince In-Reply-To: <43B0BB44.40708@roq.com> MIME-Version: 1.0 References: <20051216064615.GA41775@gslin.org> <20051216070609.GA3216@il.fontys.nl> <20051216204322.GA66409@gslin.org> <43A35A1E.3020408@samsco.org> <3aaaa3a0512221304p3a35b1ecj@mail.gmail.com> <43B0BB44.40708@roq.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Benchmark MySQL Performance On FreeBSD And Linux X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2005 06:02:41 -0000 > > Make sure your compile your MySQL dynamically which is done by default, > if you include 'BUILD_STATIC=3Dyes' you ruin your ability to change > threading libs. > > portupgrade -N -m 'BUILD_OPTIMIZED=3Dyes' > /usr/ports/databases/mysql41-server > Or upgrade > portupgrade -R -m 'BUILD_OPTIMIZED=3Dyes'/var/db/pkg/mysql-server-4.1.14 > > > Use 'ldd' to find which threading library you are using. > > ldd /usr/local/libexec/mysqld > /usr/local/libexec/mysqld: > libz.so.3 =3D> /lib/libz.so.3 (0x800978000) > libwrap.so.4 =3D> /usr/lib/libwrap.so.4 (0x800a8b000) > libcrypt.so.3 =3D> /lib/libcrypt.so.3 (0x800b94000) > libstdc++.so.5 =3D> /usr/lib/libstdc++.so.5 (0x800cad000) > libm.so.4 =3D> /lib/libm.so.4 (0x800ea4000) > libpthread.so.2 =3D> */usr/lib/libpthread.so.2* (0x800fc0000) > libc.so.6 =3D> /lib/libc.so.6 (0x8010eb000) > > Put something like this into your /etc/libmap.conf to change your > threading library, and restart the service in question for the library > change to take effect. > [/usr/local/libexec/mysqld] > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > > # ldd /usr/local/libexec/mysqld > /usr/local/libexec/mysqld: > libz.so.3 =3D> /lib/libz.so.3 (0x800978000) > libwrap.so.4 =3D> /usr/lib/libwrap.so.4 (0x800a8b000) > libcrypt.so.3 =3D> /lib/libcrypt.so.3 (0x800b94000) > libstdc++.so.5 =3D> /usr/lib/libstdc++.so.5 (0x800cad000) > libm.so.4 =3D> /lib/libm.so.4 (0x800ea4000) > libpthread.so.2 =3D> */usr/lib/libthr.so.2* (0x800fc0000) > libc.so.6 =3D> /lib/libc.so.6 (0x8010d7000) > > > > Ok thanks just 2 more questions. (a) if I build dynamic there is no performance penalty? since the Makefile states build static for performance. (b) I have no /etc/libmap.conf it doesnt exist, so if I just create one and enter what you say it will work? Chris From owner-freebsd-performance@FreeBSD.ORG Wed Dec 28 08:22:49 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C25F216A41F for ; Wed, 28 Dec 2005 08:22:49 +0000 (GMT) (envelope-from oxy@field.hu) Received: from dumballah.tvnet.hu (dumballah.tvnet.hu [195.38.96.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7C4DF43D5C for ; Wed, 28 Dec 2005 08:22:48 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (localhost.localdomain [127.0.0.1]) by dumballah.tvnet.hu (Postfix) with ESMTP id 3A346101815 for ; Wed, 28 Dec 2005 09:22:47 +0100 (CET) Received: from dumballah.tvnet.hu ([127.0.0.1]) by localhost (dumballah.tvnet.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 09110-37 for ; Wed, 28 Dec 2005 09:22:45 +0100 (CET) Received: from oxy (dsl217-197-189-131.pool.tvnet.hu [217.197.189.131]) by dumballah.tvnet.hu (Postfix) with SMTP id 8D7C1101777 for ; Wed, 28 Dec 2005 09:22:45 +0100 (CET) Message-ID: <000901c60b87$e40b8ad0$0201a8c0@oxy> From: "OxY" To: Date: Wed, 28 Dec 2005 09:22:49 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-2"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by amavisd-new at tvnet.hu Subject: em0 and tunnel performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2005 08:22:49 -0000 hi! i have an intel etherexpress gigabit card on 100mbit port and fbsd 6.0-p1.. my box is a proxy, it's traffic is 100/100mbit constant.. unfortunately i have to use tunnel and that is the source of my problem. when i enable the tunnel and the packages are going through it the performance is poor.. it seems that it's not full duplex, the speed is around 50/50mbit, and have 10-15% packet loss.. i tried to enable/disable polling, and used these sysctl vars: kern.polling.enable=1 kern.ipc.somaxconn=2048 net.inet.tcp.inflight.enable=0 net.inet.ip.portrange.randomized=0 net.inet.tcp.sendspace=65536 kern.ipc.maxsockbuf=786432 net.inet.tcp.delayed_ack=0 net.inet.tcp.newreno=0 net.inet.ip.intr_queue_maxlen=1024 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.ip.rtmaxcache=1024 without tunnel the speed is 100/100mbit, great. ps: with linux its all OK, but i wanna use fbsd :) From owner-freebsd-performance@FreeBSD.ORG Wed Dec 28 12:38:35 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ACAA716A41F for ; Wed, 28 Dec 2005 12:38:35 +0000 (GMT) (envelope-from oxy@field.hu) Received: from dumballah.tvnet.hu (dumballah.tvnet.hu [195.38.96.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id CC90B43D68 for ; Wed, 28 Dec 2005 12:38:34 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (localhost.localdomain [127.0.0.1]) by dumballah.tvnet.hu (Postfix) with ESMTP id 2EC0D1019CF for ; Wed, 28 Dec 2005 13:38:33 +0100 (CET) Received: from dumballah.tvnet.hu ([127.0.0.1]) by localhost (dumballah.tvnet.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 08910-27 for ; Wed, 28 Dec 2005 13:38:32 +0100 (CET) Received: from oxy (dsl217-197-189-131.pool.tvnet.hu [217.197.189.131]) by dumballah.tvnet.hu (Postfix) with SMTP id C6A621019CB for ; Wed, 28 Dec 2005 13:38:32 +0100 (CET) Message-ID: <000401c60bab$9fd494a0$0201a8c0@oxy> From: "OxY" To: References: <000901c60b87$e40b8ad0$0201a8c0@oxy> Date: Wed, 28 Dec 2005 13:38:37 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-2"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by amavisd-new at tvnet.hu Subject: Re: em0 and tunnel performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2005 12:38:35 -0000 after digging deeper it found out that the whole em0 driver worth nothing... when i set the media to auto it's operating in half duplex (i found out after a couple of benchmarks with iperf), when i set it to 100basetx and full duplex manually the result is worse... any idea? ----- Original Message ----- From: "OxY" To: Sent: Wednesday, December 28, 2005 9:22 AM Subject: em0 and tunnel performance problem > hi! > > i have an intel etherexpress gigabit card on 100mbit port and fbsd > 6.0-p1.. > my box is a proxy, it's traffic is 100/100mbit constant.. > unfortunately i have to use tunnel and that is the source of my problem. > when i enable the tunnel and the packages are going through it the > performance is poor.. > it seems that it's not full duplex, the speed is around 50/50mbit, and > have 10-15% packet loss.. > i tried to enable/disable polling, and used these sysctl vars: > > kern.polling.enable=1 > kern.ipc.somaxconn=2048 > net.inet.tcp.inflight.enable=0 > net.inet.ip.portrange.randomized=0 > net.inet.tcp.sendspace=65536 > kern.ipc.maxsockbuf=786432 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.newreno=0 > net.inet.ip.intr_queue_maxlen=1024 > net.inet.tcp.blackhole=2 > net.inet.udp.blackhole=1 > net.inet.ip.rtmaxcache=1024 > > without tunnel the speed is 100/100mbit, great. > > ps: with linux its all OK, but i wanna use fbsd :) > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Dec 28 21:09:17 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1630B16A41F for ; Wed, 28 Dec 2005 21:09:17 +0000 (GMT) (envelope-from sean@sean.gigave.com) Received: from mailhost.gigave.com (mailhost.gigave.com [38.113.228.14]) by mx1.FreeBSD.org (Postfix) with ESMTP id D736F43D55 for ; Wed, 28 Dec 2005 21:09:16 +0000 (GMT) (envelope-from sean@sean.gigave.com) Date: Wed, 28 Dec 2005 13:09:15 -0800 From: Sean Chittenden To: OxY Message-ID: <20051228210915.GY28159@mailhost.gigave.com> References: <000901c60b87$e40b8ad0$0201a8c0@oxy> <000401c60bab$9fd494a0$0201a8c0@oxy> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <000401c60bab$9fd494a0$0201a8c0@oxy> Message-Id: <20051228210916.C32C1217BBE@mailhost.gigave.com> Cc: freebsd-performance@freebsd.org Subject: Re: em0 and tunnel performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2005 21:09:17 -0000 > after digging deeper it found out that the whole em0 driver worth > nothing... when i set the media to auto it's operating in half > duplex (i found out after a couple of benchmarks with iperf), when i > set it to 100basetx and full duplex manually the result is worse... > any idea? Explicitly set the port speed on both the switch and client side, then report back to the list. As it stands, your environment is not setup correctly, but this kind of misconfiguration seems to be a common source of performance problems for some admins. You can *never* rely on, or use auto negotiation. Its very common to have the switch be set to auto, the PC to be set to 100 FDX, and have the switch settle on 100 half-duplex (Cisco switches in particular). netstat -i will probably show you that you have line errors on your em0 interface. If you're running on Cisco switches, see the following: http://www.cisco.com/warp/public/473/3.html If you're using a different brand of switches, see their respective documentation. Point being, auto-negotiation should never be used in a production environment (though GigE handles this better). -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Wed Dec 28 21:22:26 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 627F416A41F for ; Wed, 28 Dec 2005 21:22:26 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id A49DA43D55 for ; Wed, 28 Dec 2005 21:22:24 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([82.37.32.39]) by multiplay.co.uk (multiplay.co.uk [85.236.96.23]) (MDaemon.PRO.v8.1.3.R) with ESMTP id md50002152707.msg for ; Wed, 28 Dec 2005 21:21:55 +0000 Message-ID: <005901c60bf4$b3cf2260$0100a8c0@multiplay.co.uk> From: "Steven Hartland" To: "Sean Chittenden" , "OxY" References: <000901c60b87$e40b8ad0$0201a8c0@oxy><000401c60bab$9fd494a0$0201a8c0@oxy> <20051228210915.GY28159@mailhost.gigave.com> Date: Wed, 28 Dec 2005 21:21:37 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Spam-Processed: multiplay.co.uk, Wed, 28 Dec 2005 21:21:55 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 82.37.32.39 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Wed, 28 Dec 2005 21:21:55 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: em0 and tunnel performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Dec 2005 21:22:26 -0000 ----- Original Message ----- From: "Sean Chittenden" > You can *never* rely on, or use auto negotiation. Its very common to > have the switch be set to auto, the PC to be set to 100 FDX, and have > the switch settle on 100 half-duplex (Cisco switches in particular). > netstat -i will probably show you that you have line errors on your > em0 interface. If you're running on Cisco switches, see the > following: In fact is also common to get errors when both are hardcoded the only resolution being to set both to auto it all depends on the hardware. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Dec 29 10:19:20 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6B9B216A41F for ; Thu, 29 Dec 2005 10:19:20 +0000 (GMT) (envelope-from boisan@gmail.com) Received: from nproxy.gmail.com (nproxy.gmail.com [64.233.182.193]) by mx1.FreeBSD.org (Postfix) with ESMTP id 977E543D5A for ; Thu, 29 Dec 2005 10:19:19 +0000 (GMT) (envelope-from boisan@gmail.com) Received: by nproxy.gmail.com with SMTP id o60so93447nfa for ; Thu, 29 Dec 2005 02:19:19 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=txqy6/mE+eI9y0PrUGVkQJKfdSNeGH0aw+rXNb1eHMSH9SA2sRx6yHrsFidPrA6lDt/bjJuoObyTMB3FRJOY5A21oos1nRLQCGtndT5/CtptToirRvnhA+7IbEd9QFnOk4jTrO7H/hl8mA+t7cAb1TXPeGk1pnnOtgDjxMxEDR4= Received: by 10.48.247.14 with SMTP id u14mr350900nfh; Thu, 29 Dec 2005 02:19:18 -0800 (PST) Received: by 10.49.41.9 with HTTP; Thu, 29 Dec 2005 02:19:18 -0800 (PST) Message-ID: Date: Thu, 29 Dec 2005 11:19:18 +0100 From: Angel Blazquez To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Subject: nfs server overload (nfsd) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2005 10:19:20 -0000 Hello, We are expecting incredible overload in a NFS server. A top shows nfsd consuming most of the CPU: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMA= ND 6000 root -8 0 1204K 660K biord 1 124:15 27.88% 27.88% nfsd 6002 root 4 0 1204K 660K *Giant 0 124:18 17.58% 17.58% nfsd 6006 root 4 0 1204K 660K *Giant 0 123:38 10.21% 10.21% nfsd 6005 root 4 0 1204K 660K *Giant 0 123:36 7.47% 7.47% nfsd 6003 root 4 0 1204K 660K *Giant 0 123:08 4.15% 4.15% nfsd 6001 root 4 0 1204K 660K *Giant 0 123:16 2.83% 2.83% nfsd Memory looks fine: Mem: 27M Active, 910M Inact, 136M Wired, 51M Cache, 112M Buf, 1828K Free Swap: 2048M Total, 72K Used, 2048M Free Typing in the nfs server (console/ssh) becomes terrible, the server does not reply well. We are running this nfs server in FreeBSD 5.3-RELEASE-p23 on a Compaq Proliant server with a Compaq Smart Array 5300 that comunicates with a array of disks: /dev/da0s1d 164G 124G 27G 82% /data0 /dev/da1s1d 131G 80G 41G 66% /data1 We have /data0 and /data1 exported: /data0 -maproot=3Droot -alldirs -network 192.168.62.0 -mask 255.255.255.0 /data1 -maproot=3Droot -alldirs -network 192.168.62.0 -mask 255.255.255.0 so a couple of incoming SMTP servers we have can deliver e-mail to those filesystems. We are running exim 4.60.0 in those other servers, 4.10-RELEASE-p5 in one of them, and FreeBSD 6.0-RELEASE #0 in the other one. If we stop exim delivering e-mail, nfs server does well, the cpu gets free, and the nfs server works fine (replies to user interaction, etc). FreeBSD 6.0 sysctl output (nfs related): vfs.nfs4.access_cache_timeout: 60 vfs.nfs4.nfsv3_commit_on_close: 0 vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.realign_test: 1294030 vfs.nfs.realign_count: 0 vfs.nfs.bufpackets: 4 vfs.nfs.reconnects: 2 vfs.nfs.iodmaxidle: 120 vfs.nfs.iodmin: 4 vfs.nfs.iodmax: 20 vfs.nfs.defect: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.diskless_valid: 0 vfs.nfs.diskless_rootpath: vfs.nfs.access_cache_timeout: 2 vfs.nfs.nfsv3_commit_on_close: 0 vfs.nfs.clean_pages_on_close: 1 vfs.nfs.nfs_directio_enable: 0 vfs.nfs.nfs_directio_allow_mmap: 1 vfs.nfsrv.nfs_privport: 0 vfs.nfsrv.async: 0 vfs.nfsrv.commit_blks: 0 vfs.nfsrv.commit_miss: 0 vfs.nfsrv.realign_test: 0 vfs.nfsrv.realign_count: 0 vfs.nfsrv.gatherdelay: 10000 vfs.nfsrv.gatherdelay_v3: 0 FreeBSD 4.10 sysctl output (nfs related): vfs.nfs.nfs_privport: 0 vfs.nfs.async: 0 vfs.nfs.commit_blks: 0 vfs.nfs.commit_miss: 0 vfs.nfs.realign_test: 84602323 vfs.nfs.realign_count: 99713 vfs.nfs.bufpackets: 4 vfs.nfs.gatherdelay: 10000 vfs.nfs.gatherdelay_v3: 0 vfs.nfs.defect: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.diskless_valid: 0 vfs.nfs.diskless_rootpath: vfs.nfs.diskless_swappath: vfs.nfs.access_cache_timeout: 2 vfs.nfs.nfsv3_commit_on_close: 0 This couple of servers mounts the filesystems with this options: 192.168.62.54:/data1 /mail nfs =20 rw,nfsv3,intr,dumbtimer,rdirplus,nosuid,nodev 0 0 192.168.62.54:/data0 /data0 nfs =20 rw,nfsv3,intr,dumbtimer,rdirplus,nosuid,nodev 0 0 On the server, sysctl nfs related output looks like this: vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.realign_test: 2694 vfs.nfs.realign_count: 0 vfs.nfs.bufpackets: 4 vfs.nfs.reconnects: 2 vfs.nfs.iodmaxidle: 120 vfs.nfs.iodmin: 4 vfs.nfs.iodmax: 20 vfs.nfs.defect: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.diskless_valid: 0 vfs.nfs.diskless_rootpath: vfs.nfs.access_cache_timeout: 2 vfs.nfs.nfsv3_commit_on_close: 0 vfs.nfs4.access_cache_timeout: 60 vfs.nfs4.nfsv3_commit_on_close: 0 vfs.nfsrv.nfs_privport: 0 vfs.nfsrv.async: 1 vfs.nfsrv.commit_blks: 579238 vfs.nfsrv.commit_miss: 413059 vfs.nfsrv.realign_test: 88269083 vfs.nfsrv.realign_count: 11961 vfs.nfsrv.gatherdelay: 10000 vfs.nfsrv.gatherdelay_v3: 0 debug.hashstat.nfsnode: 65536 5 1 0 Thanks in advance, Best regards, Angel Blazquez From owner-freebsd-performance@FreeBSD.ORG Thu Dec 29 21:58:07 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C6E416A420 for ; Thu, 29 Dec 2005 21:58:07 +0000 (GMT) (envelope-from mv@roq.com) Received: from p4.roq.com (ns1.ecoms.com [207.44.130.137]) by mx1.FreeBSD.org (Postfix) with ESMTP id E65E643D7F for ; Thu, 29 Dec 2005 21:57:59 +0000 (GMT) (envelope-from mv@roq.com) Received: from p4.roq.com (localhost.roq.com [127.0.0.1]) by p4.roq.com (Postfix) with ESMTP id 347374D104 for ; Thu, 29 Dec 2005 21:58:11 +0000 (GMT) Received: from [192.168.0.2] (ppp157-158.static.internode.on.net [150.101.157.158]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by p4.roq.com (Postfix) with ESMTP id 6F18A4D088 for ; Thu, 29 Dec 2005 21:58:10 +0000 (GMT) Message-ID: <43B45BE6.7000401@roq.com> Date: Fri, 30 Dec 2005 08:57:58 +1100 From: Michael Vince User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.7.12) Gecko/20051208 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <20051216064615.GA41775@gslin.org> <20051216070609.GA3216@il.fontys.nl> <20051216204322.GA66409@gslin.org> <43A35A1E.3020408@samsco.org> <3aaaa3a0512221304p3a35b1ecj@mail.gmail.com> <43B0BB44.40708@roq.com> <3aaaa3a0512272202u715ccb4fo@mail.gmail.com> In-Reply-To: <3aaaa3a0512272202u715ccb4fo@mail.gmail.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Subject: Re: Benchmark MySQL Performance On FreeBSD And Linux X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2005 21:58:07 -0000 Chris wrote: >>Make sure your compile your MySQL dynamically which is done by default, >>if you include 'BUILD_STATIC=yes' you ruin your ability to change >>threading libs. >> >>portupgrade -N -m 'BUILD_OPTIMIZED=yes' >>/usr/ports/databases/mysql41-server >>Or upgrade >>portupgrade -R -m 'BUILD_OPTIMIZED=yes'/var/db/pkg/mysql-server-4.1.14 >> >> >>Use 'ldd' to find which threading library you are using. >> >>ldd /usr/local/libexec/mysqld >>/usr/local/libexec/mysqld: >> libz.so.3 => /lib/libz.so.3 (0x800978000) >> libwrap.so.4 => /usr/lib/libwrap.so.4 (0x800a8b000) >> libcrypt.so.3 => /lib/libcrypt.so.3 (0x800b94000) >> libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x800cad000) >> libm.so.4 => /lib/libm.so.4 (0x800ea4000) >> libpthread.so.2 => */usr/lib/libpthread.so.2* (0x800fc0000) >> libc.so.6 => /lib/libc.so.6 (0x8010eb000) >> >>Put something like this into your /etc/libmap.conf to change your >>threading library, and restart the service in question for the library >>change to take effect. >>[/usr/local/libexec/mysqld] >>libpthread.so.2 libthr.so.2 >>libpthread.so libthr.so >> >># ldd /usr/local/libexec/mysqld >>/usr/local/libexec/mysqld: >> libz.so.3 => /lib/libz.so.3 (0x800978000) >> libwrap.so.4 => /usr/lib/libwrap.so.4 (0x800a8b000) >> libcrypt.so.3 => /lib/libcrypt.so.3 (0x800b94000) >> libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x800cad000) >> libm.so.4 => /lib/libm.so.4 (0x800ea4000) >> libpthread.so.2 => */usr/lib/libthr.so.2* (0x800fc0000) >> libc.so.6 => /lib/libc.so.6 (0x8010d7000) >> >> >> >>Ok thanks just 2 more questions. >> >> > >(a) if I build dynamic there is no performance penalty? since the Makefile >states build static for performance. >(b) I have no /etc/libmap.conf it doesnt exist, so if I just create one and >enter what you say it will work? > >Chris > > Yes check the URL below http://dev.mysql.com/doc/refman/4.1/en/compile-and-link-options.html The first line of the Optimizing the MySQL Server in the Compiling section says "You obtain the fastest executables when you link with |-static|." And then again further down " If you link dynamically (without |-static|), the result is 13% slower on Linux. Note that you still can use a dynamically linked MySQL library for your client applications. It is the server that is most critical for performance." It would be great if some one could do the benchmarks on FreeBSD to verify that. > (b) I have no /etc/libmap.conf it doesnt exist, so if I just create > one and enter what you say it will work? Yes, its never there by default so just create it. FYI there is a man page on libmap.conf Mike