From owner-freebsd-performance@FreeBSD.ORG Mon Aug 10 08:47:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 221C1106566C for ; Mon, 10 Aug 2009 08:47:18 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx44.mail.ru (mx44.mail.ru [94.100.176.58]) by mx1.freebsd.org (Postfix) with ESMTP id 9D3778FC30 for ; Mon, 10 Aug 2009 08:47:17 +0000 (UTC) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx44.mail.ru (mPOP.Fallback_MX) with ESMTP id 5D67C30AE56 for ; Mon, 10 Aug 2009 12:30:07 +0400 (MSD) Received: from [91.190.115.253] (port=59538 helo=pstation) by mx38.mail.ru with asmtp id 1MaQGX-0008xL-00; Mon, 10 Aug 2009 12:30:01 +0400 Date: Mon, 10 Aug 2009 12:32:27 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <117233873703.20090810123227@mail.ru> To: Invernizzi Fabrizio In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-performance@freebsd.org Subject: Re[2]: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2009 08:47:18 -0000 Hello Invernizzi, First of all I didn't catch for what task you are tuning. For forwarding or for transfer? In any case you should start with minimal configuration (no netgraph modules, no any firewall). Choose a test similiar to your typical load. Provide us with test result, cpu consumption and output of ifconfig netstat -s netstat -m debug values from driver (sysctl (?).debug)). Wednesday, August 05, 2009, 2:13:23 PM, you wrote: IF> No improvement with kern.ipc.nmbclusters=262144 and 1.8.6 driver :<((((( IF> ++fabrizio IF> ------------------------------------------------------------------ IF> Telecom Italia IF> Fabrizio INVERNIZZI IF> Technology - TILAB IF> Accesso Fisso e Trasporto IF> Via Reiss Romoli, 274 10148 Torino IF> Tel. +39 011 2285497 IF> Mob. +39 3316001344 IF> Fax +39 06 41867287 IF> ________________________________ IF> From: Jack Vogel [mailto:jfvogel@gmail.com] IF> Sent: marted́ 4 agosto 2009 18.42 IF> To: Julian Elischer IF> Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev IF> Subject: Re: Test on 10GBE Intel based network card IF> Your nmbclusters is very low, you list it twice so I'm assuming the second value is IF> what it ends up being, 32K :( IF> I would set it to: IF> kern.ipc.nmbclusters=262144 IF> Also, I thought you were using the current driver, but now it looks like you are IF> using something fairly old, use my latest code which is 1.8.8 IF> Jack IF> On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > wrote: IF> Invernizzi Fabrizio wrote: IF> The limitation that you see is about the max number of packets that IF> FreeBSD can handle - it looks like your best performance is reached at IF> 64 byte packets? IF> If you are meaning in term of Packet per second, you are right. These IF> are the packet per second measured during tests: IF> 64 byte: 610119 Pps IF> 512 byte: 516917 Pps IF> 1492 byte: 464962 Pps IF> Am I correct that the maximum you can reach is around 639,000 packets IF> per second? IF> Yes, as you can see the maximum is 610119 Pps. IF> Where does this limit come from? IF> ah that's the whole point of tuning :-) IF> there are severalpossibities: IF> 1/ the card's interrupts are probably attache dto aonly 1 cpu, IF> so that cpu can do no more work IF> This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm IF> last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 IF> 124 processes: 13 running, 73 sleeping, 38 waiting IF> CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle IF> Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free IF> Swap: 2048M Total, 2048M Free IF> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND IF> 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 IF> 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 IF> 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 IF> 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 IF> 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq IF> 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq IF> 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq IF> 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq IF> .... IF> It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. IF> 2/ if more than 1 cpu is working, it may be that there is a lock in IF> heavy contention somewhere. IF> This I think is the problem. I am trying to understand how to IF> 1- see where the heavy contention is (context switching? Some limiting setting?) IF> 2- mitigate it IF> there ia a lock profiling tool that right now I can't remember the name of.. IF> look it up with google :-) FreeBSD lock profiling tool IF> ah, first hit... IF> http://blogs.epfl.ch/article/23832 IF> is the machine still responsive to other networks while IF> running at maximum capacity on this network? (make sure that IF> the other networks are on a differnet CPU (hmm I can't remember how to IF> do that :-). -- Best regards, Anthony mailto:ap00@mail.ru