From owner-freebsd-performance@FreeBSD.ORG Fri Aug 14 21:45:25 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C30E4106568D for ; Fri, 14 Aug 2009 21:45:25 +0000 (UTC) (envelope-from cpardo@fastsoft.com) Received: from HQ-ES.FASTSOFT.COM (hq-es.fastsoft.com [38.102.243.86]) by mx1.freebsd.org (Postfix) with ESMTP id A43268FC61 for ; Fri, 14 Aug 2009 21:45:25 +0000 (UTC) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Fri, 14 Aug 2009 14:33:25 -0700 Message-ID: In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoVIm0LnEUSix6PQmaZQulzWV0J9QAksHgAAdwK7iA= References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local><0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com><36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local><4A77094C.8030308@elischer.org><36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local><4A785F20.8050807@elischer.org><2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> From: "Carlos Pardo" To: "Jack Vogel" Cc: freebsd-performance@freebsd.org Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2009 21:45:25 -0000 Hi Jack, I have a quick question. We are getting too many missed packets per = minute running about 3Gbs traffic. We can not use frame control in our = application. We are assuming that there is no way to improve upon the = problem since it seems to be a hardware limitation with the receive = FIFO. We are using the Intel=AE 82598EB 10 Gigabit Ethernet Controller. = When can we expect the next generation card from Intel? Thanks for any = information you may provide. =20 Typical error count "ix0: Missed Packets =3D 81174" after a few minutes. Best Regards, Cpardo -----Original Message----- From: owner-freebsd-performance@freebsd.org = [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi = Fabrizio Sent: Wednesday, August 05, 2009 3:13 AM To: Jack Vogel; Julian Elischer Cc: freebsd-performance@freebsd.org; Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver = :<((((( ++fabrizio ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 ________________________________ From: Jack Vogel [mailto:jfvogel@gmail.com] Sent: marted=EC 4 agosto 2009 18.42 To: Julian Elischer Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card Your nmbclusters is very low, you list it twice so I'm assuming the = second value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=3D262144 Also, I thought you were using the current driver, but now it looks like = you are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer = > wrote: Invernizzi Fabrizio wrote: The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps Am I correct that the maximum you can reach is around 639,000 packets per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a = 64byte-long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 = up = 0+20:36:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: = cpu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: = cpu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: = cpu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: = cpu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 = rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 = rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 = rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 = rxq .... It looks like the 4 rxq processes are bound to the 4 available cores = with equal distribution. 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting = setting?) 2- mitigate it there ia a lock profiling tool that right now I can't remember the name = of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle = persone indicate. La diffusione, copia o qualsiasi altra azione = derivante dalla conoscenza di queste informazioni sono rigorosamente = vietate. Qualora abbiate ricevuto questo documento per errore siete = cortesemente pregati di darne immediata comunicazione al mittente e di = provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain = privileged information intended for the addressee(s) only. = Dissemination, copying, printing or use by anybody else is unauthorised. = If you are not the intended recipient, please delete this message and = any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@freebsd.org = mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle = persone indicate. La diffusione, copia o qualsiasi altra azione = derivante dalla conoscenza di queste informazioni sono rigorosamente = vietate. Qualora abbiate ricevuto questo documento per errore siete = cortesemente pregati di darne immediata comunicazione al mittente e di = provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain = privileged information intended for the addressee(s) only. = Dissemination, copying, printing or use by anybody else is unauthorised. = If you are not the intended recipient, please delete this message and = any attachments and advise the sender by return e-mail, Thanks. [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. = Non stampare questa mail se non =E8 necessario.