From owner-freebsd-performance@FreeBSD.ORG Mon Aug 17 21:52:47 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0F921065692 for ; Mon, 17 Aug 2009 21:52:47 +0000 (UTC) (envelope-from cpardo@fastsoft.com) Received: from HQ-ES.FASTSOFT.COM (hq-es.fastsoft.com [38.102.243.86]) by mx1.freebsd.org (Postfix) with ESMTP id 863998FC60 for ; Mon, 17 Aug 2009 21:52:47 +0000 (UTC) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Mon, 17 Aug 2009 14:52:46 -0700 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcodLLyVq4Fk8a+CSGuBRYfobfuBlQAq2hMgAGq58yA= From: "Carlos Pardo" To: Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Aug 2009 21:52:47 -0000 Hi Jack, =20 Thanks for the quick response. We can not use LRO because of the way we = accelerate on the WAN ports. We just moved from 7.0 to 8.0 to use your = latest driver (1.8.8). One thing we do not understand in 8.0. We are = having insane numbers for XON/XOFF Rcvd counters with essentially no = traffic. Driver version 1.2.16 works fine. Who should we contact for = help? =20 ix0: Std Mbuf Failed =3D 0 ix0: Missed Packets =3D 0 ix0: Receive length errors =3D 0 ix0: Crc errors =3D 0 ix0: Driver dropped packets =3D 0 ix0: watchdog timeouts =3D 0 ix0: XON Rcvd =3D 7950055973552 ix0: XON Xmtd =3D 0 ix0: XOFF Rcvd =3D 7950055973552 ix0: XOFF Xmtd =3D 0 ix0: Total Packets Rcvd =3D 2149 ix0: Good Packets Rcvd =3D 2149 ix0: Good Packets Xmtd =3D 1001 ix0: TSO Transmissions =3D 0 ix1: Std Mbuf Failed =3D 0 ix1: Missed Packets =3D 0 ix1: Receive length errors =3D 0 ix1: Crc errors =3D 0 ix1: Driver dropped packets =3D 0 ix1: watchdog timeouts =3D 0 ix1: XON Rcvd =3D 7946320044993 ix1: XON Xmtd =3D 0 ix1: XOFF Rcvd =3D 7946320044993 ix1: XOFF Xmtd =3D 0 ix1: Total Packets Rcvd =3D 1002 ix1: Good Packets Rcvd =3D 1002 ix1: Good Packets Xmtd =3D 1588 ix1: TSO Transmissions =3D 0 =20 Regards, =20 C Pardo =20 From: Jack Vogel [mailto:jfvogel@gmail.com]=20 Sent: Friday, August 14, 2009 3:15 PM To: Carlos Pardo Cc: freebsd-performance@freebsd.org Subject: Re: Test on 10GBE Intel based network card =20 I've talked over the issues with the guy on our team who has been most involved in 10G performance, he asserts that 3Gbs will saturate a single cpu with a small packet size, this is why you need multiqueue across multiple cores. He was dubious about the FIFO assertion, its a relative thing, if you can keep the thing drained it won't be a problem, doing = that is a complex combination of factors, the cpu, the bus, the memory.... If you don't deal with the systemic issues just cuz you go from an 82598 to a 82599 is not going to solve things. What about LRO, are/can you use that? I never saw an answer about the forwarding question, you can't use LRO in that case of course. Regards, Jack On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo = wrote: Hi Jack, I have a quick question. We are getting too many missed packets per = minute running about 3Gbs traffic. We can not use frame control in our = application. We are assuming that there is no way to improve upon the = problem since it seems to be a hardware limitation with the receive = FIFO. We are using the Intel=AE 82598EB 10 Gigabit Ethernet Controller. = When can we expect the next generation card from Intel? Thanks for any = information you may provide. Typical error count "ix0: Missed Packets =3D 81174" after a few minutes. Best Regards, Cpardo -----Original Message----- From: owner-freebsd-performance@freebsd.org = [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi = Fabrizio Sent: Wednesday, August 05, 2009 3:13 AM To: Jack Vogel; Julian Elischer Cc: freebsd-performance@freebsd.org; Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver = :<((((( ++fabrizio ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 ________________________________ From: Jack Vogel [mailto:jfvogel@gmail.com] Sent: marted=EC 4 agosto 2009 18.42 To: Julian Elischer Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card Your nmbclusters is very low, you list it twice so I'm assuming the = second value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=3D262144 Also, I thought you were using the current driver, but now it looks like = you are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer = > wrote: Invernizzi Fabrizio wrote: The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps Am I correct that the maximum you can reach is around 639,000 packets per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a = 64byte-long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 = up = 0+20:36:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: = cpu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: = cpu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: = cpu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: = cpu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 = rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 = rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 = rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 = rxq .... It looks like the 4 rxq processes are bound to the 4 available cores = with equal distribution. 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting = setting?) 2- mitigate it there ia a lock profiling tool that right now I can't remember the name = of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle = persone indicate. La diffusione, copia o qualsiasi altra azione = derivante dalla conoscenza di queste informazioni sono rigorosamente = vietate. Qualora abbiate ricevuto questo documento per errore siete = cortesemente pregati di darne immediata comunicazione al mittente e di = provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain = privileged information intended for the addressee(s) only. = Dissemination, copying, printing or use by anybody else is unauthorised. = If you are not the intended recipient, please delete this message and = any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@freebsd.org = mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle = persone indicate. La diffusione, copia o qualsiasi altra azione = derivante dalla conoscenza di queste informazioni sono rigorosamente = vietate. Qualora abbiate ricevuto questo documento per errore siete = cortesemente pregati di darne immediata comunicazione al mittente e di = provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain = privileged information intended for the addressee(s) only. = Dissemination, copying, printing or use by anybody else is unauthorised. = If you are not the intended recipient, please delete this message and = any attachments and advise the sender by return e-mail, Thanks. [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. = Non stampare questa mail se non =E8 necessario. =20 From owner-freebsd-performance@FreeBSD.ORG Mon Aug 17 22:03:38 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 801DC1065691 for ; Mon, 17 Aug 2009 22:03:38 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yw0-f199.google.com (mail-yw0-f199.google.com [209.85.211.199]) by mx1.freebsd.org (Postfix) with ESMTP id 145F98FC52 for ; Mon, 17 Aug 2009 22:03:37 +0000 (UTC) Received: by ywh37 with SMTP id 37so4690960ywh.28 for ; Mon, 17 Aug 2009 15:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=CuJ/lFG34LW02JEF2kAHyUDbdrpa+Ux+VDO5zwQg0Lo=; b=K5+BgfqzuSHW4sODdmzjMhW373zMFEqDaofwBJ/Gge5r4A6zUwiefs0ajwwPGdP5FZ 2ymblb90dsLvN41l+3HUcoiEhjR0clnE61ukNkLSHUsOAOpujvN6Xa8KY7R4FvxkZ96N 1Lm1dvF5LNnajKBYuxzMvZaxmjIIFKRqoXJlo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=N1WE+TJVGEqJXXo3XGCCjOhBBWvwwydQBYnZsqDaC8/xMKCRDuwOLfhumkZz/TCClt AtP0d74ILCTXNMRWF2M2xHzmU1ZWWONtUBsMpt9ju2yogdApw/uIVhlZQAKQHdGjV8Ao wFKtz8862wSQf/oC9XTkAuswEiSA8b3T5finE= MIME-Version: 1.0 Received: by 10.101.17.6 with SMTP id u6mr4158144ani.78.1250546616139; Mon, 17 Aug 2009 15:03:36 -0700 (PDT) In-Reply-To: References: Date: Mon, 17 Aug 2009 15:03:36 -0700 Message-ID: <2a41acea0908171503r3613d430ib154cd3445eb1309@mail.gmail.com> From: Jack Vogel To: Carlos Pardo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Aug 2009 22:03:38 -0000 Who ya gonna call, why me of course, its my driver :) Hmmm, the numbers on those look bogus, like some uninitialized variables. You did say you aren't using flow control, right? Jack On Mon, Aug 17, 2009 at 2:52 PM, Carlos Pardo wrote: > Hi Jack, > > > > Thanks for the quick response. We can not use LRO because of the way we > accelerate on the WAN ports. We just moved from 7.0 to 8.0 to use your > latest driver (1.8.8). One thing we do not understand in 8.0. We are havi= ng > insane numbers for XON/XOFF Rcvd counters with essentially no traffic. > Driver version 1.2.16 works fine. Who should we contact for help? > > > > ix0: Std Mbuf Failed =3D 0 > > ix0: Missed Packets =3D 0 > > ix0: Receive length errors =3D 0 > > ix0: Crc errors =3D 0 > > ix0: Driver dropped packets =3D 0 > > ix0: watchdog timeouts =3D 0 > > *ix0: XON Rcvd =3D 7950055973552* > > ix0: XON Xmtd =3D 0 > > *ix0: XOFF Rcvd =3D 7950055973552* > > ix0: XOFF Xmtd =3D 0 > > ix0: Total Packets Rcvd =3D 2149 > > ix0: Good Packets Rcvd =3D 2149 > > ix0: Good Packets Xmtd =3D 1001 > > ix0: TSO Transmissions =3D 0 > > ix1: Std Mbuf Failed =3D 0 > > ix1: Missed Packets =3D 0 > > ix1: Receive length errors =3D 0 > > ix1: Crc errors =3D 0 > > ix1: Driver dropped packets =3D 0 > > ix1: watchdog timeouts =3D 0 > > *ix1: XON Rcvd =3D 7946320044993* > > ix1: XON Xmtd =3D 0 > > *ix1: XOFF Rcvd =3D 7946320044993* > > ix1: XOFF Xmtd =3D 0 > > ix1: Total Packets Rcvd =3D 1002 > > ix1: Good Packets Rcvd =3D 1002 > > ix1: Good Packets Xmtd =3D 1588 > > ix1: TSO Transmissions =3D 0 > > > > Regards, > > > > C Pardo > > > > *From:* Jack Vogel [mailto:jfvogel@gmail.com] > *Sent:* Friday, August 14, 2009 3:15 PM > *To:* Carlos Pardo > *Cc:* freebsd-performance@freebsd.org > *Subject:* Re: Test on 10GBE Intel based network card > > > > I've talked over the issues with the guy on our team who has been most > involved in 10G performance, he asserts that 3Gbs will saturate a single > cpu with a small packet size, this is why you need multiqueue across > multiple cores. He was dubious about the FIFO assertion, its a relative > thing, if you can keep the thing drained it won't be a problem, doing tha= t > is a complex combination of factors, the cpu, the bus, the memory.... > > If you don't deal with the systemic issues just cuz you go from an 82598 > to a 82599 is not going to solve things. > > What about LRO, are/can you use that? I never saw an answer about the > forwarding question, you can't use LRO in that case of course. > > Regards, > > Jack > > On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo wrote= : > > Hi Jack, > > I have a quick question. We are getting too many missed packets per minut= e > running about 3Gbs traffic. We can not use frame control in our applicati= on. > We are assuming that there is no way to improve upon the problem since it > seems to be a hardware limitation with the receive FIFO. We are using the > Intel=AE 82598EB 10 Gigabit Ethernet Controller. When can we expect the n= ext > generation card from Intel? Thanks for any information you may provide. > > Typical error count "ix0: Missed Packets =3D 81174" after a few minutes. > > Best Regards, > > Cpardo > > > > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto: > owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi Fabrizio > > Sent: Wednesday, August 05, 2009 3:13 AM > To: Jack Vogel; Julian Elischer > Cc: freebsd-performance@freebsd.org; Stefan Lambrev > Subject: RE: Test on 10GBE Intel based network card > > No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver :<((((= ( > > ++fabrizio > > ------------------------------------------------------------------ > Telecom Italia > Fabrizio INVERNIZZI > Technology - TILAB > Accesso Fisso e Trasporto > Via Reiss Romoli, 274 10148 Torino > Tel. +39 011 2285497 > Mob. +39 3316001344 > Fax +39 06 41867287 > > > ________________________________ > From: Jack Vogel [mailto:jfvogel@gmail.com] > Sent: marted=EC 4 agosto 2009 18.42 > To: Julian Elischer > Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev > Subject: Re: Test on 10GBE Intel based network card > > Your nmbclusters is very low, you list it twice so I'm assuming the secon= d > value is > what it ends up being, 32K :( > > I would set it to: > > kern.ipc.nmbclusters=3D262144 > > Also, I thought you were using the current driver, but now it looks like > you are > using something fairly old, use my latest code which is 1.8.8 > > Jack > > On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > wrote: > Invernizzi Fabrizio wrote: > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? > If you are meaning in term of Packet per second, you are right. These > are the packet per second measured during tests: > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > Am I correct that the maximum you can reach is around 639,000 packets > per second? > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > ah that's the whole point of tuning :-) > there are severalpossibities: > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a > 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 > up > 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAN= D > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: > cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: > cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: > cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: > cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with > equal distribution. > > > > 2/ if more than 1 cpu is working, it may be that there is a lock in > heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting > setting?) > 2- mitigate it > > > > there ia a lock profiling tool that right now I can't remember the name > of.. > > look it up with google :-) FreeBSD lock profiling tool > > ah, first hit... > > http://blogs.epfl.ch/article/23832 > > > > is the machine still responsive to other networks while > running at maximum capacity on this network? (make sure that > the other networks are on a differnet CPU (hmm I can't remember how to > do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > > freebsd-performance@freebsd.org > mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org freebsd-performance-unsubscribe@freebsd.org>" > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. > Non stampare questa mail se non =E8 necessario. > > > From owner-freebsd-performance@FreeBSD.ORG Tue Aug 18 07:21:26 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79DF2106568C for ; Tue, 18 Aug 2009 07:21:26 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 74D268FC16 for ; Tue, 18 Aug 2009 07:21:25 +0000 (UTC) Received: from GRFHUB702BA020.griffon.local (10.188.101.112) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Tue, 18 Aug 2009 09:21:23 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by GRFHUB702BA020.griffon.local ([10.188.101.112]) with mapi; Tue, 18 Aug 2009 09:21:23 +0200 From: Invernizzi Fabrizio To: Jack Vogel , Carlos Pardo Date: Tue, 18 Aug 2009 09:21:13 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcofhqcaDm2rdhgnQnmKlXapbp2TrgATKc9w Message-ID: <36A93B31228D3B49B691AD31652BCAE9A4569679F5@GRFMBX702BA020.griffon.local> References: <2a41acea0908171503r3613d430ib154cd3445eb1309@mail.gmail.com> In-Reply-To: <2a41acea0908171503r3613d430ib154cd3445eb1309@mail.gmail.com> Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-performance@freebsd.org" Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Aug 2009 07:21:26 -0000 Hi I am using ixgbe 1.8.6 on FreeBSD 7.2-RELEASE (amd64). INT-64# sysctl -a | grep dev.ix | grep desc dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Vers= ion - 1.8.6 dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Vers= ion - 1.8.6 I see same strange big number of XON/XOFF Rcvd. ix0: XON Rcvd =3D 5828048552040 ix0: XON Xmtd =3D 0 XOFF Rcvd =3D 5828048552040 ix0: XOFF Xmtd =3D 0 Flow control disabled. INT-64# sysctl -a | grep dev.ix | grep flow_control dev.ix.0.flow_control: 0 dev.ix.1.flow_control: 0 Fabrizio > -----Original Message----- > From: owner-freebsd-performance@freebsd.org > [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Jack Vogel > Sent: marted=EC 18 agosto 2009 0.04 > To: Carlos Pardo > Cc: freebsd-performance@freebsd.org > Subject: Re: Test on 10GBE Intel based network card > > Who ya gonna call, why me of course, its my driver :) > > Hmmm, the numbers on those look bogus, like some > uninitialized variables. > You did say you aren't using flow control, right? > > Jack > > > On Mon, Aug 17, 2009 at 2:52 PM, Carlos Pardo > wrote: > > > Hi Jack, > > > > > > > > Thanks for the quick response. We can not use LRO because > of the way > > we accelerate on the WAN ports. We just moved from 7.0 to > 8.0 to use > > your latest driver (1.8.8). One thing we do not understand > in 8.0. We > > are having insane numbers for XON/XOFF Rcvd counters with > essentially no traffic. > > Driver version 1.2.16 works fine. Who should we contact for help? > > > > > > > > ix0: Std Mbuf Failed =3D 0 > > > > ix0: Missed Packets =3D 0 > > > > ix0: Receive length errors =3D 0 > > > > ix0: Crc errors =3D 0 > > > > ix0: Driver dropped packets =3D 0 > > > > ix0: watchdog timeouts =3D 0 > > > > *ix0: XON Rcvd =3D 7950055973552* > > > > ix0: XON Xmtd =3D 0 > > > > *ix0: XOFF Rcvd =3D 7950055973552* > > > > ix0: XOFF Xmtd =3D 0 > > > > ix0: Total Packets Rcvd =3D 2149 > > > > ix0: Good Packets Rcvd =3D 2149 > > > > ix0: Good Packets Xmtd =3D 1001 > > > > ix0: TSO Transmissions =3D 0 > > > > ix1: Std Mbuf Failed =3D 0 > > > > ix1: Missed Packets =3D 0 > > > > ix1: Receive length errors =3D 0 > > > > ix1: Crc errors =3D 0 > > > > ix1: Driver dropped packets =3D 0 > > > > ix1: watchdog timeouts =3D 0 > > > > *ix1: XON Rcvd =3D 7946320044993* > > > > ix1: XON Xmtd =3D 0 > > > > *ix1: XOFF Rcvd =3D 7946320044993* > > > > ix1: XOFF Xmtd =3D 0 > > > > ix1: Total Packets Rcvd =3D 1002 > > > > ix1: Good Packets Rcvd =3D 1002 > > > > ix1: Good Packets Xmtd =3D 1588 > > > > ix1: TSO Transmissions =3D 0 > > > > > > > > Regards, > > > > > > > > C Pardo > > > > > > > > *From:* Jack Vogel [mailto:jfvogel@gmail.com] > > *Sent:* Friday, August 14, 2009 3:15 PM > > *To:* Carlos Pardo > > *Cc:* freebsd-performance@freebsd.org > > *Subject:* Re: Test on 10GBE Intel based network card > > > > > > > > I've talked over the issues with the guy on our team who > has been most > > involved in 10G performance, he asserts that 3Gbs will saturate a > > single cpu with a small packet size, this is why you need > multiqueue > > across multiple cores. He was dubious about the FIFO > assertion, its a > > relative thing, if you can keep the thing drained it won't be a > > problem, doing that is a complex combination of factors, > the cpu, the bus, the memory.... > > > > If you don't deal with the systemic issues just cuz you go from an > > 82598 to a 82599 is not going to solve things. > > > > What about LRO, are/can you use that? I never saw an answer > about the > > forwarding question, you can't use LRO in that case of course. > > > > Regards, > > > > Jack > > > > On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo > wrote: > > > > Hi Jack, > > > > I have a quick question. We are getting too many missed packets per > > minute running about 3Gbs traffic. We can not use frame > control in our application. > > We are assuming that there is no way to improve upon the > problem since > > it seems to be a hardware limitation with the receive FIFO. We are > > using the Intel=AE 82598EB 10 Gigabit Ethernet Controller. > When can we > > expect the next generation card from Intel? Thanks for any > information you may provide. > > > > Typical error count "ix0: Missed Packets =3D 81174" after a > few minutes. > > > > Best Regards, > > > > Cpardo > > > > > > > > -----Original Message----- > > From: owner-freebsd-performance@freebsd.org [mailto: > > owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi > > Fabrizio > > > > Sent: Wednesday, August 05, 2009 3:13 AM > > To: Jack Vogel; Julian Elischer > > Cc: freebsd-performance@freebsd.org; Stefan Lambrev > > Subject: RE: Test on 10GBE Intel based network card > > > > No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver > > :<((((( > > > > ++fabrizio > > > > ------------------------------------------------------------------ > > Telecom Italia > > Fabrizio INVERNIZZI > > Technology - TILAB > > Accesso Fisso e Trasporto > > Via Reiss Romoli, 274 10148 Torino > > Tel. +39 011 2285497 > > Mob. +39 3316001344 > > Fax +39 06 41867287 > > > > > > ________________________________ > > From: Jack Vogel [mailto:jfvogel@gmail.com] > > Sent: marted=EC 4 agosto 2009 18.42 > > To: Julian Elischer > > Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan > > Lambrev > > Subject: Re: Test on 10GBE Intel based network card > > > > Your nmbclusters is very low, you list it twice so I'm assuming the > > second value is what it ends up being, 32K :( > > > > I would set it to: > > > > kern.ipc.nmbclusters=3D262144 > > > > Also, I thought you were using the current driver, but now it looks > > like you are using something fairly old, use my latest > code which is > > 1.8.8 > > > > Jack > > > > On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > > > wrote: > > Invernizzi Fabrizio wrote: > > The limitation that you see is about the max number of packets that > > FreeBSD can handle - it looks like your best performance is > reached at > > 64 byte packets? > > If you are meaning in term of Packet per second, you are > right. These > > are the packet per second measured during tests: > > 64 byte: 610119 Pps > > 512 byte: 516917 Pps > > 1492 byte: 464962 Pps > > > > > > Am I correct that the maximum you can reach is around > 639,000 packets > > per second? > > Yes, as you can see the maximum is 610119 Pps. > > Where does this limit come from? > > ah that's the whole point of tuning :-) there are > severalpossibities: > > 1/ the card's interrupts are probably attache dto aonly 1 > cpu, so that > > cpu can do no more work > > > > This seems not to be the problem. See below a top snapshot during a > > 64byte-long packet storm > > > > last pid: 8552; load averages: 0.40, 0.09, 0.03 > > up > > 0+20:36:58 09:40:29 > > 124 processes: 13 running, 73 sleeping, 38 waiting > > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, > 1.5% idle > > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M > > Free > > Swap: 2048M Total, 2048M Free > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME > WCPU COMMAND > > 11 root 1 171 ki31 0K 16K RUN 3 20.2H > 51.17% idle: > > cpu3 > > 14 root 1 171 ki31 0K 16K RUN 0 20.2H > 50.88% idle: > > cpu0 > > 12 root 1 171 ki31 0K 16K RUN 2 20.2H > 50.49% idle: > > cpu2 > > 13 root 1 171 ki31 0K 16K RUN 1 20.2H > 50.10% idle: > > cpu1 > > 42 root 1 -68 - 0K 16K RUN 1 14:20 > 36.47% ix0 rxq > > 38 root 1 -68 - 0K 16K CPU0 0 14:15 > 36.08% ix0 rxq > > 44 root 1 -68 - 0K 16K CPU2 2 14:08 > 34.47% ix0 rxq > > 40 root 1 -68 - 0K 16K CPU3 3 13:42 > 32.37% ix0 rxq > > .... > > > > It looks like the 4 rxq processes are bound to the 4 > available cores > > with equal distribution. > > > > > > > > 2/ if more than 1 cpu is working, it may be that there is a lock in > > heavy contention somewhere. > > > > This I think is the problem. I am trying to understand how to > > 1- see where the heavy contention is (context switching? > Some limiting > > setting?) > > 2- mitigate it > > > > > > > > there ia a lock profiling tool that right now I can't remember the > > name of.. > > > > look it up with google :-) FreeBSD lock profiling tool > > > > ah, first hit... > > > > http://blogs.epfl.ch/article/23832 > > > > > > > > is the machine still responsive to other networks while running at > > maximum capacity on this network? (make sure that the other > networks > > are on a differnet CPU (hmm I can't remember how to do that :-). > > > > > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > > alle persone indicate. La diffusione, copia o qualsiasi > altra azione > > derivante dalla conoscenza di queste informazioni sono > rigorosamente > > vietate. Qualora abbiate ricevuto questo documento per errore siete > > cortesemente pregati di darne immediata comunicazione al > mittente e di > > provvedere alla sua distruzione, Grazie. > > > > This e-mail and any attachments is confidential and may contain > > privileged information intended for the addressee(s) only. > > Dissemination, copying, printing or use by anybody else is > > unauthorised. If you are not the intended recipient, please delete > > this message and any attachments and advise the sender by > return e-mail, Thanks. > > > > _______________________________________________ > > > > > freebsd-performance@freebsd.org > > > > mailing list > > > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > > > To unsubscribe, send any mail to " > > freebsd-performance-unsubscribe@freebsd.org > freebsd-performance-unsubscribe@freebsd.org>" > > > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > > alle persone indicate. La diffusione, copia o qualsiasi > altra azione > > derivante dalla conoscenza di queste informazioni sono > rigorosamente > > vietate. Qualora abbiate ricevuto questo documento per errore siete > > cortesemente pregati di darne immediata comunicazione al > mittente e di > > provvedere alla sua distruzione, Grazie. > > > > This e-mail and any attachments is confidential and may contain > > privileged information intended for the addressee(s) only. > > Dissemination, copying, printing or use by anybody else is > > unauthorised. If you are not the intended recipient, please delete > > this message and any attachments and advise the sender by > return e-mail, Thanks. > > > > > [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta > l'ambiente. > > Non stampare questa mail se non =E8 necessario. > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Wed Aug 19 10:13:41 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81E00106568F for ; Wed, 19 Aug 2009 10:13:41 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG702BA020.telecomitalia.it (grfedg702ba020.telecomitalia.it [156.54.233.201]) by mx1.freebsd.org (Postfix) with ESMTP id D2C3B8FC15 for ; Wed, 19 Aug 2009 10:13:40 +0000 (UTC) Content-Type: multipart/mixed; boundary="_845582a1-7dd3-48da-9f34-4f3673aeb813_" Received: from GRFHUB701BA020.griffon.local (10.188.101.111) by GRFEDG702BA020.telecomitalia.it (10.188.45.101) with Microsoft SMTP Server (TLS) id 8.1.340.0; Wed, 19 Aug 2009 12:13:38 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by grfhub701ba020.griffon.local ([10.188.101.111]) with mapi; Wed, 19 Aug 2009 12:13:38 +0200 From: Invernizzi Fabrizio To: "freebsd-performance@freebsd.org" Date: Wed, 19 Aug 2009 12:13:37 +0200 Thread-Topic: Strange CPU distributionat very high level bandwidth Thread-Index: AcogtbfR8rs1r71cRa6H+ejVJ74Tug== Message-ID: <36A93B31228D3B49B691AD31652BCAE9A456967AF4@GRFMBX702BA020.griffon.local> Accept-Language: it-IT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT, en-US MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Strange CPU distributionat very high level bandwidth X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Aug 2009 10:13:41 -0000 --_845582a1-7dd3-48da-9f34-4f3673aeb813_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all i am going on with some performance tests on a 10gbe network card with Free= BSD. I am doing this test: I send UDP traffic to be forwarded to the other port = of the card on both the card ports. Using 1492-long packets i am uppering the number of packets per second i s= ent In order to see wich is the maximum bandwidth (or pps) the system can s= upport without losses. The limit seems to be about 1890Mbps per port (3870 Mbps total). Looking more in deep the CPU behaviour i see this : - uppering the sent pps results in uppering the intterrupt time (about 90= %) - when i am very strict to the limit, interrupt time falls to about 10% a= nd CPU is always (85%) in system (rx/tx driver procedure) Questions: - Is not the AIM intended to contrast this behaviour to limit interrupts se= nt to CPU? (nothing changes if i disable it) - Why does the system start loosing pkts in that condition? - Why does the system seem to perform better when it is managing more conte= xt switches? These are my system details: - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 cache = size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core installed. - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit Ethern= et PCI Express Server Adapter Intel(r) based (chip 82598EB). - FreeBSD 7.2-RELEASE (64 bit) Driver ixgbe-1.8.6 hw.intr_storm_threshold:2000000 dev.ix.0.low_latency: 128 dev.ix.0.ave_latency: 400 dev.ix.0.bulk_latency: 1200 dev.ix.1.low_latency: 128 dev.ix.1.ave_latency: 400 dev.ix.1.bulk_latency: 1200 ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. No= n stampare questa mail se non ? necessario. --_845582a1-7dd3-48da-9f34-4f3673aeb813_-- From owner-freebsd-performance@FreeBSD.ORG Wed Aug 19 11:14:00 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB5C7106568B for ; Wed, 19 Aug 2009 11:14:00 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 7DD608FC6B for ; Wed, 19 Aug 2009 11:14:00 +0000 (UTC) Received: from [217.150.130.134] (helo=unknown) by marvin.harmless.hu with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Mdiqr-000Ado-9B; Wed, 19 Aug 2009 12:57:09 +0200 Date: Wed, 19 Aug 2009 12:57:07 +0200 From: Gergely CZUCZY To: Invernizzi Fabrizio Message-ID: <20090819125707.0000396e@unknown> In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A456967AF4@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A456967AF4@GRFMBX702BA020.griffon.local> Organization: Harmless Digital Bt X-Mailer: Claws Mail 3.7.1 (GTK+ 2.16.0; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: "freebsd-performance@freebsd.org" Subject: Re: Strange CPU distributionat very high level bandwidth X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Aug 2009 11:14:00 -0000 Hello, Just a question. May I ask how many pps is this traffic (packet per second). Forward performance actually depends on the pps rate and not on the bandwidth usage as far as my experience goes. As I've calculated by your given data, it might be around 166Kpps, but i might be wrong there. On Wed, 19 Aug 2009 12:13:37 +0200 Invernizzi Fabrizio wrote: > Hi all > > i am going on with some performance tests on a 10gbe network card > with FreeBSD. > > I am doing this test: I send UDP traffic to be forwarded to the other > port of the card on both the card ports. Using 1492-long packets i am > uppering the number of packets per second i sent In order to see > wich is the maximum bandwidth (or pps) the system can support without > losses. > > The limit seems to be about 1890Mbps per port (3870 Mbps total). > Looking more in deep the CPU behaviour i see this : > - uppering the sent pps results in uppering the intterrupt time > (about 90%) > - when i am very strict to the limit, interrupt time falls to about > 10% and CPU is always (85%) in system (rx/tx driver procedure) > > Questions: > - Is not the AIM intended to contrast this behaviour to limit > interrupts sent to CPU? (nothing changes if i disable it) > - Why does the system start loosing pkts in that condition? > - Why does the system seem to perform better when it is managing more > context switches? > > > > These are my system details: > > - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: 1333 MHz, L2 > cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core > installed. > > - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) 10 Gigabit > Ethernet PCI Express Server Adapter Intel(r) based (chip 82598EB). > > - FreeBSD 7.2-RELEASE (64 bit) > > Driver ixgbe-1.8.6 > > hw.intr_storm_threshold:2000000 > > dev.ix.0.low_latency: 128 > dev.ix.0.ave_latency: 400 > dev.ix.0.bulk_latency: 1200 > dev.ix.1.low_latency: 128 > dev.ix.1.ave_latency: 400 > dev.ix.1.bulk_latency: 1200 > > ------------------------------------------------------------------ > Telecom Italia > Fabrizio INVERNIZZI > Technology - TILAB > Accesso Fisso e Trasporto > Via Reiss Romoli, 274 10148 Torino > Tel. +39 011 2285497 > Mob. +39 3316001344 > Fax +39 06 41867287 > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > alle persone indicate. La diffusione, copia o qualsiasi altra azione > derivante dalla conoscenza di queste informazioni sono rigorosamente > vietate. Qualora abbiate ricevuto questo documento per errore siete > cortesemente pregati di darne immediata comunicazione al mittente e > di provvedere alla sua distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain > privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is > unauthorised. If you are not the intended recipient, please delete > this message and any attachments and advise the sender by return > e-mail, Thanks. > > [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta > l'ambiente. Non stampare questa mail se non ? necessario. > > -- Sincerely, Gergely CZUCZY Harmless Digital Bt +36-30-9702963 From owner-freebsd-performance@FreeBSD.ORG Wed Aug 19 11:24:14 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 566311065690 for ; Wed, 19 Aug 2009 11:24:14 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG702BA020.telecomitalia.it (grfedg702ba020.telecomitalia.it [156.54.233.201]) by mx1.freebsd.org (Postfix) with ESMTP id D81988FC6D for ; Wed, 19 Aug 2009 11:24:13 +0000 (UTC) Received: from GRFHUB702BA020.griffon.local (10.188.101.112) by GRFEDG702BA020.telecomitalia.it (10.188.45.101) with Microsoft SMTP Server (TLS) id 8.1.340.0; Wed, 19 Aug 2009 13:24:12 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by GRFHUB702BA020.griffon.local ([10.188.101.112]) with mapi; Wed, 19 Aug 2009 13:24:12 +0200 From: Invernizzi Fabrizio To: Gergely CZUCZY Date: Wed, 19 Aug 2009 13:24:09 +0200 Thread-Topic: Strange CPU distributionat very high level bandwidth Thread-Index: Acogu9Nt7FwB6np+SGWf+gFU3I3n+wAA2Rnw Message-ID: <36A93B31228D3B49B691AD31652BCAE9A456967AFF@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A456967AF4@GRFMBX702BA020.griffon.local> <20090819125707.0000396e@unknown> In-Reply-To: <20090819125707.0000396e@unknown> Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-performance@freebsd.org" Subject: RE: Strange CPU distributionat very high level bandwidth X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Aug 2009 11:24:14 -0000 There is an error in my previous email: the upper limit I see is 5722 Mbps = (total). I am sending 239700 pps per interface, 1492-bytes long. Fabrizio > -----Original Message----- > From: Gergely CZUCZY [mailto:phoemix@harmless.hu] > Sent: mercoled=EC 19 agosto 2009 12.57 > To: Invernizzi Fabrizio > Cc: freebsd-performance@freebsd.org > Subject: Re: Strange CPU distributionat very high level bandwidth > > Hello, > > Just a question. May I ask how many pps is this traffic > (packet per second). Forward performance actually depends on > the pps rate and not on the bandwidth usage as far as my > experience goes. As I've calculated by your given data, it > might be around 166Kpps, but i might be wrong there. > > On Wed, 19 Aug 2009 12:13:37 +0200 > Invernizzi Fabrizio wrote: > > > Hi all > > > > i am going on with some performance tests on a 10gbe > network card with > > FreeBSD. > > > > I am doing this test: I send UDP traffic to be forwarded to > the other > > port of the card on both the card ports. Using 1492-long > packets i am > > uppering the number of packets per second i sent In order > to see wich > > is the maximum bandwidth (or pps) the system can support without > > losses. > > > > The limit seems to be about 1890Mbps per port (3870 Mbps total). > > Looking more in deep the CPU behaviour i see this : > > - uppering the sent pps results in uppering the intterrupt time > > (about 90%) > > - when i am very strict to the limit, interrupt time > falls to about > > 10% and CPU is always (85%) in system (rx/tx driver procedure) > > > > Questions: > > - Is not the AIM intended to contrast this behaviour to limit > > interrupts sent to CPU? (nothing changes if i disable it) > > - Why does the system start loosing pkts in that condition? > > - Why does the system seem to perform better when it is > managing more > > context switches? > > > > > > > > These are my system details: > > > > - HP 380 G5 (XEON X5420, CPU speed: 2.50GHz, BUS speed: > 1333 MHz, L2 > > cache size: 12 MB, L2 cache speed: 2,5 GHz) with 1 quad-core > > installed. > > > > - Network card: Silicom PE10G2i-LR - Dual Port Fiber (LR) > 10 Gigabit > > Ethernet PCI Express Server Adapter Intel(r) based (chip 82598EB). > > > > - FreeBSD 7.2-RELEASE (64 bit) > > > > Driver ixgbe-1.8.6 > > > > hw.intr_storm_threshold:2000000 > > > > dev.ix.0.low_latency: 128 > > dev.ix.0.ave_latency: 400 > > dev.ix.0.bulk_latency: 1200 > > dev.ix.1.low_latency: 128 > > dev.ix.1.ave_latency: 400 > > dev.ix.1.bulk_latency: 1200 > > > > ------------------------------------------------------------------ > > Telecom Italia > > Fabrizio INVERNIZZI > > Technology - TILAB > > Accesso Fisso e Trasporto > > Via Reiss Romoli, 274 10148 Torino > > Tel. +39 011 2285497 > > Mob. +39 3316001344 > > Fax +39 06 41867287 > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > > alle persone indicate. La diffusione, copia o qualsiasi > altra azione > > derivante dalla conoscenza di queste informazioni sono > rigorosamente > > vietate. Qualora abbiate ricevuto questo documento per errore siete > > cortesemente pregati di darne immediata comunicazione al > mittente e di > > provvedere alla sua distruzione, Grazie. > > > > This e-mail and any attachments is confidential and may contain > > privileged information intended for the addressee(s) only. > > Dissemination, copying, printing or use by anybody else is > > unauthorised. If you are not the intended recipient, please delete > > this message and any attachments and advise the sender by return > > e-mail, Thanks. > > > > [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta > > l'ambiente. Non stampare questa mail se non ? necessario. > > > > > > > > -- > Sincerely, > Gergely CZUCZY > Harmless Digital Bt > > +36-30-9702963 > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From owner-freebsd-performance@FreeBSD.ORG Thu Aug 20 23:35:48 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8AC061065672 for ; Thu, 20 Aug 2009 23:35:48 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 14F308FC57 for ; Thu, 20 Aug 2009 23:35:47 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1MeGpY-0000Nh-SI for freebsd-performance@freebsd.org; Fri, 21 Aug 2009 01:14:04 +0200 Received: from 93-141-109-227.adsl.net.t-com.hr ([93.141.109.227]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 21 Aug 2009 01:14:04 +0200 Received: from ivoras by 93-141-109-227.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 21 Aug 2009 01:14:04 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Fri, 21 Aug 2009 01:13:33 +0200 Lines: 78 Message-ID: References: <36A93B31228D3B49B691AD31652BCAE9A456967AF4@GRFMBX702BA020.griffon.local> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig1D5DFF69EFFC281F77A992CB" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 93-141-109-227.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.22 (Windows/20090605) In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A456967AF4@GRFMBX702BA020.griffon.local> X-Enigmail-Version: 0.96.0 Sender: news Subject: Re: Strange CPU distributionat very high level bandwidth X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Aug 2009 23:35:48 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1D5DFF69EFFC281F77A992CB Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Invernizzi Fabrizio wrote: > Hi all >=20 > i am going on with some performance tests on a 10gbe network card with = FreeBSD. >=20 > I am doing this test: I send UDP traffic to be forwarded to the other p= ort of the card on both the card ports. > Using 1492-long packets i am uppering the number of packets per second= i sent In order to see wich is the maximum bandwidth (or pps) the system= can support without losses. >=20 > The limit seems to be about 1890Mbps per port (3870 Mbps total). > Looking more in deep the CPU behaviour i see this : > - uppering the sent pps results in uppering the intterrupt time (abou= t 90%) > - when i am very strict to the limit, interrupt time falls to about 1= 0% and CPU is always (85%) in system (rx/tx driver procedure) >=20 > Questions: > - Is not the AIM intended to contrast this behaviour to limit interrupt= s sent to CPU? (nothing changes if i disable it) > - Why does the system start loosing pkts in that condition? > - Why does the system seem to perform better when it is managing more c= ontext switches? >=20 > - FreeBSD 7.2-RELEASE (64 bit) One idea for you, not directly tied to forwarding as is but to the recent development of multithreaded packet acceptance code, is to use 8.x (currently in BETA so usual precautions about debugging being enabled apply) and then play with netisr and worker thread settings. See the source here: http://svn.freebsd.org/viewvc/base/head/sys/net/netisr.c?view=3Dmarkup&pa= threv=3D195078 and the comments starting at "Three direct dispatch policies are supporte= d". The code is experimental and thus disabled in 8.0 unless a combination of the following loader tunables are set: net.isr.direct_force net.isr.direct net.isr.maxthreads net.isr.bindthreads I think you can start simply by turning off net.isr.direct_force and then start increasing net.isr.maxthreads until the benefits (if any) go away. Since it is experimental code, your benchmarks would be nice to hav= e. --------------enig1D5DFF69EFFC281F77A992CB Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqN2J0ACgkQldnAQVacBciBiACgwx2WawE6JmJmGzKuF8Vy0lMq 3jkAoIfLa5mWZ+DV87UxyHOzojxozWe7 =eyO+ -----END PGP SIGNATURE----- --------------enig1D5DFF69EFFC281F77A992CB-- From owner-freebsd-performance@FreeBSD.ORG Fri Aug 21 08:15:55 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF862106568E for ; Fri, 21 Aug 2009 08:15:55 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG702BA020.telecomitalia.it (grfedg702ba020.telecomitalia.it [156.54.233.201]) by mx1.freebsd.org (Postfix) with ESMTP id 47BFB8FC62 for ; Fri, 21 Aug 2009 08:15:55 +0000 (UTC) Received: from GRFHUB702BA020.griffon.local (10.188.101.112) by GRFEDG702BA020.telecomitalia.it (10.188.45.101) with Microsoft SMTP Server (TLS) id 8.1.340.0; Fri, 21 Aug 2009 10:15:52 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.11]) by GRFHUB702BA020.griffon.local ([10.188.101.112]) with mapi; Fri, 21 Aug 2009 10:15:51 +0200 From: Invernizzi Fabrizio To: Ivan Voras , "freebsd-performance@freebsd.org" Date: Fri, 21 Aug 2009 10:15:46 +0200 Thread-Topic: Strange CPU distributionat very high level bandwidth Thread-Index: Acoh7wlzPrst/QZzTueUdgkiEjVfgAASDhUQ Message-ID: <36A93B31228D3B49B691AD31652BCAE9A45770696C@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A456967AF4@GRFMBX702BA020.griffon.local> In-Reply-To: Accept-Language: it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Subject: RE: Strange CPU distributionat very high level bandwidth X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Aug 2009 08:15:55 -0000 Thanks for your suggestion. I hope to have time to do some tests on 8.0 and send some result on the ML= next week. ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 > -----Original Message----- > From: owner-freebsd-performance@freebsd.org > [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Ivan Voras > Sent: venerd=EC 21 agosto 2009 1.14 > To: freebsd-performance@freebsd.org > Subject: Re: Strange CPU distributionat very high level bandwidth > > Invernizzi Fabrizio wrote: > > Hi all > > > > i am going on with some performance tests on a 10gbe > network card with FreeBSD. > > > > I am doing this test: I send UDP traffic to be forwarded to > the other port of the card on both the card ports. > > Using 1492-long packets i am uppering the number of > packets per second i sent In order to see wich is the maximum > bandwidth (or pps) the system can support without losses. > > > > The limit seems to be about 1890Mbps per port (3870 Mbps total). > > Looking more in deep the CPU behaviour i see this : > > - uppering the sent pps results in uppering the > intterrupt time (about 90%) > > - when i am very strict to the limit, interrupt time > falls to about > > 10% and CPU is always (85%) in system (rx/tx driver procedure) > > > > Questions: > > - Is not the AIM intended to contrast this behaviour to limit > > interrupts sent to CPU? (nothing changes if i disable it) > > - Why does the system start loosing pkts in that condition? > > - Why does the system seem to perform better when it is > managing more context switches? > > > > > - FreeBSD 7.2-RELEASE (64 bit) > > One idea for you, not directly tied to forwarding as is but > to the recent development of multithreaded packet acceptance > code, is to use 8.x (currently in BETA so usual precautions > about debugging being enabled apply) and then play with > netisr and worker thread settings. > > See the source here: > > http://svn.freebsd.org/viewvc/base/head/sys/net/netisr.c?view=3D markup&pathrev=3D195078 > > and the comments starting at "Three direct dispatch policies > are supported". > > The code is experimental and thus disabled in 8.0 unless a > combination of the following loader tunables are set: > > net.isr.direct_force > net.isr.direct > net.isr.maxthreads > net.isr.bindthreads > > I think you can start simply by turning off > net.isr.direct_force and then start increasing > net.isr.maxthreads until the benefits (if any) go away. Since > it is experimental code, your benchmarks would be nice to have. > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks.