Date: Mon, 17 Aug 2009 15:03:36 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Carlos Pardo <cpardo@fastsoft.com> Cc: freebsd-performance@freebsd.org Subject: Re: Test on 10GBE Intel based network card Message-ID: <2a41acea0908171503r3613d430ib154cd3445eb1309@mail.gmail.com> In-Reply-To: <D13CB108B048BD47B69C0CA1E0B5C032CF1D94@hq-es.FASTSOFT.COM> References: <D13CB108B048BD47B69C0CA1E0B5C032CF1D94@hq-es.FASTSOFT.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
Who ya gonna call, why me of course, its my driver :) Hmmm, the numbers on those look bogus, like some uninitialized variables. You did say you aren't using flow control, right? Jack On Mon, Aug 17, 2009 at 2:52 PM, Carlos Pardo <cpardo@fastsoft.com> wrote: > Hi Jack, > > > > Thanks for the quick response. We can not use LRO because of the way we > accelerate on the WAN ports. We just moved from 7.0 to 8.0 to use your > latest driver (1.8.8). One thing we do not understand in 8.0. We are havi= ng > insane numbers for XON/XOFF Rcvd counters with essentially no traffic. > Driver version 1.2.16 works fine. Who should we contact for help? > > > > ix0: Std Mbuf Failed =3D 0 > > ix0: Missed Packets =3D 0 > > ix0: Receive length errors =3D 0 > > ix0: Crc errors =3D 0 > > ix0: Driver dropped packets =3D 0 > > ix0: watchdog timeouts =3D 0 > > *ix0: XON Rcvd =3D 7950055973552* > > ix0: XON Xmtd =3D 0 > > *ix0: XOFF Rcvd =3D 7950055973552* > > ix0: XOFF Xmtd =3D 0 > > ix0: Total Packets Rcvd =3D 2149 > > ix0: Good Packets Rcvd =3D 2149 > > ix0: Good Packets Xmtd =3D 1001 > > ix0: TSO Transmissions =3D 0 > > ix1: Std Mbuf Failed =3D 0 > > ix1: Missed Packets =3D 0 > > ix1: Receive length errors =3D 0 > > ix1: Crc errors =3D 0 > > ix1: Driver dropped packets =3D 0 > > ix1: watchdog timeouts =3D 0 > > *ix1: XON Rcvd =3D 7946320044993* > > ix1: XON Xmtd =3D 0 > > *ix1: XOFF Rcvd =3D 7946320044993* > > ix1: XOFF Xmtd =3D 0 > > ix1: Total Packets Rcvd =3D 1002 > > ix1: Good Packets Rcvd =3D 1002 > > ix1: Good Packets Xmtd =3D 1588 > > ix1: TSO Transmissions =3D 0 > > > > Regards, > > > > C Pardo > > > > *From:* Jack Vogel [mailto:jfvogel@gmail.com] > *Sent:* Friday, August 14, 2009 3:15 PM > *To:* Carlos Pardo > *Cc:* freebsd-performance@freebsd.org > *Subject:* Re: Test on 10GBE Intel based network card > > > > I've talked over the issues with the guy on our team who has been most > involved in 10G performance, he asserts that 3Gbs will saturate a single > cpu with a small packet size, this is why you need multiqueue across > multiple cores. He was dubious about the FIFO assertion, its a relative > thing, if you can keep the thing drained it won't be a problem, doing tha= t > is a complex combination of factors, the cpu, the bus, the memory.... > > If you don't deal with the systemic issues just cuz you go from an 82598 > to a 82599 is not going to solve things. > > What about LRO, are/can you use that? I never saw an answer about the > forwarding question, you can't use LRO in that case of course. > > Regards, > > Jack > > On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo <cpardo@fastsoft.com> wrote= : > > Hi Jack, > > I have a quick question. We are getting too many missed packets per minut= e > running about 3Gbs traffic. We can not use frame control in our applicati= on. > We are assuming that there is no way to improve upon the problem since it > seems to be a hardware limitation with the receive FIFO. We are using the > Intel=AE 82598EB 10 Gigabit Ethernet Controller. When can we expect the n= ext > generation card from Intel? Thanks for any information you may provide. > > Typical error count "ix0: Missed Packets =3D 81174" after a few minutes. > > Best Regards, > > Cpardo > > > > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto: > owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi Fabrizio > > Sent: Wednesday, August 05, 2009 3:13 AM > To: Jack Vogel; Julian Elischer > Cc: freebsd-performance@freebsd.org; Stefan Lambrev > Subject: RE: Test on 10GBE Intel based network card > > No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver :<((((= ( > > ++fabrizio > > ------------------------------------------------------------------ > Telecom Italia > Fabrizio INVERNIZZI > Technology - TILAB > Accesso Fisso e Trasporto > Via Reiss Romoli, 274 10148 Torino > Tel. +39 011 2285497 > Mob. +39 3316001344 > Fax +39 06 41867287 > > > ________________________________ > From: Jack Vogel [mailto:jfvogel@gmail.com] > Sent: marted=EC 4 agosto 2009 18.42 > To: Julian Elischer > Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev > Subject: Re: Test on 10GBE Intel based network card > > Your nmbclusters is very low, you list it twice so I'm assuming the secon= d > value is > what it ends up being, 32K :( > > I would set it to: > > kern.ipc.nmbclusters=3D262144 > > Also, I thought you were using the current driver, but now it looks like > you are > using something fairly old, use my latest code which is 1.8.8 > > Jack > > On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian@elischer.org > <mailto:julian@elischer.org>> wrote: > Invernizzi Fabrizio wrote: > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? > If you are meaning in term of Packet per second, you are right. These > are the packet per second measured during tests: > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > Am I correct that the maximum you can reach is around 639,000 packets > per second? > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > ah that's the whole point of tuning :-) > there are severalpossibities: > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a > 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 > up > 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAN= D > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: > cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: > cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: > cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: > cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with > equal distribution. > > > > 2/ if more than 1 cpu is working, it may be that there is a lock in > heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting > setting?) > 2- mitigate it > > > > there ia a lock profiling tool that right now I can't remember the name > of.. > > look it up with google :-) FreeBSD lock profiling tool > > ah, first hit... > > http://blogs.epfl.ch/article/23832 > > > > is the machine still responsive to other networks while > running at maximum capacity on this network? (make sure that > the other networks are on a differnet CPU (hmm I can't remember how to > do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > > freebsd-performance@freebsd.org<mailto:freebsd-performance@freebsd.org> > mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org<mailto: > freebsd-performance-unsubscribe@freebsd.org>" > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. > Non stampare questa mail se non =E8 necessario. > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2a41acea0908171503r3613d430ib154cd3445eb1309>