From owner-freebsd-performance@FreeBSD.ORG Fri Aug 14 22:15:30 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7726106568C for ; Fri, 14 Aug 2009 22:15:30 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yw0-f199.google.com (mail-yw0-f199.google.com [209.85.211.199]) by mx1.freebsd.org (Postfix) with ESMTP id 1C7E18FC61 for ; Fri, 14 Aug 2009 22:15:29 +0000 (UTC) Received: by ywh37 with SMTP id 37so2647130ywh.28 for ; Fri, 14 Aug 2009 15:15:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=rPcxQCDy1wYA7ChG/LFrNBvmNRAlck6eK7nQzs3JiaY=; b=ZHb92VapXs9cRSUqZ+zdNYZXzZAfC7bOYNe2y7mHa7i8RGC5vjW7JRV0FbD8LCnfcW V63Gz2qEX8dfTL9PrN9dUnEgKQwlC/yoWsLTxeVaDn8mWQs9PzUPcoNgq4Nw3MS1KntH xS65HqFxLjxyqA8C8SG0rPlAVwGi2SeqBa8fY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=OoCZGztgbZbtA4dOQDMmlrI7CxedWyG8fscQjgHklUPgJDTTNr4O9xBIeM4qauK3Go ege5qxBoHiM6lzBDlklvNno2zKgN2Kutd+M0k1k6UkBzF5G19ho8kHABqKPXhe8k9Esi OsWFxKIxIaCJRLD2/RvW/necimLGANtQSpWcw= MIME-Version: 1.0 Received: by 10.100.107.3 with SMTP id f3mr2143056anc.87.1250288129258; Fri, 14 Aug 2009 15:15:29 -0700 (PDT) In-Reply-To: References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> Date: Fri, 14 Aug 2009 15:15:29 -0700 Message-ID: <2a41acea0908141515o45b7c74g40dbddc32d4b754b@mail.gmail.com> From: Jack Vogel To: Carlos Pardo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2009 22:15:30 -0000 I've talked over the issues with the guy on our team who has been most involved in 10G performance, he asserts that 3Gbs will saturate a single cpu with a small packet size, this is why you need multiqueue across multiple cores. He was dubious about the FIFO assertion, its a relative thing, if you can keep the thing drained it won't be a problem, doing that is a complex combination of factors, the cpu, the bus, the memory.... If you don't deal with the systemic issues just cuz you go from an 82598 to a 82599 is not going to solve things. What about LRO, are/can you use that? I never saw an answer about the forwarding question, you can't use LRO in that case of course. Regards, Jack On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo wrote: > Hi Jack, > > I have a quick question. We are getting too many missed packets per minut= e > running about 3Gbs traffic. We can not use frame control in our applicati= on. > We are assuming that there is no way to improve upon the problem since it > seems to be a hardware limitation with the receive FIFO. We are using the > Intel=AE 82598EB 10 Gigabit Ethernet Controller. When can we expect the n= ext > generation card from Intel? Thanks for any information you may provide. > > Typical error count "ix0: Missed Packets =3D 81174" after a few minutes. > > Best Regards, > > Cpardo > > > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto: > owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi Fabrizio > Sent: Wednesday, August 05, 2009 3:13 AM > To: Jack Vogel; Julian Elischer > Cc: freebsd-performance@freebsd.org; Stefan Lambrev > Subject: RE: Test on 10GBE Intel based network card > > No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver :<((((= ( > > ++fabrizio > > ------------------------------------------------------------------ > Telecom Italia > Fabrizio INVERNIZZI > Technology - TILAB > Accesso Fisso e Trasporto > Via Reiss Romoli, 274 10148 Torino > Tel. +39 011 2285497 > Mob. +39 3316001344 > Fax +39 06 41867287 > > > ________________________________ > From: Jack Vogel [mailto:jfvogel@gmail.com] > Sent: marted=EC 4 agosto 2009 18.42 > To: Julian Elischer > Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev > Subject: Re: Test on 10GBE Intel based network card > > Your nmbclusters is very low, you list it twice so I'm assuming the secon= d > value is > what it ends up being, 32K :( > > I would set it to: > > kern.ipc.nmbclusters=3D262144 > > Also, I thought you were using the current driver, but now it looks like > you are > using something fairly old, use my latest code which is 1.8.8 > > Jack > > > On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > wrote: > Invernizzi Fabrizio wrote: > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? > If you are meaning in term of Packet per second, you are right. These > are the packet per second measured during tests: > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > Am I correct that the maximum you can reach is around 639,000 packets > per second? > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > ah that's the whole point of tuning :-) > there are severalpossibities: > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a > 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 > up > 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAN= D > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: > cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: > cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: > cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: > cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with > equal distribution. > > > > 2/ if more than 1 cpu is working, it may be that there is a lock in > heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting > setting?) > 2- mitigate it > > > > there ia a lock profiling tool that right now I can't remember the name > of.. > > look it up with google :-) FreeBSD lock profiling tool > > ah, first hit... > > http://blogs.epfl.ch/article/23832 > > > > is the machine still responsive to other networks while > running at maximum capacity on this network? (make sure that > the other networks are on a differnet CPU (hmm I can't remember how to > do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org freebsd-performance-unsubscribe@freebsd.org>" > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. > Non stampare questa mail se non =E8 necessario. > >