From owner-freebsd-performance@FreeBSD.ORG Tue Aug 4 07:55:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97D3E10656A5 for ; Tue, 4 Aug 2009 07:55:18 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFEDG701BA020.telecomitalia.it (grfedg701ba020.telecomitalia.it [156.54.233.200]) by mx1.freebsd.org (Postfix) with ESMTP id 18DEA8FC16 for ; Tue, 4 Aug 2009 07:55:18 +0000 (UTC) (envelope-from fabrizio.invernizzi@telecomitalia.it) Received: from GRFHUB701BA020.griffon.local (10.188.101.111) by GRFEDG701BA020.telecomitalia.it (10.188.45.100) with Microsoft SMTP Server (TLS) id 8.1.340.0; Tue, 4 Aug 2009 09:55:16 +0200 Received: from GRFMBX702BA020.griffon.local ([10.188.101.12]) by grfhub701ba020.griffon.local ([10.188.101.111]) with mapi; Tue, 4 Aug 2009 09:55:14 +0200 From: Invernizzi Fabrizio To: Julian Elischer Date: Tue, 4 Aug 2009 09:55:13 +0200 Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoUU1WjCypibGa2RsmNsqF8ZFQatgAhLlrw Message-ID: <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> In-Reply-To: <4A77094C.8030308@elischer.org> Accept-Language: it-IT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: it-IT, en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-performance@freebsd.org" , Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 07:55:19 -0000 > >> The limitation that you see is about the max number of packets that > >> FreeBSD can handle - it looks like your best performance is reached at > >> 64 byte packets? > > > > If you are meaning in term of Packet per second, you are right. These > are the packet per second measured during tests: > > > > 64 byte: 610119 Pps > > 512 byte: 516917 Pps > > 1492 byte: 464962 Pps > > > > > >> Am I correct that the maximum you can reach is around 639,000 packets > >> per second? > > > > Yes, as you can see the maximum is 610119 Pps. > > Where does this limit come from? > > ah that's the whole point of tuning :-) > there are severalpossibities: > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a 64byte-= long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 = up 0+20:3= 6:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: c= pu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: c= pu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: c= pu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: c= pu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq .... It looks like the 4 rxq processes are bound to the 4 available cores with e= qual distribution. > 2/ if more than 1 cpu is working, it may be that there is a lock in > heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting sett= ing?) 2- mitigate it > > is the machine still responsive to other networks while > running at maximum capacity on this network? (make sure that > the other networks are on a differnet CPU (hmm I can't remember how to > do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks.