Date: Wed, 5 Aug 2009 12:13:23 +0200 From: Invernizzi Fabrizio <fabrizio.invernizzi@telecomitalia.it> To: Jack Vogel <jfvogel@gmail.com>, Julian Elischer <julian@elischer.org> Cc: "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org>, Stefan Lambrev <stefan.lambrev@moneybookers.com> Subject: RE: Test on 10GBE Intel based network card Message-ID: <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> In-Reply-To: <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--_f70d3edc-3e50-4297-b616-09fd857ffdd3_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver :<(((((
++fabrizio
------------------------------------------------------------------
Telecom Italia
Fabrizio INVERNIZZI
Technology - TILAB
Accesso Fisso e Trasporto
Via Reiss Romoli, 274 10148 Torino
Tel. +39 011 2285497
Mob. +39 3316001344
Fax +39 06 41867287
________________________________
From: Jack Vogel [mailto:jfvogel@gmail.com]
Sent: marted=EC 4 agosto 2009 18.42
To: Julian Elischer
Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev
Subject: Re: Test on 10GBE Intel based network card
Your nmbclusters is very low, you list it twice so I'm assuming the second =
value is
what it ends up being, 32K :(
I would set it to:
kern.ipc.nmbclusters=3D262144
Also, I thought you were using the current driver, but now it looks like yo=
u are
using something fairly old, use my latest code which is 1.8.8
Jack
On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian@elischer.org<mailto=
:julian@elischer.org>> wrote:
Invernizzi Fabrizio wrote:
The limitation that you see is about the max number of packets that
FreeBSD can handle - it looks like your best performance is reached at
64 byte packets?
If you are meaning in term of Packet per second, you are right. These
are the packet per second measured during tests:
64 byte: 610119 Pps
512 byte: 516917 Pps
1492 byte: 464962 Pps
Am I correct that the maximum you can reach is around 639,000 packets
per second?
Yes, as you can see the maximum is 610119 Pps.
Where does this limit come from?
ah that's the whole point of tuning :-)
there are severalpossibities:
1/ the card's interrupts are probably attache dto aonly 1 cpu,
so that cpu can do no more work
This seems not to be the problem. See below a top snapshot during a 64byte-=
long packet storm
last pid: 8552; load averages: 0.40, 0.09, 0.03 =
up 0+20:3=
6:58 09:40:29
124 processes: 13 running, 73 sleeping, 38 waiting
CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle
Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free
Swap: 2048M Total, 2048M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cp=
u3
14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cp=
u0
12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cp=
u2
13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cp=
u1
42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq
38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq
44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq
40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq
....
It looks like the 4 rxq processes are bound to the 4 available cores with e=
qual distribution.
2/ if more than 1 cpu is working, it may be that there is a lock in
heavy contention somewhere.
This I think is the problem. I am trying to understand how to
1- see where the heavy contention is (context switching? Some limiting sett=
ing?)
2- mitigate it
there ia a lock profiling tool that right now I can't remember the name of.=
.
look it up with google :-) FreeBSD lock profiling tool
ah, first hit...
http://blogs.epfl.ch/article/23832
is the machine still responsive to other networks while
running at maximum capacity on this network? (make sure that
the other networks are on a differnet CPU (hmm I can't remember how to
do that :-).
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per=
sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall=
a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb=
iate ricevuto questo documento per errore siete cortesemente pregati di dar=
ne immediata comunicazione al mittente e di provvedere alla sua distruzione=
, Grazie.
This e-mail and any attachments is confidential and may contain privileged =
information intended for the addressee(s) only. Dissemination, copying, pri=
nting or use by anybody else is unauthorised. If you are not the intended r=
ecipient, please delete this message and any attachments and advise the sen=
der by return e-mail, Thanks.
_______________________________________________
freebsd-performance@freebsd.org<mailto:freebsd-performance@freebsd.org> mai=
ling list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.o=
rg<mailto:freebsd-performance-unsubscribe@freebsd.org>"
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per=
sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall=
a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb=
iate ricevuto questo documento per errore siete cortesemente pregati di dar=
ne immediata comunicazione al mittente e di provvedere alla sua distruzione=
, Grazie.
This e-mail and any attachments is confidential and may contain privileged =
information intended for the addressee(s) only. Dissemination, copying, pri=
nting or use by anybody else is unauthorised. If you are not the intended r=
ecipient, please delete this message and any attachments and advise the sen=
der by return e-mail, Thanks.
[cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. No=
n stampare questa mail se non =E8 necessario.
--_f70d3edc-3e50-4297-b616-09fd857ffdd3_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?36A93B31228D3B49B691AD31652BCAE9A45696743F>