Skip site navigation (1)Skip section navigation (2) 
Date:      Wed, 03 Oct 2007 14:43:18 +0400
From:      Vladimir Ivanov <wawa@yandex-team.ru>
To:        Bruce Evans <brde@optusnet.com.au>
Cc:        "freebsd-net@freebsd.org" <freebsd-net@FreeBSD.org>, Jack Vogel <jfvogel@gmail.com>
Subject:   Re: SMPable version of EM driver
Message-ID:  <47037246.2070400@yandex-team.ru>
In-Reply-To: <20071003111737.U14276@delplex.bde.org>
References:  <46B07931.3080300@yandex-team.ru> <2a41acea0708010923m7b21095ajc2ee84c37e0d5354@mail.gmail.com> <470280F6.9070009@yandex-team.ru> <20071003111737.U14276@delplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help 

[-- Attachment #1 --]
Bruce Evans wrote:
> On Tue, 2 Oct 2007, Vladimir Ivanov wrote:
>
>> Main improvement of this version: driver does not use TX interrupts 
>> at all. So, interrupt rate reduced significantly.
>
> Polling for anything is a bug IMO.  Buggy hardware may work better 
> with it,
> but em is not buggy :-).
The driver does not use polling. We've disabled TX interrupts because we 
seem interrupt hook is too strange and ineffective place to make 
watchdog calculations and queue cleaning. We can do it from application 
context much easier.

RX queue procedure uses another technique. We send wakeup message to RX 
kernel threads and mask RX interrupts. Each RX thread parses RX queue 
while it isn't empty.  After completion RX kernel thread unmask 
interrupt. This hint let us avoid both RX interrupt storm and additional 
latency (due to admin's throttling).  RX interrupt is being masked if 
and only if there are no threads to handle interrupt. Also, the driver 
behave itself like polling mode under heavy load.

But the major benefit of our patchset is SMP.
> For bge, I tune the interrupt moderation parameters to reduce the tx
> interrupt rate to almost as low as possible without doing polling.
> The rate is either 1 interrupt per second if the tx is almost inactive
> or 1 interrupt every 384 packets if the tx is active.  -current mistunes
> these parameters to 150 (microseconds) and 10 (descriptos).  Old tuning
> of 150 and 128 only loses a little compared with 1000000 and 384.  (150
> gives 6667 interrupts per second under load.  This interrupt rate is
> quite manageable and is about the same rate as you have to use with
> polling to get the same throughput but lower efficiency as with
> interrupts.  128 for the descriptor limit causes in a max interrupt rate
> of only a few hundred per second except with tiny packets, but 10 is
> excessively small and requires a rate of up to 140000 per second to keep
> up with tiny packets.  140000 isn't manageable.)
>
> em has more/better interrupt parameters with non-broken defaults so I 
> haven't
> needed to tune them.  For bge, I implement dynamic rx interrupt 
> moderation
> in software where em has it in hardware.  10000 interrupts/second for rx
> is a good limit.  IIRC, em uses 8000 which is a bit low for a max, and
> is missing a sysctl for easy tuning.
I've spent a lot of time for em tuning. This way has a limit.
:-)
>
> Bruce

Regards,
PS: we have published newest 1.16 revision. Just small tuning fix.

-- 
Vladimir Ivanov
Network Operations Center
OOO "Yandex"
t: +7 495 739-7000
f: +7 495 739-7070
@: noc@yandex.net (corporate)
  wawa@yandex-team.ru (personal)
www: www.yandex.ru
-- 


[-- Attachment #2 --]
0	*H
010	+0	*H
'00I
rJN0
	*H
0b10	UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA0
070704151345Z
080703151345Z0E10UThawte Freemail Member1"0 	*H
	wawa@yandex-team.ru0"0
	*H
0
ۨZ~M[d#{ȲeP|z?)!S}hamN0lؾߏJ{B6%rM4lqC%Qdi4uòI
ߥ2'Bj{
]λh~/DhG pXCBRIܕ}φAԝ=TΪ\7oƏ7f/wo?pXB{MC˥|m&ug<
YY00.0U0wawa@yandex-team.ru0U00
	*H
Vbop兵1U{ph5PϹ(;fr`&Blo
.Vْ3z>䜺:'1<ۥusxK\(oe0,43tU#-$0?0
0
	*H
010	UZA10UWestern Cape10U	Cape Town10U
Thawte Consulting1(0&UCertification Services Division1$0"UThawte Personal Freemail CA1+0)	*H
	personal-freemail@thawte.com0
030717000000Z
130716235959Z0b10	UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA00
	*H
0Ħ<UsUNʙZhup[v:aQP
0cZ,p+Z?qV˯<6$*+w=+>@dקe*TH<a@dr`00U00CU<0:08642http://crl.thawte.com/ThawtePersonalFreemailCA.crl0U0)U"0 010UPrivateLabel2-1380
	*H
HP.
fgCL!6-6/P p<ab:~t%Pb'qW%ݩ9 Oe_N4[5MwV!x!5$F]_eO1Q0M0v0b10	UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA
rJN0	+0	*H
	1	*H
0	*H
	1
071003104318Z0#	*H
	1r>xP*Ep9 h{]S]0R	*H
	1E0C0
*H
0*H
0
*H
@0+0
*H
(0
	*H
`wV~/eZ?hhBTug#\$XV7IbRg扰<F92xG+}k&/R%ɓV\kkSGf4F#>m
_g&ƑyLC!\B=A$e#Pś=;;Wjby
0U?R#D˜(+QԧP>]7-“a->dcJBܵu

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47037246.2070400>