Date: Sun, 21 Jun 2009 13:49:37 +1000 (EST) From: Bruce Evans <brde@optusnet.com.au> To: Igor Sysoev <is@rambler-co.ru> Cc: freebsd-net@FreeBSD.org Subject: Re: bge interrupt coalescing sysctls Message-ID: <20090621131437.B1458@besplex.bde.org> In-Reply-To: <20090618141925.GG60354@rambler-co.ru> References: <20090610123301.GE40250@rambler-co.ru> <20090611114120.I21056@delplex.bde.org> <20090618141925.GG60354@rambler-co.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 18 Jun 2009, Igor Sysoev wrote: > On Thu, Jun 11, 2009 at 11:54:29AM +1000, Bruce Evans wrote: > >> On Wed, 10 Jun 2009, Igor Sysoev wrote: >> ... >>> I has left only static coalescing parameters in the patch >>> and has added a loader tunable to set number of receive descriptors and >>> read only sysctl to read the tunable. I usually use these parameters: >>> >>> /boot/loader.conf: >>> hw.bge.rxd=512 >>> >>> /etc/sysctl.conf: >>> dev.bge.0.rx_coal_ticks=500 >>> dev.bge.0.tx_coal_ticks=10000 >>> dev.bge.0.rx_max_coal_bds=64 >> >> These rx settings give to high a latency for me. > > Probably, however, I use this on a host that has 6000 packets/s. 6000 is not very high, so you might be able to tune for latency at no significant cost. The normal (unpatched) value for rx_coal_ticks is 150 uS. This tends to give an interrupt rate of 6666 packets/S, and this is a good default maximum rate (it takes about 100000 interrupts/S to eat up 2GHz of CPU). If packets normally arrive at 6000/S, then you can tune for minimal latency = no rx interrupt moderation at all by setting rx_coal_ticks to 1, without affecting the interrupt rate at all (there will always by 6000 interrupts/S, but with rx_coal_ticks=1 the packets will br processed with minimal delay (more like 20 uS than 1 uS) instead of after 150 uS (default). Your settings should give about 2000 interrupts/S, with 3 packets handled every interrupt after a delay of up to 500 uS. >>> dev.bge.0.tx_max_coal_bds=128 >>> # apply the above parameters >>> dev.bge.0.program_coal=1 >>> >>> Could anyone commit it ? >> >> Not me, sorry. >> >> The patch is quite clean. If I committed then I would commit the >> dynamic coalescing configuration separately anyway. > > So have you any objections if some one else will commit this patch ? No objections. >> You can probably make hw.bge.rxd a sysctl too (it would take a down/up >> to get it changed, but that is already needed for too many parameters >> in network drivers anyway). I should use a sysctl for the ifq length >> too. This could be done at a high level for each driver. Limiting >> queue lengths may be a good way to reduce cache misses, while increasing >> them is sometimes good for reducing packet loss. > > Do you mean simple command sequence: > > sysctl hw.bge.rxd=512 > ifconfig down > ifconfig up > > or SYSCTL_ADD_PROC for hw.bge.rxd ? The simple command sequence. It's painful to write a SYSCTL_ADD_PROC() for every sysctl, even if the procedure is trival, and the procedure would be highly nontrivial for at least reducing rxd since for reduction it would have to wait for descriptors to become inactive, and decouple them without races... I made program_coal a SYSCTL_PROC() because I wanted to change the settings a lot for testing, but I didn't make the individual coal settings SYSCTL_PROC()s since there is no need to change 1 at a time and it might be wrong to change 1 at a time (they have some interactions). Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090621131437.B1458>