Date: Tue, 22 Apr 2008 00:20:38 +0400 From: Igor Sysoev <is@rambler-co.ru> To: freebsd-net@FreeBSD.org Subject: Re: bge loader tunables Message-ID: <20080421202038.GK54256@rambler-co.ru> In-Reply-To: <20071117194615.L67319@delplex.bde.org> References: <20071116154019.GE93422@rambler-co.ru> <20071117065908.T65479@delplex.bde.org> <20071117071053.GA18091@rambler-co.ru> <20071117194615.L67319@delplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Nov 17, 2007 at 09:13:50PM +1100, Bruce Evans wrote:
> On Sat, 17 Nov 2007, Igor Sysoev wrote:
>
> >On Sat, Nov 17, 2007 at 08:30:58AM +1100, Bruce Evans wrote:
> >
> >>On Fri, 16 Nov 2007, Igor Sysoev wrote:
> >>
> >>>The attached patch creates the following bge loader tunables:
> >>
> >>I plan to commit old work to do this using sysctls. Tunables are
> >>harder to use and aren't needed since changes to the defaults aren't
> >>needed for booting. I also implemented dynamic tuning for rx coal
> >>parameters so that the sysctls are mostly not needed. Ask for patches
> >>if you want to test this extensively.
> >
> >Yes, I can test your patches on 6.2 and 7.0.
> >Now bge set the coalescing parameters at attach time.
> >Do the sysctl's allow to change them on-the-fly ?
> >How does rx dynamic tuning work ?
> >Could it be turned off ?
>
> OK, the patch is enclosed at the end, in 2 versions:
> - all my patches for bge (with lots of debugging cruft and half-baked
> fixes for 5705+ sysctls.
> - edited version with only the coalescing parameter changes.
>
> I haven't used it under 6.2, but have used a similar version in ~5.2,
> and it should work in 6.2 except for the 5705+ sysctl fixes.
>
> bge actually sets parameters at init time, and it initializes whenever the
> link is brought back up, so the parameters can be changed using
> "ifconfig bgeN down up". Several network drivers have interrupt moderation
> parameters that can be changed in this way, but it is painful to change
> the link status like that, so I have a sysctl dev.bge.N.program_coal to
> apply the current parameters to the hardware. The other sysctls to change
> the parameters don't apply immediately, except the one for the rx tuning
> max interrupt rate, since applying the changed parameters to the hardware
> takes more code than a SYSCTL_INT(), and it is sometimes necessary to
> change all the parameters together atomically.
>
> Dynamic tuning works by monitoring the current rx packet rate and
> increasing the active rx_max_coal_bds so that the ratio <rx packet
> rate> / rx_max_coal_bds is usually <= the specified max rx interrupt
> rate. rx_coal_ticks is set to the constant value of the inverse of
> the specified max rx interrupt rate (in ticks) on transition to dynamic
> mode but IIRC is not changed when the dynamic rate is changed (not
> always changing it automatically allows adjusting it independently of
> the rate but is often not what is wanted). The transition has some
> bias towards lower latency over too many interrupts, so that short
> bursts don't increase the latency. I think this simple algorithm is
> good enough provided the load (in rx packets/second) doesn't oscillate
> rapidly.
>
> Dynamic tuning requires efficient reprogramming of at least one of the
> hardware coal registers so that the tuning can respond rapidly to changes.
> I have 2 methods for this:
> - bge_careful_coal = 1 avoids using uses a potentially very long
> busy-wait loop in the interrupt handler by giving up on reprogramming
> the host coalescing engine (HCE) if the HCE seems to be busy. Docs
> seem to require waiting for up to several milliseconds for the HCE
> to stablilize, and it is not clear if it is possible for the HCE to
> never stabilize because packets are streaming in. (I don't have
> proper docs.) This seems to always work (the HCE is never busy)
> for rx_max_coal_bds, but something near here didn't work for
> changing rx_coal_ticks in an old version.
> - bge_careful_coal = 0 avoids the loop by writing to the rx_max_coal_bds
> register without waiting for the HCE. This seems to work too. It
> isn't critical for the HCE to see the change immediately or even
> for it to be seen at all (missed changes might do more than give a
> huge interrupt rate for too long), but it is important for the
> change to not break the engine.
> There is no sysctl for this of for some other hackish parameters. The
> source must be edited to change this from 1 to 0.
>
> Dynamic tuning is turned off by setting the dynamic max interrupt
> frequency to 0. Then rx_coal_ticks is reset to 150, and the active
> rx_max_coal_bds is restored to the static value.
Finally I have tested your second (without debug stuff) patch in
production environment (~45K in/out packets) on FreeBSD 7.0-STABLE.
I think it should be commited.
I use my usual static settings in /etc/sysctl.conf:
dev.bge.0.dyncoal_max_intr_freq=0
#
dev.bge.0.rx_coal_ticks=500
dev.bge.0.tx_coal_ticks=10000
dev.bge.0.rx_max_coal_bds=64
dev.bge.0.tx_max_coal_bds=128
# apply the above parameters
dev.bge.0.program_coal=0
and have about only 1700-1900 interrupts per second.
The only issue was at boot time:
dev.bge.0.dyncoal_max_intr_freq: 10000 -> 0
dev.bge.0.rx_coal_ticks: 0 -> 500
dev.bge.0.tx_coal_ticks: 1000000 -> 10000
dev.bge.0.rx_max_coal_bds: 128 -> 64
dev.bge.0.tx_max_coal_bds: 384 -> 128
...
bge0: flags =8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
...
Local package initialization:
...
dev.bge.0.rx_coal_ticks: 150 -> 500
When disabling dyncoal_max_intr_freq at bge UPing resets rx_coal_ticks to 150.
--
Igor Sysoev
http://sysoev.ru/en/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080421202038.GK54256>