From owner-freebsd-net@FreeBSD.ORG Mon Apr 21 20:38:51 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C96FB106566C for ; Mon, 21 Apr 2008 20:38:51 +0000 (UTC) (envelope-from is@rambler-co.ru) Received: from relay0.rambler.ru (relay0.rambler.ru [81.19.66.187]) by mx1.freebsd.org (Postfix) with ESMTP id 6550E8FC12 for ; Mon, 21 Apr 2008 20:38:51 +0000 (UTC) (envelope-from is@rambler-co.ru) Received: from localhost (is1.park.rambler.ru [81.19.64.121]) by relay0.rambler.ru (Postfix) with ESMTP id 7B51C6274 for ; Tue, 22 Apr 2008 00:21:17 +0400 (MSD) Date: Tue, 22 Apr 2008 00:20:38 +0400 From: Igor Sysoev To: freebsd-net@FreeBSD.org Message-ID: <20080421202038.GK54256@rambler-co.ru> References: <20071116154019.GE93422@rambler-co.ru> <20071117065908.T65479@delplex.bde.org> <20071117071053.GA18091@rambler-co.ru> <20071117194615.L67319@delplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <20071117194615.L67319@delplex.bde.org> User-Agent: Mutt/1.5.13 (2006-08-11) Cc: Subject: Re: bge loader tunables X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Apr 2008 20:38:52 -0000 On Sat, Nov 17, 2007 at 09:13:50PM +1100, Bruce Evans wrote: > On Sat, 17 Nov 2007, Igor Sysoev wrote: > > >On Sat, Nov 17, 2007 at 08:30:58AM +1100, Bruce Evans wrote: > > > >>On Fri, 16 Nov 2007, Igor Sysoev wrote: > >> > >>>The attached patch creates the following bge loader tunables: > >> > >>I plan to commit old work to do this using sysctls. Tunables are > >>harder to use and aren't needed since changes to the defaults aren't > >>needed for booting. I also implemented dynamic tuning for rx coal > >>parameters so that the sysctls are mostly not needed. Ask for patches > >>if you want to test this extensively. > > > >Yes, I can test your patches on 6.2 and 7.0. > >Now bge set the coalescing parameters at attach time. > >Do the sysctl's allow to change them on-the-fly ? > >How does rx dynamic tuning work ? > >Could it be turned off ? > > OK, the patch is enclosed at the end, in 2 versions: > - all my patches for bge (with lots of debugging cruft and half-baked > fixes for 5705+ sysctls. > - edited version with only the coalescing parameter changes. > > I haven't used it under 6.2, but have used a similar version in ~5.2, > and it should work in 6.2 except for the 5705+ sysctl fixes. > > bge actually sets parameters at init time, and it initializes whenever the > link is brought back up, so the parameters can be changed using > "ifconfig bgeN down up". Several network drivers have interrupt moderation > parameters that can be changed in this way, but it is painful to change > the link status like that, so I have a sysctl dev.bge.N.program_coal to > apply the current parameters to the hardware. The other sysctls to change > the parameters don't apply immediately, except the one for the rx tuning > max interrupt rate, since applying the changed parameters to the hardware > takes more code than a SYSCTL_INT(), and it is sometimes necessary to > change all the parameters together atomically. > > Dynamic tuning works by monitoring the current rx packet rate and > increasing the active rx_max_coal_bds so that the ratio rate> / rx_max_coal_bds is usually <= the specified max rx interrupt > rate. rx_coal_ticks is set to the constant value of the inverse of > the specified max rx interrupt rate (in ticks) on transition to dynamic > mode but IIRC is not changed when the dynamic rate is changed (not > always changing it automatically allows adjusting it independently of > the rate but is often not what is wanted). The transition has some > bias towards lower latency over too many interrupts, so that short > bursts don't increase the latency. I think this simple algorithm is > good enough provided the load (in rx packets/second) doesn't oscillate > rapidly. > > Dynamic tuning requires efficient reprogramming of at least one of the > hardware coal registers so that the tuning can respond rapidly to changes. > I have 2 methods for this: > - bge_careful_coal = 1 avoids using uses a potentially very long > busy-wait loop in the interrupt handler by giving up on reprogramming > the host coalescing engine (HCE) if the HCE seems to be busy. Docs > seem to require waiting for up to several milliseconds for the HCE > to stablilize, and it is not clear if it is possible for the HCE to > never stabilize because packets are streaming in. (I don't have > proper docs.) This seems to always work (the HCE is never busy) > for rx_max_coal_bds, but something near here didn't work for > changing rx_coal_ticks in an old version. > - bge_careful_coal = 0 avoids the loop by writing to the rx_max_coal_bds > register without waiting for the HCE. This seems to work too. It > isn't critical for the HCE to see the change immediately or even > for it to be seen at all (missed changes might do more than give a > huge interrupt rate for too long), but it is important for the > change to not break the engine. > There is no sysctl for this of for some other hackish parameters. The > source must be edited to change this from 1 to 0. > > Dynamic tuning is turned off by setting the dynamic max interrupt > frequency to 0. Then rx_coal_ticks is reset to 150, and the active > rx_max_coal_bds is restored to the static value. Finally I have tested your second (without debug stuff) patch in production environment (~45K in/out packets) on FreeBSD 7.0-STABLE. I think it should be commited. I use my usual static settings in /etc/sysctl.conf: dev.bge.0.dyncoal_max_intr_freq=0 # dev.bge.0.rx_coal_ticks=500 dev.bge.0.tx_coal_ticks=10000 dev.bge.0.rx_max_coal_bds=64 dev.bge.0.tx_max_coal_bds=128 # apply the above parameters dev.bge.0.program_coal=0 and have about only 1700-1900 interrupts per second. The only issue was at boot time: dev.bge.0.dyncoal_max_intr_freq: 10000 -> 0 dev.bge.0.rx_coal_ticks: 0 -> 500 dev.bge.0.tx_coal_ticks: 1000000 -> 10000 dev.bge.0.rx_max_coal_bds: 128 -> 64 dev.bge.0.tx_max_coal_bds: 384 -> 128 ... bge0: flags =8843 metric 0 mtu 1500 options=9b ... Local package initialization: ... dev.bge.0.rx_coal_ticks: 150 -> 500 When disabling dyncoal_max_intr_freq at bge UPing resets rx_coal_ticks to 150. -- Igor Sysoev http://sysoev.ru/en/