From owner-freebsd-net@FreeBSD.ORG Sun Jun 21 03:49:42 2009 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E265F1065672 for ; Sun, 21 Jun 2009 03:49:42 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail01.syd.optusnet.com.au (mail01.syd.optusnet.com.au [211.29.132.182]) by mx1.freebsd.org (Postfix) with ESMTP id 808968FC18 for ; Sun, 21 Jun 2009 03:49:42 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c122-106-159-184.carlnfd1.nsw.optusnet.com.au [122.106.159.184]) by mail01.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n5L3nbOh029869 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 21 Jun 2009 13:49:40 +1000 Date: Sun, 21 Jun 2009 13:49:37 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Igor Sysoev In-Reply-To: <20090618141925.GG60354@rambler-co.ru> Message-ID: <20090621131437.B1458@besplex.bde.org> References: <20090610123301.GE40250@rambler-co.ru> <20090611114120.I21056@delplex.bde.org> <20090618141925.GG60354@rambler-co.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@FreeBSD.org Subject: Re: bge interrupt coalescing sysctls X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jun 2009 03:49:43 -0000 On Thu, 18 Jun 2009, Igor Sysoev wrote: > On Thu, Jun 11, 2009 at 11:54:29AM +1000, Bruce Evans wrote: > >> On Wed, 10 Jun 2009, Igor Sysoev wrote: >> ... >>> I has left only static coalescing parameters in the patch >>> and has added a loader tunable to set number of receive descriptors and >>> read only sysctl to read the tunable. I usually use these parameters: >>> >>> /boot/loader.conf: >>> hw.bge.rxd=512 >>> >>> /etc/sysctl.conf: >>> dev.bge.0.rx_coal_ticks=500 >>> dev.bge.0.tx_coal_ticks=10000 >>> dev.bge.0.rx_max_coal_bds=64 >> >> These rx settings give to high a latency for me. > > Probably, however, I use this on a host that has 6000 packets/s. 6000 is not very high, so you might be able to tune for latency at no significant cost. The normal (unpatched) value for rx_coal_ticks is 150 uS. This tends to give an interrupt rate of 6666 packets/S, and this is a good default maximum rate (it takes about 100000 interrupts/S to eat up 2GHz of CPU). If packets normally arrive at 6000/S, then you can tune for minimal latency = no rx interrupt moderation at all by setting rx_coal_ticks to 1, without affecting the interrupt rate at all (there will always by 6000 interrupts/S, but with rx_coal_ticks=1 the packets will br processed with minimal delay (more like 20 uS than 1 uS) instead of after 150 uS (default). Your settings should give about 2000 interrupts/S, with 3 packets handled every interrupt after a delay of up to 500 uS. >>> dev.bge.0.tx_max_coal_bds=128 >>> # apply the above parameters >>> dev.bge.0.program_coal=1 >>> >>> Could anyone commit it ? >> >> Not me, sorry. >> >> The patch is quite clean. If I committed then I would commit the >> dynamic coalescing configuration separately anyway. > > So have you any objections if some one else will commit this patch ? No objections. >> You can probably make hw.bge.rxd a sysctl too (it would take a down/up >> to get it changed, but that is already needed for too many parameters >> in network drivers anyway). I should use a sysctl for the ifq length >> too. This could be done at a high level for each driver. Limiting >> queue lengths may be a good way to reduce cache misses, while increasing >> them is sometimes good for reducing packet loss. > > Do you mean simple command sequence: > > sysctl hw.bge.rxd=512 > ifconfig down > ifconfig up > > or SYSCTL_ADD_PROC for hw.bge.rxd ? The simple command sequence. It's painful to write a SYSCTL_ADD_PROC() for every sysctl, even if the procedure is trival, and the procedure would be highly nontrivial for at least reducing rxd since for reduction it would have to wait for descriptors to become inactive, and decouple them without races... I made program_coal a SYSCTL_PROC() because I wanted to change the settings a lot for testing, but I didn't make the individual coal settings SYSCTL_PROC()s since there is no need to change 1 at a time and it might be wrong to change 1 at a time (they have some interactions). Bruce