From owner-freebsd-net Mon Oct 7 11:52:18 2002 Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E666737B401 for ; Mon, 7 Oct 2002 11:52:16 -0700 (PDT) Received: from isilon.com (isilon.com [65.101.129.58]) by mx1.FreeBSD.org (Postfix) with ESMTP id 02A3643E4A for ; Mon, 7 Oct 2002 11:52:16 -0700 (PDT) (envelope-from bbaumann@isilon.com) Received: from localhost (localhost [127.0.0.1]) by isilon.com (8.12.2/8.11.1) with ESMTP id g97IqF23056011 for ; Mon, 7 Oct 2002 11:52:15 -0700 (PDT) (envelope-from bbaumann@isilon.com) Date: Mon, 7 Oct 2002 11:52:15 -0700 (PDT) From: Bill Baumann To: freebsd-net@FreeBSD.ORG Subject: Re: Help with net.inet.ip.intr_queue_maxlen In-Reply-To: <3D9F8002.70500@expertcity.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org My thought is that the net.inet.ip.intr_queue_maxlen should be set to the maximum receive queue length of your NIC. Depending upon traffic (bursty short packets being the worst), other kernel operations being slow, and NIC interrupt coalesce or polling times, your NIC's rx buffer can fill significantly before the NIC's receive queue is be processed. Thats why we have large queues at this level! Unfortuantely, the default for net.inet.ip.intr_queue_maxlen is not nearly large enough to to accomidate but small percentage of the NICs rx queue. The default is 50, but should be more like 500. I think the bge driver's rx queue can handle 512 1500MTU plus 256 9000MTU frames. So the NIC can receive 768 before dropping frames, but the IP stack can only handle the first 50. As far as I know none of the drivers know to do anything about this problem. Due to the above, I can image scenarios where you would experience significant packet loss with a low CPU utilization. You may also want to consider decreasing interrupt coalescing parameters in the NIC or polling times. This will lower the latencies, but increase the CPU overhead attributed to the NIC. Good luck! On Sat, 5 Oct 2002, Steve Francis wrote: > Can someone help me with net.inet.ip.intr_queue_maxlen tuning? > > Firstly, its the "size of the IP input queue", per the source. > > So does that mean after the NIC has received the packet, the interupt > from the NIC has been processed and the packet retrieved from the NIC, > then the packet is placed in this queue, before the IP stack looks at it? > i.e. its unrelated to interupt coalescing or polling, or NIC > performance, as they have already occurred in order to put the packet > into the queue. Yes? > > I am getting incrementing net.inet.ip.intr_queue_drops at around 8,000 > pps (increasing drops at rate of 10 or so per second.) > Yet, if my statement above about what the queue is, is correct, then it > just means that the system was busy doing stuff before it had a chance > to process the incoming packets, so there was no room for new ones to > enter the queue. But as the system was only 50% busy, then if I increase > the input queue, I should be able to avoid these drops, correct? At > least until the system gets a lot busier. > > Is there a sane upper recommended limit to the queue length? > > Or am I way off base here? > Thanks > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-net" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message