From owner-freebsd-current@FreeBSD.ORG Tue Mar 9 20:49:33 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F7EB106564A for ; Tue, 9 Mar 2010 20:49:33 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.155]) by mx1.freebsd.org (Postfix) with ESMTP id AD01B8FC1B for ; Tue, 9 Mar 2010 20:49:32 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 22so2381601fge.13 for ; Tue, 09 Mar 2010 12:49:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=TiMF9qHnFmYvM/Iaf7o+TKUJN5MUH2kQB+Evw4h5HdE=; b=k2LdWFKo2MX0AVnH7P9ymtVVYQ7cPFPtX8kX4KY/WYYSpzJyfoQyL8JttbyUtCOUd9 Qea5uziedJf8kB2wKCsd+i/I8oLsZ/7I0AVW+H+levOP6zMaTpOAzqftAkx9TZXWJxa/ c4WCd5wYgHPihYhNjeuuf6E1YQB1pnx6iXxXo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=HMsrG/KEcigV4tLomYVP0alyXtgu16fidDhP+TQ4SjiyWZ6a6YHNw7X5a0288SdrXR o5AML/kUW02vZnGgJnaXV7qO5xqSwqSq3YngqFWFwWUuHvtGrKoveeNURxHXyRnWQE1w C/A6A8eL0WiJYzeZQO4Ir+9NY6EA4K4uXgJr4= Received: by 10.87.62.1 with SMTP id p1mr1101864fgk.42.1268167771571; Tue, 09 Mar 2010 12:49:31 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 12sm280379fgg.19.2010.03.09.12.49.28 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 09 Mar 2010 12:49:30 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Tue, 9 Mar 2010 12:49:27 -0800 From: Pyun YongHyeon Date: Tue, 9 Mar 2010 12:49:27 -0800 To: Ian FREISLICH Message-ID: <20100309204927.GN1311@michelle.cdnetworks.com> References: <20100305215539.GG14818@michelle.cdnetworks.com> <20100305210435.GF14818@michelle.cdnetworks.com> <20100305184046.GD14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 20:49:33 -0000 On Tue, Mar 09, 2010 at 03:31:55PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > On Mon, Mar 08, 2010 at 04:45:20PM +0200, Ian FREISLICH wrote: > > > Pyun YongHyeon wrote: > > > > On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > > > > > Pyun YongHyeon wrote: > > > > > > Thanks for the info. Frankly, I have no idea how to explain the > > > > > > issue given that you have no heavy load. > > > > > > > > > > How many cores would be involved in handling the traffic and runnig > > > > > PF rules on this machine? There are 4x > > > > > CPU: Quad-Core AMD Opteron(tm) Processor 8354 (2194.51-MHz K8-class CPU > ) > > > > > In this server. I'm also using carp extensively. > > > > > > > > > > > > > pf(4) uses a single lock for processing, number of core would have > > > > no much benefit. > > > > > > What's interesting is the effect on CPU utilisation and interrupt > > > generation that net.inet.ip.fastforwarding has: > > > > > > net.inet.ip.fastforwarding=1 > > > interrupt rate is around 10000/s per bce interface > > > cpu 8.0% interrupt > > > > > > > Yes, this is one of intentional change of the patch. Stock bce(4) > > seems to generate too much interrupts on BCM5709 so I rewrote > > interrupt handling with the help of David. sysctl nodes are also > > exported to control interrupt moderation so you can change them if > > you want. Default value was tuned to generate interrupts less than > > 10k per second and try to minimize latencies. > > Can you explain the tunables please - I'm guessing it's these: > > dev.bce.$i.tx_quick_cons_trip_int This value controls the number of TX Quick BD Chain entries that must be completed before a status block is generated during an interrupt. > dev.bce.$i.tx_quick_cons_trip This value controls the number of TX Quick BD Chain entries that must be completed before a status block is generated. Setting this to 0 disables TX Quick BD Chain consumption from generating status blocks. > dev.bce.$i.tx_ticks_int This value controls the number of 1us ticks that will be counted for status block updates generated due to TX activity during interrupt processing. Setting this value to 0 disables the TX timer feature during interrupts. > dev.bce.$i.tx_ticks This value controls the number of 1us ticks that will be counted before a status block update is generated due to TX activity. Setting this value to 0 disables the TX timer feature. > dev.bce.$i.rx_quick_cons_trip_int This value controls the number of RX Quick BD entries that must be completed before a status block is generated during interrupt processing. > dev.bce.$i.rx_quick_cons_trip This value controls the number of RX Quick BD Chain entries that must be completed before a status block is generated. Setting this to 0 disables RX Event consumption from generating status blocks. > dev.bce.$i.rx_ticks_int This value controls the number of 1us ticks that will be counted for status block updates generated due to RX activity during interrupt processing. Setting this value to 0 disables the RX timer feature during interrupts. > dev.bce.$i.rx_ticks > This value controls the number of 1us ticks that will be counted before a status block update is generated due to RX activity. Setting this value to 0 disables the RX timer feature.