From owner-freebsd-net@FreeBSD.ORG Sun Oct 14 16:14:29 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13A2616A46C for ; Sun, 14 Oct 2007 16:14:29 +0000 (UTC) (envelope-from wawa@yandex-team.ru) Received: from relanium.yandex.ru (relanium.yandex.ru [213.180.193.88]) by mx1.freebsd.org (Postfix) with ESMTP id 7FEBE13C45D for ; Sun, 14 Oct 2007 16:14:28 +0000 (UTC) (envelope-from wawa@yandex-team.ru) Received: from [87.250.227.231] (v3-227-231.yandex.net [87.250.227.231]) by relanium.yandex.ru (8.14.1/8.14.1) with ESMTP id l9EGEGed054005 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 14 Oct 2007 20:14:21 +0400 (MSD) (envelope-from wawa@yandex-team.ru) Message-ID: <4712405B.1080808@yandex-team.ru> Date: Sun, 14 Oct 2007 20:14:19 +0400 From: Vladimir Ivanov Organization: Yandex LLC User-Agent: Thunderbird 1.5.0.13 (X11/20070824) MIME-Version: 1.0 To: "freebsd-net@freebsd.org" References: <46B07931.3080300@yandex-team.ru> <55e8a96c0708010626x7a433c21hacc124c7c70af471@mail.gmail.com> <46B09802.7080908@yandex-team.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Antivirus: Dr.Web (R) for Mail Servers on relanium.yandex.ru host X-Antivirus-Code: 100000 Subject: Re: SMPable version of EM driver X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 16:14:29 -0000 Hi, Mike Tancsa wrote: > On Wed, 01 Aug 2007 18:26:10 +0400, in sentex.lists.freebsd.net you > wrote: > > >> Bill Marquette wrote: >> >>> [skip] >>> What type of performance differences are you seeing with these >>> changes? Is this with FreeBSD acting as a router/firewall, or purely >>> >>> >> RX queue is being processed w/more than one thread. >> TX queue thread isn't locked with RX anymore. >> >> Extra CPU time can be used by e.g. IPFW firewall or routing and so on. >> >> > > Hi, > I am interested in trying your version of the em driver. On > one of my routers, I am seeing > > kernel: em2: Missed Packets = 953 > kernel: em2: Receive No Buffers = 128 > kernel: em2: RX overruns = 7 > kernel: em2: Good Packets Rcvd = 62453961 > kernel: em2: Good Packets Xmtd = 31935910 > > This is with the em driver currently in the RELENG_6 tree (version > 6.6.6).. Previous versions were the same. > I notice that you have some different defaults as well > > dev.em.0.rx_int_delay: 0 > dev.em.0.tx_int_delay: 67108 > dev.em.0.rx_abs_int_delay: 1000 > dev.em.0.tx_abs_int_delay: 67108 > > vs > > dev.em.1.rx_int_delay: 0 > dev.em.1.tx_int_delay: 66 > dev.em.1.rx_abs_int_delay: 66 > dev.em.1.tx_abs_int_delay: 66 > dev.em.1.rx_processing_limit: 100 > > What are these tuned for ? Hi pps ? Low latency ? > We've both problems and even more: we need low latency, we've huge pps, we've to run firewall and so on Tuning can not solve them. Actually our rx/tx timeout defaults mostly are meaningless because: 1) we do not use TX interrupts et all 2) we use explicit SYSCTL (see dev.em.N.rx_kthread_priority) for tuning RX threads' priority instead of rx_processing_limit. 3) we mask rx interrupts if aren't ready to catch that's why we do not need interrupt pending/throttling. > Thanks for any info, > > ---Mike > > >> Also: >> + RX and TX use different priority value. System seems to be more stable >> if RX scheduled w/less priority. >> + RX/TX stay masked if there is no thread ready to catch interrupt. >> >> >>> as a server? Any chance you are using the pf filtering engine (which >>> I believe is still under giant in releng_6) with this? Thanks >>> >>> >> I have been talked that GIANT is a big problem for pf driver and they >> can not fix it easy. >> >> Regards, >> -- >> [skip] WBR, PS: your personal e-mail doesn't work -- Vladimir Ivanov Network Operations Center OOO "Yandex" t: +7 495 739-7000 f: +7 495 739-7070 @: noc@yandex.net (corporate) wawa@yandex-team.ru (personal) www: www.yandex.ru --