From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 13:47:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0148F1065676 for ; Sun, 4 Oct 2009 13:47:26 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx33.mail.ru (mx33.mail.ru [94.100.176.47]) by mx1.freebsd.org (Postfix) with ESMTP id B4A928FC13 for ; Sun, 4 Oct 2009 13:47:25 +0000 (UTC) Received: from [217.25.27.27] (port=38280 helo=[217.25.27.27]) by mx33.mail.ru with asmtp id 1MuRQq-000OxJ-00 for freebsd-net@freebsd.org; Sun, 04 Oct 2009 17:47:24 +0400 Message-ID: <4AC8A76B.3050502@mail.ru> Date: Sun, 04 Oct 2009 18:47:23 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Subject: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rihad List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 13:47:26 -0000 Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP users online limited by dummynet pipes of various speeds. According to netstat -s output around 500-1000 packets are being dropped every second (this accounts for wasting around 7-12 mbit/s worth of traffic according to systat -ifstat): # while :; do netstat -z -s 2>/dev/null | fgrep -w "output packets dropped"; sleep 1; done 16824 output packets dropped due to no bufs, etc. 548 output packets dropped due to no bufs, etc. 842 output packets dropped due to no bufs, etc. 709 output packets dropped due to no bufs, etc. 652 output packets dropped due to no bufs, etc. ^C Pipes have been created like this: ipfw pipe 1024 config bw 1024kbit/s mask dst-ip 0xffffffff queue 350KBytes etc., and then assigned to users by application (ipfw tablearg). I've tried playing with the queue setting, from as little as 1 slot to as much as 4096KBytes - packets are still being dropped, more or less. Should I somehow calculate the proper queue value for the given pipe width? The manpage says 50 slots is typical for Ethernet devices (not mentioning whether it's 10, 100 or 1000 mbit/s), and that's it. sysctls: kern.ipc.nmbclusters=50000 net.inet.ip.dummynet.io_fast=1 Polling can't be enabled with bce. Any hints? Should I provide any further info? Thanks. From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 14:42:32 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 821FD1065679 for ; Sun, 4 Oct 2009 14:42:32 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 4844D8FC1C for ; Sun, 4 Oct 2009 14:42:32 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 025AA730DA; Sun, 4 Oct 2009 16:49:10 +0200 (CEST) Date: Sun, 4 Oct 2009 16:49:09 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091004144909.GA42503@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC8A76B.3050502@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 14:42:32 -0000 On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: > Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell > PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP > users online limited by dummynet pipes of various speeds. According to > netstat -s output around 500-1000 packets are being dropped every second > (this accounts for wasting around 7-12 mbit/s worth of traffic according > to systat -ifstat): what kind of packets are you seeing as dropped ? please give the output of 'netstat -s output | grep drop' At those speeds you might be hitting various limits with your config (e.g. 50k nmbclusters is probably way too small for 4k users -- means you have an average of 10-15 buffers per user; the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, which is quite high besides the fact that in order to scale to 4k users you would need over 1GB of kernel memory just for the buffers). I'd most likely suspect the nmbclusters argument. netstat -m might give you some useful stats. cheers luigi > # while :; do netstat -z -s 2>/dev/null | fgrep -w "output packets > dropped"; sleep 1; done > 16824 output packets dropped due to no bufs, etc. > 548 output packets dropped due to no bufs, etc. > 842 output packets dropped due to no bufs, etc. > 709 output packets dropped due to no bufs, etc. > 652 output packets dropped due to no bufs, etc. > ^C > > Pipes have been created like this: > ipfw pipe 1024 config bw 1024kbit/s mask dst-ip 0xffffffff queue 350KBytes > etc., and then assigned to users by application (ipfw tablearg). > > I've tried playing with the queue setting, from as little as 1 slot to > as much as 4096KBytes - packets are still being dropped, more or less. > Should I somehow calculate the proper queue value for the given pipe > width? The manpage says 50 slots is typical for Ethernet devices (not > mentioning whether it's 10, 100 or 1000 mbit/s), and that's it. > > sysctls: > kern.ipc.nmbclusters=50000 > net.inet.ip.dummynet.io_fast=1 > > Polling can't be enabled with bce. > > Any hints? Should I provide any further info? > > Thanks. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 14:53:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 268D4106568D for ; Sun, 4 Oct 2009 14:53:26 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id D85C18FC15 for ; Sun, 4 Oct 2009 14:53:25 +0000 (UTC) Received: from [217.25.27.27] (port=40194 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MuSSi-000DdI-00; Sun, 04 Oct 2009 18:53:24 +0400 Message-ID: <4AC8B6E3.2070101@mail.ru> Date: Sun, 04 Oct 2009 19:53:23 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> In-Reply-To: <20091004144909.GA42503@onelab2.iet.unipi.it> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 14:53:26 -0000 Luigi Rizzo wrote: > On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: >> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell >> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP >> users online limited by dummynet pipes of various speeds. According to >> netstat -s output around 500-1000 packets are being dropped every second >> (this accounts for wasting around 7-12 mbit/s worth of traffic according >> to systat -ifstat): > > what kind of packets are you seeing as dropped ? > please give the output of 'netstat -s output | grep drop' > The output packets, like I said: # netstat -s output | grep drop 2 connections closed (including 2 drops) 0 embryonic connections dropped 2 connections dropped by rexmit timeout 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 connections dropped by keepalive 0 dropped 2 dropped due to no socket 0 dropped due to full socket buffers 0 fragments dropped (dup or out of space) 2 fragments dropped after timeout 7538 output packets dropped due to no bufs, etc. The statistics are zeroed every 15 seconds in another window as I'm investigating the issue, but the rate is around 500-1000 lost packets every second at the current ~530 mbit/s load. > At those speeds you might be hitting various limits with your > config (e.g. 50k nmbclusters is probably way too small for I bet it isn't: 1967/5009/6976/50000 mbuf clusters in use (current/cache/total/max) > 4k users -- means you have an average of 10-15 buffers per user; > the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, > which is quite high besides the fact that in order to scale to 4k users > you would need over 1GB of kernel memory just for the buffers). Aha. Can you be more specific about the kernel memory stuff? Which setting needs tweaking? I have another similar box with 2 em GigE interfaces working @220-230 mbit/s and virtually no out-of-bufs dropped packets as with bce @500-600 mbit. It too has 350KBytes dummynet queue sizes. And it too has adequate mbuf load: 3071/10427/13498/25600 mbuf clusters in use (current/cache/total/max) From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 14:56:00 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B6921065670 for ; Sun, 4 Oct 2009 14:56:00 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id BCFEF8FC17 for ; Sun, 4 Oct 2009 14:55:59 +0000 (UTC) Received: from [217.25.27.27] (port=41510 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1MuSVC-0002ht-00; Sun, 04 Oct 2009 18:55:58 +0400 Message-ID: <4AC8B77D.6070703@mail.ru> Date: Sun, 04 Oct 2009 19:55:57 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> In-Reply-To: <20091004144909.GA42503@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 14:56:00 -0000 Luigi Rizzo wrote: > On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: >> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell >> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP >> users online limited by dummynet pipes of various speeds. According to >> netstat -s output around 500-1000 packets are being dropped every second >> (this accounts for wasting around 7-12 mbit/s worth of traffic according >> to systat -ifstat): > > At those speeds you might be hitting various limits with your > config (e.g. 50k nmbclusters is probably way too small for > 4k users -- means you have an average of 10-15 buffers per user; > the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, > which is quite high besides the fact that in order to scale to 4k users > you would need over 1GB of kernel memory just for the buffers). top output: Mem: 2037M Active, 1248M Inact, 450M Wired, 184M Cache, 214M Buf, 17M Free I guess we're quite far from reaching 1GB of kernel memory. From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 15:04:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF3421065670 for ; Sun, 4 Oct 2009 15:04:39 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id A497F8FC08 for ; Sun, 4 Oct 2009 15:04:39 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 955AF730DA; Sun, 4 Oct 2009 17:11:17 +0200 (CEST) Date: Sun, 4 Oct 2009 17:11:17 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091004151117.GA42877@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> <4AC8B77D.6070703@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC8B77D.6070703@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 15:04:40 -0000 On Sun, Oct 04, 2009 at 07:55:57PM +0500, rihad wrote: > Luigi Rizzo wrote: > >On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: > >>Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell > >>PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP > >>users online limited by dummynet pipes of various speeds. According to > >>netstat -s output around 500-1000 packets are being dropped every second > >>(this accounts for wasting around 7-12 mbit/s worth of traffic according > >>to systat -ifstat): > > > >At those speeds you might be hitting various limits with your > >config (e.g. 50k nmbclusters is probably way too small for > >4k users -- means you have an average of 10-15 buffers per user; > >the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, > >which is quite high besides the fact that in order to scale to 4k users > >you would need over 1GB of kernel memory just for the buffers). > > > top output: > Mem: 2037M Active, 1248M Inact, 450M Wired, 184M Cache, 214M Buf, 17M Free > > I guess we're quite far from reaching 1GB of kernel memory. of course, you'd have to configure also 500k nmbclusters (and then probably this would not fit) From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 15:08:40 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9517910656C6 for ; Sun, 4 Oct 2009 15:08:40 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 2381D8FC26 for ; Sun, 4 Oct 2009 15:08:39 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 0A0C7730DA; Sun, 4 Oct 2009 17:15:18 +0200 (CEST) Date: Sun, 4 Oct 2009 17:15:18 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091004151518.GB42877@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> <4AC8B6E3.2070101@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC8B6E3.2070101@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 15:08:40 -0000 On Sun, Oct 04, 2009 at 07:53:23PM +0500, rihad wrote: > Luigi Rizzo wrote: > >On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: > >>Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell > >>PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP > >>users online limited by dummynet pipes of various speeds. According to > >>netstat -s output around 500-1000 packets are being dropped every second > >>(this accounts for wasting around 7-12 mbit/s worth of traffic according > >>to systat -ifstat): > > > >what kind of packets are you seeing as dropped ? > >please give the output of 'netstat -s output | grep drop' > > > The output packets, like I said: > # netstat -s output | grep drop > 2 connections closed (including 2 drops) > 0 embryonic connections dropped > 2 connections dropped by rexmit timeout > 0 connections dropped by persist timeout > 0 Connections (fin_wait_2) dropped because of timeout > 0 connections dropped by keepalive > 0 dropped > 2 dropped due to no socket > 0 dropped due to full socket buffers > 0 fragments dropped (dup or out of space) > 2 fragments dropped after timeout > 7538 output packets dropped due to no bufs, etc. > > The statistics are zeroed every 15 seconds in another window as I'm > investigating the issue, but the rate is around 500-1000 lost packets > every second at the current ~530 mbit/s load. > > >At those speeds you might be hitting various limits with your > >config (e.g. 50k nmbclusters is probably way too small for > > I bet it isn't: > 1967/5009/6976/50000 mbuf clusters in use (current/cache/total/max) > > >4k users -- means you have an average of 10-15 buffers per user; > >the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, > >which is quite high besides the fact that in order to scale to 4k users > >you would need over 1GB of kernel memory just for the buffers). > Aha. Can you be more specific about the kernel memory stuff? Which > setting needs tweaking? > > > I have another similar box with 2 em GigE interfaces working @220-230 > mbit/s and virtually no out-of-bufs dropped packets as with bce @500-600 > mbit. It too has 350KBytes dummynet queue sizes. And it too has adequate > mbuf load: > 3071/10427/13498/25600 mbuf clusters in use (current/cache/total/max) I think a quick way to tell if the problem is in dummynet/ipfw or elsewhere would be to reconfigure the pipes (for short times, e.g. 1-2 minutes while you test things) as # first, try to remove the shaping to see if the drops # are still present or not ipfw pipe XX delete; ipfw pipe XX config // no buffering # second, do more traffic aggregation to see if the number of # pipes influences the drops. These are several different # configs to be tried. ipfw pipe XX delete; ipfw pipe XX config bw 500Mbits/s ipfw pipe XX delete; ipfw pipe XX config bw 50Mbits/s mask src-ip 0xffffff00 ipfw pipe XX delete; ipfw pipe XX config bw 5Mbits/s mask src-ip 0xfffffff0 and see if things change. If losses persist even removing dummynet, then of course it is a device problem. Also note that dummynet introduces some burstiness in the output, which might saturate the output queue in the card (no idea what is used by bce). This particular phenomenon could be reduced by raising HZ to 2000 or 4000. cheers luigi From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 15:29:03 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39B3C1065676 for ; Sun, 4 Oct 2009 15:29:03 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx74.mail.ru (mx74.mail.ru [94.100.176.89]) by mx1.freebsd.org (Postfix) with ESMTP id E951E8FC0C for ; Sun, 4 Oct 2009 15:29:02 +0000 (UTC) Received: from [217.25.27.27] (port=60765 helo=[217.25.27.27]) by mx74.mail.ru with asmtp id 1MuT1A-0001Pp-00; Sun, 04 Oct 2009 19:29:00 +0400 Message-ID: <4AC8BF3B.10601@mail.ru> Date: Sun, 04 Oct 2009 20:28:59 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> <4AC8B6E3.2070101@mail.ru> <20091004151518.GB42877@onelab2.iet.unipi.it> In-Reply-To: <20091004151518.GB42877@onelab2.iet.unipi.it> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 15:29:03 -0000 Luigi Rizzo wrote: > On Sun, Oct 04, 2009 at 07:53:23PM +0500, rihad wrote: >> Luigi Rizzo wrote: >>> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: >>>> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell >>>> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP >>>> users online limited by dummynet pipes of various speeds. According to >>>> netstat -s output around 500-1000 packets are being dropped every second >>>> (this accounts for wasting around 7-12 mbit/s worth of traffic according >>>> to systat -ifstat): >>> what kind of packets are you seeing as dropped ? >>> please give the output of 'netstat -s output | grep drop' >>> >> The output packets, like I said: >> # netstat -s output | grep drop >> 2 connections closed (including 2 drops) >> 0 embryonic connections dropped >> 2 connections dropped by rexmit timeout >> 0 connections dropped by persist timeout >> 0 Connections (fin_wait_2) dropped because of timeout >> 0 connections dropped by keepalive >> 0 dropped >> 2 dropped due to no socket >> 0 dropped due to full socket buffers >> 0 fragments dropped (dup or out of space) >> 2 fragments dropped after timeout >> 7538 output packets dropped due to no bufs, etc. >> >> The statistics are zeroed every 15 seconds in another window as I'm >> investigating the issue, but the rate is around 500-1000 lost packets >> every second at the current ~530 mbit/s load. >> >>> At those speeds you might be hitting various limits with your >>> config (e.g. 50k nmbclusters is probably way too small for >> I bet it isn't: >> 1967/5009/6976/50000 mbuf clusters in use (current/cache/total/max) >> >>> 4k users -- means you have an average of 10-15 buffers per user; >>> the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering, >>> which is quite high besides the fact that in order to scale to 4k users >>> you would need over 1GB of kernel memory just for the buffers). >> Aha. Can you be more specific about the kernel memory stuff? Which >> setting needs tweaking? >> >> >> I have another similar box with 2 em GigE interfaces working @220-230 >> mbit/s and virtually no out-of-bufs dropped packets as with bce @500-600 >> mbit. It too has 350KBytes dummynet queue sizes. And it too has adequate >> mbuf load: >> 3071/10427/13498/25600 mbuf clusters in use (current/cache/total/max) > > I think a quick way to tell if the problem is in dummynet/ipfw or elsewhere > would be to reconfigure the pipes (for short times, e.g. 1-2 minutes > while you test things) as > > # first, try to remove the shaping to see if the drops > # are still present or not > ipfw pipe XX delete; ipfw pipe XX config // no buffering > > # second, do more traffic aggregation to see if the number of > # pipes influences the drops. These are several different > # configs to be tried. > ipfw pipe XX delete; ipfw pipe XX config bw 500Mbits/s > ipfw pipe XX delete; ipfw pipe XX config bw 50Mbits/s mask src-ip 0xffffff00 > ipfw pipe XX delete; ipfw pipe XX config bw 5Mbits/s mask src-ip 0xfffffff0 > > and see if things change. If losses persist even removing dummynet, > then of course it is a device problem. > Also note that dummynet introduces some burstiness in the output, > which might saturate the output queue in the card (no idea what is > used by bce). This particular phenomenon could be reduced by raising > HZ to 2000 or 4000. > Thanks for the tip. although I took an easier route by simply doing "ipfw add allow ip from any to any" before the pipe rules, and the buf drop rate instantly became 0. So the problem is dummynet/ipfw. Should I go to setting HZ to 2000? Mine is 1000. I somehow don't think the change would improve things. Maybe there's another way not involving a machine reboot? This is a production machine ;-( From owner-freebsd-net@FreeBSD.ORG Sun Oct 4 17:40:55 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF73D1065679; Sun, 4 Oct 2009 17:40:55 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A76F38FC1A; Sun, 4 Oct 2009 17:40:55 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n94HetqF060197; Sun, 4 Oct 2009 17:40:55 GMT (envelope-from gavin@freefall.freebsd.org) Received: (from gavin@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n94HetQ6060185; Sun, 4 Oct 2009 17:40:55 GMT (envelope-from gavin) Date: Sun, 4 Oct 2009 17:40:55 GMT Message-Id: <200910041740.n94HetQ6060185@freefall.freebsd.org> To: gavin@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: gavin@FreeBSD.org Cc: Subject: Re: bin/139346: [patch] arp(8) add option to remove static entries listed in file X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 17:40:55 -0000 Old Synopsis: [patch] add arp option to remove static entries listed in file New Synopsis: [patch] arp(8) add option to remove static entries listed in file Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: gavin Responsible-Changed-When: Sun Oct 4 17:38:34 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). To submitter: Could you also please provide a patch to update the man page? Thanks! http://www.freebsd.org/cgi/query-pr.cgi?pr=139346 From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 03:23:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9BFA3106566B for ; Mon, 5 Oct 2009 03:23:26 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id CC3098FC13 for ; Mon, 5 Oct 2009 03:23:25 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n952tLJE053119; Mon, 5 Oct 2009 10:55:21 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n952tLL2053117; Mon, 5 Oct 2009 10:55:21 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 10:55:21 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005025521.GA52702@svzserv.kemerovo.su> References: <4AC8A76B.3050502@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC8A76B.3050502@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 03:23:26 -0000 On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: > sysctls: > kern.ipc.nmbclusters=50000 > net.inet.ip.dummynet.io_fast=1 I guess you should also try to increase pipes length: net.inet.ip.dummynet.hash_size=65536 net.inet.ip.dummynet.pipe_slot_limit=1000 And reconfigure pipes like this: ipfw pipe NNN config bw ... queue 1000 And default 'taildrop' policy of dummynet pipes may be guilty, you'd use GRED to prevent excessive drops. Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 04:40:03 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69D51106568F for ; Mon, 5 Oct 2009 04:40:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 57A428FC19 for ; Mon, 5 Oct 2009 04:40:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n954e2jV017951 for ; Mon, 5 Oct 2009 04:40:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n954e2Ja017950; Mon, 5 Oct 2009 04:40:02 GMT (envelope-from gnats) Date: Mon, 5 Oct 2009 04:40:02 GMT Message-Id: <200910050440.n954e2Ja017950@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: "Dyadchenko Mihail, Siberian Networks" Cc: Subject: Re: kern/134931: [route] [fib] Route messages sent to all socket listeners regardless of setfib X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "Dyadchenko Mihail, Siberian Networks" List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 04:40:03 -0000 The following reply was made to PR kern/134931; it has been noted by GNATS. From: "Dyadchenko Mihail, Siberian Networks" To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/134931: [route] [fib] Route messages sent to all socket listeners regardless of setfib Date: Mon, 5 Oct 2009 11:30:32 +0700 Hello. Thanks, 7.2-Release was patched and recompiled successful, test with route add + route monitor also looks normal. May be on this week I will test in in real network. -- Dyadchenko Mihail Senior System Administrator, Siberian Networks Tel. (383) 205 0000 Fax. (383) 201-13-54 M.Dyadchenko@sibset-team.ru http://www.sibset.ru/ From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 06:03:47 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21135106568B for ; Mon, 5 Oct 2009 06:03:47 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id DAE698FC13 for ; Mon, 5 Oct 2009 06:03:46 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id DDEF6730DA; Mon, 5 Oct 2009 08:10:25 +0200 (CEST) Date: Mon, 5 Oct 2009 08:10:25 +0200 From: Luigi Rizzo To: Eugene Grosbein Message-ID: <20091005061025.GB55845@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091005025521.GA52702@svzserv.kemerovo.su> User-Agent: Mutt/1.4.2.3i Cc: rihad , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 06:03:47 -0000 On Mon, Oct 05, 2009 at 10:55:21AM +0800, Eugene Grosbein wrote: > On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: > > > sysctls: > > kern.ipc.nmbclusters=50000 > > net.inet.ip.dummynet.io_fast=1 > > I guess you should also try to increase pipes length: > > net.inet.ip.dummynet.hash_size=65536 > net.inet.ip.dummynet.pipe_slot_limit=1000 in fact, i forgot to ask, we'd need to know the output of "ipfw pipe show" just to get the idea if there is any known reason for the drop (e.g. queues too short, etc.) cheers luigi > And reconfigure pipes like this: > > ipfw pipe NNN config bw ... queue 1000 > > And default 'taildrop' policy of dummynet pipes may be guilty, > you'd use GRED to prevent excessive drops. > > Eugene Grosbein > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 08:51:11 2009 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 925A2106566B for ; Mon, 5 Oct 2009 08:51:11 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 074F28FC1D for ; Mon, 5 Oct 2009 08:51:10 +0000 (UTC) Received: (qmail 20015 invoked from network); 5 Oct 2009 07:52:26 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 5 Oct 2009 07:52:26 -0000 Message-ID: <4AC9AD41.2070200@freebsd.org> Date: Mon, 05 Oct 2009 10:24:33 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: Kevin Oberman References: <20091002231826.066291CC0E@ptavv.es.net> In-Reply-To: <20091002231826.066291CC0E@ptavv.es.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: net@freebsd.org Subject: Re: Unusual TCP options can cause FreeBSD to issue a reset X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 08:51:11 -0000 Kevin Oberman wrote: > I have a system that is unable to connect to a FreeBSD system due to > the odd formatting of the TCP options. The options contains only the > timestamp which, if recommendations in RFC1323 are followed, are > preceded by two NOP bytes to put the timestamp values on 4 byte > boundaries. > > This system sends the 12 byte timestamp option alone, followed by two > zero bytes to pad it. This meets the requirements of RFC793 and 1323 is > explicit that stacks must accept this, even though it results in > sub-optimal performance and does not meet the recommendations in 1323 > appendix A. Just this alone should not cause a reset from FreeBSD. > Any idea if this is hard to fix? I see in on both 7.2 and 8.0. Can you post a detailed tcpdump of a failing connect? And please enable net.inet.tcp.log_debug which should give a better explanation on why FreeBSD thinks the connection is bad. -- Andre From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 08:53:23 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2518B1065676 for ; Mon, 5 Oct 2009 08:53:23 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx30.mail.ru (mx30.mail.ru [94.100.176.44]) by mx1.freebsd.org (Postfix) with ESMTP id B27DC8FC19 for ; Mon, 5 Oct 2009 08:53:22 +0000 (UTC) Received: from [217.25.27.27] (port=47720 helo=[217.25.27.27]) by mx30.mail.ru with asmtp id 1MujJo-0008p2-00; Mon, 05 Oct 2009 12:53:21 +0400 Message-ID: <4AC9B400.9020400@mail.ru> Date: Mon, 05 Oct 2009 13:53:20 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> In-Reply-To: <20091005061025.GB55845@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 08:53:23 -0000 Luigi Rizzo wrote: > On Mon, Oct 05, 2009 at 10:55:21AM +0800, Eugene Grosbein wrote: >> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote: >> >>> sysctls: >>> kern.ipc.nmbclusters=50000 >>> net.inet.ip.dummynet.io_fast=1 >> I guess you should also try to increase pipes length: >> >> net.inet.ip.dummynet.hash_size=65536 >> net.inet.ip.dummynet.pipe_slot_limit=1000 > > in fact, i forgot to ask, we'd need to know the output of > "ipfw pipe show" just to get the idea if there is any > known reason for the drop (e.g. queues too short, etc.) > The pipes are fine, each normally having 100-120 concurrent consumers (i.e. active users). (The max number of consumers allowed per pipe is hash_size*max_chain_len, normally 64*16==1024.) kern.ipc.nmbclusters=111111 And these are by default: #net.inet.ip.dummynet.hash_size=64 #net.inet.ip.dummynet.max_chain_len=16 Yesterday I set up a cronjob logging the number of drops in the past 15 minutes: */15 * * * * (echo -n "$(date) "; netstat -z -s 2>/dev/null | fgrep -w "output packets dropped") >> /tmp/bufs.log And here's bufs.log: Sun Oct 4 21:45:00 AZST 2009 418869 output packets dropped due to no bufs, etc. Sun Oct 4 22:00:00 AZST 2009 851693 output packets dropped due to no bufs, etc. Sun Oct 4 22:15:01 AZST 2009 932885 output packets dropped due to no bufs, etc. Sun Oct 4 22:30:00 AZST 2009 890522 output packets dropped due to no bufs, etc. Sun Oct 4 22:45:00 AZST 2009 1065931 output packets dropped due to no bufs, etc. Sun Oct 4 23:00:00 AZST 2009 937863 output packets dropped due to no bufs, etc. Sun Oct 4 23:15:01 AZST 2009 1018822 output packets dropped due to no bufs, etc. Sun Oct 4 23:30:00 AZST 2009 981922 output packets dropped due to no bufs, etc. Sun Oct 4 23:45:00 AZST 2009 1015124 output packets dropped due to no bufs, etc. Mon Oct 5 00:00:01 AZST 2009 1123926 output packets dropped due to no bufs, etc. Mon Oct 5 00:15:01 AZST 2009 948161 output packets dropped due to no bufs, etc. Mon Oct 5 00:30:00 AZST 2009 937277 output packets dropped due to no bufs, etc. Mon Oct 5 00:45:00 AZST 2009 875218 output packets dropped due to no bufs, etc. Mon Oct 5 01:00:00 AZST 2009 803527 output packets dropped due to no bufs, etc. Mon Oct 5 01:15:00 AZST 2009 728639 output packets dropped due to no bufs, etc. Mon Oct 5 01:30:00 AZST 2009 626154 output packets dropped due to no bufs, etc. Mon Oct 5 01:45:00 AZST 2009 519441 output packets dropped due to no bufs, etc. Mon Oct 5 02:00:00 AZST 2009 371098 output packets dropped due to no bufs, etc. Mon Oct 5 02:15:00 AZST 2009 681243 output packets dropped due to no bufs, etc. Mon Oct 5 02:30:00 AZST 2009 562909 output packets dropped due to no bufs, etc. Mon Oct 5 02:45:00 AZST 2009 426734 output packets dropped due to no bufs, etc. Mon Oct 5 03:00:00 AZST 2009 344619 output packets dropped due to no bufs, etc. Mon Oct 5 03:15:00 AZST 2009 90006 output packets dropped due to no bufs, etc. Mon Oct 5 03:30:00 AZST 2009 17064 output packets dropped due to no bufs, etc. Mon Oct 5 03:45:00 AZST 2009 3851 output packets dropped due to no bufs, etc. Mon Oct 5 04:00:00 AZST 2009 1323 output packets dropped due to no bufs, etc. Mon Oct 5 04:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 04:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 04:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:30:01 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 05:45:01 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:15:01 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 06:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 07:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 08:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:30:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 09:45:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 10:00:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 10:15:00 AZST 2009 0 output packets dropped due to no bufs, etc. Mon Oct 5 10:30:00 AZST 2009 177 output packets dropped due to no bufs, etc. Mon Oct 5 10:45:00 AZST 2009 1701 output packets dropped due to no bufs, etc. Mon Oct 5 11:00:01 AZST 2009 19933 output packets dropped due to no bufs, etc. Mon Oct 5 11:15:00 AZST 2009 30003 output packets dropped due to no bufs, etc. Mon Oct 5 11:30:00 AZST 2009 56712 output packets dropped due to no bufs, etc. Mon Oct 5 11:45:00 AZST 2009 78721 output packets dropped due to no bufs, etc. Mon Oct 5 12:00:01 AZST 2009 112518 output packets dropped due to no bufs, etc. Mon Oct 5 12:15:00 AZST 2009 7229 output packets dropped due to no bufs, etc. Mon Oct 5 12:30:01 AZST 2009 24965 output packets dropped due to no bufs, etc. Mon Oct 5 12:45:00 AZST 2009 75900 output packets dropped due to no bufs, etc. Mon Oct 5 13:00:00 AZST 2009 45002 output packets dropped due to no bufs, etc. Mon Oct 5 13:15:00 AZST 2009 67161 output packets dropped due to no bufs, etc. Mon Oct 5 13:30:00 AZST 2009 112591 output packets dropped due to no bufs, etc. As you can see the drops gradually went away completely at about 4:00 a.m., and started coming up at about 10:30 a.m., although at a lower rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the "quiet" times, is about 200-330 mbit/s 5 minute average, without a single drop. But after that, in come the drops, no matter how high I set the queue. Should I try 10000 slots? 20000? I'm really sure there are plenty of heavy downloaders between 4a.m. and 10a.m., but still without a single drop before approx. 330 mbit/s! Strange, isn't it? This makes me believe I'm hitting some other global memory limit, rather than per-user limit, at around 340-350 mbit/s, causing drops. top and netstat -m are OK right now, though: Mem: 1870M Active, 1220M Inact, 481M Wired, 201M Cache, 214M Buf, 164M Free 15595/5385/20980/111112 mbuf clusters in use (current/cache/total/max) From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 09:01:08 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1F5B106566B for ; Mon, 5 Oct 2009 09:01:08 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 359478FC12 for ; Mon, 5 Oct 2009 09:01:07 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n959122t070683; Mon, 5 Oct 2009 17:01:02 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95912UL070682; Mon, 5 Oct 2009 17:01:02 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 17:01:02 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005090102.GA70430@svzserv.kemerovo.su> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9B400.9020400@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 09:01:08 -0000 On Mon, Oct 05, 2009 at 01:53:20PM +0500, rihad wrote: > As you can see the drops gradually went away completely at about 4:00 > a.m., and started coming up at about 10:30 a.m., although at a lower > rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at > 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the > "quiet" times, is about 200-330 mbit/s 5 minute average, without a > single drop. But after that, in come the drops, no matter how high I set > the queue. Should I try 10000 slots? 20000? First switch from taildrop (default) to GRED, it is designed to fight your problem. Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 09:29:04 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2DD4106573E for ; Mon, 5 Oct 2009 09:29:01 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id CA8518FC1B for ; Mon, 5 Oct 2009 09:29:00 +0000 (UTC) Received: from [217.25.27.27] (port=44411 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1MujsJ-000FnN-00; Mon, 05 Oct 2009 13:28:59 +0400 Message-ID: <4AC9BC5A.50902@mail.ru> Date: Mon, 05 Oct 2009 14:28:58 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> In-Reply-To: <20091005090102.GA70430@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 09:29:04 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 01:53:20PM +0500, rihad wrote: > >> As you can see the drops gradually went away completely at about 4:00 >> a.m., and started coming up at about 10:30 a.m., although at a lower >> rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at >> 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the >> "quiet" times, is about 200-330 mbit/s 5 minute average, without a >> single drop. But after that, in come the drops, no matter how high I set >> the queue. Should I try 10000 slots? 20000? > > First switch from taildrop (default) to GRED, it is designed to fight > your problem. > Oh, I almost forgot... Right now I've googled up and am reading this intro: http://www-rp.lip6.fr/~sf/WebSF/PapersWeb/iscc01.ps So turning to GRED would turn my FreeBSD router from dumb into a smart router that knows TCP? I thought pushing bits around at a lower level, and a sufficient queue size were enough. Still not sure why increasing queue size as high as I want doesn't completely eliminate drops. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 09:48:32 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6D471065672 for ; Mon, 5 Oct 2009 09:48:32 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx48.mail.ru (mx48.mail.ru [94.100.176.62]) by mx1.freebsd.org (Postfix) with ESMTP id 62C368FC33 for ; Mon, 5 Oct 2009 09:48:32 +0000 (UTC) Received: from [217.25.27.27] (port=63094 helo=[217.25.27.27]) by mx48.mail.ru with asmtp id 1MukBC-000DkE-00; Mon, 05 Oct 2009 13:48:30 +0400 Message-ID: <4AC9C0ED.3020904@mail.ru> Date: Mon, 05 Oct 2009 14:48:29 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> In-Reply-To: <20091005090102.GA70430@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 09:48:32 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 01:53:20PM +0500, rihad wrote: > >> As you can see the drops gradually went away completely at about 4:00 >> a.m., and started coming up at about 10:30 a.m., although at a lower >> rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at >> 10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the >> "quiet" times, is about 200-330 mbit/s 5 minute average, without a >> single drop. But after that, in come the drops, no matter how high I set >> the queue. Should I try 10000 slots? 20000? > > First switch from taildrop (default) to GRED, it is designed to fight > your problem. > red | gred w_q/min_th/max_th/max_p Make use of the RED (Random Early Detection) queue management algo- rithm. w_q and max_p are floating point numbers between 0 and 1 (0 not included), while min_th and max_th are integer numbers specify- ing thresholds for queue management (thresholds are computed in bytes if the queue has been defined in bytes, in slots otherwise). The dummynet(4) also supports the gentle RED variant (gred). Do you or someone else know what w_q and max_p are? There's just too much info for me to grasp here: http://www.icir.org/floyd/red.html From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 09:56:02 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E086106566B for ; Mon, 5 Oct 2009 09:56:02 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id DE7428FC14 for ; Mon, 5 Oct 2009 09:56:01 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n959u07e073805; Mon, 5 Oct 2009 17:56:00 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n959u0EE073804; Mon, 5 Oct 2009 17:56:00 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 17:56:00 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005095600.GA73335@svzserv.kemerovo.su> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9BC5A.50902@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 09:56:02 -0000 On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote: > Oh, I almost forgot... Right now I've googled up and am reading this > intro: http://www-rp.lip6.fr/~sf/WebSF/PapersWeb/iscc01.ps > > So turning to GRED would turn my FreeBSD router from dumb into a smart > router that knows TCP? I thought pushing bits around at a lower level, > and a sufficient queue size were enough. No, it will still deal with IP packets but more clever. > Still not sure why increasing queue size as high as I want doesn't > completely eliminate drops. The goal is to make sources of traffic to slow down, this is the only way to descrease drops - any finite queue may be overhelmed with traffic. Taildrop does not really help with this. GRED does much better. Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 09:58:07 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 295AE1065672 for ; Mon, 5 Oct 2009 09:58:07 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id E09138FC21 for ; Mon, 5 Oct 2009 09:58:06 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 2390B730DA; Mon, 5 Oct 2009 12:04:46 +0200 (CEST) Date: Mon, 5 Oct 2009 12:04:46 +0200 From: Luigi Rizzo To: Eugene Grosbein Message-ID: <20091005100446.GA60244@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091005095600.GA73335@svzserv.kemerovo.su> User-Agent: Mutt/1.4.2.3i Cc: rihad , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 09:58:07 -0000 On Mon, Oct 05, 2009 at 05:56:00PM +0800, Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote: > > > Oh, I almost forgot... Right now I've googled up and am reading this > > intro: http://www-rp.lip6.fr/~sf/WebSF/PapersWeb/iscc01.ps > > > > So turning to GRED would turn my FreeBSD router from dumb into a smart > > router that knows TCP? I thought pushing bits around at a lower level, > > and a sufficient queue size were enough. > > No, it will still deal with IP packets but more clever. > > > Still not sure why increasing queue size as high as I want doesn't > > completely eliminate drops. > > The goal is to make sources of traffic to slow down, this is the only > way to descrease drops - any finite queue may be overhelmed with traffic. > Taildrop does not really help with this. GRED does much better. i think the first problem here is figure out _why_ we have the drops, as the original poster said that queues are configured with a very large amount of buffer (and i think there is a misconfiguration somewhere because the mbuf stats do not make sense) cheers luigi From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 09:59:19 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE1EF106566B for ; Mon, 5 Oct 2009 09:59:19 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 4A5498FC0C for ; Mon, 5 Oct 2009 09:59:18 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n959xGq6073972; Mon, 5 Oct 2009 17:59:16 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n959xGpH073971; Mon, 5 Oct 2009 17:59:16 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 17:59:16 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005095916.GB73335@svzserv.kemerovo.su> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9C0ED.3020904@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9C0ED.3020904@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 09:59:19 -0000 On Mon, Oct 05, 2009 at 02:48:29PM +0500, rihad wrote: > >First switch from taildrop (default) to GRED, it is designed to fight > >your problem. > > red | gred w_q/min_th/max_th/max_p > Make use of the RED (Random Early Detection) queue > management algo- > rithm. w_q and max_p are floating point numbers between 0 > and 1 (0 > not included), while min_th and max_th are integer numbers > specify- > ing thresholds for queue management (thresholds are computed in > bytes if the queue has been defined in bytes, in slots > otherwise). > The dummynet(4) also supports the gentle RED variant (gred). > > Do you or someone else know what w_q and max_p are? > > There's just too much info for me to grasp here: > http://www.icir.org/floyd/red.html http://docs.freebsd.org/cgi/getmsg.cgi?fetch=126518+0+archive/2009/freebsd-net/20091004.freebsd-net Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 10:05:36 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C3321065679 for ; Mon, 5 Oct 2009 10:05:36 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id A9E7D8FC29 for ; Mon, 5 Oct 2009 10:05:35 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95A5WF2074408; Mon, 5 Oct 2009 18:05:32 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95A5Wlt074407; Mon, 5 Oct 2009 18:05:32 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 18:05:32 +0800 From: Eugene Grosbein To: Luigi Rizzo Message-ID: <20091005100532.GC73335@svzserv.kemerovo.su> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091005100446.GA60244@onelab2.iet.unipi.it> User-Agent: Mutt/1.4.2.3i Cc: rihad , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 10:05:36 -0000 On Mon, Oct 05, 2009 at 12:04:46PM +0200, Luigi Rizzo wrote: > > The goal is to make sources of traffic to slow down, this is the only > > way to descrease drops - any finite queue may be overhelmed with traffic. > > Taildrop does not really help with this. GRED does much better. > > i think the first problem here is figure out _why_ we have > the drops, as the original poster said that queues are configured > with a very large amount of buffer (and i think there is a > misconfiguration somewhere because the mbuf stats do not make > sense) That may be very simple, f.e. wide uplink channel and policy that dictates slower client speeds. Any taildrop queue would drop lots of packets. Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 10:21:01 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D3B6D1065672 for ; Mon, 5 Oct 2009 10:21:01 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id 8CF728FC08 for ; Mon, 5 Oct 2009 10:21:00 +0000 (UTC) Received: from [217.25.27.27] (port=32491 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1Mukgc-0005mZ-00; Mon, 05 Oct 2009 14:20:59 +0400 Message-ID: <4AC9C88A.5050509@mail.ru> Date: Mon, 05 Oct 2009 15:20:58 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> In-Reply-To: <20091005100532.GC73335@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 10:21:01 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 12:04:46PM +0200, Luigi Rizzo wrote: > >>> The goal is to make sources of traffic to slow down, this is the only >>> way to descrease drops - any finite queue may be overhelmed with traffic. >>> Taildrop does not really help with this. GRED does much better. >> i think the first problem here is figure out _why_ we have >> the drops, as the original poster said that queues are configured >> with a very large amount of buffer (and i think there is a >> misconfiguration somewhere because the mbuf stats do not make >> sense) > > That may be very simple, f.e. wide uplink channel and policy that > dictates slower client speeds. Any taildrop queue would drop lots > of packets. > If uplink is e.g. 100 mbit/s, but data is fed to client by dummynet at 1 mbit/s, doesn't the _client's_ TCP software know to slow things down to not overwhelm 1 mbit/s? Where has TCP slow-start gone? My router box isn't some application proxy that starts downloading at full 100 mbit/s thus quickly filling client's 1 mbit/s link. It's just a router. Although it doesn't yet make sense to me, I'll try going to GRED soon. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 10:52:41 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9C0B1065672 for ; Mon, 5 Oct 2009 10:52:41 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx33.mail.ru (mx33.mail.ru [94.100.176.47]) by mx1.freebsd.org (Postfix) with ESMTP id 94D9D8FC1A for ; Mon, 5 Oct 2009 10:52:41 +0000 (UTC) Received: from [217.25.27.27] (port=30017 helo=[217.25.27.27]) by mx33.mail.ru with asmtp id 1MulBI-0003bj-00; Mon, 05 Oct 2009 14:52:40 +0400 Message-ID: <4AC9CFF7.3090208@mail.ru> Date: Mon, 05 Oct 2009 15:52:39 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> In-Reply-To: <20091005095600.GA73335@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 10:52:41 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote: > >> Still not sure why increasing queue size as high as I want doesn't >> completely eliminate drops. > > The goal is to make sources of traffic to slow down, this is the only > way to descrease drops - any finite queue may be overhelmed with traffic. > Taildrop does not really help with this. GRED does much better. > Alright, so I changed to gred by adding to each config command: ipfw ... gred 0.002/900/1000/0.1 queue 1000 and reconfigured. Still around 300-400 drops per second, which was typical at this load level before with taildrop anyway. There are around 3-5 mbit/s being wasted according to systat -ifstat. Should I now increase slots to 5-10-20k? Very strange. "ipfw pipe show" correctly shows that gred is at work. For example: 00512: 512.000 Kbit/s 0 ms 1000 sl. 79 queues (64 buckets) GRED w_q 0.001999 min_th 900 max_th 1000 max_p 0.099991 mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 ... From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:00:47 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFF51106566B for ; Mon, 5 Oct 2009 11:00:47 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 72E768FC15 for ; Mon, 5 Oct 2009 11:00:47 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id E40A8730DA; Mon, 5 Oct 2009 13:07:26 +0200 (CEST) Date: Mon, 5 Oct 2009 13:07:26 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091005110726.GA62598@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9CFF7.3090208@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Eugene Grosbein Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:00:47 -0000 On Mon, Oct 05, 2009 at 03:52:39PM +0500, rihad wrote: > Eugene Grosbein wrote: > >On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote: > > > >>Still not sure why increasing queue size as high as I want doesn't > >>completely eliminate drops. > > > >The goal is to make sources of traffic to slow down, this is the only > >way to descrease drops - any finite queue may be overhelmed with traffic. > >Taildrop does not really help with this. GRED does much better. > > > > Alright, so I changed to gred by adding to each config command: > ipfw ... gred 0.002/900/1000/0.1 queue 1000 > and reconfigured. Still around 300-400 drops per second, which was > typical at this load level before with taildrop anyway. There are around > 3-5 mbit/s being wasted according to systat -ifstat. > > Should I now increase slots to 5-10-20k? > Very strange. > > "ipfw pipe show" correctly shows that gred is at work. For example: > 00512: 512.000 Kbit/s 0 ms 1000 sl. 79 queues (64 buckets) > GRED w_q 0.001999 min_th 900 max_th 1000 max_p 0.099991 > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > ... you keep omitting the important info i.e. whether individual pipes have drops, significant queue lenghts and so on. i am giving up! From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:06:57 2009 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D83E106568D for ; Mon, 5 Oct 2009 11:06:57 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7A7998FC19 for ; Mon, 5 Oct 2009 11:06:57 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n95B6vf8088749 for ; Mon, 5 Oct 2009 11:06:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n95B6uOW088747 for freebsd-net@FreeBSD.org; Mon, 5 Oct 2009 11:06:56 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 5 Oct 2009 11:06:56 GMT Message-Id: <200910051106.n95B6uOW088747@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-net@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:06:57 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/139346 net [patch] arp(8) add option to remove static entries lis o kern/139268 net [if_bridge] [patch] allow if_bridge to forward just VL o kern/139204 net [arp] DHCP server replies rejected, ARP entry lost bef o kern/139162 net [fwip] [panic] 8.0-RC1 panics if using IP over firewir o kern/139145 net [ip6] IPv6 blackhole / reject routes broken o kern/139117 net [lagg] + wlan boot timing (EBUSY) o kern/139113 net [arp] removing IP alias doesn't delete permanent arp e o kern/139058 net [ipfilter] mbuf cluster leak on FreeBSD 7.2 o kern/138999 net [libc] lighttpd/php-cgi with freebsd sendfile(2) enabl o kern/138850 net [dummynet] dummynet doesn't work correctly on a bridge o kern/138782 net [panic] sbflush_internal: cc 0 || mb 0xffffff004127b00 o kern/138739 net [wpi] wpi(4) does not work very well under 8.0-BETA4 o kern/138694 net [bge] FreeBSD 6.3 release does not recognize Broadcom o amd64/138688 net [rum] possibly broken on 8 Beta 4 amd64: able to wpa a o kern/138678 net [lo] FreeBSD does not assign linklocal address to loop o kern/138676 net [route] after buildworld not work local routes [regres f kern/138666 net [multicast] [panic] not working multicast through igmp o kern/138660 net [igb] igb driver troubles in 8.0-BETA4 o kern/138652 net TCP window scaling value calculated incorrectly? o kern/138620 net [lagg] [patch] lagg port bpf-writes blocked o kern/138427 net [wpi] [panic] Kernel panic after trying set monitor wl o kern/138407 net [gre] gre(4) interface does not come up after reboot o kern/138390 net [gif] [patch] NULL pointer dereference in gif_input() o kern/138378 net [altq] [patch] Memory leak in hfsc_class_modify() in f o kern/138332 net [tun] [lor] ifconfig tun0 destroy causes LOR on 8.0-BE o kern/138266 net [panic] kernel panic when udp benchmark test used as r o kern/138177 net [ipfilter] FreeBSD crashing repeatedly in ip_nat.c:257 o kern/138130 net [netinet] [patch] Resource leak in LibAliasRefreshModu o kern/138046 net [tcp] tcp sockets stay in SYN_SENT even after receivin o kern/137881 net [netgraph] [panic] ng_pppoe fatal trap 12 o bin/137841 net [patch] wpa_supplicant(8) cannot verify SHA256 signed p kern/137795 net [sctp] [panic] mtx_lock() of destroyed mutex o kern/137776 net [rum] panic in rum(4) driver on 8.0-BETA2 o kern/137775 net [netgraph] [patch] Add XMIT_FAILOVER to ng_one2many o bin/137641 net ifconfig(8): various problems with "vlan_device.vlan_i o kern/137592 net [ath] panic - 7-STABLE (Aug 7, 2009 UTC) crashes on ne o bin/137484 net [patch] Integer overflow in wpa_supplicant(8) base64 e o kern/137392 net [ip] [panic] crash in ip_nat.c line 2577 o kern/137372 net [ral] FreeBSD doesn't support wireless interface from o kern/137317 net [tcp] logs full of syncache problems o kern/137292 net [ste] DFE-580TX not working properly o kern/137279 net [bge] [panic] Page fault (fatal trap 12) NFS server w/ o kern/137170 net [ath] atheros AR9285 not recognised o kern/137089 net [lagg] lagg falsely triggers IPv6 duplicate address de o bin/136994 net [patch] ifconfig(8) print carp mac address o kern/136943 net [wpi] [lor] wpi0_com_lock / wpi0 o kern/136911 net [netgraph] [panic] system panic on kldload ng_bpf.ko t o kern/136876 net [bge] bge will not resume properly after suspend o kern/136836 net [ath] atheros card stops functioning after about 12 ho o kern/136618 net [pf][stf] panic on cloning interface without unit numb o kern/136482 net [age] Attansic L1 Gigabit Ethernet recieves multicasts o kern/136426 net [panic] spawning several dhclients in parallel panics o kern/136168 net [em] em driver initialization fails on Intel 5000PSL m o kern/135836 net [bce] bce BCM5709 Watchdog after warm boot - ok after o kern/135502 net [periodic] Warning message raised by rtfree function i o kern/135222 net [igb] low speed routing between two igb interfaces o kern/135067 net [patch] [fib] Incorrect KASSERTs in sys/net/route.c o kern/134956 net [em] FreeBSD 7.1 & 7.2, Intel PRO/1000 PT Quad Port Se o kern/134931 net [route] [fib] Route messages sent to all socket listen o kern/134658 net [bce] bce driver fails on PowerEdge m610 blade. o kern/134583 net [hang] Machine with jail freezes after random amount o o kern/134531 net [route] [panic] kernel crash related to routes/zebra o kern/134401 net [msk] [panic] Kernel Fatal trap 12: page fault while i o kern/134369 net [route] [ip6] IPV6 in Head broken for routing table up o kern/134168 net [ral] ral driver problem on RT2525 2.4GHz transceiver o kern/134157 net [dummynet] dummynet loads cpu for 100% and make a syst o kern/134079 net [em] "em0: Invalid MAC address" in FreeBSD-Current ( 8 o kern/133969 net [dummynet] [panic] Fatal trap 12: page fault while in o kern/133968 net [dummynet] [panic] dummynet kernel panic o kern/133902 net [tun] Killing tun0 iface ssh tunnel causes Panic Strin o kern/133786 net [netinet] [patch] ip_input might cause kernel panic o kern/133736 net [udp] ip_id not protected ... o kern/133613 net [wpi] [panic] kernel panic in wpi(4) o kern/133595 net [panic] Kernel Panic at pcpu.h:195 o kern/133572 net [ppp] [hang] incoming PPTP connection hangs the system o kern/133490 net [bpf] [panic] 'kmem_map too small' panic on Dell r900 o kern/133328 net [bge] [panic] Kernel panics with Windows7 client o kern/133235 net [netinet] [patch] Process SIOCDLIFADDR command incorre o kern/133218 net [carp] [hang] use of carp(4) causes system to freeze f kern/133213 net arp and sshd errors on 7.1-PRERELEASE o kern/133204 net [msk] msk driver timeouts o kern/133060 net [ipsec] [pfsync] [panic] Kernel panic with ipsec + pfs o kern/132991 net [bge] if_bge low performance problem f bin/132911 net ip6fw(8): argument type of fill_icmptypes is wrong and o kern/132889 net [ndis] [panic] NDIS kernel crash on load BCM4321 AGN d o kern/132885 net [wlan] 802.1x broken after SVN rev 189592 o conf/132851 net [fib] [patch] allow to setup fib for service running f o kern/132832 net [netinet] [patch] tcp_output() might generate invalid o bin/132798 net [patch] ggatec(8): ggated/ggatec connection slowdown p o kern/132734 net [ifmib] [panic] panic in net/if_mib.c o kern/132722 net [ath] Wifi ath0 associates fine with AP, but DHCP or I o kern/132705 net [libwrap] [patch] libwrap - infinite loop if hosts.all o kern/132672 net [ndis] [panic] ndis with rt2860.sys causes kernel pani o kern/132669 net [xl] 3c905-TX send DUP! in reply on ping (sometime) o kern/132625 net [iwn] iwn drivers don't support setting country o kern/132554 net [ipl] There is no ippool start script/ipfilter magic t o kern/132354 net [nat] Getting some packages to ipnat(8) causes crash o kern/132285 net [carp] alias gives incorrect hash in dmesg o kern/132277 net [crypto] [ipsec] poor performance using cryptodevice f o kern/132107 net [carp] carp(4) advskew setting ignored when carp IP us o kern/131781 net [ndis] ndis keeps dropping the link o kern/131776 net [wi] driver fails to init o kern/131753 net [altq] [panic] kernel panic in hfsc_dequeue o bin/131567 net [socket] [patch] Update for regression/sockets/unix_cm o kern/131549 net ifconfig(8) can't clear 'monitor' mode on the wireless o kern/131536 net [netinet] [patch] kernel does allow manipulation of su o bin/131365 net route(8): route add changes interpretation of network o kern/131162 net [ath] Atheros driver bugginess and kernel crashes o kern/131153 net [iwi] iwi doesn't see a wireless network f kern/131087 net [ipw] [panic] ipw / iwi - no sent/received packets; iw f kern/130820 net [ndis] wpa_supplicant(8) returns 'no space on device' o kern/130628 net [nfs] NFS / rpc.lockd deadlock on 7.1-R o conf/130555 net [rc.d] [patch] No good way to set ipfilter variables a o kern/130525 net [ndis] [panic] 64 bit ar5008 ndisgen-erated driver cau o kern/130311 net [wlan_xauth] [panic] hostapd restart causing kernel pa o kern/130109 net [ipfw] Can not set fib for packets originated from loc f kern/130059 net [panic] Leaking 50k mbufs/hour o kern/129750 net [ath] Atheros AR5006 exits on "cannot map register spa f kern/129719 net [nfs] [panic] Panic during shutdown, tcp_ctloutput: in o kern/129580 net [ndis] Netgear WG311v3 (ndis) causes kenel trap at boo o kern/129517 net [ipsec] [panic] double fault / stack overflow o kern/129508 net [carp] [panic] Kernel panic with EtherIP (may be relat o kern/129352 net [xl] [patch] xl0 watchdog timeout o kern/129219 net [ppp] Kernel panic when using kernel mode ppp o kern/129197 net [panic] 7.0 IP stack related panic o kern/129135 net [vge] vge driver on a VIA mini-ITX not working o bin/128954 net ifconfig(8) deletes valid routes o kern/128917 net [wpi] [panic] if_wpi and wpa+tkip causing kernel panic o kern/128884 net [msk] if_msk page fault while in kernel mode o kern/128840 net [igb] page fault under load with igb/LRO o bin/128602 net [an] wpa_supplicant(8) crashes with an(4) o kern/128598 net [bluetooth] WARNING: attempt to net_add_domain(bluetoo o kern/128448 net [nfs] 6.4-RC1 Boot Fails if NFS Hostname cannot be res o conf/128334 net [request] use wpa_cli in the "WPA DHCP" situation o bin/128295 net [patch] ifconfig(8) does not print TOE4 or TOE6 capabi o bin/128001 net wpa_supplicant(8), wlan(4), and wi(4) issues o kern/127928 net [tcp] [patch] TCP bandwidth gets squeezed every time t o kern/127834 net [ixgbe] [patch] wrong error counting o kern/127826 net [iwi] iwi0 driver has reduced performance and connecti o kern/127815 net [gif] [patch] if_gif does not set vlan attributes from o kern/127724 net [rtalloc] rtfree: 0xc5a8f870 has 1 refs f bin/127719 net [arp] arp: Segmentation fault (core dumped) s kern/127587 net [bge] [request] if_bge(4) doesn't support BCM576X fami f kern/127528 net [icmp]: icmp socket receives icmp replies not owned by o bin/127192 net routed(8) removes the secondary alias IP of interface f kern/127145 net [wi]: prism (wi) driver crash at bigger traffic o kern/127102 net [wpi] Intel 3945ABG low throughput o kern/127057 net [udp] Unable to send UDP packet via IPv6 socket to IPv o kern/127050 net [carp] ipv6 does not work on carp interfaces [regressi o kern/126945 net [carp] CARP interface destruction with ifconfig destro o kern/126924 net [an] [patch] printf -> device_printf and simplify prob o kern/126895 net [patch] [ral] Add antenna selection (marked as TBD) o kern/126874 net [vlan]: Zebra problem if ifconfig vlanX destroy o bin/126822 net wpa_supplicant(8): WPA PSK does not work in adhoc mode o kern/126714 net [carp] CARP interface renaming makes system no longer o kern/126695 net rtfree messages and network disruption upon use of if_ o kern/126688 net [ixgbe] [patch] 1.4.7 ixgbe driver panic with 4GB and o kern/126475 net [ath] [panic] ath pcmcia card inevitably panics under o kern/126339 net [ipw] ipw driver drops the connection o kern/126214 net [ath] txpower problem with Atheros wifi card o kern/126075 net [inet] [patch] internet control accesses beyond end of o bin/125922 net [patch] Deadlock in arp(8) o kern/125920 net [arp] Kernel Routing Table loses Ethernet Link status o kern/125845 net [netinet] [patch] tcp_lro_rx() should make use of hard o kern/125816 net [carp] [if_bridge] carp stuck in init when using bridg f kern/125502 net [ral] ifconfig ral0 scan produces no output unless in o kern/125258 net [socket] socket's SO_REUSEADDR option does not work o kern/125239 net [gre] kernel crash when using gre o kern/124767 net [iwi] Wireless connection using iwi0 driver (Intel 220 o kern/124753 net [ieee80211] net80211 discards power-save queue packets o kern/124341 net [ral] promiscuous mode for wireless device ral0 looses o kern/124160 net [libc] connect(2) function loops indefinitely o kern/124127 net [msk] watchdog timeout (missed Tx interrupts) -- recov o kern/124021 net [ip6] [panic] page fault in nd6_output() o kern/123968 net [rum] [panic] rum driver causes kernel panic with WPA. p kern/123961 net [vr] [patch] Allow vr interface to handle vlans o kern/123892 net [tap] [patch] No buffer space available o kern/123890 net [ppp] [panic] crash & reboot on work with PPP low-spee o kern/123858 net [stf] [patch] stf not usable behind a NAT o kern/123796 net [ipf] FreeBSD 6.1+VPN+ipnat+ipf: port mapping does not o bin/123633 net ifconfig(8) doesn't set inet and ether address in one f kern/123617 net [tcp] breaking connection when client downloading file o kern/123603 net [tcp] tcp_do_segment and Received duplicate SYN o kern/123559 net [iwi] iwi periodically disassociates/associates [regre o bin/123465 net [ip6] route(8): route add -inet6 -interfac o kern/123463 net [ipsec] [panic] repeatable crash related to ipsec-tool o kern/123429 net [nfe] [hang] "ifconfig nfe up" causes a hard system lo o kern/123347 net [bge] bge1: watchdog timeout -- linkstate changed to D o conf/123330 net [nsswitch.conf] Enabling samba wins in nsswitch.conf c o kern/123256 net [wpi] panic: blockable sleep lock with wpi(4) f kern/123172 net [bce] Watchdog timeout problems with if_bce o kern/123160 net [ip] Panic and reboot at sysctl kern.polling.enable=0 o kern/122989 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/122954 net [lagg] IPv6 EUI64 incorrectly chosen for lagg devices o kern/122928 net [em] interface watchdog timeouts and stops receiving p f kern/122839 net [multicast] FreeBSD 7 multicast routing problem o kern/122780 net [lagg] tcpdump on lagg interface during high pps wedge o kern/122772 net [em] em0 taskq panic, tcp reassembly bug causes radix o kern/122743 net [mbuf] [panic] vm_page_unwire: invalid wire count: 0 o kern/122697 net [ath] Atheros card is not well supported o kern/122685 net It is not visible passing packets in tcpdump(1) o kern/122551 net [bge] Broadcom 5715S no carrier on HP BL460c blade usi o kern/122319 net [wi] imposible to enable ad-hoc demo mode with Orinoco o kern/122290 net [netgraph] [panic] Netgraph related "kmem_map too smal f kern/122252 net [ipmi] [bge] IPMI problem with BCM5704 (does not work o kern/122195 net [ed] Alignment problems in if_ed o kern/122058 net [em] [panic] Panic on em1: taskq o kern/122033 net [ral] [lor] Lock order reversal in ral0 at bootup [reg o bin/121895 net [patch] rtsol(8)/rtsold(8) doesn't handle managed netw o kern/121872 net [wpi] driver fails to attach on a fujitsu-siemens s711 s kern/121774 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/121706 net [netinet] [patch] "rtfree: 0xc4383870 has 1 refs" emit o kern/121624 net [em] [regression] Intel em WOL fails after upgrade to o kern/121555 net [panic] Fatal trap 12: current process = 12 (swi1: net o kern/121443 net [gif] [lor] icmp6_input/nd6_lookup o kern/121437 net [vlan] Routing to layer-2 address does not work on VLA o bin/121359 net [patch] ppp(8): fix local stack overflow in ppp o kern/121298 net [em] [panic] Fatal trap 12: page fault while in kernel o kern/121257 net [tcp] TSO + natd -> slow outgoing tcp traffic o kern/121181 net [panic] Fatal trap 3: breakpoint instruction fault whi o kern/121080 net [bge] IPv6 NUD problem on multi address config on bge0 o kern/120966 net [rum] kernel panic with if_rum and WPA encryption p docs/120945 net [patch] ip6(4) man page lacks documentation for TCLASS o kern/120566 net [request]: ifconfig(8) make order of arguments more fr o kern/120304 net [netgraph] [patch] netgraph source assumes 32-bit time o kern/120266 net [udp] [panic] gnugk causes kernel panic when closing U o kern/120232 net [nfe] [patch] Bring in nfe(4) to RELENG_6 o kern/120130 net [carp] [panic] carp causes kernel panics in any conste o bin/120060 net routed(8) deletes link-level routes in the presence of o kern/119945 net [rum] [panic] rum device in hostap mode, cause kernel o kern/119791 net [nfs] UDP NFS mount of aliased IP addresses from a Sol o kern/119617 net [nfs] nfs error on wpa network when reseting/shutdown f kern/119516 net [ip6] [panic] _mtx_lock_sleep: recursed on non-recursi o kern/119432 net [arp] route add -host -iface causes arp e o kern/119225 net [wi] 7.0-RC1 no carrier with Prism 2.5 wifi card [regr a bin/118987 net ifconfig(8): ifconfig -l (address_family) does not wor o sparc/118932 net [panic] 7.0-BETA4/sparc-64 kernel panic in rip_output a kern/118879 net [bge] [patch] bge has checksum problems on the 5703 ch o kern/118727 net [netgraph] [patch] [request] add new ng_pf module a kern/118238 net [bce] [patch] bce driver shows "no carrier" on Intel S s kern/117717 net [panic] Kernel panic with Bittorrent client. o kern/117448 net [carp] 6.2 kernel crash [regression] o kern/117423 net [vlan] Duplicate IP on different interfaces o bin/117339 net [patch] route(8): loading routing management commands o kern/117271 net [tap] OpenVPN TAP uses 99% CPU on releng_6 when if_tap o kern/117043 net [em] Intel PWLA8492MT Dual-Port Network adapter EEPROM o kern/116837 net [tun] [panic] [patch] ifconfig tunX destroy: panic o kern/116747 net [ndis] FreeBSD 7.0-CURRENT crash with Dell TrueMobile o bin/116643 net [patch] [request] fstat(1): add INET/INET6 socket deta o kern/116328 net [bge]: Solid hang with bge interface o kern/116185 net [iwi] if_iwi driver leads system to reboot o kern/115239 net [ipnat] panic with 'kmem_map too small' using ipnat o kern/115019 net [netgraph] ng_ether upper hook packet flow stops on ad o kern/115002 net [wi] if_wi timeout. failed allocation (busy bit). ifco o kern/114915 net [patch] [pcn] pcn (sys/pci/if_pcn.c) ethernet driver f o kern/113895 net [xl] xl0 fails on 6.2-RELEASE but worked fine on 5.5-R o kern/113432 net [ucom] WARNING: attempt to net_add_domain(netgraph) af o kern/112722 net [ipsec] [udp] IP v4 udp fragmented packet reject o kern/112686 net [patm] patm driver freezes System (FreeBSD 6.2-p4) i38 o kern/112570 net [bge] packet loss with bge driver on BCM5704 chipset o bin/112557 net [patch] ppp(8) lock file should not use symlink name o kern/112528 net [nfs] NFS over TCP under load hangs with "impossible p o kern/111457 net [ral] ral(4) freeze o kern/110140 net [ipw] ipw fails under load o kern/109733 net [bge] bge link state issues [regression] o kern/109470 net [wi] Orinoco Classic Gold PC Card Can't Channel Hop o kern/109308 net [pppd] [panic] Multiple panics kernel ppp suspected [r o kern/109251 net [re] [patch] if_re cardbus card won't attach o bin/108895 net pppd(8): PPPoE dead connections on 6.2 [regression] o kern/108542 net [bce] Huge network latencies with 6.2-RELEASE / STABLE o kern/107944 net [wi] [patch] Forget to unlock mutex-locks o kern/107850 net [bce] bce driver link negotiation is faulty o conf/107035 net [patch] bridge(8): bridge interface given in rc.conf n o kern/106438 net [ipf] ipfilter: keep state does not seem to allow repl o kern/106316 net [dummynet] dummynet with multipass ipfw drops packets o kern/106243 net [nve] double fault panic in if_nve.c on high loads o kern/105945 net Address can disappear from network interface s kern/105943 net Network stack may modify read-only mbuf chain copies o bin/105925 net problems with ifconfig(8) and vlan(4) [regression] o kern/105348 net [ath] ath device stopps TX o kern/104851 net [inet6] [patch] On link routes not configured when usi o kern/104751 net [netgraph] kernel panic, when getting info about my tr o kern/104485 net [bge] Broadcom BCM5704C: Intermittent on newer chip ve o kern/103191 net Unpredictable reboot o kern/103135 net [ipsec] ipsec with ipfw divert (not NAT) encodes a pac o conf/102502 net [netgraph] [patch] ifconfig name does't rename netgrap o kern/102035 net [plip] plip networking disables parallel port printing o kern/101948 net [ipf] [panic] Kernel Panic Trap No 12 Page Fault - cau o kern/100709 net [libc] getaddrinfo(3) should return TTL info o kern/100519 net [netisr] suggestion to fix suboptimal network polling o kern/98978 net [ipf] [patch] ipfilter drops OOW packets under 6.1-Rel o kern/98597 net [inet6] Bug in FreeBSD 6.1 IPv6 link-local DAD procedu o bin/98218 net wpa_supplicant(8) blacklist not working f bin/97392 net ppp(8) hangs instead terminating o kern/97306 net [netgraph] NG_L2TP locks after connection with failed f kern/96268 net [socket] TCP socket performance drops by 3000% if pack o kern/96030 net [bfe] [patch] Install hangs with Broadcomm 440x NIC in o kern/95519 net [ral] ral0 could not map mbuf o kern/95288 net [pppd] [tty] [panic] if_ppp panic in sys/kern/tty_subr o kern/95277 net [netinet] [patch] IP Encapsulation mask_match() return o kern/95267 net packet drops periodically appear s kern/94863 net [bge] [patch] hack to get bge(4) working on IBM e326m o kern/94162 net [bge] 6.x kenel stale with bge(4) o kern/93886 net [ath] Atheros/D-Link DWL-G650 long delay to associate f kern/93378 net [tcp] Slow data transfer in Postfix and Cyrus IMAP (wo o kern/93019 net [ppp] ppp and tunX problems: no traffic after restarti o kern/92880 net [libc] [patch] almost rewritten inet_network(3) functi f kern/92552 net A serious bug in most network drivers from 5.X to 6.X s kern/92279 net [dc] Core faults everytime I reboot, possible NIC issu o kern/92090 net [bge] bge0: watchdog timeout -- resetting o kern/91859 net [ndis] if_ndis does not work with Asus WL-138 s kern/91777 net [ipf] [patch] wrong behaviour with skip rule inside an o kern/91594 net [em] FreeBSD > 5.4 w/ACPI fails to detect Intel Pro/10 o kern/91364 net [ral] [wep] WF-511 RT2500 Card PCI and WEP o kern/91311 net [aue] aue interface hanging o kern/90890 net [vr] Problems with network: vr0: tx shutdown timeout s kern/90086 net [hang] 5.4p8 on supermicro P8SCT hangs during boot if f kern/88082 net [ath] [panic] cts protection for ath0 causes panic o kern/87521 net [ipf] [panic] using ipfilter "auth" keyword leads to k o kern/87506 net [vr] [patch] Fix alias support on vr interfaces s kern/86920 net [ndis] ifconfig: SIOCS80211: Invalid argument [regress o kern/86103 net [ipf] Illegal NAT Traversal in IPFilter o kern/85780 net 'panic: bogus refcnt 0' in routing/ipv6 o bin/85445 net ifconfig(8): deprecated keyword to ifconfig inoperativ o kern/85266 net [xe] [patch] xe(4) driver does not recognise Xircom XE o kern/84202 net [ed] [patch] Holtek HT80232 PCI NIC recognition on Fre o bin/82975 net route change does not parse classfull network as given o kern/82497 net [vge] vge(4) on AMD64 only works when loaded late, not f kern/81644 net [vge] vge(4) does not work properly when loaded as a K s kern/81147 net [net] [patch] em0 reinitialization while adding aliase o kern/80853 net [ed] [patch] add support for Compex RL2000/ISA in PnP o kern/79895 net [ipf] 5.4-RC2 breaks ipfilter NAT when using netgraph f kern/79262 net [dc] Adaptec ANA-6922 not fully supported o bin/79228 net [patch] extend arp(8) to be able to create blackhole r o kern/78090 net [ipf] ipf filtering on bridged packets doesn't work if p kern/77913 net [wi] [patch] Add the APDL-325 WLAN pccard to wi(4) o kern/77341 net [ip6] problems with IPV6 implementation o kern/77273 net [ipf] ipfilter breaks ipv6 statefull filtering on 5.3 s kern/77195 net [ipf] [patch] ipfilter ioctl SIOCGNATL does not match o kern/75873 net Usability problem with non-RFC-compliant IP spoof prot s kern/75407 net [an] an(4): no carrier after short time f kern/73538 net [bge] problem with the Broadcom BCM5788 Gigabit Ethern o kern/71469 net default route to internet magically disappears with mu o kern/70904 net [ipf] ipfilter ipnat problem with h323 proxy support o kern/64556 net [sis] [patch] if_sis short cable fix problems with Net s kern/60293 net [patch] FreeBSD arp poison patch o kern/54383 net [nfs] [patch] NFS root configurations without dynamic f i386/45773 net [bge] Softboot causes autoconf failure on Broadcom 570 s bin/41647 net ifconfig(8) doesn't accept lladdr along with inet addr s kern/39937 net ipstealth issue a kern/38554 net [patch] changing interface ipaddress doesn't seem to w o kern/35442 net [sis] [patch] Problem transmitting runts in if_sis dri o kern/34665 net [ipf] [hang] ipfilter rcmd proxy "hangs". o kern/31647 net [libc] socket calls can return undocumented EINVAL o kern/30186 net [libc] getaddrinfo(3) does not handle incorrect servna o kern/27474 net [ipf] [ppp] Interactive use of user PPP and ipfilter c o conf/23063 net [arp] [patch] for static ARP tables in rc.network 357 problems total. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:20:03 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A11B1065692 for ; Mon, 5 Oct 2009 11:20:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 586A28FC12 for ; Mon, 5 Oct 2009 11:20:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n95BK2Qd000890 for ; Mon, 5 Oct 2009 11:20:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n95BK2di000889; Mon, 5 Oct 2009 11:20:02 GMT (envelope-from gnats) Date: Mon, 5 Oct 2009 11:20:02 GMT Message-Id: <200910051120.n95BK2di000889@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Carlos Cc: Subject: Re: kern/137776: [rum] panic in rum(4) driver on 8.0-BETA2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Carlos List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:20:03 -0000 The following reply was made to PR kern/137776; it has been noted by GNATS. From: Carlos To: bug-followup@freebsd.org, flz@freebsd.org Cc: Subject: Re: kern/137776: [rum] panic in rum(4) driver on 8.0-BETA2 Date: Mon, 5 Oct 2009 12:52:32 +0200 hi list, I would like to say sorry for my poor english i want to say that I can reproduce this panic always. If i run wine + WoW (World of Warcraft), the internet connection is lost and then if i run /etc/rc.d/netif restart the panic always appear. if you need make tests i can make it on my pc and give you the results of the test. I am only a normal user, not a developer user. If I have to apply patch (maybe) I need some help in how to apply it. My system is running 8.0-RC1 FreeBSD 8.0-RC1 #0: Thu Sep 17 20:45:19 UTC 2009 r...@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 and this is my panic: kgdb /boot/kernel/kernel.symbols vmcore.1 GNU gdb 6.1.1 [FreeBSD] This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x20 fault code = supervisor read, page not present instruction pointer = 0x20:0xc09408a8 stack pointer = 0x28:0xe89809a4 frame pointer = 0x28:0xe89809c0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2321 (wpa_supplicant) trap number = 12 panic: page fault cpuid = 1 Uptime: 3m18s Physical memory: 2022 MB Dumping 222 MB: 207 191 175 159 143 127 111 95 79 63 47 31 15 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_journal.ko Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from /boot/kernel/snd_hda.ko.symbols...done. done. Loaded symbols for /boot/kernel/snd_hda.ko Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done. done. Loaded symbols for /boot/kernel/sound.ko Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /boot/kernel/coretemp.ko.symbols...done. done. Loaded symbols for /boot/kernel/coretemp.ko Reading symbols from /boot/modules/nvidia.ko...done. Loaded symbols for /boot/modules/nvidia.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/atapicam.ko...Reading symbols from /boot/kernel/atapicam.ko.symbols...done. done. Loaded symbols for /boot/kernel/atapicam.ko Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/linprocfs.ko Reading symbols from /usr/local/modules/fuse.ko...done. Loaded symbols for /usr/local/modules/fuse.ko Reading symbols from /usr/local/modules/rtc.ko...done. Loaded symbols for /usr/local/modules/rtc.ko #0 doadump () at pcpu.h:246 246 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:246 #1 0xc08823c7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xc08826b9 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0xc0bb346c in trap_fatal (frame=0xe8980964, eva=32) at /usr/src/sys/i386/i386/trap.c:933 #4 0xc0bb36f0 in trap_pfault (frame=0xe8980964, usermode=0, eva=32) at /usr/src/sys/i386/i386/trap.c:846 #5 0xc0bb40d5 in trap (frame=0xe8980964) at /usr/src/sys/i386/i386/trap.c:528 #6 0xc0b96a4b in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #7 0xc09408a8 in ieee80211_crypto_encap (ni=0xc6b5d000, m=0xc776dc00) at /usr/src/sys/net80211/ieee80211_crypto.c:560 #8 0xc07ca32b in rum_start (ifp=0xc67e2400) at /usr/src/sys/dev/usb/wlan/if_rum.c:1216 #9 0xc0923582 in if_start (ifp=0xc67e2400) at /usr/src/sys/net/if.c:3242 #10 0xc092750b in if_transmit (ifp=0xc67e2400, m=0xc6a60c00) at /usr/src/sys/net/if.c:3254 #11 0xc0963082 in ieee80211_start (ifp=0xc6409400) at /usr/src/sys/net80211/ieee80211_output.c:362 #12 0xc0923582 in if_start (ifp=0xc6409400) at /usr/src/sys/net/if.c:3242 #13 0xc092750b in if_transmit (ifp=0xc6409400, m=0xc6acc200) at /usr/src/sys/net/if.c:3254 #14 0xc092bd90 in ether_output_frame (ifp=0xc6409400, m=0xc6acc200) at /usr/src/sys/net/if_ethersubr.c:452 #15 0xc092c77b in ether_output (ifp=0xc6409400, m=0xc6acc200, dst=0xe8980b98, ro=0x0) at /usr/src/sys/net/if_ethersubr.c:423 #16 0xc09626fd in ieee80211_output (ifp=0xc6409400, m=0xc6acc200, dst=0xe8980b98, ro=0x0) at /usr/src/sys/net80211/ieee80211_output.c:406 #17 0xc092034b in bpfwrite (dev=0xc61df200, uio=0xe8980c58, ioflag=0) at /usr/src/sys/net/bpf.c:889 #18 0xc0806b4f in devfs_write_f (fp=0xc68c8310, uio=0xe8980c58, cred=0xc77d8e00, flags=0, td=0xc68586c0) at /usr/src/sys/fs/devfs/devfs_vnops.c:1509 #19 0xc08be3f7 in dofilewrite (td=0xc68586c0, fd=6, fp=0xc68c8310, auio=0xe8980c58, offset=-1, flags=0) at file.h:239 #20 0xc08be6e8 in kern_writev (td=0xc68586c0, fd=6, auio=0xe8980c58) at /usr/src/sys/kern/sys_generic.c:446 #21 0xc08be76f in write (td=0xc68586c0, uap=0xe8980cf8) at /usr/src/sys/kern/sys_generic.c:362 #22 0xc0bb3a35 in syscall (frame=0xe8980d38) at /usr/src/sys/i386/i386/trap.c:1073 #23 0xc0b96ab0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261 #24 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:29:06 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 035C4106568D for ; Mon, 5 Oct 2009 11:29:06 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 9223A8FC1C for ; Mon, 5 Oct 2009 11:29:05 +0000 (UTC) Received: from [217.25.27.27] (port=60065 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MulkV-0006cZ-00; Mon, 05 Oct 2009 15:29:03 +0400 Message-ID: <4AC9D87E.7000005@mail.ru> Date: Mon, 05 Oct 2009 16:29:02 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> In-Reply-To: <20091005110726.GA62598@onelab2.iet.unipi.it> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:29:06 -0000 Luigi Rizzo wrote: > On Mon, Oct 05, 2009 at 03:52:39PM +0500, rihad wrote: >> Eugene Grosbein wrote: >>> On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote: >>> >>>> Still not sure why increasing queue size as high as I want doesn't >>>> completely eliminate drops. >>> The goal is to make sources of traffic to slow down, this is the only >>> way to descrease drops - any finite queue may be overhelmed with traffic. >>> Taildrop does not really help with this. GRED does much better. >>> >> Alright, so I changed to gred by adding to each config command: >> ipfw ... gred 0.002/900/1000/0.1 queue 1000 >> and reconfigured. Still around 300-400 drops per second, which was >> typical at this load level before with taildrop anyway. There are around >> 3-5 mbit/s being wasted according to systat -ifstat. >> >> Should I now increase slots to 5-10-20k? >> Very strange. >> >> "ipfw pipe show" correctly shows that gred is at work. For example: >> 00512: 512.000 Kbit/s 0 ms 1000 sl. 79 queues (64 buckets) >> GRED w_q 0.001999 min_th 900 max_th 1000 max_p 0.099991 >> mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 >> ... > > you keep omitting the important info i.e. whether individual > pipes have drops, significant queue lenghts and so on. > Sorry. Almost everyone has 0 in the last Drp column, but some have above zero. I'm not just sure how this can be helpful to anyone. 05120: 5.120 Mbit/s 0 ms 5000 sl. 66 queues (64 buckets) GRED w_q 0.001999 min_th 4500 max_th 5000 max_p 0.099991 mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 1 131 0 0 0 1 ip 0.0.0.0/0 39 53360 0 0 0 2 ip 0.0.0.0/0 382206 418022848 0 0 0 3 ip 0.0.0.0/0 34 2008 0 0 0 4 ip 0.0.0.0/0 4868510 6277077787 15 20452 9 5 ip 0.0.0.0/0 14 16675 0 0 0 5 ip 0.0.0.0/0 3 4158 0 0 0 6 ip 0.0.0.0/0 38 43576 0 0 0 7 ip 0.0.0.0/0 1265954 1475400663 0 0 0 8 ip 0.0.0.0/0 1081461 1247681879 0 0 749 9 ip 0.0.0.0/0 6186589 8737048919 0 0 19243 10 ip 0.0.0.0/0 21607 5636447 0 0 5 11 ip 0.0.0.0/0 437 94576 0 0 0 12 ip 0.0.0.0/0 22915 18634779 0 0 0 13 ip 0.0.0.0/0 557988 688051579 0 0 0 14 ip 0.0.0.0/0 50339 65685647 0 0 0 15 ip 0.0.0.0/0 554835 546223485 0 0 140 16 ip 0.0.0.0/0 32 13104 0 0 0 17 ip 0.0.0.0/0 2034099 2719966792 0 0 0 18 ip 0.0.0.0/0 282 36551 0 0 0 19 ip 0.0.0.0/0 8351766 8947643162 0 0 0 20 ip 0.0.0.0/0 4 624 0 0 0 21 ip 0.0.0.0/0 22391 29922375 0 0 0 22 ip 0.0.0.0/0 9 424 0 0 0 23 ip 0.0.0.0/0 750322 935365326 0 0 0 24 ip 0.0.0.0/0 1 40 0 0 0 25 ip 0.0.0.0/0 3617690 3501375619 0 0 602 26 ip 0.0.0.0/0 12116 12039435 0 0 0 27 ip 0.0.0.0/0 524311 653399507 0 0 8 28 ip 0.0.0.0/0 3 417 0 0 0 29 ip 0.0.0.0/0 16 2034 0 0 0 30 ip 0.0.0.0/0 64 82661 3 4432 0 31 ip 0.0.0.0/0 946389 1175221367 0 0 66 32 ip 0.0.0.0/0 1 168 0 0 0 32 ip 0.0.0.0/0 28 41776 0 0 0 33 ip 0.0.0.0/0 6 6433 0 0 0 34 ip 0.0.0.0/0 1 536 0 0 0 35 ip 0.0.0.0/0 2021 2641048 0 0 0 36 ip 0.0.0.0/0 350 264039 0 0 0 37 ip 0.0.0.0/0 167578 137763107 0 0 0 38 ip 0.0.0.0/0 250404 128905757 0 0 0 39 ip 0.0.0.0/0 385139 287006012 0 0 0 40 ip 0.0.0.0/0 49 68696 0 0 0 41 ip 0.0.0.0/0 23 1813 0 0 0 42 ip 0.0.0.0/0 129 135256 0 0 0 43 ip 0.0.0.0/0 3232 2191027 0 0 0 44 ip 0.0.0.0/0 27935157 24307287646 0 0 18802 45 ip 0.0.0.0/0 2166 212635 0 0 0 46 ip 0.0.0.0/0 1127307 1392467620 0 0 3 47 ip 0.0.0.0/0 1216900 1258200836 0 0 0 48 ip 0.0.0.0/0 2 2984 1 1492 0 49 ip 0.0.0.0/0 1 112 0 0 0 50 ip 0.0.0.0/0 1409 326389 0 0 0 51 ip 0.0.0.0/0 46674 47291021 10 14920 0 52 ip 0.0.0.0/0 86667 66834983 0 0 0 53 ip 0.0.0.0/0 434998 302827189 0 0 0 54 ip 0.0.0.0/0 542 277669 0 0 0 55 ip 0.0.0.0/0 1088072 919495021 0 0 0 56 ip 0.0.0.0/0 64 81240 0 0 0 57 ip 0.0.0.0/0 41028 59193278 0 0 0 58 ip 0.0.0.0/0 1 210 0 0 0 59 ip 0.0.0.0/0 4 310 0 0 0 60 ip 0.0.0.0/0 2 2984 0 0 0 61 ip 0.0.0.0/0 42874 36616688 0 0 0 62 ip 0.0.0.0/0 4 498 0 0 0 63 ip 0.0.0.0/0 530137 717027403 0 0 0 From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:30:40 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C3941065695 for ; Mon, 5 Oct 2009 11:30:40 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 05C8C8FC15 for ; Mon, 5 Oct 2009 11:30:39 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95BUb0Y079010; Mon, 5 Oct 2009 19:30:37 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95BUbR2079009; Mon, 5 Oct 2009 19:30:37 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 19:30:37 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005113037.GA77999@svzserv.kemerovo.su> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9C88A.5050509@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:30:40 -0000 On Mon, Oct 05, 2009 at 03:20:58PM +0500, rihad wrote: > >>>Taildrop does not really help with this. GRED does much better. > >>i think the first problem here is figure out _why_ we have > >>the drops, as the original poster said that queues are configured > >>with a very large amount of buffer (and i think there is a > >>misconfiguration somewhere because the mbuf stats do not make > >>sense) > > > >That may be very simple, f.e. wide uplink channel and policy that > >dictates slower client speeds. Any taildrop queue would drop lots > >of packets. > > > If uplink is e.g. 100 mbit/s, but data is fed to client by dummynet at 1 > mbit/s, doesn't the _client's_ TCP software know to slow things down to > not overwhelm 1 mbit/s? That's not client's TCP software feeding your router with traffic but server side. > Where has TCP slow-start gone? My router box > isn't some application proxy that starts downloading at full 100 mbit/s > thus quickly filling client's 1 mbit/s link. It's just a router. While there is no or little competition for bandwidth from the router to clients, TCP would work just fine. I suspect your shaping policy makes heavy competition between clients. In this case, TCP behaves not-so-well without help of router's good shaping algorythms and taildrop is not good one. > Although it doesn't yet make sense to me, I'll try going to GRED soon. "Works for me" :-) Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:46:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4723E1065676 for ; Mon, 5 Oct 2009 11:46:39 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id A80358FC12 for ; Mon, 5 Oct 2009 11:46:37 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95Bka6c079927; Mon, 5 Oct 2009 19:46:36 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95Bkagr079926; Mon, 5 Oct 2009 19:46:36 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 19:46:36 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005114636.GB77999@svzserv.kemerovo.su> References: <4AC8A76B.3050502@mail.ru> <20091004144909.GA42503@onelab2.iet.unipi.it> <4AC8B6E3.2070101@mail.ru> <20091004151518.GB42877@onelab2.iet.unipi.it> <4AC8BF3B.10601@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC8BF3B.10601@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:46:39 -0000 On Sun, Oct 04, 2009 at 08:28:59PM +0500, rihad wrote: > Thanks for the tip. although I took an easier route by simply doing > "ipfw add allow ip from any to any" before the pipe rules, and the buf > drop rate instantly became 0. So the problem is dummynet/ipfw. You should also estimate volume of non-TCP traffic that is generally has not flow control capabilities of TCP. Or, try something like this: ipfw add 100 skipto 200 tcp from any to any # direct only TCP to dummynet ipfw add 150 allow ip from any to any # pass non-TCP ipfw add 200 ... # here dummynet rules go And take a look at drop counters. Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:48:03 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFEB21065694 for ; Mon, 5 Oct 2009 11:48:03 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 789C48FC14 for ; Mon, 5 Oct 2009 11:48:03 +0000 (UTC) Received: from [217.25.27.27] (port=15276 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1Mum2q-000ETS-00; Mon, 05 Oct 2009 15:48:00 +0400 Message-ID: <4AC9DCEF.7080502@mail.ru> Date: Mon, 05 Oct 2009 16:47:59 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> <20091005113037.GA77999@svzserv.kemerovo.su> In-Reply-To: <20091005113037.GA77999@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:48:03 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 03:20:58PM +0500, rihad wrote: > >>>>> Taildrop does not really help with this. GRED does much better. >>>> i think the first problem here is figure out _why_ we have >>>> the drops, as the original poster said that queues are configured >>>> with a very large amount of buffer (and i think there is a >>>> misconfiguration somewhere because the mbuf stats do not make >>>> sense) >>> That may be very simple, f.e. wide uplink channel and policy that >>> dictates slower client speeds. Any taildrop queue would drop lots >>> of packets. >>> >> If uplink is e.g. 100 mbit/s, but data is fed to client by dummynet at 1 >> mbit/s, doesn't the _client's_ TCP software know to slow things down to >> not overwhelm 1 mbit/s? > > That's not client's TCP software feeding your router with traffic > but server side. > Yup, I meant client-side decreasing congestion window, delaying ACKs etc. etc., typical for TCP congestion control. Or...? GRED didn't solve the problem :( From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:50:13 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54FD01065670 for ; Mon, 5 Oct 2009 11:50:13 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx48.mail.ru (mx48.mail.ru [94.100.176.62]) by mx1.freebsd.org (Postfix) with ESMTP id 0E1A48FC0C for ; Mon, 5 Oct 2009 11:50:12 +0000 (UTC) Received: from [217.25.27.27] (port=45380 helo=[217.25.27.27]) by mx48.mail.ru with asmtp id 1Mum4x-0009Po-00; Mon, 05 Oct 2009 15:50:11 +0400 Message-ID: <4AC9DD72.9060802@mail.ru> Date: Mon, 05 Oct 2009 16:50:10 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> <20091005113037.GA77999@svzserv.kemerovo.su> In-Reply-To: <20091005113037.GA77999@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:50:13 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 03:20:58PM +0500, rihad wrote: >> Where has TCP slow-start gone? My router box >> isn't some application proxy that starts downloading at full 100 mbit/s >> thus quickly filling client's 1 mbit/s link. It's just a router. > > While there is no or little competition for bandwidth from the router > to clients, TCP would work just fine. I suspect your shaping policy > makes heavy competition between clients. In this case, TCP behaves > not-so-well without help of router's good shaping algorythms > and taildrop is not good one. > Nothing fancy (i.e. no competition). Only tons of per-user pipes simulating the given throughput. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 11:57:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAF741065694 for ; Mon, 5 Oct 2009 11:57:39 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 4090C8FC23 for ; Mon, 5 Oct 2009 11:57:38 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 3FFD8730DA; Mon, 5 Oct 2009 14:04:18 +0200 (CEST) Date: Mon, 5 Oct 2009 14:04:18 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091005120418.GA63131@onelab2.iet.unipi.it> References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9D87E.7000005@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 11:57:40 -0000 On Mon, Oct 05, 2009 at 04:29:02PM +0500, rihad wrote: > Luigi Rizzo wrote: ... > >you keep omitting the important info i.e. whether individual > >pipes have drops, significant queue lenghts and so on. > > > Sorry. Almost everyone has 0 in the last Drp column, but some have above > zero. I'm not just sure how this can be helpful to anyone. because you were complaining about 'dummynet causing drops and waste of bandwidth'. Now, drops could be due to either 1) some saturation in the dummynet machine (memory shortage, cpu shortage, etc.) which cause unwanted drops; 2) intentional drops introduced by dummynet because a flow exceeds its queue size. These drops are those shown in the 'Drop' column in 'ipfw pipe show' (they are cumulative, so you should do an 'ipfw pipe delete; ipfw pipe 5120 config ...' whenever you want to re-run the stats, or compute the differences between subsequent reads, to figure out what happens. If all drops you are seeing are of type 2, then there is nothing you can do to remove them: you set a bandwidth limit, the client is sending faster than it should, perhaps with UDP so even RED/GRED won't help you, and you see the drops once the queue starts to fill up. Examples below: the entries in bucket 4 and 44 If you are seeing drops that are not listed in 'pipe show' then yun need to investigate where the packets are lost, again it could be on the output queue of the interface (due to the burstiness introduced by dummynet), or shortage of mbufs (but this did not seem to be the case from your previous stats) or something else. It's all up to you to run measurements, possibly without omitting potentially significant data (e.g. sysctl -a net.inet.ip) or making assumptions (e.g. you have configured 5000 slots per queue, but with only 50k mbufs in total there is no chance to guarantee 5000 slots to each queue -- all you will achieve is give a lot of slots to the greedy nodes, and very little to the other ones) cheers luigi > 05120: 5.120 Mbit/s 0 ms 5000 sl. 66 queues (64 buckets) > GRED w_q 0.001999 min_th 4500 max_th 5000 max_p 0.099991 > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 0 ip 0.0.0.0/0 1 131 0 0 0 > 1 ip 0.0.0.0/0 39 53360 0 0 0 > 2 ip 0.0.0.0/0 382206 418022848 0 > 0 0 > 3 ip 0.0.0.0/0 34 2008 0 0 0 > 4 ip 0.0.0.0/0 4868510 6277077787 15 > 20452 9 > 5 ip 0.0.0.0/0 14 16675 0 0 0 > 5 ip 0.0.0.0/0 3 4158 0 0 0 > 6 ip 0.0.0.0/0 38 43576 0 0 0 > 7 ip 0.0.0.0/0 1265954 1475400663 0 > 0 0 > 8 ip 0.0.0.0/0 1081461 1247681879 0 > 0 749 > 9 ip 0.0.0.0/0 6186589 8737048919 0 > 0 19243 > 10 ip 0.0.0.0/0 21607 5636447 0 0 5 > 11 ip 0.0.0.0/0 437 94576 0 0 0 > 12 ip 0.0.0.0/0 22915 18634779 0 0 0 > 13 ip 0.0.0.0/0 557988 688051579 0 > 0 0 > 14 ip 0.0.0.0/0 50339 65685647 0 0 0 > 15 ip 0.0.0.0/0 554835 546223485 0 > 0 140 > 16 ip 0.0.0.0/0 32 13104 0 0 0 > 17 ip 0.0.0.0/0 2034099 2719966792 0 > 0 0 > 18 ip 0.0.0.0/0 282 36551 0 0 0 > 19 ip 0.0.0.0/0 8351766 8947643162 0 > 0 0 > 20 ip 0.0.0.0/0 4 624 0 0 0 > 21 ip 0.0.0.0/0 22391 29922375 0 0 0 > 22 ip 0.0.0.0/0 9 424 0 0 0 > 23 ip 0.0.0.0/0 750322 935365326 0 > 0 0 > 24 ip 0.0.0.0/0 1 40 0 0 0 > 25 ip 0.0.0.0/0 3617690 3501375619 0 > 0 602 > 26 ip 0.0.0.0/0 12116 12039435 0 0 0 > 27 ip 0.0.0.0/0 524311 653399507 0 > 0 8 > 28 ip 0.0.0.0/0 3 417 0 0 0 > 29 ip 0.0.0.0/0 16 2034 0 0 0 > 30 ip 0.0.0.0/0 64 82661 3 4432 0 > 31 ip 0.0.0.0/0 946389 1175221367 0 > 0 66 > 32 ip 0.0.0.0/0 1 168 0 0 0 > 32 ip 0.0.0.0/0 28 41776 0 0 0 > 33 ip 0.0.0.0/0 6 6433 0 0 0 > 34 ip 0.0.0.0/0 1 536 0 0 0 > 35 ip 0.0.0.0/0 2021 2641048 0 0 0 > 36 ip 0.0.0.0/0 350 264039 0 0 0 > 37 ip 0.0.0.0/0 167578 137763107 0 > 0 0 > 38 ip 0.0.0.0/0 250404 128905757 0 > 0 0 > 39 ip 0.0.0.0/0 385139 287006012 0 > 0 0 > 40 ip 0.0.0.0/0 49 68696 0 0 0 > 41 ip 0.0.0.0/0 23 1813 0 0 0 > 42 ip 0.0.0.0/0 129 135256 0 0 0 > 43 ip 0.0.0.0/0 3232 2191027 0 0 0 > 44 ip 0.0.0.0/0 27935157 24307287646 0 > 0 18802 > 45 ip 0.0.0.0/0 2166 212635 0 0 0 > 46 ip 0.0.0.0/0 1127307 1392467620 0 > 0 3 > 47 ip 0.0.0.0/0 1216900 1258200836 0 > 0 0 > 48 ip 0.0.0.0/0 2 2984 1 1492 0 > 49 ip 0.0.0.0/0 1 112 0 0 0 > 50 ip 0.0.0.0/0 1409 326389 0 0 0 > 51 ip 0.0.0.0/0 46674 47291021 10 14920 0 > 52 ip 0.0.0.0/0 86667 66834983 0 0 0 > 53 ip 0.0.0.0/0 434998 302827189 0 > 0 0 > 54 ip 0.0.0.0/0 542 277669 0 0 0 > 55 ip 0.0.0.0/0 1088072 919495021 0 > 0 0 > 56 ip 0.0.0.0/0 64 81240 0 0 0 > 57 ip 0.0.0.0/0 41028 59193278 0 0 0 > 58 ip 0.0.0.0/0 1 210 0 0 0 > 59 ip 0.0.0.0/0 4 310 0 0 0 > 60 ip 0.0.0.0/0 2 2984 0 0 0 > 61 ip 0.0.0.0/0 42874 36616688 0 0 0 > 62 ip 0.0.0.0/0 4 498 0 0 0 > 63 ip 0.0.0.0/0 530137 717027403 0 > 0 0 From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 12:00:59 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D00001065693 for ; Mon, 5 Oct 2009 12:00:59 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 393DD8FC20 for ; Mon, 5 Oct 2009 12:00:58 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95C0vK3080879; Mon, 5 Oct 2009 20:00:57 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95C0vTk080878; Mon, 5 Oct 2009 20:00:57 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 20:00:57 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005120057.GA79942@svzserv.kemerovo.su> References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> <20091005113037.GA77999@svzserv.kemerovo.su> <4AC9DD72.9060802@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9DD72.9060802@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 12:00:59 -0000 On Mon, Oct 05, 2009 at 04:50:10PM +0500, rihad wrote: > >>Where has TCP slow-start gone? My router box > >>isn't some application proxy that starts downloading at full 100 mbit/s > >>thus quickly filling client's 1 mbit/s link. It's just a router. > > > >While there is no or little competition for bandwidth from the router > >to clients, TCP would work just fine. I suspect your shaping policy > >makes heavy competition between clients. In this case, TCP behaves > >not-so-well without help of router's good shaping algorythms > >and taildrop is not good one. > > > > Nothing fancy (i.e. no competition). Only tons of per-user pipes > simulating the given throughput. You've mentioned previously: "The pipes are fine, each normally having 100-120 concurrent consumers (i.e. active users)." This IS competition between TCP flows inside each pipe. Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 12:12:14 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15A71106568D for ; Mon, 5 Oct 2009 12:12:14 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx40.mail.ru (mx40.mail.ru [94.100.176.54]) by mx1.freebsd.org (Postfix) with ESMTP id C269F8FC18 for ; Mon, 5 Oct 2009 12:12:13 +0000 (UTC) Received: from [217.25.27.27] (port=2804 helo=[217.25.27.27]) by mx40.mail.ru with asmtp id 1MumQF-000JYM-00; Mon, 05 Oct 2009 16:12:12 +0400 Message-ID: <4AC9E29B.6080908@mail.ru> Date: Mon, 05 Oct 2009 17:12:11 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> In-Reply-To: <20091005120418.GA63131@onelab2.iet.unipi.it> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 12:12:14 -0000 Luigi Rizzo wrote: > On Mon, Oct 05, 2009 at 04:29:02PM +0500, rihad wrote: >> Luigi Rizzo wrote: > ... >>> you keep omitting the important info i.e. whether individual >>> pipes have drops, significant queue lenghts and so on. >>> >> Sorry. Almost everyone has 0 in the last Drp column, but some have above >> zero. I'm not just sure how this can be helpful to anyone. > > because you were complaining about 'dummynet causing drops and > waste of bandwidth'. > Now, drops could be due to either > 1) some saturation in the dummynet machine (memory shortage, cpu > shortage, etc.) which cause unwanted drops; > I too think the box is hitting some other global limit and dropping packets. If not, then how come that between 4a.m. and 10a.m. when the traffic load is at 250-330 mbit/s there isn't a single drop? > 2) intentional drops introduced by dummynet because a flow exceeds > its queue size. These drops are those shown in the 'Drop' > column in 'ipfw pipe show' (they are cumulative, so you > should do an 'ipfw pipe delete; ipfw pipe 5120 config ...' > whenever you want to re-run the stats, or compute the > differences between subsequent reads, to figure out what > happens. > > If all drops you are seeing are of type 2, then there is nothing > you can do to remove them: you set a bandwidth limit, the > client is sending faster than it should, perhaps with UDP > so even RED/GRED won't help you, and you see the drops > once the queue starts to fill up. > Examples below: the entries in bucket 4 and 44 > Then I guess I'm left with increasing slots and see how it goes. Currently it's set to 10000 for each pipe. Thanks for yours and Eugene's efforts, I appreciate it. > If you are seeing drops that are not listed in 'pipe show' > then yun need to investigate where the packets are lost, > again it could be on the output queue of the interface > (due to the burstiness introduced by dummynet), or shortage > of mbufs (but this did not seem to be the case from your > previous stats) or something else. > This indeed is not a problem, proved by the fact that, like I said, short-circuiting "ipfw allow ip from any to any" before dummynet pipe rules instantly eliminates all drops, and bce0 and bce1 load evens out (bce0 used for input, and bce1 for output). > It's all up to you to run measurements, possibly > without omitting potentially significant data > (e.g. sysctl -a net.inet.ip) > or making assumptions (e.g. you have configured > 5000 slots per queue, but with only 50k mbufs in total > there is no chance to guarantee 5000 slots to each > queue -- all you will achieve is give a lot of slots > to the greedy nodes, and very little to the other ones) > Well, I've been monitoring this stuff. It has never reached above 20000 mbufs (111111 is the current limit). From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 12:18:32 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 731C81065670 for ; Mon, 5 Oct 2009 12:18:32 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx39.mail.ru (mx39.mail.ru [94.100.176.53]) by mx1.freebsd.org (Postfix) with ESMTP id 2BFDF8FC18 for ; Mon, 5 Oct 2009 12:18:31 +0000 (UTC) Received: from [217.25.27.27] (port=9312 helo=[217.25.27.27]) by mx39.mail.ru with asmtp id 1MumWM-0000Tt-00; Mon, 05 Oct 2009 16:18:30 +0400 Message-ID: <4AC9E415.9040801@mail.ru> Date: Mon, 05 Oct 2009 17:18:29 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> <20091005113037.GA77999@svzserv.kemerovo.su> <4AC9DD72.9060802@mail.ru> <20091005120057.GA79942@svzserv.kemerovo.su> In-Reply-To: <20091005120057.GA79942@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 12:18:32 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 04:50:10PM +0500, rihad wrote: > >>>> Where has TCP slow-start gone? My router box >>>> isn't some application proxy that starts downloading at full 100 mbit/s >>>> thus quickly filling client's 1 mbit/s link. It's just a router. >>> While there is no or little competition for bandwidth from the router >>> to clients, TCP would work just fine. I suspect your shaping policy >>> makes heavy competition between clients. In this case, TCP behaves >>> not-so-well without help of router's good shaping algorythms >>> and taildrop is not good one. >>> >> Nothing fancy (i.e. no competition). Only tons of per-user pipes >> simulating the given throughput. > > You've mentioned previously: "The pipes are fine, each normally having > 100-120 concurrent consumers (i.e. active users)." > This IS competition between TCP flows inside each pipe. > Well, each user gets instantiated with a new copy of the pipe. Each such user counts towards the limit imposed by hash_size*max_chain_len for that pipe only. It would have been competition had I used dst-ip dst-ip 0xffffff00 or similar and not dst-ip 0xffffffff, _then_ all 256 users (determined by the mask) would compete for the pipe's bandwidth. So the only competition is in the uplink at our main Cisco, I guess. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 12:25:51 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95068106568B for ; Mon, 5 Oct 2009 12:25:51 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 3E30A8FC19 for ; Mon, 5 Oct 2009 12:25:50 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 6EEC0730DA; Mon, 5 Oct 2009 14:32:30 +0200 (CEST) Date: Mon, 5 Oct 2009 14:32:30 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091005123230.GA64167@onelab2.iet.unipi.it> References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9E29B.6080908@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 12:25:51 -0000 On Mon, Oct 05, 2009 at 05:12:11PM +0500, rihad wrote: > Luigi Rizzo wrote: > >On Mon, Oct 05, 2009 at 04:29:02PM +0500, rihad wrote: > >>Luigi Rizzo wrote: > >... > >>>you keep omitting the important info i.e. whether individual > >>>pipes have drops, significant queue lenghts and so on. > >>> > >>Sorry. Almost everyone has 0 in the last Drp column, but some have above > >>zero. I'm not just sure how this can be helpful to anyone. > > > >because you were complaining about 'dummynet causing drops and > >waste of bandwidth'. > >Now, drops could be due to either > >1) some saturation in the dummynet machine (memory shortage, cpu > > shortage, etc.) which cause unwanted drops; > > > I too think the box is hitting some other global limit and dropping > packets. If not, then how come that between 4a.m. and 10a.m. when the > traffic load is at 250-330 mbit/s there isn't a single drop? there may be different reasons, e.g. the big offenders were idle when you saw no drops. You still do not have enough information on which packets are dropped and where, so you cannot prove your assumptions. Also, below: 1. increasing the queue size won't help at all. Those who overflow a queue of 1000 slots will also overflow a queue of 10k slots. 2. your test with 'ipfw allow ip from any to any' does not prove that the interface queue is not saturating, because you also remove the burstiness that dummynet introduces, and so the queue is driven differently. good luck luigi > >2) intentional drops introduced by dummynet because a flow exceeds > > its queue size. These drops are those shown in the 'Drop' > > column in 'ipfw pipe show' (they are cumulative, so you > > should do an 'ipfw pipe delete; ipfw pipe 5120 config ...' > > whenever you want to re-run the stats, or compute the > > differences between subsequent reads, to figure out what > > happens. > > > >If all drops you are seeing are of type 2, then there is nothing > >you can do to remove them: you set a bandwidth limit, the > >client is sending faster than it should, perhaps with UDP > >so even RED/GRED won't help you, and you see the drops > >once the queue starts to fill up. > >Examples below: the entries in bucket 4 and 44 > > > Then I guess I'm left with increasing slots and see how it goes. > Currently it's set to 10000 for each pipe. Thanks for yours and Eugene's > efforts, I appreciate it. > > >If you are seeing drops that are not listed in 'pipe show' > >then yun need to investigate where the packets are lost, > >again it could be on the output queue of the interface > >(due to the burstiness introduced by dummynet), or shortage > >of mbufs (but this did not seem to be the case from your > >previous stats) or something else. > > > This indeed is not a problem, proved by the fact that, like I said, > short-circuiting "ipfw allow ip from any to any" before dummynet pipe > rules instantly eliminates all drops, and bce0 and bce1 load evens out > (bce0 used for input, and bce1 for output). > > >It's all up to you to run measurements, possibly > >without omitting potentially significant data > >(e.g. sysctl -a net.inet.ip) > >or making assumptions (e.g. you have configured > >5000 slots per queue, but with only 50k mbufs in total > >there is no chance to guarantee 5000 slots to each > >queue -- all you will achieve is give a lot of slots > >to the greedy nodes, and very little to the other ones) > > > Well, I've been monitoring this stuff. It has never reached above 20000 > mbufs (111111 is the current limit). From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 13:08:50 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5AAEA106566B for ; Mon, 5 Oct 2009 13:08:50 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id 12E078FC0C for ; Mon, 5 Oct 2009 13:08:50 +0000 (UTC) Received: from [217.25.27.27] (port=32840 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1MunJ2-0009te-00; Mon, 05 Oct 2009 17:08:48 +0400 Message-ID: <4AC9EFDF.4080302@mail.ru> Date: Mon, 05 Oct 2009 18:08:47 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> In-Reply-To: <20091005123230.GA64167@onelab2.iet.unipi.it> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 13:08:50 -0000 Luigi Rizzo wrote: > On Mon, Oct 05, 2009 at 05:12:11PM +0500, rihad wrote: >> Luigi Rizzo wrote: >>> On Mon, Oct 05, 2009 at 04:29:02PM +0500, rihad wrote: >>>> Luigi Rizzo wrote: >>> ... >>>>> you keep omitting the important info i.e. whether individual >>>>> pipes have drops, significant queue lenghts and so on. >>>>> >>>> Sorry. Almost everyone has 0 in the last Drp column, but some have above >>>> zero. I'm not just sure how this can be helpful to anyone. >>> because you were complaining about 'dummynet causing drops and >>> waste of bandwidth'. >>> Now, drops could be due to either >>> 1) some saturation in the dummynet machine (memory shortage, cpu >>> shortage, etc.) which cause unwanted drops; >>> >> I too think the box is hitting some other global limit and dropping >> packets. If not, then how come that between 4a.m. and 10a.m. when the >> traffic load is at 250-330 mbit/s there isn't a single drop? > > there may be different reasons, e.g. the big offenders were > idle when you saw no drops. You still do not have enough > information on which packets are dropped and where, > so you cannot prove your assumptions. > > Also, below: > 1. increasing the queue size won't help at all. Those > who overflow a queue of 1000 slots will also overflow > a queue of 10k slots. > > 2. your test with 'ipfw allow ip from any to any' does not > prove that the interface queue is not saturating, because > you also remove the burstiness that dummynet introduces, > and so the queue is driven differently. > There's one thing I noticed: net.inet.ip.dummynet.io_pkt_drop doesn't grow! But still around 400 packets dropped per second. net.inet.ip.dummynet.tick_lost is always zero net.inet.ip.dummynet.tick_diff: grows at about 50 per second. net.inet.ip.dummynet.tick_adjustment: grows at about 5 per second. How do I investigate and fix this burstiness issue? $ netstat -i Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll bce0 1500 00:1d:09:xx:xx:xx 24777049059 0 75426020 0 0 bce0 1500 xx.xx.xx.xx/xx my.hostname 159293969 - 75282225 - - bce1 1500 00:1d:09:xx:xx:xx 724725 0 24514919344 0 0 bce1 1500 192.168.94.0 local.hostname 656243 - 83024869 - - From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 13:56:36 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E3E710656CA for ; Mon, 5 Oct 2009 13:56:36 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id BB8DE8FC15 for ; Mon, 5 Oct 2009 13:56:35 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95DuWqa089724; Mon, 5 Oct 2009 21:56:32 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95DuWsW089723; Mon, 5 Oct 2009 21:56:32 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 21:56:32 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005135632.GA89194@svzserv.kemerovo.su> References: <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> <20091005113037.GA77999@svzserv.kemerovo.su> <4AC9DD72.9060802@mail.ru> <20091005120057.GA79942@svzserv.kemerovo.su> <4AC9E415.9040801@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9E415.9040801@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 13:56:36 -0000 On Mon, Oct 05, 2009 at 05:18:29PM +0500, rihad wrote: > >You've mentioned previously: "The pipes are fine, each normally having > >100-120 concurrent consumers (i.e. active users)." > >This IS competition between TCP flows inside each pipe. > > > Well, each user gets instantiated with a new copy of the pipe. Each such > user counts towards the limit imposed by hash_size*max_chain_len for > that pipe only. It would have been competition had I used dst-ip dst-ip > 0xffffff00 or similar and not dst-ip 0xffffffff, _then_ all 256 users > (determined by the mask) would compete for the pipe's bandwidth. So the > only competition is in the uplink at our main Cisco, I guess. Hmm, yes, you are rigth. I've missed 'mask'. Try to disable net.inet.ip.dummynet.io_fast to see if there is a bug in 'fast' dummynet mode. Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 14:03:54 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E6921065679 for ; Mon, 5 Oct 2009 14:03:54 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx33.mail.ru (mx33.mail.ru [94.100.176.47]) by mx1.freebsd.org (Postfix) with ESMTP id 1721C8FC21 for ; Mon, 5 Oct 2009 14:03:53 +0000 (UTC) Received: from [217.25.27.27] (port=49042 helo=[217.25.27.27]) by mx33.mail.ru with asmtp id 1MuoAK-0003NR-00; Mon, 05 Oct 2009 18:03:52 +0400 Message-ID: <4AC9FCC7.1080108@mail.ru> Date: Mon, 05 Oct 2009 19:03:51 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> <20091005113037.GA77999@svzserv.kemerovo.su> <4AC9DD72.9060802@mail.ru> <20091005120057.GA79942@svzserv.kemerovo.su> <4AC9E415.9040801@mail.ru> <20091005135632.GA89194@svzserv.kemerovo.su> In-Reply-To: <20091005135632.GA89194@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 14:03:54 -0000 Eugene Grosbein wrote: > Try to disable net.inet.ip.dummynet.io_fast to see if there is a bug > in 'fast' dummynet mode. > Already tried that, no difference. Interestingly, now I checked sysctls and found out that net.inet.ip.dummynet.io_pkt_fast counter is still growing despite io_fast being disabled. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 14:04:15 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A23F1065694 for ; Mon, 5 Oct 2009 14:04:15 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 4EBA48FC1D for ; Mon, 5 Oct 2009 14:04:13 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95E49eh090173; Mon, 5 Oct 2009 22:04:09 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95E49oc090172; Mon, 5 Oct 2009 22:04:09 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 22:04:09 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005140409.GB89194@svzserv.kemerovo.su> References: <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC9EFDF.4080302@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 14:04:15 -0000 On Mon, Oct 05, 2009 at 06:08:47PM +0500, rihad wrote: > How do I investigate and fix this burstiness issue? Please also show: sysctl net.isr sysctl net.inet.ip.intr_queue_maxlen Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 14:30:20 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B430106566B for ; Mon, 5 Oct 2009 14:30:20 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 23C108FC13 for ; Mon, 5 Oct 2009 14:30:19 +0000 (UTC) Received: from [217.25.27.27] (port=8315 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MuoZs-000K9O-00; Mon, 05 Oct 2009 18:30:17 +0400 Message-ID: <4ACA02F7.6040409@mail.ru> Date: Mon, 05 Oct 2009 19:30:15 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> In-Reply-To: <20091005140409.GB89194@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 14:30:20 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 06:08:47PM +0500, rihad wrote: > >> How do I investigate and fix this burstiness issue? > > Please also show: > > sysctl net.isr > sysctl net.inet.ip.intr_queue_maxlen > net.isr.swi_count: 65461359 net.isr.drop: 0 net.isr.queued: 32843752 net.isr.deferred: 0 net.isr.directed: -723075002 net.isr.count: -723074001 net.isr.direct: 1 net.inet.ip.intr_queue_maxlen: 50 From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 14:38:39 2009 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E579106568D; Mon, 5 Oct 2009 14:38:39 +0000 (UTC) (envelope-from oberman@es.net) Received: from mailgw.es.net (mail1.es.net [IPv6:2001:400:201:1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 033DA8FC19; Mon, 5 Oct 2009 14:38:39 +0000 (UTC) Received: from ptavv.es.net (ptavv.es.net [IPv6:2001:400:910::29]) by mailgw.es.net (8.14.3/8.14.3) with ESMTP id n95Ecb0v030599 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 5 Oct 2009 07:38:38 -0700 Received: from ptavv.es.net (ptavv.es.net [127.0.0.1]) by ptavv.es.net (Tachyon Server) with ESMTP id C370F1CC2E; Mon, 5 Oct 2009 07:38:37 -0700 (PDT) To: Andre Oppermann In-reply-to: Your message of "Mon, 05 Oct 2009 10:24:33 +0200." <4AC9AD41.2070200@freebsd.org> Date: Mon, 05 Oct 2009 07:38:37 -0700 From: "Kevin Oberman" Message-Id: <20091005143837.C370F1CC2E@ptavv.es.net> X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.8161:2.4.5, 1.2.40, 4.0.166 definitions=2009-10-05_08:2009-09-29, 2009-10-05, 2009-10-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-0908210000 definitions=main-0910050070 Cc: net@freebsd.org Subject: Re: Unusual TCP options can cause FreeBSD to issue a reset X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 14:38:39 -0000 > Date: Mon, 05 Oct 2009 10:24:33 +0200 > From: Andre Oppermann > > Kevin Oberman wrote: > > I have a system that is unable to connect to a FreeBSD system due to > > the odd formatting of the TCP options. The options contains only the > > timestamp which, if recommendations in RFC1323 are followed, are > > preceded by two NOP bytes to put the timestamp values on 4 byte > > boundaries. > > > > This system sends the 12 byte timestamp option alone, followed by two > > zero bytes to pad it. This meets the requirements of RFC793 and 1323 is > > explicit that stacks must accept this, even though it results in > > sub-optimal performance and does not meet the recommendations in 1323 > > appendix A. > > Just this alone should not cause a reset from FreeBSD. > > > Any idea if this is hard to fix? I see in on both 7.2 and 8.0. > > Can you post a detailed tcpdump of a failing connect? And please enable > net.inet.tcp.log_debug which should give a better explanation on why > FreeBSD thinks the connection is bad. Thanks!. The debug output made the issue clear and it is not a FreeBSD problem. It is with the remote system and the timestamps used, I see the following timestamps: SYN----->159082 0 SYNACK-->57062695 159082 ACK----->159082 0 Clearly, this is bogus. Sorry for the noise and the bad analysis on Friday. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 14:50:40 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 197811065695 for ; Mon, 5 Oct 2009 14:50:40 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 77BBB8FC0A for ; Mon, 5 Oct 2009 14:50:39 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95EobmF092762; Mon, 5 Oct 2009 22:50:37 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95EobY4092761; Mon, 5 Oct 2009 22:50:37 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 22:50:37 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005145037.GA92519@svzserv.kemerovo.su> References: <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACA02F7.6040409@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 14:50:40 -0000 On Mon, Oct 05, 2009 at 07:30:15PM +0500, rihad wrote: > >>How do I investigate and fix this burstiness issue? > > > >Please also show: > > > >sysctl net.isr > >sysctl net.inet.ip.intr_queue_maxlen > > net.isr.swi_count: 65461359 > net.isr.drop: 0 > net.isr.queued: 32843752 > net.isr.deferred: 0 > net.isr.directed: -723075002 > net.isr.count: -723074001 > net.isr.direct: 1 > > net.inet.ip.intr_queue_maxlen: 50 What is CPU load in when the load is maximum? Eugene From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 15:07:21 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38C331065696 for ; Mon, 5 Oct 2009 15:07:21 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id E54E48FC1C for ; Mon, 5 Oct 2009 15:07:20 +0000 (UTC) Received: from [217.25.27.27] (port=10307 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1Mup9j-000CZ3-00; Mon, 05 Oct 2009 19:07:19 +0400 Message-ID: <4ACA0BA6.70706@mail.ru> Date: Mon, 05 Oct 2009 20:07:18 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091005145037.GA92519@svzserv.kemerovo.su> In-Reply-To: <20091005145037.GA92519@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 15:07:21 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 07:30:15PM +0500, rihad wrote: > >>>> How do I investigate and fix this burstiness issue? >>> Please also show: >>> >>> sysctl net.isr >>> sysctl net.inet.ip.intr_queue_maxlen >> net.isr.swi_count: 65461359 >> net.isr.drop: 0 >> net.isr.queued: 32843752 >> net.isr.deferred: 0 >> net.isr.directed: -723075002 >> net.isr.count: -723074001 >> net.isr.direct: 1 >> >> net.inet.ip.intr_queue_maxlen: 50 > > What is CPU load in when the load is maximum? > It has 2 quad-cores, so I'm not sure. Here's the output of top -S: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 1 171 ki31 0K 16K CPU7 7 120.2H 98.39% idle: cpu7 12 root 1 171 ki31 0K 16K CPU6 6 120.3H 95.75% idle: cpu6 16 root 1 171 ki31 0K 16K CPU2 2 115.5H 93.99% idle: cpu2 14 root 1 171 ki31 0K 16K CPU4 4 114.6H 91.89% idle: cpu4 18 root 1 171 ki31 0K 16K RUN 0 97.7H 88.96% idle: cpu0 13 root 1 171 ki31 0K 16K CPU5 5 117.5H 85.35% idle: cpu5 15 root 1 171 ki31 0K 16K CPU3 3 110.2H 79.79% idle: cpu3 29 root 1 -68 - 0K 16K WAIT 1 59.1H 54.05% irq256: bce0 17 root 1 171 ki31 0K 16K CPU1 1 63.5H 42.29% idle: cpu1 467 root 1 -68 - 0K 16K - 3 21.4H 11.57% dummynet 19 root 1 -32 - 0K 16K WAIT 4 268:29 3.66% swi4: clock sio 31 root 1 -68 - 0K 16K WAIT 2 310:19 2.98% irq257: bce1 30 root 1 -64 - 0K 16K WAIT 6 40:02 0.29% irq16: mfi0 3 root 1 -8 - 0K 16K - 5 55:42 0.00% g_up 4 root 1 -8 - 0K 16K - 1 51:22 0.00% g_down 22 root 1 44 - 0K 16K - 3 13:30 0.00% yarrow 21 root 1 -44 - 0K 16K WAIT 0 9:09 0.00% swi1: net 51 root 1 20 - 0K 16K syncer 2 7:25 0.00% syncer 599 root 1 44 0 5660K 1132K select 0 1:40 0.00% syslogd 52 root 1 -16 - 0K 16K sdflus 5 0:34 0.00% softdepflush 2 root 1 -8 - 0K 16K - 2 0:31 0.00% g_event From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 15:42:41 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 406F5106566B for ; Mon, 5 Oct 2009 15:42:41 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 9EB798FC0C for ; Mon, 5 Oct 2009 15:42:39 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n95FgaDw095964; Mon, 5 Oct 2009 23:42:36 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n95FgaDb095963; Mon, 5 Oct 2009 23:42:36 +0800 (KRAST) (envelope-from eugen) Date: Mon, 5 Oct 2009 23:42:36 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091005154236.GA95635@svzserv.kemerovo.su> References: <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091005145037.GA92519@svzserv.kemerovo.su> <4ACA0BA6.70706@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACA0BA6.70706@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 15:42:41 -0000 On Mon, Oct 05, 2009 at 08:07:18PM +0500, rihad wrote: > >What is CPU load in when the load is maximum? > > > It has 2 quad-cores, so I'm not sure. Here's the output of top -S: There is a rumour about FreeBSD's shedulers... That they are not so good for 8 cores and that you may get MORE speed by disabling 4 cores if it's possible for your system. Or even using uniprocessor kernel. Only rumour, though :-) Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 16:09:54 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9651B1065695 for ; Mon, 5 Oct 2009 16:09:54 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id 4F03D8FC0C for ; Mon, 5 Oct 2009 16:09:54 +0000 (UTC) Received: from [217.25.27.27] (port=59166 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1Muq8F-000Pwq-00; Mon, 05 Oct 2009 20:09:52 +0400 Message-ID: <4ACA1A4D.4070801@mail.ru> Date: Mon, 05 Oct 2009 21:09:49 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091005145037.GA92519@svzserv.kemerovo.su> <4ACA0BA6.70706@mail.ru> <20091005154236.GA95635@svzserv.kemerovo.su> In-Reply-To: <20091005154236.GA95635@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 16:09:54 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 08:07:18PM +0500, rihad wrote: > >>> What is CPU load in when the load is maximum? >>> >> It has 2 quad-cores, so I'm not sure. Here's the output of top -S: > > There is a rumour about FreeBSD's shedulers... > That they are not so good for 8 cores and that you may get MORE speed > by disabling 4 cores if it's possible for your system. > Or even using uniprocessor kernel. > > Only rumour, though :-) > I'd really like to try this as the last option ;-) It's 21:07 where I live, and we're again wasting 9-10 mbit/s w/ 4k users online. systat -ifstat: bce1 in 0.000 Mb/s 0.003 Mb/s 49.004 MB out 470.102 Mb/s 470.102 Mb/s 22.637 TB bce0 in 479.754 Mb/s 479.754 Mb/s 22.858 TB out 0.148 Mb/s 0.207 Mb/s 6.950 GB From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:03:14 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8E421065696 for ; Mon, 5 Oct 2009 17:03:14 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outP.internet-mail-service.net (outp.internet-mail-service.net [216.240.47.239]) by mx1.freebsd.org (Postfix) with ESMTP id AC9868FC1C for ; Mon, 5 Oct 2009 17:03:14 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 1DEBFB98A1; Mon, 5 Oct 2009 10:03:48 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 941B52D601B; Mon, 5 Oct 2009 10:03:13 -0700 (PDT) Message-ID: <4ACA26D4.2080102@elischer.org> Date: Mon, 05 Oct 2009 10:03:16 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> In-Reply-To: <20091005100446.GA60244@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: rihad , Eugene Grosbein , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:03:14 -0000 Luigi Rizzo wrote: >> Taildrop does not really help with this. GRED does much better. > > i think the first problem here is figure out _why_ we have > the drops, as the original poster said that queues are configured > with a very large amount of buffer (and i think there is a > misconfiguration somewhere because the mbuf stats do not make > sense) it all depends on the characteristics of the traffic you need different queue lengths if it is just a small number of high speeed sessions (and mayne a large number of slow speed sessions), or if it is a larger number of medium speed sessions. Is it possible to know what sessions are losing packets? From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:18:09 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01C0E106566B for ; Mon, 5 Oct 2009 17:18:09 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outW.internet-mail-service.net (outw.internet-mail-service.net [216.240.47.246]) by mx1.freebsd.org (Postfix) with ESMTP id D649A8FC14 for ; Mon, 5 Oct 2009 17:18:08 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 15B31B987B; Mon, 5 Oct 2009 10:18:16 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id EB3F92D6018; Mon, 5 Oct 2009 10:18:07 -0700 (PDT) Message-ID: <4ACA2A52.4050704@elischer.org> Date: Mon, 05 Oct 2009 10:18:10 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> In-Reply-To: <4AC9D87E.7000005@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:18:09 -0000 rihad wrote: > Luigi Rizzo wrote: >> On Mon, Oct 05, 2009 at 03:52:39PM +0500, rihad wrote: >>> Eugene Grosbein wrote: >>>> On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote: >>>> >>>>> Still not sure why increasing queue size as high as I want doesn't >>>>> completely eliminate drops. >>>> The goal is to make sources of traffic to slow down, this is the only >>>> way to descrease drops - any finite queue may be overhelmed with >>>> traffic. >>>> Taildrop does not really help with this. GRED does much better. >>>> >>> Alright, so I changed to gred by adding to each config command: >>> ipfw ... gred 0.002/900/1000/0.1 queue 1000 >>> and reconfigured. Still around 300-400 drops per second, which was >>> typical at this load level before with taildrop anyway. There are >>> around 3-5 mbit/s being wasted according to systat -ifstat. >>> >>> Should I now increase slots to 5-10-20k? >>> Very strange. >>> >>> "ipfw pipe show" correctly shows that gred is at work. For example: >>> 00512: 512.000 Kbit/s 0 ms 1000 sl. 79 queues (64 buckets) >>> GRED w_q 0.001999 min_th 900 max_th 1000 max_p 0.099991 >>> mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 >>> ... >> >> you keep omitting the important info i.e. whether individual >> pipes have drops, significant queue lenghts and so on. >> > Sorry. Almost everyone has 0 in the last Drp column, but some have above > zero. I'm not just sure how this can be helpful to anyone. This tells us there are just a few sessions with VERY LARGE WINDOWS that are trying to push the link too fast. They are flooding the firewall and geting all the drops. (unless the problem is that the hash/mask is putting a lot of session on the same slot.) > > 05120: 5.120 Mbit/s 0 ms 5000 sl. 66 queues (64 buckets) > GRED w_q 0.001999 min_th 4500 max_th 5000 max_p 0.099991 > mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 > BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes > Pkt/Byte Drp > 0 ip 0.0.0.0/0 1 131 0 0 0 > 1 ip 0.0.0.0/0 39 53360 0 0 0 > 2 ip 0.0.0.0/0 382206 418022848 0 0 0 > 3 ip 0.0.0.0/0 34 2008 0 0 0 > 4 ip 0.0.0.0/0 4868510 6277077787 15 > 20452 9 > 5 ip 0.0.0.0/0 14 16675 0 0 0 > 5 ip 0.0.0.0/0 3 4158 0 0 0 > 6 ip 0.0.0.0/0 38 43576 0 0 0 > 7 ip 0.0.0.0/0 1265954 1475400663 0 > 0 0 > 8 ip 0.0.0.0/0 1081461 1247681879 0 > 0 749 > 9 ip 0.0.0.0/0 6186589 8737048919 0 > 0 19243 > 10 ip 0.0.0.0/0 21607 5636447 0 0 5 > 11 ip 0.0.0.0/0 437 94576 0 0 0 > 12 ip 0.0.0.0/0 22915 18634779 0 0 0 > 13 ip 0.0.0.0/0 557988 688051579 0 0 0 > 14 ip 0.0.0.0/0 50339 65685647 0 0 0 > 15 ip 0.0.0.0/0 554835 546223485 0 0 140 > 16 ip 0.0.0.0/0 32 13104 0 0 0 > 17 ip 0.0.0.0/0 2034099 2719966792 0 > 0 0 > 18 ip 0.0.0.0/0 282 36551 0 0 0 > 19 ip 0.0.0.0/0 8351766 8947643162 0 > 0 0 > 20 ip 0.0.0.0/0 4 624 0 0 0 > 21 ip 0.0.0.0/0 22391 29922375 0 0 0 > 22 ip 0.0.0.0/0 9 424 0 0 0 > 23 ip 0.0.0.0/0 750322 935365326 0 0 0 > 24 ip 0.0.0.0/0 1 40 0 0 0 > 25 ip 0.0.0.0/0 3617690 3501375619 0 > 0 602 > 26 ip 0.0.0.0/0 12116 12039435 0 0 0 > 27 ip 0.0.0.0/0 524311 653399507 0 0 8 > 28 ip 0.0.0.0/0 3 417 0 0 0 > 29 ip 0.0.0.0/0 16 2034 0 0 0 > 30 ip 0.0.0.0/0 64 82661 3 4432 0 > 31 ip 0.0.0.0/0 946389 1175221367 0 0 66 > 32 ip 0.0.0.0/0 1 168 0 0 0 > 32 ip 0.0.0.0/0 28 41776 0 0 0 > 33 ip 0.0.0.0/0 6 6433 0 0 0 > 34 ip 0.0.0.0/0 1 536 0 0 0 > 35 ip 0.0.0.0/0 2021 2641048 0 0 0 > 36 ip 0.0.0.0/0 350 264039 0 0 0 > 37 ip 0.0.0.0/0 167578 137763107 0 0 0 > 38 ip 0.0.0.0/0 250404 128905757 0 0 0 > 39 ip 0.0.0.0/0 385139 287006012 0 0 0 > 40 ip 0.0.0.0/0 49 68696 0 0 0 > 41 ip 0.0.0.0/0 23 1813 0 0 0 > 42 ip 0.0.0.0/0 129 135256 0 0 0 > 43 ip 0.0.0.0/0 3232 2191027 0 0 0 > 44 ip 0.0.0.0/0 27935157 24307287646 0 > 0 18802 > 45 ip 0.0.0.0/0 2166 212635 0 0 0 > 46 ip 0.0.0.0/0 1127307 1392467620 0 > 0 3 > 47 ip 0.0.0.0/0 1216900 1258200836 0 > 0 0 > 48 ip 0.0.0.0/0 2 2984 1 1492 0 > 49 ip 0.0.0.0/0 1 112 0 0 0 > 50 ip 0.0.0.0/0 1409 326389 0 0 0 > 51 ip 0.0.0.0/0 46674 47291021 10 14920 0 > 52 ip 0.0.0.0/0 86667 66834983 0 0 0 > 53 ip 0.0.0.0/0 434998 302827189 0 0 0 > 54 ip 0.0.0.0/0 542 277669 0 0 0 > 55 ip 0.0.0.0/0 1088072 919495021 0 0 0 > 56 ip 0.0.0.0/0 64 81240 0 0 0 > 57 ip 0.0.0.0/0 41028 59193278 0 0 0 > 58 ip 0.0.0.0/0 1 210 0 0 0 > 59 ip 0.0.0.0/0 4 310 0 0 0 > 60 ip 0.0.0.0/0 2 2984 0 0 0 > 61 ip 0.0.0.0/0 42874 36616688 0 0 0 > 62 ip 0.0.0.0/0 4 498 0 0 0 > 63 ip 0.0.0.0/0 530137 717027403 0 0 0 > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:23:57 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5B8E1065672 for ; Mon, 5 Oct 2009 17:23:57 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx39.mail.ru (mx39.mail.ru [94.100.176.53]) by mx1.freebsd.org (Postfix) with ESMTP id 8DC638FC0C for ; Mon, 5 Oct 2009 17:23:57 +0000 (UTC) Received: from [217.25.27.27] (port=15152 helo=[217.25.27.27]) by mx39.mail.ru with asmtp id 1MurHw-000HKg-00; Mon, 05 Oct 2009 21:23:56 +0400 Message-ID: <4ACA2BA6.40709@mail.ru> Date: Mon, 05 Oct 2009 22:23:50 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <4ACA26D4.2080102@elischer.org> In-Reply-To: <4ACA26D4.2080102@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Eugene Grosbein , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:23:57 -0000 Julian Elischer wrote: > Luigi Rizzo wrote: > >>> Taildrop does not really help with this. GRED does much better. >> >> i think the first problem here is figure out _why_ we have >> the drops, as the original poster said that queues are configured >> with a very large amount of buffer (and i think there is a >> misconfiguration somewhere because the mbuf stats do not make >> sense) > > it all depends on the characteristics of the traffic > > you need different queue lengths if it is just a small number of high > speeed sessions (and mayne a large number of slow speed sessions), > or if it is a larger number of medium speed sessions. > > Is it possible to know what sessions are losing packets? > Yes, of course, by running ipfw pipe show ;-) There's one confusing thing, though: net.inet.ip.dummynet.io_pkt_drop isn't increasing while around 800-1000 packets per second are being dropped right now. And so "ipfw pipe show" Drp column wouldn't grow either. So it's either not dummynet dropping packets, or a bug (?). From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:25:54 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F9C6106566B for ; Mon, 5 Oct 2009 17:25:54 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outP.internet-mail-service.net (outp.internet-mail-service.net [216.240.47.239]) by mx1.freebsd.org (Postfix) with ESMTP id 5F2C48FC0A for ; Mon, 5 Oct 2009 17:25:54 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 2DFF1B985A; Mon, 5 Oct 2009 10:25:56 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 92ED92D601A; Mon, 5 Oct 2009 10:25:53 -0700 (PDT) Message-ID: <4ACA2C24.2060205@elischer.org> Date: Mon, 05 Oct 2009 10:25:56 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> In-Reply-To: <4AC9E29B.6080908@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:25:54 -0000 rihad wrote: > Luigi Rizzo wrote: >> On Mon, Oct 05, 2009 at 04:29:02PM +0500, rihad wrote: >>> Luigi Rizzo wrote: >> ... >>>> you keep omitting the important info i.e. whether individual >>>> pipes have drops, significant queue lenghts and so on. >>>> >>> Sorry. Almost everyone has 0 in the last Drp column, but some have >>> above zero. I'm not just sure how this can be helpful to anyone. >> >> because you were complaining about 'dummynet causing drops and >> waste of bandwidth'. >> Now, drops could be due to either >> 1) some saturation in the dummynet machine (memory shortage, cpu >> shortage, etc.) which cause unwanted drops; >> > I too think the box is hitting some other global limit and dropping > packets. If not, then how come that between 4a.m. and 10a.m. when the > traffic load is at 250-330 mbit/s there isn't a single drop? > >> 2) intentional drops introduced by dummynet because a flow exceeds >> its queue size. These drops are those shown in the 'Drop' >> column in 'ipfw pipe show' (they are cumulative, so you >> should do an 'ipfw pipe delete; ipfw pipe 5120 config ...' >> whenever you want to re-run the stats, or compute the >> differences between subsequent reads, to figure out what >> happens. >> >> If all drops you are seeing are of type 2, then there is nothing >> you can do to remove them: you set a bandwidth limit, the >> client is sending faster than it should, perhaps with UDP >> so even RED/GRED won't help you, and you see the drops >> once the queue starts to fill up. >> Examples below: the entries in bucket 4 and 44 >> > Then I guess I'm left with increasing slots and see how it goes. > Currently it's set to 10000 for each pipe. Thanks for yours and Eugene's > efforts, I appreciate it. > >> If you are seeing drops that are not listed in 'pipe show' >> then yun need to investigate where the packets are lost, >> again it could be on the output queue of the interface >> (due to the burstiness introduced by dummynet), or shortage >> of mbufs (but this did not seem to be the case from your >> previous stats) or something else. >> > This indeed is not a problem, proved by the fact that, like I said, > short-circuiting "ipfw allow ip from any to any" before dummynet pipe > rules instantly eliminates all drops, and bce0 and bce1 load evens out > (bce0 used for input, and bce1 for output). no it could be a problem because dummy net releases all the packets for a slot that are going ot be let for a tick out at once, instead of having them arrive spread out through the tick. also it does one pipe at a time which means that related packets arrive at once followed by packets from other sessions.. this may produce differences in some cases. > >> It's all up to you to run measurements, possibly >> without omitting potentially significant data >> (e.g. sysctl -a net.inet.ip) >> or making assumptions (e.g. you have configured >> 5000 slots per queue, but with only 50k mbufs in total >> there is no chance to guarantee 5000 slots to each >> queue -- all you will achieve is give a lot of slots >> to the greedy nodes, and very little to the other ones) >> > Well, I've been monitoring this stuff. It has never reached above 20000 > mbufs (111111 is the current limit). > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:27:13 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F15041065672 for ; Mon, 5 Oct 2009 17:27:13 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outH.internet-mail-service.net (outh.internet-mail-service.net [216.240.47.231]) by mx1.freebsd.org (Postfix) with ESMTP id D0EDF8FC25 for ; Mon, 5 Oct 2009 17:27:13 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 345D7B985A; Mon, 5 Oct 2009 10:27:18 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 2A0052D6011; Mon, 5 Oct 2009 10:27:13 -0700 (PDT) Message-ID: <4ACA2C74.9030206@elischer.org> Date: Mon, 05 Oct 2009 10:27:16 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <20091005100532.GC73335@svzserv.kemerovo.su> <4AC9C88A.5050509@mail.ru> <20091005113037.GA77999@svzserv.kemerovo.su> <4AC9DD72.9060802@mail.ru> <20091005120057.GA79942@svzserv.kemerovo.su> <4AC9E415.9040801@mail.ru> In-Reply-To: <4AC9E415.9040801@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Eugene Grosbein Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:27:14 -0000 rihad wrote: > Eugene Grosbein wrote: >> On Mon, Oct 05, 2009 at 04:50:10PM +0500, rihad wrote: >> >>>>> Where has TCP slow-start gone? My router box isn't some application >>>>> proxy that starts downloading at full 100 mbit/s thus quickly >>>>> filling client's 1 mbit/s link. It's just a router. >>>> While there is no or little competition for bandwidth from the router >>>> to clients, TCP would work just fine. I suspect your shaping policy >>>> makes heavy competition between clients. In this case, TCP behaves >>>> not-so-well without help of router's good shaping algorythms >>>> and taildrop is not good one. >>>> >>> Nothing fancy (i.e. no competition). Only tons of per-user pipes >>> simulating the given throughput. >> >> You've mentioned previously: "The pipes are fine, each normally having >> 100-120 concurrent consumers (i.e. active users)." >> This IS competition between TCP flows inside each pipe. >> > Well, each user gets instantiated with a new copy of the pipe. Each such > user counts towards the limit imposed by hash_size*max_chain_len for > that pipe only. It would have been competition had I used dst-ip dst-ip > 0xffffff00 or similar and not dst-ip 0xffffffff, _then_ all 256 users > (determined by the mask) would compete for the pipe's bandwidth. So the > only competition is in the uplink at our main Cisco, I guess. yesssss, so try running your interfaces slower. :-) > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:28:37 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C4681065670 for ; Mon, 5 Oct 2009 17:28:37 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outO.internet-mail-service.net (outo.internet-mail-service.net [216.240.47.238]) by mx1.freebsd.org (Postfix) with ESMTP id 3C46F8FC1A for ; Mon, 5 Oct 2009 17:28:36 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 7AA3CB985A; Mon, 5 Oct 2009 10:28:42 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id ABA0C2D6015; Mon, 5 Oct 2009 10:28:35 -0700 (PDT) Message-ID: <4ACA2CC6.70201@elischer.org> Date: Mon, 05 Oct 2009 10:28:38 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> In-Reply-To: <4AC9EFDF.4080302@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:28:37 -0000 rihad wrote: > Luigi Rizzo wrote: >> On Mon, Oct 05, 2009 at 05:12:11PM +0500, rihad wrote: >>> Luigi Rizzo wrote: >>>> On Mon, Oct 05, 2009 at 04:29:02PM +0500, rihad wrote: >>>>> Luigi Rizzo wrote: >>>> ... >>>>>> you keep omitting the important info i.e. whether individual >>>>>> pipes have drops, significant queue lenghts and so on. >>>>>> >>>>> Sorry. Almost everyone has 0 in the last Drp column, but some have >>>>> above zero. I'm not just sure how this can be helpful to anyone. >>>> because you were complaining about 'dummynet causing drops and >>>> waste of bandwidth'. >>>> Now, drops could be due to either >>>> 1) some saturation in the dummynet machine (memory shortage, cpu >>>> shortage, etc.) which cause unwanted drops; >>>> >>> I too think the box is hitting some other global limit and dropping >>> packets. If not, then how come that between 4a.m. and 10a.m. when the >>> traffic load is at 250-330 mbit/s there isn't a single drop? >> >> there may be different reasons, e.g. the big offenders were >> idle when you saw no drops. You still do not have enough >> information on which packets are dropped and where, >> so you cannot prove your assumptions. >> >> Also, below: >> 1. increasing the queue size won't help at all. Those >> who overflow a queue of 1000 slots will also overflow >> a queue of 10k slots. >> > >> 2. your test with 'ipfw allow ip from any to any' does not >> prove that the interface queue is not saturating, because >> you also remove the burstiness that dummynet introduces, >> and so the queue is driven differently. >> > > There's one thing I noticed: > net.inet.ip.dummynet.io_pkt_drop doesn't grow! But still around 400 > packets dropped per second. > net.inet.ip.dummynet.tick_lost is always zero > net.inet.ip.dummynet.tick_diff: grows at about 50 per second. > net.inet.ip.dummynet.tick_adjustment: grows at about 5 per second. > > How do I investigate and fix this burstiness issue? higher Hz rate? > > > $ netstat -i > Name Mtu Network Address Ipkts Ierrs Opkts > Oerrs Coll > bce0 1500 00:1d:09:xx:xx:xx 24777049059 0 75426020 > 0 0 > bce0 1500 xx.xx.xx.xx/xx my.hostname 159293969 - 75282225 > - - > bce1 1500 00:1d:09:xx:xx:xx 724725 0 24514919344 > 0 0 > bce1 1500 192.168.94.0 local.hostname 656243 - 83024869 - - > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:33:44 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 997CD1065672 for ; Mon, 5 Oct 2009 17:33:44 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id 51C5A8FC19 for ; Mon, 5 Oct 2009 17:33:44 +0000 (UTC) Received: from [217.25.27.27] (port=56179 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1MurRO-000Gan-00; Mon, 05 Oct 2009 21:33:42 +0400 Message-ID: <4ACA2DDB.50809@mail.ru> Date: Mon, 05 Oct 2009 22:33:15 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <4ACA2A52.4050704@elischer.org> In-Reply-To: <4ACA2A52.4050704@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:33:44 -0000 Julian Elischer wrote: > rihad wrote: >> Luigi Rizzo wrote: >>> On Mon, Oct 05, 2009 at 03:52:39PM +0500, rihad wrote: >>>> Eugene Grosbein wrote: >>>>> On Mon, Oct 05, 2009 at 02:28:58PM +0500, rihad wrote: >>>>> >>>>>> Still not sure why increasing queue size as high as I want doesn't >>>>>> completely eliminate drops. >>>>> The goal is to make sources of traffic to slow down, this is the only >>>>> way to descrease drops - any finite queue may be overhelmed with >>>>> traffic. >>>>> Taildrop does not really help with this. GRED does much better. >>>>> >>>> Alright, so I changed to gred by adding to each config command: >>>> ipfw ... gred 0.002/900/1000/0.1 queue 1000 >>>> and reconfigured. Still around 300-400 drops per second, which was >>>> typical at this load level before with taildrop anyway. There are >>>> around 3-5 mbit/s being wasted according to systat -ifstat. >>>> >>>> Should I now increase slots to 5-10-20k? >>>> Very strange. >>>> >>>> "ipfw pipe show" correctly shows that gred is at work. For example: >>>> 00512: 512.000 Kbit/s 0 ms 1000 sl. 79 queues (64 buckets) >>>> GRED w_q 0.001999 min_th 900 max_th 1000 max_p 0.099991 >>>> mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 >>>> ... >>> >>> you keep omitting the important info i.e. whether individual >>> pipes have drops, significant queue lenghts and so on. >>> >> Sorry. Almost everyone has 0 in the last Drp column, but some have >> above zero. I'm not just sure how this can be helpful to anyone. > > > This tells us there are just a few sessions with VERY LARGE WINDOWS > that are trying to push the link too fast. They are flooding the > firewall and geting all the drops. (unless the problem is that the > hash/mask is putting a lot of session on the same slot.) > > All masks are dst-ip 0xffffffff. I doubt very strongly that it's due to such offenders, because all the way from 4a.m. to 10a.m. at the load of 200-330 mbit/s there are zero drops. There are no doubt heavy downloaders leaving their PCs on in the night. The drops begin as soon as the load crosses 340-360 mbit/s or some such, and it only worsens as the load approaches 500-530 mbit/s. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:49:42 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC30B106566B for ; Mon, 5 Oct 2009 17:49:42 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outE.internet-mail-service.net (oute.internet-mail-service.net [216.240.47.228]) by mx1.freebsd.org (Postfix) with ESMTP id CE6598FC1B for ; Mon, 5 Oct 2009 17:49:42 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 6137E22E7; Mon, 5 Oct 2009 10:49:42 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 0B9B72D6012; Mon, 5 Oct 2009 10:49:41 -0700 (PDT) Message-ID: <4ACA31B9.5090406@elischer.org> Date: Mon, 05 Oct 2009 10:49:45 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091005145037.GA92519@svzserv.kemerovo.su> <4ACA0BA6.70706@mail.ru> <20091005154236.GA95635@svzserv.kemerovo.su> In-Reply-To: <20091005154236.GA95635@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: rihad , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:49:43 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 08:07:18PM +0500, rihad wrote: > >>> What is CPU load in when the load is maximum? >>> >> It has 2 quad-cores, so I'm not sure. Here's the output of top -S: > > There is a rumour about FreeBSD's shedulers... > That they are not so good for 8 cores and that you may get MORE speed > by disabling 4 cores if it's possible for your system. > Or even using uniprocessor kernel. > > Only rumour, though :-) > true for 6.x, less true for 7.x, and not true for 8.x > Eugene Grosbein > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:56:42 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2BF9106568B for ; Mon, 5 Oct 2009 17:56:42 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outQ.internet-mail-service.net (outq.internet-mail-service.net [216.240.47.240]) by mx1.freebsd.org (Postfix) with ESMTP id B5E308FC15 for ; Mon, 5 Oct 2009 17:56:42 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 41380D4E0; Mon, 5 Oct 2009 10:57:00 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 7DD8E2D6013; Mon, 5 Oct 2009 10:56:41 -0700 (PDT) Message-ID: <4ACA335C.2060806@elischer.org> Date: Mon, 05 Oct 2009 10:56:44 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <4ACA26D4.2080102@elischer.org> <4ACA2BA6.40709@mail.ru> In-Reply-To: <4ACA2BA6.40709@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Eugene Grosbein , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:56:42 -0000 rihad wrote: > Julian Elischer wrote: >> Luigi Rizzo wrote: >> >>>> Taildrop does not really help with this. GRED does much better. >>> >>> i think the first problem here is figure out _why_ we have >>> the drops, as the original poster said that queues are configured >>> with a very large amount of buffer (and i think there is a >>> misconfiguration somewhere because the mbuf stats do not make >>> sense) >> >> it all depends on the characteristics of the traffic >> >> you need different queue lengths if it is just a small number of high >> speeed sessions (and mayne a large number of slow speed sessions), >> or if it is a larger number of medium speed sessions. >> >> Is it possible to know what sessions are losing packets? >> > Yes, of course, by running ipfw pipe show ;-) > There's one confusing thing, though: net.inet.ip.dummynet.io_pkt_drop > isn't increasing while around 800-1000 packets per second are being > dropped right now. And so "ipfw pipe show" Drp column wouldn't grow > either. So it's either not dummynet dropping packets, or a bug (?). I suspect your interface queues. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 17:56:46 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F2AF1106568B for ; Mon, 5 Oct 2009 17:56:46 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id AB6A88FC1D for ; Mon, 5 Oct 2009 17:56:46 +0000 (UTC) Received: from [217.25.27.27] (port=6243 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1Murnh-00010s-00; Mon, 05 Oct 2009 21:56:45 +0400 Message-ID: <4ACA3359.9040604@mail.ru> Date: Mon, 05 Oct 2009 22:56:41 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> In-Reply-To: <4ACA2CC6.70201@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 17:56:47 -0000 Julian Elischer wrote: >> How do I investigate and fix this burstiness issue? > > higher Hz rate? > Hmm, mine is 1000. I'll try bumping it up to 2000 (via /boot/loader.conf) but since a reboot is required I think it'll have to wait for a while. From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 18:03:36 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 177661065694 for ; Mon, 5 Oct 2009 18:03:36 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx40.mail.ru (mx40.mail.ru [94.100.176.54]) by mx1.freebsd.org (Postfix) with ESMTP id C48118FC1A for ; Mon, 5 Oct 2009 18:03:35 +0000 (UTC) Received: from [217.25.27.27] (port=39766 helo=[217.25.27.27]) by mx40.mail.ru with asmtp id 1MuruH-000B8W-00; Mon, 05 Oct 2009 22:03:33 +0400 Message-ID: <4ACA34F3.60901@mail.ru> Date: Mon, 05 Oct 2009 23:03:31 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <4ACA26D4.2080102@elischer.org> <4ACA2BA6.40709@mail.ru> <4ACA335C.2060806@elischer.org> In-Reply-To: <4ACA335C.2060806@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Eugene Grosbein , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 18:03:36 -0000 Julian Elischer wrote: > rihad wrote: >> Julian Elischer wrote: >>> Luigi Rizzo wrote: >>> >>> Is it possible to know what sessions are losing packets? >>> >> Yes, of course, by running ipfw pipe show ;-) >> There's one confusing thing, though: net.inet.ip.dummynet.io_pkt_drop >> isn't increasing while around 800-1000 packets per second are being >> dropped right now. And so "ipfw pipe show" Drp column wouldn't grow >> either. So it's either not dummynet dropping packets, or a bug (?). > > > I suspect your interface queues. > You mean in hardware? Any way to tweak those? From owner-freebsd-net@FreeBSD.ORG Mon Oct 5 18:54:48 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0C341065692 for ; Mon, 5 Oct 2009 18:54:48 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outC.internet-mail-service.net (outc.internet-mail-service.net [216.240.47.226]) by mx1.freebsd.org (Postfix) with ESMTP id D328E8FC16 for ; Mon, 5 Oct 2009 18:54:47 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id E8C14B98E3; Mon, 5 Oct 2009 11:54:47 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 24DB72D6022; Mon, 5 Oct 2009 11:54:47 -0700 (PDT) Message-ID: <4ACA40FA.7010207@elischer.org> Date: Mon, 05 Oct 2009 11:54:50 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC8A76B.3050502@mail.ru> <20091005025521.GA52702@svzserv.kemerovo.su> <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <20091005100446.GA60244@onelab2.iet.unipi.it> <4ACA26D4.2080102@elischer.org> <4ACA2BA6.40709@mail.ru> <4ACA335C.2060806@elischer.org> <4ACA34F3.60901@mail.ru> In-Reply-To: <4ACA34F3.60901@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Eugene Grosbein , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 18:54:49 -0000 rihad wrote: > Julian Elischer wrote: >> rihad wrote: >>> Julian Elischer wrote: >>>> Luigi Rizzo wrote: >>>> >>>> Is it possible to know what sessions are losing packets? >>>> >>> Yes, of course, by running ipfw pipe show ;-) >>> There's one confusing thing, though: net.inet.ip.dummynet.io_pkt_drop >>> isn't increasing while around 800-1000 packets per second are being >>> dropped right now. And so "ipfw pipe show" Drp column wouldn't grow >>> either. So it's either not dummynet dropping packets, or a bug (?). >> >> >> I suspect your interface queues. >> > > You mean in hardware? Any way to tweak those? no, there is (usually) a software queue before the hardware queue. look at /sys/net/if_ethersubr.c From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 04:21:58 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE720106568D for ; Tue, 6 Oct 2009 04:21:58 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 56CD28FC17 for ; Tue, 6 Oct 2009 04:21:57 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n964Ls9e006299; Tue, 6 Oct 2009 12:21:54 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n964Lrk0006298; Tue, 6 Oct 2009 12:21:53 +0800 (KRAST) (envelope-from eugen) Date: Tue, 6 Oct 2009 12:21:53 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091006042153.GA5857@svzserv.kemerovo.su> References: <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACA02F7.6040409@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 04:21:59 -0000 On Mon, Oct 05, 2009 at 07:30:15PM +0500, rihad wrote: > >>How do I investigate and fix this burstiness issue? > > > >Please also show: > > > >sysctl net.isr > >sysctl net.inet.ip.intr_queue_maxlen > > net.isr.swi_count: 65461359 > net.isr.drop: 0 > net.isr.queued: 32843752 > net.isr.deferred: 0 > net.isr.directed: -723075002 > net.isr.count: -723074001 > net.isr.direct: 1 > > net.inet.ip.intr_queue_maxlen: 50 Try to increase net.inet.ip.intr_queue_maxlen uptio 4096. Eugene From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 04:23:43 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A408106568B for ; Tue, 6 Oct 2009 04:23:43 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id D6D4C8FC0A for ; Tue, 6 Oct 2009 04:23:42 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n964NZvo006352; Tue, 6 Oct 2009 12:23:35 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n964NZ0K006351; Tue, 6 Oct 2009 12:23:35 +0800 (KRAST) (envelope-from eugen) Date: Tue, 6 Oct 2009 12:23:35 +0800 From: Eugene Grosbein To: Julian Elischer Message-ID: <20091006042335.GB5857@svzserv.kemerovo.su> References: <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091005145037.GA92519@svzserv.kemerovo.su> <4ACA0BA6.70706@mail.ru> <20091005154236.GA95635@svzserv.kemerovo.su> <4ACA31B9.5090406@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACA31B9.5090406@elischer.org> User-Agent: Mutt/1.4.2.3i Cc: rihad , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 04:23:43 -0000 On Mon, Oct 05, 2009 at 10:49:45AM -0700, Julian Elischer wrote: > >There is a rumour about FreeBSD's shedulers... > >That they are not so good for 8 cores and that you may get MORE speed > >by disabling 4 cores if it's possible for your system. > >Or even using uniprocessor kernel. > >Only rumour, though :-) > true for 6.x, less true for 7.x, and not true for 8.x That's great. For both of ULE and BSD? Eugene From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 04:38:41 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29393106566B for ; Tue, 6 Oct 2009 04:38:41 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id D5C738FC12 for ; Tue, 6 Oct 2009 04:38:40 +0000 (UTC) Received: from [217.25.27.27] (port=47775 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1Mv1ot-000K1T-00; Tue, 06 Oct 2009 08:38:39 +0400 Message-ID: <4ACAC9CE.7080900@mail.ru> Date: Tue, 06 Oct 2009 09:38:38 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091006042153.GA5857@svzserv.kemerovo.su> In-Reply-To: <20091006042153.GA5857@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 04:38:41 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 07:30:15PM +0500, rihad wrote: > >>>> How do I investigate and fix this burstiness issue? >>> Please also show: >>> >>> sysctl net.isr >>> sysctl net.inet.ip.intr_queue_maxlen >> net.isr.swi_count: 65461359 >> net.isr.drop: 0 >> net.isr.queued: 32843752 >> net.isr.deferred: 0 >> net.isr.directed: -723075002 >> net.isr.count: -723074001 >> net.isr.direct: 1 >> >> net.inet.ip.intr_queue_maxlen: 50 > > Try to increase net.inet.ip.intr_queue_maxlen uptio 4096. > You sure? Packets are never dropped once I add "allow ip from any to any" before pipes, effectively turning dummynet off. Yet I've doubled it for starters (50->100) let's see if it works in an hour or so, when it's 10:30 here. From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 05:21:20 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9C27106566B for ; Tue, 6 Oct 2009 05:21:20 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx2.mail.ru (mx2.mail.ru [94.100.176.16]) by mx1.freebsd.org (Postfix) with ESMTP id 5D3558FC12 for ; Tue, 6 Oct 2009 05:21:19 +0000 (UTC) Received: from [217.25.27.27] (port=11279 helo=[217.25.27.27]) by mx2.mail.ru with asmtp id 1Mv2UA-0007bG-00; Tue, 06 Oct 2009 09:21:18 +0400 Message-ID: <4ACAD3CD.4000908@mail.ru> Date: Tue, 06 Oct 2009 10:21:17 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091006042153.GA5857@svzserv.kemerovo.su> <4ACAC9CE.7080900@mail.ru> In-Reply-To: <4ACAC9CE.7080900@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 05:21:20 -0000 > Eugene Grosbein wrote: >> Try to increase net.inet.ip.intr_queue_maxlen uptio 4096. >> > You sure? Packets are never dropped once I add "allow ip from any to > any" before pipes, effectively turning dummynet off. Yet I've doubled it > for starters (50->100) let's see if it works in an hour or so, when it's > 10:30 here. > Besides, net.inet.ip.intr_queue_maxlen: Maximum size of the IP input queue but the drops have been occurring on the _output_. I'll see in about 10 minutes if it's true or not. I've also set up kern.hz=2000 in loader.conf the effects of which will be seen in an hour or so, when I'm permitted to reboot the box. From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 06:38:55 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49EC8106566B for ; Tue, 6 Oct 2009 06:38:55 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id 031B88FC0C for ; Tue, 6 Oct 2009 06:38:54 +0000 (UTC) Received: from [217.25.27.27] (port=46837 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1Mv3hF-000BbP-00; Tue, 06 Oct 2009 10:38:53 +0400 Message-ID: <4ACAE5FC.7040605@mail.ru> Date: Tue, 06 Oct 2009 11:38:52 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <20091005140409.GB89194@svzserv.kemerovo.su> <4ACA02F7.6040409@mail.ru> <20091006042153.GA5857@svzserv.kemerovo.su> In-Reply-To: <20091006042153.GA5857@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 06:38:55 -0000 Eugene Grosbein wrote: > On Mon, Oct 05, 2009 at 07:30:15PM +0500, rihad wrote: > >>>> How do I investigate and fix this burstiness issue? >>> Please also show: >>> >>> sysctl net.isr >>> sysctl net.inet.ip.intr_queue_maxlen >> net.isr.swi_count: 65461359 >> net.isr.drop: 0 >> net.isr.queued: 32843752 >> net.isr.deferred: 0 >> net.isr.directed: -723075002 >> net.isr.count: -723074001 >> net.isr.direct: 1 >> >> net.inet.ip.intr_queue_maxlen: 50 > > Try to increase net.inet.ip.intr_queue_maxlen uptio 4096. > Apparently increasing the input queue had no effect on the output drops. Besides, net.inet.ip.intr_queue_drops has only reached 70 for 6 days of uptime. I'll soon be rebooting to change HZ 1000 -> 2000, and we'll see how that goes. There must be some global limit, not nmbclusters, not per-user output queues, not the interface input queue, but something else that triggers at around 360-370 mbit/s... From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 08:26:23 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13EF9106566B for ; Tue, 6 Oct 2009 08:26:23 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id BFF998FC1E for ; Tue, 6 Oct 2009 08:26:22 +0000 (UTC) Received: from [217.25.27.27] (port=38681 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1Mv5NE-000HMG-00; Tue, 06 Oct 2009 12:26:21 +0400 Message-ID: <4ACAFF2A.1000206@mail.ru> Date: Tue, 06 Oct 2009 13:26:18 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> In-Reply-To: <4ACA2CC6.70201@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 08:26:23 -0000 Julian Elischer wrote: > rihad wrote: >> Luigi Rizzo wrote: >>> 2. your test with 'ipfw allow ip from any to any' does not >>> prove that the interface queue is not saturating, because >>> you also remove the burstiness that dummynet introduces, >>> and so the queue is driven differently. >>> >> >> How do I investigate and fix this burstiness issue? > > higher Hz rate? > Rebooted with HZ=2000 10 minutes ago. Due to application design the ipfw table (pipe tablearg) was flushed, so there are now 350 (and increasing at a rate 1 per 1-2 seconds as I type this) or so users in the table, and not 4k as normally would be. The box is servicing 450+ mbit/s without a single drop. I want to monitor how things change once the number of users in ipfw tables gradually increases up to several thousands. From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 09:20:03 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9A3F106568B for ; Tue, 6 Oct 2009 09:20:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7E0AE8FC13 for ; Tue, 6 Oct 2009 09:20:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n969K319071483 for ; Tue, 6 Oct 2009 09:20:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n969K3K9071480; Tue, 6 Oct 2009 09:20:03 GMT (envelope-from gnats) Date: Tue, 6 Oct 2009 09:20:03 GMT Message-Id: <200910060920.n969K3K9071480@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Gaurav Goel Cc: Subject: Re: kern/138652: TCP window scaling value calculated incorrectly? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Gaurav Goel List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 09:20:03 -0000 The following reply was made to PR kern/138652; it has been noted by GNATS. From: Gaurav Goel To: bug-followup@freebsd.org Cc: Subject: Re: kern/138652: TCP window scaling value calculated incorrectly? Date: Tue, 6 Oct 2009 14:47:52 +0530 --000feaf371bd7882aa047540b470 Content-Type: text/plain; charset=UTF-8 Hi, May I know if someone is working on the bug. I have provided the solution too, but didn't get any response. Thanks, Gaurav Goel On Thu, Sep 10, 2009 at 7:00 PM, Gaurav Goel wrote: > Dear Gavin, > > Please find the fix for the problem: > > *Replace* > * while (tp->request_r_scale < TCP_MAX_WINSHIFT && > (TCP_MAXWIN << tp->request_r_scale) < sb_max) > tp->request_r_scale++;* > > *With* > *unsigned int new_TCP_MAXWIN = TCP_MAXWIN; > while (tp->request_r_scale < TCP_MAX_WINSHIFT) > { > if(new_TCP_MAXWIN < sb_max) > tp->request_r_scale++; > else > break;** > ** new_TCP_MAXWIN <<=1;** > ** new_TCP_MAXWIN |=1;** > **}* > > Please inform me if I am right/wrong. > > Thanks, > Gaurav Goel > > > On Wed, Sep 9, 2009 at 7:59 PM, wrote: > >> Old Synopsis: TCP window scaling value >> New Synopsis: TCP window scaling value calculated incorrectly? >> >> State-Changed-From-To: feedback->open >> State-Changed-By: gavin >> State-Changed-When: Wed Sep 9 14:24:24 UTC 2009 >> State-Changed-Why: >> Over to maintainer(s) for investigation >> >> >> Responsible-Changed-From-To: gavin->freebsd-net >> Responsible-Changed-By: gavin >> Responsible-Changed-When: Wed Sep 9 14:24:24 UTC 2009 >> Responsible-Changed-Why: >> Feedback was received, thanks! >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=138652 >> > > > > -- > Gaurav Goel > -- Gaurav Goel --000feaf371bd7882aa047540b470 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi,

May I know if someone is working on the bug. I have provided the= solution too, but didn't get any response.

Thanks,
Gaurav Go= el

On Thu, Sep 10, 2009 at 7:00 PM, Gaura= v Goel <gaurav= 0287@gmail.com> wrote:
Dear Gavin,
Please find the fix for the problem:

Replace
<= span style=3D"color: rgb(153, 0, 0); background-color: rgb(255, 255, 255);"= >=C2=A0=C2=A0=C2=A0 while (tp->request_r_scale < TCP_MAX_WINSHIFT &am= p;&
unsigne= d int new_TCP_MAXWIN =3D TCP_MAXWIN;
while (tp->request_r_scale &= lt; TCP_MAX_WINSHIFT)
{
=C2=A0=C2=A0=C2=A0 if(new_TCP_MAXWIN < sb_max)=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 tp->request_r_scale++;
=C2=A0=C2=A0=C2=A0 else
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 break;
=C2=A0=C2=A0=C2=A0 new_TCP_MAXWIN <<=3D1;=
=C2=A0=C2=A0=C2=A0 new_TCP_MAXWIN |=3D1;
}

Please in= form me if I am right/wrong.

Thanks,
Gaurav Goel
<= div class=3D"h5">

On Wed, Sep 9, 2009 at = 7:59 PM, <gavin@freebsd.org> wrote:
Old Synopsis: TCP= window scaling value
New Synopsis: TCP window scaling value calculated incorrectly?

State-Changed-From-To: feedback->open
State-Changed-By: gavin
State-Changed-When: Wed Sep 9 14:24:24 UTC 2009
State-Changed-Why:
Over to maintainer(s) for investigation


Responsible-Changed-From-To: gavin->freebsd-net
Responsible-Changed-By: gavin
Responsible-Changed-When: Wed Sep 9 14:24:24 UTC 2009
Responsible-Changed-Why:
Feedback was received, thanks!

http://www.freebsd.org/cgi/query-pr.cgi?pr=3D138652



--
Gaurav Goel



--
Gaurav Goel
--000feaf371bd7882aa047540b470-- From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 09:21:43 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 773EF1065672 for ; Tue, 6 Oct 2009 09:21:43 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id 2E2DA8FC17 for ; Tue, 6 Oct 2009 09:21:43 +0000 (UTC) Received: from [217.25.27.27] (port=19823 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1Mv6Eo-000DXv-00; Tue, 06 Oct 2009 13:21:42 +0400 Message-ID: <4ACB0C22.4000008@mail.ru> Date: Tue, 06 Oct 2009 14:21:38 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <20091005061025.GB55845@onelab2.iet.unipi.it> <4AC9B400.9020400@mail.ru> <20091005090102.GA70430@svzserv.kemerovo.su> <4AC9BC5A.50902@mail.ru> <20091005095600.GA73335@svzserv.kemerovo.su> <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> In-Reply-To: <4ACAFF2A.1000206@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 09:21:43 -0000 rihad wrote: > Julian Elischer wrote: >> rihad wrote: >>> Luigi Rizzo wrote: >>>> 2. your test with 'ipfw allow ip from any to any' does not >>>> prove that the interface queue is not saturating, because >>>> you also remove the burstiness that dummynet introduces, >>>> and so the queue is driven differently. >>>> >>> >>> How do I investigate and fix this burstiness issue? >> >> higher Hz rate? >> > > Rebooted with HZ=2000 10 minutes ago. Due to application design the ipfw > table (pipe tablearg) was flushed, so there are now 350 (and increasing > at a rate 1 per 1-2 seconds as I type this) or so users in the table, > and not 4k as normally would be. The box is servicing 450+ mbit/s > without a single drop. I want to monitor how things change once the > number of users in ipfw tables gradually increases up to several thousands. > It starts dropping packets at around 2000 online users (ipfw table load). I've set up a shell script to monitor this: # while :; do ipfw table 0 list | wc -l; netstat -s 2>/dev/null |fgrep -w 'output packets dropped'; sleep 10; done ... # all zeroes above this 1999 0 output packets dropped due to no bufs, etc. 2001 0 output packets dropped due to no bufs, etc. 2008 0 output packets dropped due to no bufs, etc. 2017 0 output packets dropped due to no bufs, etc. 2027 156 output packets dropped due to no bufs, etc. 2037 156 output packets dropped due to no bufs, etc. 2045 156 output packets dropped due to no bufs, etc. 2372 202 output packets dropped due to no bufs, etc. 2377 207 output packets dropped due to no bufs, etc. 2391 338 output packets dropped due to no bufs, etc. 2402 394 output packets dropped due to no bufs, etc. 2415 531 output packets dropped due to no bufs, etc. 2421 725 output packets dropped due to no bufs, etc. Is there some limit on the number of IP addresses in an ipfw table? From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 09:27:28 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 550021065679 for ; Tue, 6 Oct 2009 09:27:28 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 160A88FC17 for ; Tue, 6 Oct 2009 09:27:28 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id B1241730DA; Tue, 6 Oct 2009 11:34:08 +0200 (CEST) Date: Tue, 6 Oct 2009 11:34:08 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091006093408.GA86830@onelab2.iet.unipi.it> References: <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACB0C22.4000008@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 09:27:28 -0000 On Tue, Oct 06, 2009 at 02:21:38PM +0500, rihad wrote: > rihad wrote: > >Julian Elischer wrote: > >>rihad wrote: > >>>Luigi Rizzo wrote: > >>>>2. your test with 'ipfw allow ip from any to any' does not > >>>> prove that the interface queue is not saturating, because > >>>> you also remove the burstiness that dummynet introduces, > >>>> and so the queue is driven differently. > >>>> > >>> > >>>How do I investigate and fix this burstiness issue? > >> > >>higher Hz rate? > >> > > > >Rebooted with HZ=2000 10 minutes ago. Due to application design the ipfw > > table (pipe tablearg) was flushed, so there are now 350 (and increasing > >at a rate 1 per 1-2 seconds as I type this) or so users in the table, > >and not 4k as normally would be. The box is servicing 450+ mbit/s > >without a single drop. I want to monitor how things change once the > >number of users in ipfw tables gradually increases up to several thousands. > > > > It starts dropping packets at around 2000 online users (ipfw table > load). I've set up a shell script to monitor this: once again: you should check which pipes are dropping packets and whether the number of drops indicated in the pipes matches the counts indicated by netstat. cheers luigi > # while :; do ipfw table 0 list | wc -l; netstat -s 2>/dev/null |fgrep > -w 'output packets dropped'; sleep 10; done > > ... # all zeroes above this > 1999 > 0 output packets dropped due to no bufs, etc. > 2001 > 0 output packets dropped due to no bufs, etc. > 2008 > 0 output packets dropped due to no bufs, etc. > 2017 > 0 output packets dropped due to no bufs, etc. > 2027 > 156 output packets dropped due to no bufs, etc. > 2037 > 156 output packets dropped due to no bufs, etc. > 2045 > 156 output packets dropped due to no bufs, etc. > 2372 > 202 output packets dropped due to no bufs, etc. > 2377 > 207 output packets dropped due to no bufs, etc. > 2391 > 338 output packets dropped due to no bufs, etc. > 2402 > 394 output packets dropped due to no bufs, etc. > 2415 > 531 output packets dropped due to no bufs, etc. > 2421 > 725 output packets dropped due to no bufs, etc. > > > Is there some limit on the number of IP addresses in an ipfw table? > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 09:34:37 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC6E9106568D for ; Tue, 6 Oct 2009 09:34:37 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 653048FC14 for ; Tue, 6 Oct 2009 09:34:37 +0000 (UTC) Received: from [217.25.27.27] (port=2945 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1Mv6RH-000IeR-00; Tue, 06 Oct 2009 13:34:35 +0400 Message-ID: <4ACB0F28.3000906@mail.ru> Date: Tue, 06 Oct 2009 14:34:32 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006093408.GA86830@onelab2.iet.unipi.it> In-Reply-To: <20091006093408.GA86830@onelab2.iet.unipi.it> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 09:34:37 -0000 Luigi Rizzo wrote: > On Tue, Oct 06, 2009 at 02:21:38PM +0500, rihad wrote: >> rihad wrote: >>> Julian Elischer wrote: >>>> rihad wrote: >>>>> Luigi Rizzo wrote: >>>>>> 2. your test with 'ipfw allow ip from any to any' does not >>>>>> prove that the interface queue is not saturating, because >>>>>> you also remove the burstiness that dummynet introduces, >>>>>> and so the queue is driven differently. >>>>>> >>>>> How do I investigate and fix this burstiness issue? >>>> higher Hz rate? >>>> >>> Rebooted with HZ=2000 10 minutes ago. Due to application design the ipfw >>> table (pipe tablearg) was flushed, so there are now 350 (and increasing >>> at a rate 1 per 1-2 seconds as I type this) or so users in the table, >>> and not 4k as normally would be. The box is servicing 450+ mbit/s >>> without a single drop. I want to monitor how things change once the >>> number of users in ipfw tables gradually increases up to several thousands. >>> >> It starts dropping packets at around 2000 online users (ipfw table >> load). I've set up a shell script to monitor this: > > once again: > you should check which pipes are dropping packets and whether > the number of drops indicated in the pipes matches the counts > indicated by netstat. > It's impossible to do so accurately, since users come and go any moment, and their pipes expire, and it's plain useless. As to the accordance of packet drop rate with net.inet.ip.dummynet.io_pkt_drop, they vary wildly: 8664 output packets dropped due to no bufs, etc. net.inet.ip.dummynet.io_pkt_drop: 111 since boottime! From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 10:07:36 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE0EE106568D for ; Tue, 6 Oct 2009 10:07:36 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 372238FC0C for ; Tue, 6 Oct 2009 10:07:35 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n96A7QtL026931; Tue, 6 Oct 2009 18:07:26 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n96A7QMV026930; Tue, 6 Oct 2009 18:07:26 +0800 (KRAST) (envelope-from eugen) Date: Tue, 6 Oct 2009 18:07:26 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091006100726.GA26426@svzserv.kemerovo.su> References: <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACB0C22.4000008@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 10:07:36 -0000 On Tue, Oct 06, 2009 at 02:21:38PM +0500, rihad wrote: > Is there some limit on the number of IP addresses in an ipfw table? No, generally handles much more. Please show your ipfw rule(s) containing 'tablearg'. Eugene From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 10:11:07 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 508751065670 for ; Tue, 6 Oct 2009 10:11:07 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 116488FC14 for ; Tue, 6 Oct 2009 10:11:06 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id F0DDF730DA; Tue, 6 Oct 2009 12:17:47 +0200 (CEST) Date: Tue, 6 Oct 2009 12:17:47 +0200 From: Luigi Rizzo To: rihad Message-ID: <20091006101747.GA87655@onelab2.iet.unipi.it> References: <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006093408.GA86830@onelab2.iet.unipi.it> <4ACB0F28.3000906@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACB0F28.3000906@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 10:11:07 -0000 On Tue, Oct 06, 2009 at 02:34:32PM +0500, rihad wrote: > Luigi Rizzo wrote: > >On Tue, Oct 06, 2009 at 02:21:38PM +0500, rihad wrote: > >>rihad wrote: > >>>Julian Elischer wrote: > >>>>rihad wrote: > >>>>>Luigi Rizzo wrote: > >>>>>>2. your test with 'ipfw allow ip from any to any' does not > >>>>>> prove that the interface queue is not saturating, because > >>>>>> you also remove the burstiness that dummynet introduces, > >>>>>> and so the queue is driven differently. > >>>>>> > >>>>>How do I investigate and fix this burstiness issue? > >>>>higher Hz rate? > >>>> > >>>Rebooted with HZ=2000 10 minutes ago. Due to application design the ipfw > >>>table (pipe tablearg) was flushed, so there are now 350 (and increasing > >>>at a rate 1 per 1-2 seconds as I type this) or so users in the table, > >>>and not 4k as normally would be. The box is servicing 450+ mbit/s > >>>without a single drop. I want to monitor how things change once the > >>>number of users in ipfw tables gradually increases up to several > >>>thousands. > >>> > >>It starts dropping packets at around 2000 online users (ipfw table > >>load). I've set up a shell script to monitor this: > > > >once again: > >you should check which pipes are dropping packets and whether > >the number of drops indicated in the pipes matches the counts > >indicated by netstat. > > > It's impossible to do so accurately, since users come and go any moment, > and their pipes expire, and it's plain useless. As to the accordance of > packet drop rate with net.inet.ip.dummynet.io_pkt_drop, they vary wildly: > > 8664 output packets dropped due to no bufs, etc. > net.inet.ip.dummynet.io_pkt_drop: 111 io_pkt_drop only reports packets dropped to errors (missing pipes, randomly forced packet drops which you don't use, no buffers and so on). The packets intentionally dropped in dummynet because queues are full are listed by 'ipfw pipe show'. Even if pipes expire, there is a difference between having partial information and completely ignoring what is available and claiming "it's plain useless". BTW at least while you try to debug the problem you can temporarily disable the pipe expire with 'sysctl net.inet.ip.dummynet.expire=0' and also you could poll the stats more frequently (say every 1-2-5 sec) to get a better idea of what happens. The one time you sent the 'pipe show' info there were clearly a few pipes with thousand packet drops -- as i said those are unavoidable and correspond to clients that systematically exceed their share (500k/1m as you set) e.g. because they are flooding the net with TCP SYN or UDP requests. This may be due to viruses, aggressive p2p, and so on. A single client can easily generate the extra 2000 packets per seconds that you are seeing. It's up to you to open your eyes looking for evidence, or close them and randomly blame one or another piece of the system. cheers luigi > since boottime! From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 13:15:07 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C52B1065672 for ; Tue, 6 Oct 2009 13:15:07 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id E7E538FC21 for ; Tue, 6 Oct 2009 13:15:06 +0000 (UTC) Received: from [217.25.27.27] (port=16015 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1Mv9sd-0002t2-00; Tue, 06 Oct 2009 17:15:03 +0400 Message-ID: <4ACB42D2.2070909@mail.ru> Date: Tue, 06 Oct 2009 18:14:58 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC9CFF7.3090208@mail.ru> <20091005110726.GA62598@onelab2.iet.unipi.it> <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> In-Reply-To: <20091006100726.GA26426@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 13:15:07 -0000 Eugene Grosbein wrote: > On Tue, Oct 06, 2009 at 02:21:38PM +0500, rihad wrote: > >> Is there some limit on the number of IP addresses in an ipfw table? > > No, generally handles much more. Please show your ipfw rule(s) > containing 'tablearg'. > 01031 x x allow ip from any to any 01040 x x skipto 1100 ip from table(127) to any out recv bce0 xmit bce1 01060 x x pipe tablearg ip from any to table(0) out recv bce0 xmit bce1 01070 x x allow ip from any to table(0) out recv bce0 xmit bce1 01100 x x pipe tablearg ip from any to table(2) out 65535 x x allow ip from any to any table(127) contains country-wide ISPs' netblocks (under 100 entries). table(0) and table(2) contain same user IP addresses, but different pipe IDs - normally around 3-4k entries each. Now please pay special attention to rule 1031. I've added it to bypass dummynet and stop packets from being dropped for now. Normally the rule isn't there. As I found out today after rebooting, drops only start occurring when the number of entries in table(0) exceeds 2000 or so (please see my previous email). Maybe it's a coincidence - I don't know. Global traffic load doesn't matter - it was approximately the same before and after the drops (around 450 mbit/s). From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 13:33:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E5E21065670 for ; Tue, 6 Oct 2009 13:33:26 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx30.mail.ru (mx30.mail.ru [94.100.176.44]) by mx1.freebsd.org (Postfix) with ESMTP id 473818FC12 for ; Tue, 6 Oct 2009 13:33:26 +0000 (UTC) Received: from [217.25.27.27] (port=1178 helo=[217.25.27.27]) by mx30.mail.ru with asmtp id 1MvAAO-000Cub-00; Tue, 06 Oct 2009 17:33:24 +0400 Message-ID: <4ACB4720.40004@mail.ru> Date: Tue, 06 Oct 2009 18:33:20 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Luigi Rizzo References: <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006093408.GA86830@onelab2.iet.unipi.it> <4ACB0F28.3000906@mail.ru> <20091006101747.GA87655@onelab2.iet.unipi.it> In-Reply-To: <20091006101747.GA87655@onelab2.iet.unipi.it> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 13:33:26 -0000 Luigi Rizzo wrote: > On Tue, Oct 06, 2009 at 02:34:32PM +0500, rihad wrote: >> Luigi Rizzo wrote: >> 8664 output packets dropped due to no bufs, etc. >> net.inet.ip.dummynet.io_pkt_drop: 111 > > io_pkt_drop only reports packets dropped to errors (missing pipes, > randomly forced packet drops which you don't use, no buffers and so on). > > The packets intentionally dropped in dummynet because queues are full > are listed by 'ipfw pipe show'. > > Even if pipes expire, there is a difference between having partial > information and completely ignoring what is available and claiming > "it's plain useless". Ok, without looking I can say: there _are_ always some users overrunning their queues that have non-zero Drp column in ipfw pipe show. But they are also there downloading like crazy when the number of online users is below 2000 or so, so how come that they're overrunning their share without getting reflected in the global netstat -s drop counter? There are 0 drops! netstat -s only starts growing when the number of online users exceeds 2000. > > BTW at least while you try to debug the problem you can temporarily > disable the pipe expire with 'sysctl net.inet.ip.dummynet.expire=0' > and also you could poll the stats more frequently (say every 1-2-5 sec) > to get a better idea of what happens. > I've tried expire=0 for a while, too. No difference whatsoever. > The one time you sent the 'pipe show' info there were clearly > a few pipes with thousand packet drops -- as i said those are > unavoidable and correspond to clients that systematically > exceed their share (500k/1m as you set) e.g. because they are flooding > the net with TCP SYN or UDP requests. This may be due to viruses, > aggressive p2p, and so on. A single client can easily generate > the extra 2000 packets per seconds that you are seeing. > Only downloads (i.e. traffic arriving from the Net) crosses the PC at hand. What makes me think this is a global problem, is that the drops _never_ happen when the number of online users in IPFW tables is below 2000 or so, no matter how high global traffic load is. Please see my previous email with ipfw rules. From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 14:22:05 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E44BC106566B for ; Tue, 6 Oct 2009 14:22:05 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 306158FC16 for ; Tue, 6 Oct 2009 14:22:04 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n96ELrM4043446; Tue, 6 Oct 2009 22:21:53 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n96ELqvA043445; Tue, 6 Oct 2009 22:21:52 +0800 (KRAST) (envelope-from eugen) Date: Tue, 6 Oct 2009 22:21:52 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091006142152.GA42350@svzserv.kemerovo.su> References: <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACB42D2.2070909@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 14:22:06 -0000 On Tue, Oct 06, 2009 at 06:14:58PM +0500, rihad wrote: > >No, generally handles much more. Please show your ipfw rule(s) > >containing 'tablearg'. > > 01031 x x allow ip from any to any > 01040 x x skipto 1100 ip from table(127) to any out > recv bce0 xmit bce1 > 01060 x x pipe tablearg ip from any to table(0) out > recv bce0 xmit bce1 > 01070 x x allow ip from any to table(0) out recv bce0 > xmit bce1 > 01100 x x pipe tablearg ip from any to table(2) out > 65535 x x allow ip from any to any > > table(127) contains country-wide ISPs' netblocks (under 100 entries). > table(0) and table(2) contain same user IP addresses, but different pipe > IDs - normally around 3-4k entries each. > > Now please pay special attention to rule 1031. I've added it to bypass > dummynet and stop packets from being dropped for now. Normally the rule > isn't there. > > As I found out today after rebooting, drops only start occurring when > the number of entries in table(0) exceeds 2000 or so (please see my > previous email). Maybe it's a coincidence - I don't know. Global traffic > load doesn't matter - it was approximately the same before and after the > drops (around 450 mbit/s). It's possible that pipe lookup by its number is inefficient and firewall keeps its lock for too much time while searching the pipe, just a guess. And packets start to drop, eh? Try setting net.isr.direct to 0 and make large net.inet.ip.intr_queue_maxlen. This way, one of your cores may run bce's thread, enqueue incoming packets and return to work immediately. The rest of processing may be performed by another kernel thread, hopefully using another core. Just to see if this changes anything. top -S should help here too. Eugene From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 15:28:42 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEBD81065672 for ; Tue, 6 Oct 2009 15:28:42 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id A6A9E8FC17 for ; Tue, 6 Oct 2009 15:28:42 +0000 (UTC) Received: from [217.25.27.27] (port=17745 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MvBxw-000HKm-00; Tue, 06 Oct 2009 19:28:40 +0400 Message-ID: <4ACB6223.1000709@mail.ru> Date: Tue, 06 Oct 2009 20:28:35 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> In-Reply-To: <20091006142152.GA42350@svzserv.kemerovo.su> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 15:28:43 -0000 Eugene Grosbein wrote: > On Tue, Oct 06, 2009 at 06:14:58PM +0500, rihad wrote: > >>> No, generally handles much more. Please show your ipfw rule(s) >>> containing 'tablearg'. >> 01031 x x allow ip from any to any >> 01040 x x skipto 1100 ip from table(127) to any out >> recv bce0 xmit bce1 >> 01060 x x pipe tablearg ip from any to table(0) out >> recv bce0 xmit bce1 >> 01070 x x allow ip from any to table(0) out recv bce0 >> xmit bce1 >> 01100 x x pipe tablearg ip from any to table(2) out >> 65535 x x allow ip from any to any >> >> table(127) contains country-wide ISPs' netblocks (under 100 entries). >> table(0) and table(2) contain same user IP addresses, but different pipe >> IDs - normally around 3-4k entries each. >> >> Now please pay special attention to rule 1031. I've added it to bypass >> dummynet and stop packets from being dropped for now. Normally the rule >> isn't there. >> >> As I found out today after rebooting, drops only start occurring when >> the number of entries in table(0) exceeds 2000 or so (please see my >> previous email). Maybe it's a coincidence - I don't know. Global traffic >> load doesn't matter - it was approximately the same before and after the >> drops (around 450 mbit/s). > > It's possible that pipe lookup by its number is inefficient > and firewall keeps its lock for too much time while searching the pipe, > just a guess. And packets start to drop, eh? > > Try setting net.isr.direct to 0 and make large net.inet.ip.intr_queue_maxlen. > This way, one of your cores may run bce's thread, enqueue incoming > packets and return to work immediately. The rest of processing may be > performed by another kernel thread, hopefully using another core. > Just to see if this changes anything. top -S should help here too. > Since this is a remote production box, I'm really scared of toggling such on/off flags I've never used before, particularly under heavy traffic loads, they're way too eager to lock up the whole system. I might try this tomorrow morning, though, first on another less critical box. p.s.: I've just tested toggling the flag on my virtual machine, it went fine. I don't think net.inet.ip.intr_queue_maxlen is relevant to this problem, as net.inet.ip.intr_queue_drops is normally zero or very close to it at all times. From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 15:40:04 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CE51106566B for ; Tue, 6 Oct 2009 15:40:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 119218FC0A for ; Tue, 6 Oct 2009 15:40:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n96Fe3iI014872 for ; Tue, 6 Oct 2009 15:40:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n96Fe3x0014870; Tue, 6 Oct 2009 15:40:03 GMT (envelope-from gnats) Date: Tue, 6 Oct 2009 15:40:03 GMT Message-Id: <200910061540.n96Fe3x0014870@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Stef Walter Cc: Subject: Re: kern/137164: [netinet] [patch] assert panic imo_match_source() X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Stef Walter List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 15:40:04 -0000 The following reply was made to PR kern/137164; it has been noted by GNATS. From: Stef Walter To: bug-followup@FreeBSD.org, jhanna@pangolin-systems.com, Bruce Simpson Cc: Subject: Re: kern/137164: [netinet] [patch] assert panic imo_match_source() Date: Tue, 06 Oct 2009 10:37:31 -0500 This is a multi-part message in MIME format. --------------090401010308070005060707 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Thanks for working on getting all these multicast fixes in. Much appreciated! Just one more thing, previous to 7.x FreeBSD would return EADDRINUSE in the case of a double IP_ADD_MEMBERSHIP. Software like quagga uses this error code to detect this condition. As patched (and MFC'd in 7.x and 8.x) EINVAL is returned instead and this confuses such software. Currently the multicast code does not remove memberships from its internal structures when an interface goes down. It's hard for userland to reliably track the condition of a multicast membership that didn't go away due to an interface going down. So software like quagga uses EADDRINUSE to track the condition. Obviously, as you Bruce mentioned, an better solution needs to be found eventually WRT to 'dynamic' interfaces and the multicast code. But for now would the attached patch be acceptable? It would prevent regressions from FreeBSD 6.x. Thanks for considering, Stef --------------090401010308070005060707 Content-Type: text/x-diff; name="freebsd-mcast-eaddrinuse.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="freebsd-mcast-eaddrinuse.patch" --- sys/netinet/in_mcast.c.orig 2009-09-30 16:43:35.000000000 +0000 +++ sys/netinet/in_mcast.c 2009-09-30 17:04:59.000000000 +0000 @@ -2010,5 +2010,5 @@ * is atomic with allocation of a membership. */ - error = EINVAL; + error = EADDRINUSE; goto out_inp_locked; } --------------090401010308070005060707-- From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 16:12:52 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF11A106566B for ; Tue, 6 Oct 2009 16:12:52 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 184468FC18 for ; Tue, 6 Oct 2009 16:12:51 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n96GCeo7050048; Wed, 7 Oct 2009 00:12:40 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n96GCecU050047; Wed, 7 Oct 2009 00:12:40 +0800 (KRAST) (envelope-from eugen) Date: Wed, 7 Oct 2009 00:12:40 +0800 From: Eugene Grosbein To: rihad Message-ID: <20091006161240.GA49940@svzserv.kemerovo.su> References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACB6223.1000709@mail.ru> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 16:12:52 -0000 On Tue, Oct 06, 2009 at 08:28:35PM +0500, rihad wrote: > I don't think net.inet.ip.intr_queue_maxlen is relevant to this problem, > as net.inet.ip.intr_queue_drops is normally zero or very close to it at > all times. When net.isr.direct is 1, this queue is used very seldom. Would you change it to 0, it will be used extensively. Eugene From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 16:25:12 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B92D106566B for ; Tue, 6 Oct 2009 16:25:12 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id 1357F8FC1B for ; Tue, 6 Oct 2009 16:25:11 +0000 (UTC) Received: from [217.25.27.27] (port=23387 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1MvCqc-000Ppr-00; Tue, 06 Oct 2009 20:25:10 +0400 Message-ID: <4ACB6F60.7090407@mail.ru> Date: Tue, 06 Oct 2009 21:25:04 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Eugene Grosbein References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> In-Reply-To: <20091006161240.GA49940@svzserv.kemerovo.su> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 16:25:12 -0000 Eugene Grosbein wrote: > On Tue, Oct 06, 2009 at 08:28:35PM +0500, rihad wrote: > >> I don't think net.inet.ip.intr_queue_maxlen is relevant to this problem, >> as net.inet.ip.intr_queue_drops is normally zero or very close to it at >> all times. > > When net.isr.direct is 1, this queue is used very seldom. > Would you change it to 0, it will be used extensively. > Ah, ok, thanks. I'll try that tomorrow. But still... > Try setting net.isr.direct to 0 and make large net.inet.ip.intr_queue_maxlen. > This way, one of your cores may run bce's thread, enqueue incoming > packets and return to work immediately. The rest of processing may be > performed by another kernel thread, hopefully using another core. > Just to see if this changes anything. top -S should help here too. It isn't the incoming bce0 losing packets, but rather the outgoing bce1, which is almost idle interrupt-wise: 29 root 1 -68 - 0K 16K CPU1 1 223:39 55.86% irq256: bce0 31 root 1 -68 - 0K 16K WAIT 2 19:27 4.10% irq257: bce1 From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 16:50:04 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57F301065670 for ; Tue, 6 Oct 2009 16:50:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2D0948FC1C for ; Tue, 6 Oct 2009 16:50:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n96Go4MI074406 for ; Tue, 6 Oct 2009 16:50:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n96Go4lj074405; Tue, 6 Oct 2009 16:50:04 GMT (envelope-from gnats) Date: Tue, 6 Oct 2009 16:50:04 GMT Message-Id: <200910061650.n96Go4lj074405@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Stanislav Sedov Cc: Subject: Re: kern/127587: [bge] [request] if_bge(4) doesn't support BCM576X family X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Stanislav Sedov List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 16:50:04 -0000 The following reply was made to PR kern/127587; it has been noted by GNATS. From: Stanislav Sedov To: Stanislav Sedov Cc: Norikatsu Shigemura , matthieu , bug-followup@FreeBSD.org Subject: Re: kern/127587: [bge] [request] if_bge(4) doesn't support BCM576X family Date: Tue, 6 Oct 2009 20:46:13 +0400 --Signature=_Tue__6_Oct_2009_20_46_13_+0400_Jz_mOzdr9=3k/ma7 Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, 2 Oct 2009 15:28:55 +0400 Stanislav Sedov mentioned: > Hello, Norikatsu! >=20 > Can you check, please, if the patch matthieu proposed > works for you? >=20 Or, even better, can you, please, the following patch attached? It should add support for newest 576x models as well as for some other recent devices. Thanks! Index: sys/dev/bge/if_bgereg.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sys/dev/bge/if_bgereg.h (revision 197707) +++ sys/dev/bge/if_bgereg.h (working copy) @@ -218,6 +218,7 @@ #define BGE_PCI_UNDI_TX_BD_PRODIDX_LO 0xAC #define BGE_PCI_ISR_MBX_HI 0xB0 #define BGE_PCI_ISR_MBX_LO 0xB4 +#define BGE_PCI_PRODID_ASICREV 0xBC =20 /* PCI Misc. Host control register */ #define BGE_PCIMISCCTL_CLEAR_INTA 0x00000001 @@ -229,6 +230,7 @@ #define BGE_PCIMISCCTL_REG_WORDSWAP 0x00000040 #define BGE_PCIMISCCTL_INDIRECT_ACCESS 0x00000080 #define BGE_PCIMISCCTL_ASICREV 0xFFFF0000 +#define BGE_PCIMISCCTL_ASICREV_SHIFT 16 =20 #define BGE_HIF_SWAP_OPTIONS (BGE_PCIMISCCTL_ENDIAN_WORDSWAP) #if BYTE_ORDER =3D=3D LITTLE_ENDIAN @@ -245,66 +247,72 @@ (BGE_HIF_SWAP_OPTIONS|BGE_PCIMISCCTL_CLEAR_INTA| \ BGE_PCIMISCCTL_MASK_PCI_INTR|BGE_PCIMISCCTL_INDIRECT_ACCESS) =20 -#define BGE_CHIPID_TIGON_I 0x40000000 -#define BGE_CHIPID_TIGON_II 0x60000000 -#define BGE_CHIPID_BCM5700_A0 0x70000000 -#define BGE_CHIPID_BCM5700_A1 0x70010000 -#define BGE_CHIPID_BCM5700_B0 0x71000000 -#define BGE_CHIPID_BCM5700_B1 0x71010000 -#define BGE_CHIPID_BCM5700_B2 0x71020000 -#define BGE_CHIPID_BCM5700_B3 0x71030000 -#define BGE_CHIPID_BCM5700_ALTIMA 0x71040000 -#define BGE_CHIPID_BCM5700_C0 0x72000000 -#define BGE_CHIPID_BCM5701_A0 0x00000000 /* grrrr */ -#define BGE_CHIPID_BCM5701_B0 0x01000000 -#define BGE_CHIPID_BCM5701_B2 0x01020000 -#define BGE_CHIPID_BCM5701_B5 0x01050000 -#define BGE_CHIPID_BCM5703_A0 0x10000000 -#define BGE_CHIPID_BCM5703_A1 0x10010000 -#define BGE_CHIPID_BCM5703_A2 0x10020000 -#define BGE_CHIPID_BCM5703_A3 0x10030000 -#define BGE_CHIPID_BCM5703_B0 0x11000000 -#define BGE_CHIPID_BCM5704_A0 0x20000000 -#define BGE_CHIPID_BCM5704_A1 0x20010000 -#define BGE_CHIPID_BCM5704_A2 0x20020000 -#define BGE_CHIPID_BCM5704_A3 0x20030000 -#define BGE_CHIPID_BCM5704_B0 0x21000000 -#define BGE_CHIPID_BCM5705_A0 0x30000000 -#define BGE_CHIPID_BCM5705_A1 0x30010000 -#define BGE_CHIPID_BCM5705_A2 0x30020000 -#define BGE_CHIPID_BCM5705_A3 0x30030000 -#define BGE_CHIPID_BCM5750_A0 0x40000000 -#define BGE_CHIPID_BCM5750_A1 0x40010000 -#define BGE_CHIPID_BCM5750_A3 0x40030000 -#define BGE_CHIPID_BCM5750_B0 0x41000000 -#define BGE_CHIPID_BCM5750_B1 0x41010000 -#define BGE_CHIPID_BCM5750_C0 0x42000000 -#define BGE_CHIPID_BCM5750_C1 0x42010000 -#define BGE_CHIPID_BCM5750_C2 0x42020000 -#define BGE_CHIPID_BCM5714_A0 0x50000000 -#define BGE_CHIPID_BCM5752_A0 0x60000000 -#define BGE_CHIPID_BCM5752_A1 0x60010000 -#define BGE_CHIPID_BCM5752_A2 0x60020000 -#define BGE_CHIPID_BCM5714_B0 0x80000000 -#define BGE_CHIPID_BCM5714_B3 0x80030000 -#define BGE_CHIPID_BCM5715_A0 0x90000000 -#define BGE_CHIPID_BCM5715_A1 0x90010000 -#define BGE_CHIPID_BCM5715_A3 0x90030000 -#define BGE_CHIPID_BCM5755_A0 0xa0000000 -#define BGE_CHIPID_BCM5755_A1 0xa0010000 -#define BGE_CHIPID_BCM5755_A2 0xa0020000 -#define BGE_CHIPID_BCM5722_A0 0xa2000000 -#define BGE_CHIPID_BCM5754_A0 0xb0000000 -#define BGE_CHIPID_BCM5754_A1 0xb0010000 -#define BGE_CHIPID_BCM5754_A2 0xb0020000 -#define BGE_CHIPID_BCM5787_A0 0xb0000000 -#define BGE_CHIPID_BCM5787_A1 0xb0010000 -#define BGE_CHIPID_BCM5787_A2 0xb0020000 -#define BGE_CHIPID_BCM5906_A1 0xc0010000 -#define BGE_CHIPID_BCM5906_A2 0xc0020000 +#define BGE_CHIPID_TIGON_I 0x4000 +#define BGE_CHIPID_TIGON_II 0x6000 +#define BGE_CHIPID_BCM5700_A0 0x7000 +#define BGE_CHIPID_BCM5700_A1 0x7001 +#define BGE_CHIPID_BCM5700_B0 0x7100 +#define BGE_CHIPID_BCM5700_B1 0x7101 +#define BGE_CHIPID_BCM5700_B2 0x7102 +#define BGE_CHIPID_BCM5700_B3 0x7103 +#define BGE_CHIPID_BCM5700_ALTIMA 0x7104 +#define BGE_CHIPID_BCM5700_C0 0x7200 +#define BGE_CHIPID_BCM5701_A0 0x0000 /* grrrr */ +#define BGE_CHIPID_BCM5701_B0 0x0100 +#define BGE_CHIPID_BCM5701_B2 0x0102 +#define BGE_CHIPID_BCM5701_B5 0x0105 +#define BGE_CHIPID_BCM5703_A0 0x1000 +#define BGE_CHIPID_BCM5703_A1 0x1001 +#define BGE_CHIPID_BCM5703_A2 0x1002 +#define BGE_CHIPID_BCM5703_A3 0x1003 +#define BGE_CHIPID_BCM5703_B0 0x1100 +#define BGE_CHIPID_BCM5704_A0 0x2000 +#define BGE_CHIPID_BCM5704_A1 0x2001 +#define BGE_CHIPID_BCM5704_A2 0x2002 +#define BGE_CHIPID_BCM5704_A3 0x2003 +#define BGE_CHIPID_BCM5704_B0 0x2100 +#define BGE_CHIPID_BCM5705_A0 0x3000 +#define BGE_CHIPID_BCM5705_A1 0x3001 +#define BGE_CHIPID_BCM5705_A2 0x3002 +#define BGE_CHIPID_BCM5705_A3 0x3003 +#define BGE_CHIPID_BCM5750_A0 0x4000 +#define BGE_CHIPID_BCM5750_A1 0x4001 +#define BGE_CHIPID_BCM5750_A3 0x4000 +#define BGE_CHIPID_BCM5750_B0 0x4100 +#define BGE_CHIPID_BCM5750_B1 0x4101 +#define BGE_CHIPID_BCM5750_C0 0x4200 +#define BGE_CHIPID_BCM5750_C1 0x4201 +#define BGE_CHIPID_BCM5750_C2 0x4202 +#define BGE_CHIPID_BCM5714_A0 0x5000 +#define BGE_CHIPID_BCM5752_A0 0x6000 +#define BGE_CHIPID_BCM5752_A1 0x6001 +#define BGE_CHIPID_BCM5752_A2 0x6002 +#define BGE_CHIPID_BCM5714_B0 0x8000 +#define BGE_CHIPID_BCM5714_B3 0x8003 +#define BGE_CHIPID_BCM5715_A0 0x9000 +#define BGE_CHIPID_BCM5715_A1 0x9001 +#define BGE_CHIPID_BCM5715_A3 0x9003 +#define BGE_CHIPID_BCM5755_A0 0xa000 +#define BGE_CHIPID_BCM5755_A1 0xa001 +#define BGE_CHIPID_BCM5755_A2 0xa002 +#define BGE_CHIPID_BCM5722_A0 0xa200 +#define BGE_CHIPID_BCM5754_A0 0xb000 +#define BGE_CHIPID_BCM5754_A1 0xb001 +#define BGE_CHIPID_BCM5754_A2 0xb002 +#define BGE_CHIPID_BCM5761_A0 0x5761000 +#define BGE_CHIPID_BCM5761_A1 0x5761100 +#define BGE_CHIPID_BCM5784_A0 0x5784000 +#define BGE_CHIPID_BCM5784_A1 0x5784100 +#define BGE_CHIPID_BCM5787_A0 0xb000 +#define BGE_CHIPID_BCM5787_A1 0xb001 +#define BGE_CHIPID_BCM5787_A2 0xb002 +#define BGE_CHIPID_BCM5906_A1 0xc001 +#define BGE_CHIPID_BCM5906_A2 0xc002 +#define BGE_CHIPID_BCM57780_A0 0x57780000 +#define BGE_CHIPID_BCM57780_A1 0x57780001 =20 /* shorthand one */ -#define BGE_ASICREV(x) ((x) >> 28) +#define BGE_ASICREV(x) ((x) >> 12) #define BGE_ASICREV_BCM5701 0x00 #define BGE_ASICREV_BCM5703 0x01 #define BGE_ASICREV_BCM5704 0x02 @@ -319,9 +327,16 @@ #define BGE_ASICREV_BCM5754 0x0b #define BGE_ASICREV_BCM5787 0x0b #define BGE_ASICREV_BCM5906 0x0c +/* Should consult BGE_PCI_PRODID_ASICREV for ChipID */ +#define BGE_ASICREV_USE_PRODID_REG 0x0f +/* BGE_PCI_PRODID_ASICREV ASIC rev. identifiers. */ +#define BGE_ASICREV_BCM5761 0x5761 +#define BGE_ASICREV_BCM5784 0x5784 +#define BGE_ASICREV_BCM5785 0x5785 +#define BGE_ASICREV_BCM57780 0x57780 =20 /* chip revisions */ -#define BGE_CHIPREV(x) ((x) >> 24) +#define BGE_CHIPREV(x) ((x) >> 8) #define BGE_CHIPREV_5700_AX 0x70 #define BGE_CHIPREV_5700_BX 0x71 #define BGE_CHIPREV_5700_CX 0x72 @@ -331,6 +346,9 @@ #define BGE_CHIPREV_5704_BX 0x21 #define BGE_CHIPREV_5750_AX 0x40 #define BGE_CHIPREV_5750_BX 0x41 +/* BGE_PCI_PRODID_ASICREV chip rev. identifiers. */ +#define BGE_CHIPREV_5761_AX 0x57611 +#define BGE_CHIPREV_5784_AX 0x57841 =20 /* PCI DMA Read/Write Control register */ #define BGE_PCIDMARWCTL_MINDMA 0x000000FF @@ -861,6 +879,7 @@ #define BGE_SDCMODE_RESET 0x00000001 #define BGE_SDCMODE_ENABLE 0x00000002 #define BGE_SDCMODE_ATTN 0x00000004 +#define BGE_SDCMODE_CDELAY 0x00000010 =20 /* Send Data completion status register */ #define BGE_SDCSTAT_ATTN 0x00000004 @@ -1378,6 +1397,9 @@ #define BGE_RDMAMODE_PCI_FIFOOREAD_ATTN 0x00000100 #define BGE_RDMAMODE_LOCWRITE_TOOBIG 0x00000200 #define BGE_RDMAMODE_ALL_ATTNS 0x000003FC +#define BGE_RDMAMODE_BD_SBD_CRPT_ATTN 0x00000800 +#define BGE_RDMAMODE_MBUF_RBD_CRPT_ATTN 0x00001000 +#define BGE_RDMAMODE_MBUF_SBD_CRPT_ATTN 0x00002000 #define BGE_RDMAMODE_FIFO_SIZE_128 0x00020000 #define BGE_RDMAMODE_FIFO_LONG_BURST 0x00030000 =20 @@ -2101,6 +2123,7 @@ #define BCOM_DEVICEID_BCM5720 0x1658 #define BCOM_DEVICEID_BCM5721 0x1659 #define BCOM_DEVICEID_BCM5722 0x165A +#define BCOM_DEVICEID_BCM5723 0x165B #define BCOM_DEVICEID_BCM5750 0x1676 #define BCOM_DEVICEID_BCM5750M 0x167C #define BCOM_DEVICEID_BCM5751 0x1677 @@ -2115,13 +2138,22 @@ #define BCOM_DEVICEID_BCM5754M 0x1672 #define BCOM_DEVICEID_BCM5755 0x167B #define BCOM_DEVICEID_BCM5755M 0x1673 +#define BCOM_DEVICEID_BCM5761 0x1681 +#define BCOM_DEVICEID_BCM5761E 0x1680 +#define BCOM_DEVICEID_BCM5761S 0x1688 +#define BCOM_DEVICEID_BCM5761SE 0x1689 +#define BCOM_DEVICEID_BCM5764 0x1684 #define BCOM_DEVICEID_BCM5780 0x166A #define BCOM_DEVICEID_BCM5780S 0x166B #define BCOM_DEVICEID_BCM5781 0x16DD #define BCOM_DEVICEID_BCM5782 0x1696 +#define BCOM_DEVICEID_BCM5784 0x1698 +#define BCOM_DEVICEID_BCM5785F 0x16a0 +#define BCOM_DEVICEID_BCM5785G 0x1699 #define BCOM_DEVICEID_BCM5786 0x169A #define BCOM_DEVICEID_BCM5787 0x169B #define BCOM_DEVICEID_BCM5787M 0x1693 +#define BCOM_DEVICEID_BCM5787F 0x167f #define BCOM_DEVICEID_BCM5788 0x169C #define BCOM_DEVICEID_BCM5789 0x169D #define BCOM_DEVICEID_BCM5901 0x170D @@ -2129,6 +2161,10 @@ #define BCOM_DEVICEID_BCM5903M 0x16FF #define BCOM_DEVICEID_BCM5906 0x1712 #define BCOM_DEVICEID_BCM5906M 0x1713 +#define BCOM_DEVICEID_BCM57760 0x1690 +#define BCOM_DEVICEID_BCM57780 0x1692 +#define BCOM_DEVICEID_BCM57788 0x1691 +#define BCOM_DEVICEID_BCM57790 0x1694 =20 /* * Alteon AceNIC PCI vendor/device ID. @@ -2179,6 +2215,14 @@ #define SUN_VENDORID 0x108e =20 /* + * Fujitsu vendor/device IDs + */ +#define FJTSU_VENDORID 0x10cf +#define FJTSU_DEVICEID_PW008GE5 0x11a1 +#define FJTSU_DEVICEID_PW008GE4 0x11a2 +#define FJTSU_DEVICEID_PP250450 0x11cc /* PRIMEPOWER250/450 LAN */ + +/* * Offset of MAC address inside EEPROM. */ #define BGE_EE_MAC_OFFSET 0x7C @@ -2558,6 +2602,7 @@ #define BGE_FLAG_5705_PLUS 0x00002000 #define BGE_FLAG_5714_FAMILY 0x00004000 #define BGE_FLAG_575X_PLUS 0x00008000 +#define BGE_FLAG_5755_PLUS 0x00010000 #define BGE_FLAG_RX_ALIGNBUG 0x00100000 #define BGE_FLAG_NO_3LED 0x00200000 #define BGE_FLAG_ADC_BUG 0x00400000 @@ -2568,8 +2613,8 @@ #define BGE_FLAG_CRC_BUG 0x08000000 #define BGE_FLAG_5788 0x20000000 uint32_t bge_chipid; - uint8_t bge_asicrev; - uint8_t bge_chiprev; + uint32_t bge_asicrev; + uint32_t bge_chiprev; uint8_t bge_asf_mode; uint8_t bge_asf_count; struct bge_ring_data bge_ldata; /* rings */ Index: sys/dev/bge/if_bge.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sys/dev/bge/if_bge.c (revision 197707) +++ sys/dev/bge/if_bge.c (working copy) @@ -170,6 +170,7 @@ { BCOM_VENDORID, BCOM_DEVICEID_BCM5720 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5721 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5722 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5723 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5750 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5750M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5751 }, @@ -184,12 +185,21 @@ { BCOM_VENDORID, BCOM_DEVICEID_BCM5754M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5755 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5755M }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761E }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761S }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761SE }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5764 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5780 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5780S }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5781 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5782 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5784 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5785F }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5785G }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5786 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5787 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5787F }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5787M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5788 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5789 }, @@ -198,11 +208,19 @@ { BCOM_VENDORID, BCOM_DEVICEID_BCM5903M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5906 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5906M }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57760 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57780 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57788 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57790 }, =20 { SK_VENDORID, SK_DEVICEID_ALTIMA }, =20 { TC_VENDORID, TC_DEVICEID_3C996 }, =20 + { FJTSU_VENDORID, FJTSU_DEVICEID_PW008GE4 }, + { FJTSU_VENDORID, FJTSU_DEVICEID_PW008GE5 }, + { FJTSU_VENDORID, FJTSU_DEVICEID_PP250450 }, + { 0, 0 } }; =20 @@ -216,6 +234,7 @@ { BCOM_VENDORID, "Broadcom" }, { SK_VENDORID, "SysKonnect" }, { TC_VENDORID, "3Com" }, + { FJTSU_VENDORID, "Fujitsu" }, =20 { 0, NULL } }; @@ -271,12 +290,18 @@ { BGE_CHIPID_BCM5755_A1, "BCM5755 A1" }, { BGE_CHIPID_BCM5755_A2, "BCM5755 A2" }, { BGE_CHIPID_BCM5722_A0, "BCM5722 A0" }, + { BGE_CHIPID_BCM5761_A0, "BCM5761 A0" }, + { BGE_CHIPID_BCM5761_A1, "BCM5761 A1" }, + { BGE_CHIPID_BCM5784_A0, "BCM5784 A0" }, + { BGE_CHIPID_BCM5784_A1, "BCM5784 A1" }, /* 5754 and 5787 share the same ASIC ID */ { BGE_CHIPID_BCM5787_A0, "BCM5754/5787 A0" },=20 { BGE_CHIPID_BCM5787_A1, "BCM5754/5787 A1" }, { BGE_CHIPID_BCM5787_A2, "BCM5754/5787 A2" }, { BGE_CHIPID_BCM5906_A1, "BCM5906 A1" }, { BGE_CHIPID_BCM5906_A2, "BCM5906 A2" }, + { BGE_CHIPID_BCM57780_A0, "BCM57780 A0" }, + { BGE_CHIPID_BCM57780_A1, "BCM57780 A1" }, =20 { 0, NULL } }; @@ -297,9 +322,13 @@ { BGE_ASICREV_BCM5780, "unknown BCM5780" }, { BGE_ASICREV_BCM5714, "unknown BCM5714" }, { BGE_ASICREV_BCM5755, "unknown BCM5755" }, + { BGE_ASICREV_BCM5761, "unknown BCM5761" }, + { BGE_ASICREV_BCM5784, "unknown BCM5784" }, + { BGE_ASICREV_BCM5785, "unknown BCM5785" }, /* 5754 and 5787 share the same ASIC ID */ { BGE_ASICREV_BCM5787, "unknown BCM5754/5787" }, { BGE_ASICREV_BCM5906, "unknown BCM5906" }, + { BGE_ASICREV_BCM57780, "unknown BCM57780" }, =20 { 0, NULL } }; @@ -309,6 +338,7 @@ #define BGE_IS_5705_PLUS(sc) ((sc)->bge_flags & BGE_FLAG_5705_PLUS) #define BGE_IS_5714_FAMILY(sc) ((sc)->bge_flags & BGE_FLAG_5714_FAMILY) #define BGE_IS_575X_PLUS(sc) ((sc)->bge_flags & BGE_FLAG_575X_PLUS) +#define BGE_IS_5755_PLUS(sc) ((sc)->bge_flags & BGE_FLAG_5755_PLUS) =20 const struct bge_revision * bge_lookup_rev(uint32_t); const struct bge_vendor * bge_lookup_vendor(uint16_t); @@ -1758,8 +1788,7 @@ val =3D BGE_WDMAMODE_ENABLE | BGE_WDMAMODE_ALL_ATTNS; =20 /* Enable host coalescing bug fix. */ - if (sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5755 || - sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5787) + if (BGE_IS_5755_PLUS(sc)) val |=3D 1 << 29; =20 /* Turn on write DMA state machine */ @@ -1768,6 +1797,12 @@ =20 /* Turn on read DMA state machine */ val =3D BGE_RDMAMODE_ENABLE | BGE_RDMAMODE_ALL_ATTNS; + if (sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5784 || + sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5785 || + sc->bge_asicrev =3D=3D BGE_ASICREV_BCM57780) + val |=3D BGE_RDMAMODE_BD_SBD_CRPT_ATTN | + BGE_RDMAMODE_MBUF_RBD_CRPT_ATTN | + BGE_RDMAMODE_MBUF_SBD_CRPT_ATTN; if (sc->bge_flags & BGE_FLAG_PCIE) val |=3D BGE_RDMAMODE_FIFO_LONG_BURST; CSR_WRITE_4(sc, BGE_RDMA_MODE, val); @@ -1790,7 +1825,10 @@ CSR_WRITE_4(sc, BGE_SBDC_MODE, BGE_SBDCMODE_ENABLE); =20 /* Turn on send data completion state machine */ - CSR_WRITE_4(sc, BGE_SDC_MODE, BGE_SDCMODE_ENABLE); + val =3D BGE_SDCMODE_ENABLE; + if (sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5761) + val |=3D BGE_SDCMODE_CDELAY; + CSR_WRITE_4(sc, BGE_SDC_MODE, val); =20 /* Turn on send data initiator state machine */ CSR_WRITE_4(sc, BGE_SDI_MODE, BGE_SDIMODE_ENABLE); @@ -1897,8 +1935,11 @@ const struct bge_vendor *v; uint32_t id; =20 - id =3D pci_read_config(dev, BGE_PCI_MISC_CTL, 4) & - BGE_PCIMISCCTL_ASICREV; + id =3D pci_read_config(dev, BGE_PCI_MISC_CTL, 4) >> + BGE_PCIMISCCTL_ASICREV_SHIFT; + if (BGE_ASICREV(id) =3D=3D BGE_ASICREV_USE_PRODID_REG) + id =3D pci_read_config(dev, + BGE_PCI_PRODID_ASICREV, 4); br =3D bge_lookup_rev(id); v =3D bge_lookup_vendor(vid); { @@ -1915,8 +1956,8 @@ br !=3D NULL ? br->br_name : "NetXtreme Ethernet Controller"); } - snprintf(buf, 96, "%s, %sASIC rev. %#04x", model, - br !=3D NULL ? "" : "unknown ", id >> 16); + snprintf(buf, 96, "%s, %sASIC rev. %#08x", model, + br !=3D NULL ? "" : "unknown ", id); device_set_desc_copy(dev, buf); if (pci_get_subvendor(dev) =3D=3D DELL_VENDORID) sc->bge_flags |=3D BGE_FLAG_NO_3LED; @@ -2411,8 +2452,11 @@ =20 /* Save various chip information. */ sc->bge_chipid =3D - pci_read_config(dev, BGE_PCI_MISC_CTL, 4) & - BGE_PCIMISCCTL_ASICREV; + pci_read_config(dev, BGE_PCI_MISC_CTL, 4) >> + BGE_PCIMISCCTL_ASICREV_SHIFT; + if (BGE_ASICREV(sc->bge_chipid) =3D=3D BGE_ASICREV_USE_PRODID_REG) + sc->bge_chipid =3D pci_read_config(dev, BGE_PCI_PRODID_ASICREV, + 4); sc->bge_asicrev =3D BGE_ASICREV(sc->bge_chipid); sc->bge_chiprev =3D BGE_CHIPREV(sc->bge_chipid); =20 @@ -2431,6 +2475,15 @@ =20 /* Save chipset family. */ switch (sc->bge_asicrev) { + case BGE_ASICREV_BCM5755: + case BGE_ASICREV_BCM5761: + case BGE_ASICREV_BCM5784: + case BGE_ASICREV_BCM5785: + case BGE_ASICREV_BCM5787: + case BGE_ASICREV_BCM57780: + sc->bge_flags |=3D BGE_FLAG_5755_PLUS | BGE_FLAG_575X_PLUS | + BGE_FLAG_5705_PLUS; + break; case BGE_ASICREV_BCM5700: case BGE_ASICREV_BCM5701: case BGE_ASICREV_BCM5703: @@ -2440,12 +2493,11 @@ case BGE_ASICREV_BCM5714_A0: case BGE_ASICREV_BCM5780: case BGE_ASICREV_BCM5714: - sc->bge_flags |=3D BGE_FLAG_5714_FAMILY /* | BGE_FLAG_JUMBO */; - /* FALLTHROUGH */ + sc->bge_flags |=3D BGE_FLAG_5714_FAMILY | BGE_FLAG_575X_PLUS | + BGE_FLAG_5705_PLUS /* | BGE_FLAG_JUMBO */; + break; case BGE_ASICREV_BCM5750: case BGE_ASICREV_BCM5752: - case BGE_ASICREV_BCM5755: - case BGE_ASICREV_BCM5787: case BGE_ASICREV_BCM5906: sc->bge_flags |=3D BGE_FLAG_575X_PLUS; /* FALLTHROUGH */ @@ -2466,6 +2518,8 @@ if (BGE_IS_5705_PLUS(sc) && !(sc->bge_flags & BGE_FLAG_ADJUST_TRIM)) { if (sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5755 || + sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5761 || + sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5784 || sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5787) { if (sc->bge_chipid !=3D BGE_CHIPID_BCM5722_A0) sc->bge_flags |=3D BGE_FLAG_JITTER_BUG; @@ -2873,8 +2927,7 @@ =20 /* Disable fastboot on controllers that support it. */ if (sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5752 || - sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5755 || - sc->bge_asicrev =3D=3D BGE_ASICREV_BCM5787) { + BGE_IS_5755_PLUS(sc)) { if (bootverbose) device_printf(sc->bge_dev, "Disabling fastboot\n"); CSR_WRITE_4(sc, BGE_FASTBOOT_PC, 0x0); @@ -4690,6 +4743,8 @@ =20 printf("Hardware Flags:\n"); if (BGE_IS_575X_PLUS(sc)) + printf(" - 5755 Plus\n"); + if (BGE_IS_575X_PLUS(sc)) printf(" - 575X Plus\n"); if (BGE_IS_5705_PLUS(sc)) printf(" - 5705 Plus\n"); --=20 Stanislav Sedov ST4096-RIPE --Signature=_Tue__6_Oct_2009_20_46_13_+0400_Jz_mOzdr9=3k/ma7 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJKy3RaAAoJEKN82nOYvCd0VlkQAIyN+KFrIrkFNdr/V7LKLPkv n7dJhjdffXSbAV/02hDmCUkyJJFc4J0xVy045Xh4mxgkl78pwp7SyL8pR8/5EFz5 lAqte2tQj13Cum1uEsWJ9h/ylm9oKrzfS8i+/nJATJ0HIumAb5oWBRNDeWgD2rI2 WT7PE/zf7tbgOHQbrYaghBqZsvyLloY31lGvfMmCW48qd3GRoe6pGX1rqNMBxJcz Ck6i1jAWra2BHqF3MZhnUMkWsUXXNMf3M+89cgcaMov44o36HkYiN8JGtrQUjfk6 NElh2SMfoOptap6/juu5ZqaXPHG0WB/UDDPYUIhR7cNMR6jUCh4XnZ7hM2RtEpDI 79aNoOlatpTP5/6XrCXbmP54adCH/fPJARNX4B/s344hOAYqrYUNNMGmWxWHTlHk wIBGi1SHWJV5nYsFBELGuR5qvqJsN0/idYPEPsfPqSmRqoXOgutLuKuyeC1e8j+g G6K1IPmPiRNFrP6RdEf5I8OGL3mDpU6jUEfdB2+Vym/Z6DR+2N17xxIOseKcSHiN /QiiGXCV6eX9zSFgGnfNQTKTLVum+P2QhBmoHzqDy9+5Jkoa4SsR6vwLoB9bBsEh m/NDA8a0ETavI35hXz9TG0LVm6Wno+CEju/09G5Arfogu5913pdZsTfg/vl7ANEq +eYk8qqEWyilDj1jyVjv =ZduT -----END PGP SIGNATURE----- --Signature=_Tue__6_Oct_2009_20_46_13_+0400_Jz_mOzdr9=3k/ma7-- From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 17:06:38 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8207106568D for ; Tue, 6 Oct 2009 17:06:38 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 7D1868FC13 for ; Tue, 6 Oct 2009 17:06:38 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id F300146B0C; Tue, 6 Oct 2009 13:06:37 -0400 (EDT) Date: Tue, 6 Oct 2009 18:06:37 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Eugene Grosbein In-Reply-To: <20091006161240.GA49940@svzserv.kemerovo.su> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: rihad , Luigi Rizzo , Julian Elischer , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 17:06:38 -0000 On Wed, 7 Oct 2009, Eugene Grosbein wrote: > On Tue, Oct 06, 2009 at 08:28:35PM +0500, rihad wrote: > >> I don't think net.inet.ip.intr_queue_maxlen is relevant to this problem, as >> net.inet.ip.intr_queue_drops is normally zero or very close to it at all >> times. > > When net.isr.direct is 1, this queue is used very seldom. Would you change > it to 0, it will be used extensively. Just to clarify this more specifically: With net.isr.direct set to 0, the netisr will always be used when processing inbound IP packets. With net.isr.direct set to 1, the netisr will only be used for special cases, such as loopback traffic, IPSEC decapsulation, and other processing types where there's a risk of recursive processing. In the default 8.0 configuration, we use one netisr thread; however, you can specify to use multiple threads at boot time. This is not the default currently because we're still researching load distribution schemes, and on current high-performance systems the hardware tends to take care of that already pretty well (i.e., most modern 10gbps cards). Also, ipfw/dummynet have fairly non-granular locking, so adding parallelism won't necessarily help currently. Robert From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 17:20:05 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0944F1065676 for ; Tue, 6 Oct 2009 17:20:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D34D28FC0A for ; Tue, 6 Oct 2009 17:20:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n96HK4Zf099716 for ; Tue, 6 Oct 2009 17:20:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n96HK44A099715; Tue, 6 Oct 2009 17:20:04 GMT (envelope-from gnats) Date: Tue, 6 Oct 2009 17:20:04 GMT Message-Id: <200910061720.n96HK44A099715@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Bruce Simpson Cc: Subject: Re: kern/137164: [netinet] [patch] assert panic imo_match_source() X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Bruce Simpson List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 17:20:05 -0000 The following reply was made to PR kern/137164; it has been noted by GNATS. From: Bruce Simpson To: Stef Walter Cc: bug-followup@FreeBSD.org, jhanna@pangolin-systems.com Subject: Re: kern/137164: [netinet] [patch] assert panic imo_match_source() Date: Tue, 06 Oct 2009 17:56:54 +0100 Hi Stef, I am about to leave on holiday for two weeks, so I probably won't have time to check this change in. I'm kind of in a last minute rush right now, so it might be better to ping syrinx@ about this change (I don't know if she is still under mentorship or not). If I can get to it during the break (no promises), so far so good. cheers, BMS From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 17:21:27 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 672CC1065693 for ; Tue, 6 Oct 2009 17:21:27 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx33.mail.ru (mx33.mail.ru [94.100.176.47]) by mx1.freebsd.org (Postfix) with ESMTP id DFB8C8FC21 for ; Tue, 6 Oct 2009 17:21:26 +0000 (UTC) Received: from [217.25.27.27] (port=34657 helo=[217.25.27.27]) by mx33.mail.ru with asmtp id 1MvDj3-0006Hj-00; Tue, 06 Oct 2009 21:21:25 +0400 Message-ID: <4ACB7C8E.6090905@mail.ru> Date: Tue, 06 Oct 2009 22:21:18 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 17:21:27 -0000 Robert Watson wrote: > and on current high-performance systems the hardware tends to > take care of that already pretty well (i.e., most modern 10gbps cards). > Do you think that us switching to 10gbps cards would solve the problem discussed? We're currently at 500-550 mbps and rising, so we might as well switch to it now without waiting to hit the ceiling. From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 17:48:57 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91F5D1065676 for ; Tue, 6 Oct 2009 17:48:57 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4B25F8FC12 for ; Tue, 6 Oct 2009 17:48:56 +0000 (UTC) Received: by lath.rinet.ru (Postfix, from userid 222) id 79689704F; Tue, 6 Oct 2009 21:30:39 +0400 (MSD) Date: Tue, 6 Oct 2009 21:30:39 +0400 From: Oleg Bulyzhin To: Luigi Rizzo Message-ID: <20091006173039.GA66473@lath.rinet.ru> References: <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006093408.GA86830@onelab2.iet.unipi.it> <4ACB0F28.3000906@mail.ru> <20091006101747.GA87655@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091006101747.GA87655@onelab2.iet.unipi.it> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: rihad , Julian Elischer , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 17:48:57 -0000 On Tue, Oct 06, 2009 at 12:17:47PM +0200, Luigi Rizzo wrote: > io_pkt_drop only reports packets dropped to errors (missing pipes, > randomly forced packet drops which you don't use, no buffers and so on). You are mistaken here. io_pkt_drop is total number of packets dropped by dummynet_io(). -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 19:50:04 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4157D1065672 for ; Tue, 6 Oct 2009 19:50:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 16E1A8FC0A for ; Tue, 6 Oct 2009 19:50:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n96Jo2M4032649 for ; Tue, 6 Oct 2009 19:50:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n96Jo2g6032639; Tue, 6 Oct 2009 19:50:02 GMT (envelope-from gnats) Date: Tue, 6 Oct 2009 19:50:02 GMT Message-Id: <200910061950.n96Jo2g6032639@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: kern/139113: commit references a PR X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 19:50:04 -0000 The following reply was made to PR kern/139113; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/139113: commit references a PR Date: Tue, 6 Oct 2009 19:45:04 +0000 (UTC) Author: qingli Date: Tue Oct 6 19:44:44 2009 New Revision: 197811 URL: http://svn.freebsd.org/changeset/base/197811 Log: MFC 197695 Previously, if an address alias is configured on an interface, and this address alias has a prefix matching that of another address configured on the same interface, then the ARP entry for the alias is not deleted from the ARP table when that address alias is removed. This patch fixes the aforementioned issue. PR: kern/139113 Reviewed by: bz Approved by: re Modified: stable/8/sys/ (props changed) stable/8/sys/amd64/include/xen/ (props changed) stable/8/sys/cddl/contrib/opensolaris/ (props changed) stable/8/sys/contrib/dev/acpica/ (props changed) stable/8/sys/contrib/pf/ (props changed) stable/8/sys/dev/xen/xenpci/ (props changed) stable/8/sys/netinet/in.c Modified: stable/8/sys/netinet/in.c ============================================================================== --- stable/8/sys/netinet/in.c Tue Oct 6 18:47:02 2009 (r197810) +++ stable/8/sys/netinet/in.c Tue Oct 6 19:44:44 2009 (r197811) @@ -1060,6 +1060,8 @@ in_scrubprefix(struct in_ifaddr *target) !(target->ia_ifp->if_flags & IFF_LOOPBACK)) { error = ifa_del_loopback_route((struct ifaddr *)target, (struct sockaddr *)&target->ia_addr); + /* remove arp cache */ + arp_ifscrub(target->ia_ifp, IA_SIN(target)->sin_addr.s_addr); } if ((target->ia_flags & IFA_ROUTE) == 0) { @@ -1082,8 +1084,6 @@ in_scrubprefix(struct in_ifaddr *target) prefix = target->ia_addr.sin_addr; mask = target->ia_sockmask.sin_addr; prefix.s_addr &= mask.s_addr; - /* remove arp cache */ - arp_ifscrub(target->ia_ifp, IA_SIN(target)->sin_addr.s_addr); } IN_IFADDR_RLOCK(); _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Oct 6 20:58:32 2009 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E7641065672 for ; Tue, 6 Oct 2009 20:58:32 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 1C5AD8FC0C for ; Tue, 6 Oct 2009 20:58:31 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id BB06F730DA; Tue, 6 Oct 2009 23:05:13 +0200 (CEST) Date: Tue, 6 Oct 2009 23:05:13 +0200 From: Luigi Rizzo To: Oleg Bulyzhin Message-ID: <20091006210513.GA14620@onelab2.iet.unipi.it> References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006093408.GA86830@onelab2.iet.unipi.it> <4ACB0F28.3000906@mail.ru> <20091006101747.GA87655@onelab2.iet.unipi.it> <20091006173039.GA66473@lath.rinet.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091006173039.GA66473@lath.rinet.ru> User-Agent: Mutt/1.4.2.3i Cc: rihad , Julian Elischer , freebsd-net@FreeBSD.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 20:58:32 -0000 On Tue, Oct 06, 2009 at 09:30:39PM +0400, Oleg Bulyzhin wrote: > On Tue, Oct 06, 2009 at 12:17:47PM +0200, Luigi Rizzo wrote: > > > io_pkt_drop only reports packets dropped to errors (missing pipes, > > randomly forced packet drops which you don't use, no buffers and so on). > > You are mistaken here. io_pkt_drop is total number of packets dropped by > dummynet_io(). hmm... you are right. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 02:43:04 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBFE91065672 for ; Wed, 7 Oct 2009 02:43:04 +0000 (UTC) (envelope-from dhorn2000@gmail.com) Received: from mail-gx0-f214.google.com (mail-gx0-f214.google.com [209.85.217.214]) by mx1.freebsd.org (Postfix) with ESMTP id 7AF1A8FC0C for ; Wed, 7 Oct 2009 02:43:04 +0000 (UTC) Received: by gxk6 with SMTP id 6so3951230gxk.13 for ; Tue, 06 Oct 2009 19:43:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=1FdSg+9kzZhBKeYCkcIqqCzTUvBPxyLEXz6viTtN85o=; b=bYR2YU4RtYRKNPCaKoY84Zjy3BT6Hr33Oq89Mgvvfwh2p3h9n2vQezw6pJrNOxjXKu N+ovXujT+Tuto5dlzONh2tcDq81kwUvWWkVdwhsNOTnwDnFcBLiMHAf47pu7bDNJ/2zO VKZoyKOZhLSsyOc5he3RnGwj1ADQwj+melD50= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=XY+HnhCjFPXAw1SES+j+C/GngX+JlCzu1GNFloN2mJR60TyLuXrZn1qICA38FlK0k1 x6eLU7j69ZKaB6M125W66ZUoFUE3w516WJEbyhu3CMjPUEyAxWk8RW4VLUznqAekZx7i P/fph9qtC9WoXLhd+Sf6vbCSfTP3FbPpAIjBQ= MIME-Version: 1.0 Received: by 10.101.113.12 with SMTP id q12mr2350305anm.124.1254883383728; Tue, 06 Oct 2009 19:43:03 -0700 (PDT) Date: Tue, 6 Oct 2009 22:43:03 -0400 Message-ID: <25ff90d60910061943g75ab90bcs785659f2c2ec40f4@mail.gmail.com> From: David Horn To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: ezwlansetup sh script (8.0) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 02:43:04 -0000 I have recently been using my laptop wifi in several new locations and decided it was time to write a user friendly interactive shell script to manage the necessary configuration file entries that govern wlan connections in freebsd 8+, and to manage wpa_supplicant, etc. This is just a first draft (no warranties!), but I wanted to get some feedback before I continue. It still needs some formatting/style changes, and likely re-wording of some of the prompts. Please let me know if: a) It eats your cat or config files b) It works and/or you find it useful c) It sucks, and you find zero value since you have memorized wpa_supplicant.conf and do not need a user friendly front end d) ok for a start, but you really want feature abc e) there is already a duplication of effort going on somewhere else # Revision=0.1 (Alpha) Works, but not well tested yet # # Requires: FreeBSD 8+ # # Designed for the mainstream end-user workstation scenarios, # not for any of the server hostap, multi-bss, lagg, vlan, gateway, WDS, or # other setups which require static configuration # # Works with: # - WEP, WPA, WPA2, NONE Authentication - Station (STA) mode ONLY # - multiple Access-Point (BSS) profiles in wpa_supplicant.conf # - single wlanX interface per wireless nic # - multiple wireless nics (e.g. iwn0 and ath0) # - with/without existing /etc/rc.conf entries # - with/without existing /etc/wpa_supplicant entries # - demand-load wireless nic kernel module + add to loader.conf # - priority field per profile setting # - Hidden SSID per profile setting # - rescan and display sorted AP list (sorted by signal strength) # - Display scan results with detected authentication values # - Ad-Hoc (As a static rc.conf entry only) + switch ibss<->bss mode # # Broken with # - WPS # - WPA-EAP/802.1x - tbd # - IBSS with encryption (WEP/WPA static key) Full sh script (ezwlansetup.sh) is here: http://sites.google.com/site/freebsdattachments/home/ezwlansetup.sh ./ezwlansetup.sh --help for usage information ---Dave H From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 07:03:30 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC188106568B for ; Wed, 7 Oct 2009 07:03:30 +0000 (UTC) (envelope-from universite@ukr.net) Received: from mary-teresa.otrada.od.ua (universite.broker.freenet6.net [IPv6:2001:5c0:1400:b::27e9]) by mx1.freebsd.org (Postfix) with ESMTP id 12E538FC14 for ; Wed, 7 Oct 2009 07:03:28 +0000 (UTC) Received: from phenom (mary-teresa.otrada.od.ua [10.0.0.10]) (authenticated bits=0) by mary-teresa.otrada.od.ua (8.14.3/8.14.3) with ESMTP id n9773N4q061678 for ; Wed, 7 Oct 2009 10:03:24 +0300 (EEST) (envelope-from universite@ukr.net) X-Authentication-Warning: mary-teresa.otrada.od.ua: Host mary-teresa.otrada.od.ua [10.0.0.10] claimed to be phenom X-AntiVirus: Checked by Dr.Web [version: 5.0, engine: 5.00.0.12182, virus records: 655901, updated: 7.10.2009] Message-ID: <4ACC3D3B.5080508@ukr.net> Date: Wed, 07 Oct 2009 10:03:23 +0300 From: "Vladislav V. Prodan" User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mary-teresa.otrada.od.ua Subject: Packages with the wrong source X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 07:03:31 -0000 I have two channels on the Internet PPPoE - tun1 and tun2. When you start ntpd, incoming packets begin to crumble with the first ISP, although requests come from the 2-nd ISP 23:40:53.548840 IP (tos 0x40, ttl 58, id 0, offset 0, flags [DF], proto UDP (17), length 76) 192.168.30.1.ntp > .ntp: [udp sum ok] NTPv3, length 48 symmetric active, Leap indicator: (0), Stratum 3, poll 4s, precision -19 Root Delay: 0.021423, Root dispersion: 0.086944, Reference-ID: Te.NeT.UA Reference Timestamp: 3460219701.661973714 (2009/08/25 23:08:21) Originator Timestamp: 3460221653.546850204 (2009/08/25 23:40:53) Receive Timestamp: 3460221653.552442133 (2009/08/25 23:40:53) Transmit Timestamp: 3460221653.552893817 (2009/08/25 23:40:53) Originator - Receive Timestamp: +0.005591936 Originator - Transmit Timestamp: +0.006043639 And these packages bandwidth demands grow to 2MBps Only helps to turn off ntpd. The only problem with the ntp packets The problem suddenly appears and as suddenly disappears. There are ideas on how to fix this problem? # uname -a FreeBSD mary-teresa.xxxxxxxxxxx 8.0-BETA4 FreeBSD 8.0-BETA4 #0: Sun Sep 13 03:05:09 EEST 2009 vlad11@mary-teresa.xxxxxxxxxxx:/usr/obj/usr/src/sys/mary-teresa.18 amd64 # ntptrace -n 192.168.30.1 192.168.30.1: stratum 3, offset -0.000105, root distance 0.012885 [2-nd ISP] 195.138.zzz.34: stratum 2, offset 0.000140, root distance 0.012610 62.149.0.30: stratum 1, offset 0.000000, root distance 0.000000, refid 'GPS' # netstat -rn | grep 192.168. 10.0.0.0/8 192.168.61.250 UGS 0 375059 vr1 192.168.0.0/16 192.168.61.250 UGS 0 48072705 vr1 192.168.60.0/23 link#3 U 0 790 vr1 192.168.60.194 link#4 UHS 0 3491 lo0 # netstat -rn | grep UGS default 89.209.95.254 UGS 0 232097628 tun1 10.0.0.0/8 192.168.61.250 UGS 0 375120 vr1 192.168.0.0/16 192.168.61.250 UGS 0 49891561 vr1 1-st ISP <--- tun1 <-- vr0 -- [router] -- vr1 --> tun2 --> 2-nd ISP # ifconfig vr0 vr0: flags=8843 metric 0 mtu 1500 options=2808 ether 00:11:95:ff:a7:c0 media: Ethernet autoselect (100baseTX ) status: active # ifconfig vr1 vr1: flags=8943 metric 0 mtu 1500 options=2808 ether 00:02:44:8b:89:40 inet 192.168.60.194 netmask 0xfffffe00 broadcast 192.168.61.255 media: Ethernet autoselect (100baseTX ) status: active # ifconfig tun1 tun1: flags=8051 metric 0 mtu 1492 inet 89.209.sss.sss --> 89.209.95.254 netmask 0xffffffff Opened by PID 1582 # ifconfig tun2 tun2: flags=8051 metric 0 mtu 1492 inet 85.238.sss.sss --> 195.138.80.134 netmask 0xffffffff Opened by PID 97047 From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 08:46:42 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB29310656A5; Wed, 7 Oct 2009 08:46:42 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id 817458FC14; Wed, 7 Oct 2009 08:46:42 +0000 (UTC) Received: from [217.25.27.27] (port=24145 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1MvSAI-000Dfn-00; Wed, 07 Oct 2009 12:46:30 +0400 Message-ID: <4ACC5563.602@mail.ru> Date: Wed, 07 Oct 2009 13:46:27 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 08:46:42 -0000 Robert Watson wrote: > > On Wed, 7 Oct 2009, Eugene Grosbein wrote: > >> On Tue, Oct 06, 2009 at 08:28:35PM +0500, rihad wrote: >> >>> I don't think net.inet.ip.intr_queue_maxlen is relevant to this >>> problem, as net.inet.ip.intr_queue_drops is normally zero or very >>> close to it at all times. >> >> When net.isr.direct is 1, this queue is used very seldom. Would you >> change it to 0, it will be used extensively. > > Just to clarify this more specifically: > > With net.isr.direct set to 0, the netisr will always be used when > processing inbound IP packets. > Maybe I've done something wrong, but setting net.isr.direct=0 on a live system hasn't yet overrun the default queue of 50 @460-470 mbit/s: $ sysctl net.inet.ip | fgrep intr net.inet.ip.intr_queue_maxlen: 50 net.inet.ip.intr_queue_drops: 0 $ sysctl net.isr.direct net.isr.direct: 0 I've yet to test how this direct=0 improves extensive dummynet drops. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 08:51:52 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3C7C1065670; Wed, 7 Oct 2009 08:51:52 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id 6A84C8FC0C; Wed, 7 Oct 2009 08:51:52 +0000 (UTC) Received: from [217.25.27.27] (port=29043 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1MvSFS-000FjI-00; Wed, 07 Oct 2009 12:51:51 +0400 Message-ID: <4ACC56A6.1030808@mail.ru> Date: Wed, 07 Oct 2009 13:51:50 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> In-Reply-To: <4ACC5563.602@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 08:51:52 -0000 rihad wrote: > I've yet to test how this direct=0 improves extensive dummynet drops. > Ooops... After a couple of minutes, suddenly: net.inet.ip.intr_queue_drops: 1284 Bumped it up a bit. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 08:59:04 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05D0C1065741 for ; Wed, 7 Oct 2009 08:59:04 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id B95AB8FC20 for ; Wed, 7 Oct 2009 08:59:03 +0000 (UTC) Received: by lath.rinet.ru (Postfix, from userid 222) id 97364704B; Wed, 7 Oct 2009 12:59:02 +0400 (MSD) Date: Wed, 7 Oct 2009 12:59:02 +0400 From: Oleg Bulyzhin To: rihad Message-ID: <20091007085902.GA88982@lath.rinet.ru> References: <4AC8A76B.3050502@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4AC8A76B.3050502@mail.ru> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 08:59:04 -0000 Please show your 'sysctl net.inet.ip' output. -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 09:01:20 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02F16106568F for ; Wed, 7 Oct 2009 09:01:20 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id CAD038FC1D for ; Wed, 7 Oct 2009 09:01:19 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id 40F1D46B1A; Wed, 7 Oct 2009 05:01:19 -0400 (EDT) Date: Wed, 7 Oct 2009 10:01:19 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: rihad In-Reply-To: <4ACC56A6.1030808@mail.ru> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 09:01:20 -0000 On Wed, 7 Oct 2009, rihad wrote: > rihad wrote: >> I've yet to test how this direct=0 improves extensive dummynet drops. > > Ooops... After a couple of minutes, suddenly: > > net.inet.ip.intr_queue_drops: 1284 > > Bumped it up a bit. Yes, I was going to suggest that moving to deferred dispatch has probably simply moved the drops to a new spot, the queue between the ithreads and the netisr thread. In your setup, how many network interfaces are in use, and what drivers? If what's happening is that you're maxing out a CPU then moving to multiple netisrs might help if your card supports generating flow IDs, but most lower-end cards don't. I have patches to generate those flow IDs in software rather than hardware, but there are some downsides to doing so, not least that it takes cache line misses on the packet that generally make up a lot of the cost of processing the packet. My experience with most reasonable cards is that letting them doing the work distribution with RSS and use multiple ithreads is a more performant strategy than using software work distribution on current systems, though. Someone has probably asked for this already, but -- could you send a snapshot of the top -SH output in the steady state? Let top run for a few minutes and then copy/paste the first 10-20 lines into an e-mail. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 09:22:57 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CEE9B1065701; Wed, 7 Oct 2009 09:22:57 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id 822228FC14; Wed, 7 Oct 2009 09:22:57 +0000 (UTC) Received: from [217.25.27.27] (port=57345 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1MvSjX-000FWG-00; Wed, 07 Oct 2009 13:22:55 +0400 Message-ID: <4ACC5DEC.1010006@mail.ru> Date: Wed, 07 Oct 2009 14:22:52 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 09:22:57 -0000 Robert Watson wrote: > On Wed, 7 Oct 2009, rihad wrote: > >> rihad wrote: >>> I've yet to test how this direct=0 improves extensive dummynet drops. >> >> Ooops... After a couple of minutes, suddenly: >> >> net.inet.ip.intr_queue_drops: 1284 >> >> Bumped it up a bit. > > Yes, I was going to suggest that moving to deferred dispatch has > probably simply moved the drops to a new spot, the queue between the > ithreads and the netisr thread. In your setup, how many network > interfaces are in use, and what drivers? > bce -- Broadcom NetXtreme II (BCM5706/BCM5708) PCI/PCIe Gigabit Ethernet adapter driver device bce compiled into a 7.1-RELEASE-p8 kernel. 2 network cards: bce0 used for ~400-500 mbit/s input, bce1 for output, i.e. acting as a smart router. It has 2 quad core CPUs. Now the probability of drops (as monitored by netstat -s's "output packets dropped due to no bufs, etc.") is definitely a function of traffic load and the number of items in a ipfw table. I've just decreased the size of the two tables from ~2600 to ~1800 each and the drops instantly went away, even though the traffic passing through the box didn't decrease, it even increased a bit due to now shaping fewer clients (luckily "ipfw pipe tablearg" passes packets failing a table lookup untouched). > If what's happening is that you're maxing out a CPU then moving to > multiple netisrs might help if your card supports generating flow IDs, > but most lower-end cards don't. I have patches to generate those flow > IDs in software rather than hardware, but there are some downsides to > doing so, not least that it takes cache line misses on the packet that > generally make up a lot of the cost of processing the packet. > > My experience with most reasonable cards is that letting them doing the > work distribution with RSS and use multiple ithreads is a more > performant strategy than using software work distribution on current > systems, though. > So should we prefer a bunch of expensive quality 10 gig cards? Any you would recommend? > Someone has probably asked for this already, but -- could you send a > snapshot of the top -SH output in the steady state? Let top run for a > few minutes and then copy/paste the first 10-20 lines into an e-mail. > Sure. Mind you: now there's only 1800 entries in each of the two ipfw tables, so any drops have stopped. But it only takes another 200-300 entries to start dropping. 155 processes: 10 running, 129 sleeping, 16 waiting CPU: 2.4% user, 0.0% nice, 2.0% system, 9.3% interrupt, 86.2% idle Mem: 1691M Active, 1491M Inact, 454M Wired, 130M Cache, 214M Buf, 170M Free Swap: 2048M Total, 12K Used, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 15 root 171 ki31 0K 16K CPU3 3 22.4H 97.85% idle: cpu3 14 root 171 ki31 0K 16K CPU4 4 23.0H 96.29% idle: cpu4 12 root 171 ki31 0K 16K CPU6 6 23.8H 94.58% idle: cpu6 16 root 171 ki31 0K 16K CPU2 2 22.5H 90.72% idle: cpu2 13 root 171 ki31 0K 16K CPU5 5 23.4H 90.58% idle: cpu5 18 root 171 ki31 0K 16K RUN 0 20.3H 85.60% idle: cpu0 17 root 171 ki31 0K 16K CPU1 1 910:03 78.37% idle: cpu1 11 root 171 ki31 0K 16K CPU7 7 23.8H 65.62% idle: cpu7 21 root -44 - 0K 16K CPU7 7 19:03 48.34% swi1: net 29 root -68 - 0K 16K WAIT 1 515:49 19.63% irq256: bce0 31 root -68 - 0K 16K WAIT 2 56:05 5.52% irq257: bce1 19 root -32 - 0K 16K WAIT 5 50:05 3.86% swi4: clock sio 983 flowtools 44 0 12112K 6440K select 0 13:20 0.15% flow-capture 465 root -68 - 0K 16K - 3 51:19 0.00% dummynet 3 root -8 - 0K 16K - 1 7:41 0.00% g_up 4 root -8 - 0K 16K - 2 7:14 0.00% g_down 30 root -64 - 0K 16K WAIT 6 5:30 0.00% irq16: mfi0 From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 09:23:49 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0C2E1065693 for ; Wed, 7 Oct 2009 09:23:49 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx34.mail.ru (mx34.mail.ru [94.100.176.48]) by mx1.freebsd.org (Postfix) with ESMTP id 7E8A08FC24 for ; Wed, 7 Oct 2009 09:23:49 +0000 (UTC) Received: from [217.25.27.27] (port=21844 helo=[217.25.27.27]) by mx34.mail.ru with asmtp id 1MvSkN-0004Qu-00; Wed, 07 Oct 2009 13:23:47 +0400 Message-ID: <4ACC5E23.8090405@mail.ru> Date: Wed, 07 Oct 2009 14:23:47 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> In-Reply-To: <20091007085902.GA88982@lath.rinet.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 09:23:49 -0000 Oleg Bulyzhin wrote: > Please show your 'sysctl net.inet.ip' output. > net.inet.ip.portrange.randomtime: 45 net.inet.ip.portrange.randomcps: 10 net.inet.ip.portrange.randomized: 1 net.inet.ip.portrange.reservedlow: 0 net.inet.ip.portrange.reservedhigh: 1023 net.inet.ip.portrange.hilast: 65535 net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.last: 65535 net.inet.ip.portrange.first: 49152 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.forwarding: 1 net.inet.ip.redirect: 1 net.inet.ip.ttl: 64 net.inet.ip.rtexpire: 3600 net.inet.ip.rtminexpire: 10 net.inet.ip.rtmaxcache: 128 net.inet.ip.sourceroute: 0 net.inet.ip.intr_queue_maxlen: 300 net.inet.ip.intr_queue_drops: 1365 net.inet.ip.accept_sourceroute: 0 net.inet.ip.keepfaith: 0 net.inet.ip.gifttl: 30 net.inet.ip.same_prefix_carp_only: 0 net.inet.ip.subnets_are_local: 0 net.inet.ip.fastforwarding: 0 net.inet.ip.fw.dyn_keepalive: 1 net.inet.ip.fw.dyn_short_lifetime: 5 net.inet.ip.fw.dyn_udp_lifetime: 10 net.inet.ip.fw.dyn_rst_lifetime: 1 net.inet.ip.fw.dyn_fin_lifetime: 1 net.inet.ip.fw.dyn_syn_lifetime: 20 net.inet.ip.fw.dyn_ack_lifetime: 300 net.inet.ip.fw.static_count: 12 net.inet.ip.fw.dyn_max: 4096 net.inet.ip.fw.dyn_count: 0 net.inet.ip.fw.curr_dyn_buckets: 256 net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.default_rule: 65535 net.inet.ip.fw.verbose_limit: 0 net.inet.ip.fw.verbose: 0 net.inet.ip.fw.debug: 1 net.inet.ip.fw.one_pass: 0 net.inet.ip.fw.autoinc_step: 100 net.inet.ip.fw.enable: 1 net.inet.ip.maxfragpackets: 3472 net.inet.ip.maxfragsperpacket: 16 net.inet.ip.fragpackets: 0 net.inet.ip.check_interface: 0 net.inet.ip.random_id: 0 net.inet.ip.sendsourcequench: 0 net.inet.ip.process_options: 1 net.inet.ip.dummynet.debug: 0 net.inet.ip.dummynet.pipe_byte_limit: 1048576 net.inet.ip.dummynet.pipe_slot_limit: 2000 net.inet.ip.dummynet.io_pkt_drop: 4515 net.inet.ip.dummynet.io_pkt_fast: 120785085 net.inet.ip.dummynet.io_pkt: 639360193 net.inet.ip.dummynet.io_fast: 1 net.inet.ip.dummynet.tick_lost: 0 net.inet.ip.dummynet.tick_diff: 4252218 net.inet.ip.dummynet.tick_adjustment: 950800 net.inet.ip.dummynet.tick_delta_sum: 173 net.inet.ip.dummynet.tick_delta: 502 net.inet.ip.dummynet.red_max_pkt_size: 1500 net.inet.ip.dummynet.red_avg_pkt_size: 512 net.inet.ip.dummynet.red_lookup_depth: 256 net.inet.ip.dummynet.max_chain_len: 16 net.inet.ip.dummynet.expire: 1 net.inet.ip.dummynet.search_steps: 710716767 net.inet.ip.dummynet.searches: 639360196 net.inet.ip.dummynet.extract_heap: 0 net.inet.ip.dummynet.ready_heap: 496 net.inet.ip.dummynet.curr_time: 181442521 net.inet.ip.dummynet.hash_size: 65536 From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 09:36:59 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 355D61065694 for ; Wed, 7 Oct 2009 09:36:59 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 073A58FC12 for ; Wed, 7 Oct 2009 09:36:59 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id A049746B0C; Wed, 7 Oct 2009 05:36:58 -0400 (EDT) Date: Wed, 7 Oct 2009 10:36:58 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: rihad In-Reply-To: <4ACC5DEC.1010006@mail.ru> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 09:36:59 -0000 On Wed, 7 Oct 2009, rihad wrote: >> snapshot of the top -SH output in the steady state? Let top run for a few >> minutes and then copy/paste the first 10-20 lines into an e-mail. >> > Sure. Mind you: now there's only 1800 entries in each of the two ipfw > tables, so any drops have stopped. But it only takes another 200-300 entries > to start dropping. Could you do the same in the net.isr.direct=1 configuration so we can compare? Robert > > 155 processes: 10 running, 129 sleeping, 16 waiting > CPU: 2.4% user, 0.0% nice, 2.0% system, 9.3% interrupt, 86.2% idle > Mem: 1691M Active, 1491M Inact, 454M Wired, 130M Cache, 214M Buf, 170M Free > Swap: 2048M Total, 12K Used, 2048M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 15 root 171 ki31 0K 16K CPU3 3 22.4H 97.85% idle: cpu3 > 14 root 171 ki31 0K 16K CPU4 4 23.0H 96.29% idle: cpu4 > 12 root 171 ki31 0K 16K CPU6 6 23.8H 94.58% idle: cpu6 > 16 root 171 ki31 0K 16K CPU2 2 22.5H 90.72% idle: cpu2 > 13 root 171 ki31 0K 16K CPU5 5 23.4H 90.58% idle: cpu5 > 18 root 171 ki31 0K 16K RUN 0 20.3H 85.60% idle: cpu0 > 17 root 171 ki31 0K 16K CPU1 1 910:03 78.37% idle: cpu1 > 11 root 171 ki31 0K 16K CPU7 7 23.8H 65.62% idle: cpu7 > 21 root -44 - 0K 16K CPU7 7 19:03 48.34% swi1: net > 29 root -68 - 0K 16K WAIT 1 515:49 19.63% irq256: bce0 > 31 root -68 - 0K 16K WAIT 2 56:05 5.52% irq257: bce1 > 19 root -32 - 0K 16K WAIT 5 50:05 3.86% swi4: clock > sio > 983 flowtools 44 0 12112K 6440K select 0 13:20 0.15% flow-capture > 465 root -68 - 0K 16K - 3 51:19 0.00% dummynet > 3 root -8 - 0K 16K - 1 7:41 0.00% g_up > 4 root -8 - 0K 16K - 2 7:14 0.00% g_down > 30 root -64 - 0K 16K WAIT 6 5:30 0.00% irq16: mfi0 > > From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 09:55:49 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDC631065676; Wed, 7 Oct 2009 09:55:49 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 7F8DC8FC15; Wed, 7 Oct 2009 09:55:49 +0000 (UTC) Received: from [217.25.27.27] (port=51618 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MvTFL-0003ME-00; Wed, 07 Oct 2009 13:55:47 +0400 Message-ID: <4ACC65A0.7030900@mail.ru> Date: Wed, 07 Oct 2009 14:55:44 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 09:55:50 -0000 Robert Watson wrote: > > On Wed, 7 Oct 2009, rihad wrote: > >>> snapshot of the top -SH output in the steady state? Let top run for >>> a few minutes and then copy/paste the first 10-20 lines into an e-mail. >>> >> Sure. Mind you: now there's only 1800 entries in each of the two ipfw >> tables, so any drops have stopped. But it only takes another 200-300 >> entries to start dropping. > > Could you do the same in the net.isr.direct=1 configuration so we can > compare? > net.isr.direct=1: last pid: 92152; load averages: 0.99, 1.18, 1.15 up 1+01:42:28 14:53:09 162 processes: 9 running, 136 sleeping, 17 waiting CPU: 2.1% user, 0.0% nice, 5.4% system, 7.0% interrupt, 85.5% idle Mem: 1693M Active, 1429M Inact, 447M Wired, 197M Cache, 214M Buf, 170M Free Swap: 2048M Total, 12K Used, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 171 ki31 0K 16K CPU6 6 24.3H 100.00% idle: cpu6 13 root 171 ki31 0K 16K CPU5 5 23.8H 95.95% idle: cpu5 14 root 171 ki31 0K 16K CPU4 4 23.4H 93.12% idle: cpu4 16 root 171 ki31 0K 16K CPU2 2 23.0H 90.19% idle: cpu2 11 root 171 ki31 0K 16K CPU7 7 24.2H 87.26% idle: cpu7 15 root 171 ki31 0K 16K CPU3 3 22.8H 86.18% idle: cpu3 18 root 171 ki31 0K 16K RUN 0 20.6H 84.96% idle: cpu0 17 root 171 ki31 0K 16K CPU1 1 933:23 47.85% idle: cpu1 29 root -68 - 0K 16K WAIT 1 522:02 46.88% irq256: bce0 465 root -68 - 0K 16K - 7 55:15 12.65% dummynet 31 root -68 - 0K 16K WAIT 2 57:29 4.74% irq257: bce1 21 root -44 - 0K 16K WAIT 0 34:55 4.64% swi1: net 19 root -32 - 0K 16K WAIT 4 51:41 3.96% swi4: clock sio 30 root -64 - 0K 16K WAIT 6 5:43 0.73% irq16: mfi0 Almost 2000 entries in the table, traffic load= 420-430 mbps, drops haven't yet started. Previous net.isr.direct=0: > >> >> 155 processes: 10 running, 129 sleeping, 16 waiting >> CPU: 2.4% user, 0.0% nice, 2.0% system, 9.3% interrupt, 86.2% idle >> Mem: 1691M Active, 1491M Inact, 454M Wired, 130M Cache, 214M Buf, 170M >> Free >> Swap: 2048M Total, 12K Used, 2048M Free >> >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 15 root 171 ki31 0K 16K CPU3 3 22.4H 97.85% idle: cpu3 >> 14 root 171 ki31 0K 16K CPU4 4 23.0H 96.29% idle: cpu4 >> 12 root 171 ki31 0K 16K CPU6 6 23.8H 94.58% idle: cpu6 >> 16 root 171 ki31 0K 16K CPU2 2 22.5H 90.72% idle: cpu2 >> 13 root 171 ki31 0K 16K CPU5 5 23.4H 90.58% idle: cpu5 >> 18 root 171 ki31 0K 16K RUN 0 20.3H 85.60% idle: cpu0 >> 17 root 171 ki31 0K 16K CPU1 1 910:03 78.37% idle: cpu1 >> 11 root 171 ki31 0K 16K CPU7 7 23.8H 65.62% idle: cpu7 >> 21 root -44 - 0K 16K CPU7 7 19:03 48.34% swi1: net >> 29 root -68 - 0K 16K WAIT 1 515:49 19.63% irq256: >> bce0 >> 31 root -68 - 0K 16K WAIT 2 56:05 5.52% irq257: >> bce1 >> 19 root -32 - 0K 16K WAIT 5 50:05 3.86% swi4: >> clock sio >> 983 flowtools 44 0 12112K 6440K select 0 13:20 0.15% >> flow-capture >> 465 root -68 - 0K 16K - 3 51:19 0.00% dummynet >> 3 root -8 - 0K 16K - 1 7:41 0.00% g_up >> 4 root -8 - 0K 16K - 2 7:14 0.00% g_down >> 30 root -64 - 0K 16K WAIT 6 5:30 0.00% irq16: mfi0 >> >> > > From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 10:05:07 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 905EB106568F for ; Wed, 7 Oct 2009 10:05:07 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4E6F28FC12 for ; Wed, 7 Oct 2009 10:05:07 +0000 (UTC) Received: by lath.rinet.ru (Postfix, from userid 222) id 22A1C704B; Wed, 7 Oct 2009 14:05:03 +0400 (MSD) Date: Wed, 7 Oct 2009 14:05:03 +0400 From: Oleg Bulyzhin To: rihad Message-ID: <20091007100503.GB88982@lath.rinet.ru> References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACC5E23.8090405@mail.ru> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 10:05:07 -0000 On Wed, Oct 07, 2009 at 02:23:47PM +0500, rihad wrote: Few questions: 1) why are you not using fastforwarding? 2) search_steps/searches ratio is not that good, are you using 'buckets' keyword in your pipe configuration? 3) you have net.inet.ip.fw.one_pass = 0, is it intended? -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 10:16:32 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67E991065692; Wed, 7 Oct 2009 10:16:32 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx40.mail.ru (mx40.mail.ru [94.100.176.54]) by mx1.freebsd.org (Postfix) with ESMTP id 249678FC15; Wed, 7 Oct 2009 10:16:31 +0000 (UTC) Received: from [217.25.27.27] (port=42136 helo=[217.25.27.27]) by mx40.mail.ru with asmtp id 1MvTZM-000Gey-00; Wed, 07 Oct 2009 14:16:28 +0400 Message-ID: <4ACC6A7B.5050808@mail.ru> Date: Wed, 07 Oct 2009 15:16:27 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> In-Reply-To: <20091007100503.GB88982@lath.rinet.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 10:16:32 -0000 Oleg Bulyzhin wrote: > On Wed, Oct 07, 2009 at 02:23:47PM +0500, rihad wrote: > > Few questions: > 1) why are you not using fastforwarding? > 2) search_steps/searches ratio is not that good, are you using 'buckets' > keyword in your pipe configuration? > 3) you have net.inet.ip.fw.one_pass = 0, is it intended? > 1) and 3): the box does traffic accounting and shaping, so I need one_pass=0 to do both ngtee and pipes. 2) Hm, I'm not using "buckets", but rather net.inet.ip.dummynet.hash_size. It's at default, 64. I've tried setting net.inet.ip.dummynet.hash_size=65536 in sysctl.conf but somehow it was still 64 after reboot, so I left it at 64. Should I make it 128? 256? Does it matter that much? The load is at approx. 70-120 consumers per pipe, so I thought 64 bucket size was enough. Here's how I've been monitoring the pipe load interactively: #!/usr/local/bin/bash while :; do ( buff="$(ipfw pipe show | grep -E '^[[:digit:]]+:' | sort -n -k8,8 -r)" online="$(mysql -Bs -u... -p... dbname -e"select count(*) from online")" clear echo "$buff" echo "$buff" | awk '{sum+=$8} END {printf "QLoad: %d/%d\n", sum, '"$online"'}' netstat -m | fgrep -w 'mbuf clusters in use' ) | dd 2>/dev/null if [ -t 0 ]; then read -s -n 1 -t 15 else sleep 15 fi done Public domain ;-) From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 10:27:40 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2BF91065696; Wed, 7 Oct 2009 10:27:40 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx30.mail.ru (mx30.mail.ru [94.100.176.44]) by mx1.freebsd.org (Postfix) with ESMTP id 7C5098FC2B; Wed, 7 Oct 2009 10:27:40 +0000 (UTC) Received: from [217.25.27.27] (port=16180 helo=[217.25.27.27]) by mx30.mail.ru with asmtp id 1MvTkA-000AZH-00; Wed, 07 Oct 2009 14:27:38 +0400 Message-ID: <4ACC6D17.7000405@mail.ru> Date: Wed, 07 Oct 2009 15:27:35 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> In-Reply-To: <4ACC5DEC.1010006@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 10:27:41 -0000 rihad wrote: > Now the probability of drops (as monitored by netstat -s's "output > packets dropped due to no bufs, etc.") is definitely a function of > traffic load and the number of items in a ipfw table. I've just > decreased the size of the two tables from ~2600 to ~1800 each and the > drops instantly went away, even though the traffic passing through the > box didn't decrease, it even increased a bit due to now shaping fewer > clients (luckily "ipfw pipe tablearg" passes packets failing a table > lookup untouched). > ~2100 users in each of the two tables, Drops have started coming in but at a a very very slow rate, like 30-150 in a single burst every 10-20 minutes. Run every 10 seconds: $ while :; do netstat -s 2>/dev/null | fgrep -w "output packets dropped"; sleep 10; done 30900 output packets dropped due to no bufs, etc. ... 250-300 lines skipped 30923 output packets dropped due to no bufs, etc. ... 50-100 lines skipped 30953 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31165 output packets dropped due to no bufs, etc. 31444 output packets dropped due to no bufs, etc. 31444 output packets dropped due to no bufs, etc. 31444 output packets dropped due to no bufs, etc. 31549 output packets dropped due to no bufs, etc. 31549 output packets dropped due to no bufs, etc. 31549 output packets dropped due to no bufs, etc. 31549 output packets dropped due to no bufs, etc. 31549 output packets dropped due to no bufs, etc. 31549 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. 31678 output packets dropped due to no bufs, etc. So the larger the number of users (increased at about 1-2 every 10 seconds as users log in and out) the shorter the pause between the bursts. net.isr.direct=0 top -SH: last pid: 2528; load averages: 0.69, 0.89, 0.96 up 1+02:15:20 15:26:01 165 processes: 12 running, 137 sleeping, 16 waiting CPU: 9.5% user, 0.0% nice, 3.8% system, 6.9% interrupt, 79.9% idle Mem: 1726M Active, 1453M Inact, 433M Wired, 178M Cache, 214M Buf, 145M Free Swap: 2048M Total, 12K Used, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 171 ki31 0K 16K RUN 6 24.8H 100.00% idle: cpu6 11 root 171 ki31 0K 16K CPU7 7 24.7H 98.29% idle: cpu7 13 root 171 ki31 0K 16K CPU5 5 24.3H 98.19% idle: cpu5 14 root 171 ki31 0K 16K CPU4 4 23.9H 95.41% idle: cpu4 15 root 171 ki31 0K 16K CPU3 3 23.3H 93.55% idle: cpu3 16 root 171 ki31 0K 16K CPU2 2 23.4H 87.06% idle: cpu2 18 root 171 ki31 0K 16K CPU0 0 21.1H 86.72% idle: cpu0 29 root -68 - 0K 16K CPU1 1 537:45 47.61% irq256: bce0 17 root 171 ki31 0K 16K RUN 1 948:22 43.12% idle: cpu1 19 root -32 - 0K 16K WAIT 4 53:10 4.25% swi4: clock sio 31 root -68 - 0K 16K WAIT 2 58:44 3.86% irq257: bce1 465 root -68 - 0K 16K CPU3 3 59:02 1.51% dummynet 21 root -44 - 0K 16K WAIT 0 34:58 0.00% swi1: net 3 root -8 - 0K 16K - 0 8:15 0.00% g_up Dummynet's WCPU is mostly 0-4%, but might jump to 6-12% sometimes, depending on which fraction of the second you look at it. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 10:35:12 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8B9B106566B; Wed, 7 Oct 2009 10:35:12 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx33.mail.ru (mx33.mail.ru [94.100.176.47]) by mx1.freebsd.org (Postfix) with ESMTP id 9CFAC8FC13; Wed, 7 Oct 2009 10:35:12 +0000 (UTC) Received: from [217.25.27.27] (port=59219 helo=[217.25.27.27]) by mx33.mail.ru with asmtp id 1MvTrS-0003RC-00; Wed, 07 Oct 2009 14:35:11 +0400 Message-ID: <4ACC6EDE.4040305@mail.ru> Date: Wed, 07 Oct 2009 15:35:10 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC6D17.7000405@mail.ru> In-Reply-To: <4ACC6D17.7000405@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 10:35:13 -0000 rihad wrote: > net.isr.direct=0 Sorry, net.isr.direct=1 I forgot to revert it back after copy'n'pasting top -SH for Mr. Robert. > top -SH: > last pid: 2528; load averages: 0.69, 0.89, 0.96 > up 1+02:15:20 15:26:01 > 165 processes: 12 running, 137 sleeping, 16 waiting > CPU: 9.5% user, 0.0% nice, 3.8% system, 6.9% interrupt, 79.9% idle > Mem: 1726M Active, 1453M Inact, 433M Wired, 178M Cache, 214M Buf, 145M Free > Swap: 2048M Total, 12K Used, 2048M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root 171 ki31 0K 16K RUN 6 24.8H 100.00% idle: cpu6 > 11 root 171 ki31 0K 16K CPU7 7 24.7H 98.29% idle: cpu7 > 13 root 171 ki31 0K 16K CPU5 5 24.3H 98.19% idle: cpu5 > 14 root 171 ki31 0K 16K CPU4 4 23.9H 95.41% idle: cpu4 > 15 root 171 ki31 0K 16K CPU3 3 23.3H 93.55% idle: cpu3 > 16 root 171 ki31 0K 16K CPU2 2 23.4H 87.06% idle: cpu2 > 18 root 171 ki31 0K 16K CPU0 0 21.1H 86.72% idle: cpu0 > 29 root -68 - 0K 16K CPU1 1 537:45 47.61% irq256: bce0 > 17 root 171 ki31 0K 16K RUN 1 948:22 43.12% idle: cpu1 > 19 root -32 - 0K 16K WAIT 4 53:10 4.25% swi4: > clock sio > 31 root -68 - 0K 16K WAIT 2 58:44 3.86% irq257: bce1 > 465 root -68 - 0K 16K CPU3 3 59:02 1.51% dummynet > 21 root -44 - 0K 16K WAIT 0 34:58 0.00% swi1: net > 3 root -8 - 0K 16K - 0 8:15 0.00% g_up > > > Dummynet's WCPU is mostly 0-4%, but might jump to 6-12% sometimes, > depending on which fraction of the second you look at it. > From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 10:45:27 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31B5D1065696 for ; Wed, 7 Oct 2009 10:45:27 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id E2A088FC18 for ; Wed, 7 Oct 2009 10:45:26 +0000 (UTC) Received: by lath.rinet.ru (Postfix, from userid 222) id 7EFDB704B; Wed, 7 Oct 2009 14:45:25 +0400 (MSD) Date: Wed, 7 Oct 2009 14:45:25 +0400 From: Oleg Bulyzhin To: rihad Message-ID: <20091007104525.GC88982@lath.rinet.ru> References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACC6A7B.5050808@mail.ru> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 10:45:27 -0000 On Wed, Oct 07, 2009 at 03:16:27PM +0500, rihad wrote: > Oleg Bulyzhin wrote: > > On Wed, Oct 07, 2009 at 02:23:47PM +0500, rihad wrote: > > > > Few questions: > > 1) why are you not using fastforwarding? > > 2) search_steps/searches ratio is not that good, are you using 'buckets' > > keyword in your pipe configuration? > > 3) you have net.inet.ip.fw.one_pass = 0, is it intended? > > > 1) and 3): the box does traffic accounting and shaping, so I need > one_pass=0 to do both ngtee and pipes. Still can not see any objection for not using fastforwarding, and usually ipfw ruleset can be rearranged for using dummynet & netgraph with one_pass=1. Could you show your 'ipfw show' output? (hide ip addresses if you wish but keep counters please). > 2) Hm, I'm not using "buckets", but rather > net.inet.ip.dummynet.hash_size. It's at default, 64. I've tried setting > net.inet.ip.dummynet.hash_size=65536 in sysctl.conf but somehow it was > still 64 after reboot, so I left it at 64. Should I make it 128? 256? > Does it matter that much? The load is at approx. 70-120 consumers per > pipe, so I thought 64 bucket size was enough. It depends on traffic pattern, try to increase it and watch search_steps/searches ratio (~1.001 is good enough) -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 10:52:58 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D02D91065670; Wed, 7 Oct 2009 10:52:58 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id 8B0998FC1C; Wed, 7 Oct 2009 10:52:58 +0000 (UTC) Received: from [217.25.27.27] (port=39366 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1MvU8e-0008eE-00; Wed, 07 Oct 2009 14:52:57 +0400 Message-ID: <4ACC7308.6070301@mail.ru> Date: Wed, 07 Oct 2009 15:52:56 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> In-Reply-To: <20091007104525.GC88982@lath.rinet.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 10:52:58 -0000 Oleg Bulyzhin wrote: > On Wed, Oct 07, 2009 at 03:16:27PM +0500, rihad wrote: >> Oleg Bulyzhin wrote: >>> On Wed, Oct 07, 2009 at 02:23:47PM +0500, rihad wrote: >>> >>> Few questions: >>> 1) why are you not using fastforwarding? >>> 2) search_steps/searches ratio is not that good, are you using 'buckets' >>> keyword in your pipe configuration? >>> 3) you have net.inet.ip.fw.one_pass = 0, is it intended? >>> >> 1) and 3): the box does traffic accounting and shaping, so I need >> one_pass=0 to do both ngtee and pipes. > Still can not see any objection for not using fastforwarding, and usually > ipfw ruleset can be rearranged for using dummynet & netgraph with one_pass=1. You probably have some special sources of documentation ;-) According to man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the packet unless one_pass=0. Or do you mean sprinkling smart skiptos here and there? ;-) > Could you show your 'ipfw show' output? (hide ip addresses if you wish but > keep counters please). > Here it is, in its whole glory: 00100 10434423 1484891105 allow ip from any to any via lo0 00200 2 14 deny ip from any to 127.0.0.0/8 00300 1 4 deny ip from 127.0.0.0/8 to any 01000 3300039938 327603104711 allow ip from any to any in 01010 26214900 421138433 allow ip from me to any out 01020 5453857 46806278 allow icmp from any to any out 01030 3268289053 327224694165 ngtee 1 ip from any to any out 01040 18681181 1089636054 skipto 1100 ip from table(127) to any out recv bce0 xmit bce1 01060 777488848 76743392754 pipe tablearg ip from any to table(0) out recv bce0 xmit bce1 01070 776831109 76682499457 allow ip from any to table(0) out recv bce0 xmit bce1 01100 13102697 808411842 pipe tablearg ip from any to table(2) out 65535 662648946 66711487830 allow ip from any to any table(127) is static in nature and is under 100 entries. table(0) and table(2) have the same IP clients' addresses but different pipe IDs. >> 2) Hm, I'm not using "buckets", but rather >> net.inet.ip.dummynet.hash_size. It's at default, 64. I've tried setting >> net.inet.ip.dummynet.hash_size=65536 in sysctl.conf but somehow it was >> still 64 after reboot, so I left it at 64. Should I make it 128? 256? >> Does it matter that much? The load is at approx. 70-120 consumers per >> pipe, so I thought 64 bucket size was enough. > It depends on traffic pattern, try to increase it and watch > search_steps/searches ratio (~1.001 is good enough) > Hm, thanks, I'll try that. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 11:23:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D15F10656A3; Wed, 7 Oct 2009 11:23:39 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id 155438FC20; Wed, 7 Oct 2009 11:23:39 +0000 (UTC) Received: from [217.25.27.27] (port=38122 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1MvUcL-000Jo6-00; Wed, 07 Oct 2009 15:23:37 +0400 Message-ID: <4ACC7A38.50809@mail.ru> Date: Wed, 07 Oct 2009 16:23:36 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> In-Reply-To: <4ACC7308.6070301@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 11:23:39 -0000 rihad wrote: > Oleg Bulyzhin wrote: >> On Wed, Oct 07, 2009 at 03:16:27PM +0500, rihad wrote: >>> Oleg Bulyzhin wrote: >>>> On Wed, Oct 07, 2009 at 02:23:47PM +0500, rihad wrote: >>>> >>>> Few questions: >>>> 1) why are you not using fastforwarding? >>>> 2) search_steps/searches ratio is not that good, are you using >>>> 'buckets' >>>> keyword in your pipe configuration? >>>> 3) you have net.inet.ip.fw.one_pass = 0, is it intended? >>>> >>> 1) and 3): the box does traffic accounting and shaping, so I need >>> one_pass=0 to do both ngtee and pipes. >> Still can not see any objection for not using fastforwarding, and usually >> ipfw ruleset can be rearranged for using dummynet & netgraph with >> one_pass=1. > > You probably have some special sources of documentation ;-) According to > man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the packet > unless one_pass=0. Or do you mean sprinkling smart skiptos here and > there? ;-) > >> Could you show your 'ipfw show' output? (hide ip addresses if you wish >> but >> keep counters please). >> > Here it is, in its whole glory: > > 00100 10434423 1484891105 allow ip from any to any via lo0 > 00200 2 14 deny ip from any to 127.0.0.0/8 > 00300 1 4 deny ip from 127.0.0.0/8 to any > 01000 3300039938 327603104711 allow ip from any to any in > 01010 26214900 421138433 allow ip from me to any out > 01020 5453857 46806278 allow icmp from any to any out > 01030 3268289053 327224694165 ngtee 1 ip from any to any out > 01040 18681181 1089636054 skipto 1100 ip from table(127) to any out > recv bce0 xmit bce1 > 01060 777488848 76743392754 pipe tablearg ip from any to table(0) out > recv bce0 xmit bce1 > 01070 776831109 76682499457 allow ip from any to table(0) out recv > bce0 xmit bce1 > 01100 13102697 808411842 pipe tablearg ip from any to table(2) out > 65535 662648946 66711487830 allow ip from any to any > > table(127) is static in nature and is under 100 entries. > table(0) and table(2) have the same IP clients' addresses but different > pipe IDs. > >>> 2) Hm, I'm not using "buckets", but rather >>> net.inet.ip.dummynet.hash_size. It's at default, 64. I've tried >>> setting net.inet.ip.dummynet.hash_size=65536 in sysctl.conf but >>> somehow it was still 64 after reboot, so I left it at 64. Should I >>> make it 128? 256? Does it matter that much? The load is at approx. >>> 70-120 consumers per pipe, so I thought 64 bucket size was enough. >> It depends on traffic pattern, try to increase it and watch >> search_steps/searches ratio (~1.001 is good enough) >> After reconfiguring all pipes with hash_size=256 (4 times as much as before), the ratio has started decreasing slowly. Run by me every 5-100 seconds: [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10639566354978963640 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10638988711274017516 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10637649664889937145 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10636898392044547569 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10634798328730542254 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10608591323771604268 [rihad@billing ~]$ echo "$(sysctl -n net.inet.ip.dummynet.search_steps)/$(sysctl -n net.inet.ip.dummynet.searches)" | bc -l 1.10600110020578292697 but the number of drops is still there. Run every minute: Wed Oct 7 11:00:44 UTC 2009 34630 output packets dropped due to no bufs, etc. Wed Oct 7 11:01:44 UTC 2009 34630 output packets dropped due to no bufs, etc. Wed Oct 7 11:02:44 UTC 2009 34729 output packets dropped due to no bufs, etc. Wed Oct 7 11:03:44 UTC 2009 34729 output packets dropped due to no bufs, etc. Wed Oct 7 11:04:44 UTC 2009 34861 output packets dropped due to no bufs, etc. Wed Oct 7 11:05:44 UTC 2009 34932 output packets dropped due to no bufs, etc. Wed Oct 7 11:06:44 UTC 2009 35499 output packets dropped due to no bufs, etc. Wed Oct 7 11:07:45 UTC 2009 35780 output packets dropped due to no bufs, etc. Wed Oct 7 11:08:45 UTC 2009 35841 output packets dropped due to no bufs, etc. Wed Oct 7 11:09:45 UTC 2009 36348 output packets dropped due to no bufs, etc. Wed Oct 7 11:10:45 UTC 2009 36568 output packets dropped due to no bufs, etc. Wed Oct 7 11:11:45 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:12:45 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:13:46 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:14:46 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:15:46 UTC 2009 36673 output packets dropped due to no bufs, etc. Wed Oct 7 11:16:46 UTC 2009 36849 output packets dropped due to no bufs, etc. Wed Oct 7 11:17:46 UTC 2009 37234 output packets dropped due to no bufs, etc. Wed Oct 7 11:18:46 UTC 2009 37949 output packets dropped due to no bufs, etc. Wed Oct 7 11:19:47 UTC 2009 38043 output packets dropped due to no bufs, etc. Wed Oct 7 11:20:47 UTC 2009 38549 output packets dropped due to no bufs, etc. 2200-2350 users online (ipfw table load). I'll wait and see if the drop rate approaches 500-1000 per second as the number of online users comes close to 3-4K. net.isr.direct=0 From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 11:29:23 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E60E1065692 for ; Wed, 7 Oct 2009 11:29:23 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from n21.bullet.mail.mud.yahoo.com (n21.bullet.mail.mud.yahoo.com [68.142.206.160]) by mx1.freebsd.org (Postfix) with SMTP id D1E528FC25 for ; Wed, 7 Oct 2009 11:29:22 +0000 (UTC) Received: from [68.142.194.243] by n21.bullet.mail.mud.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 Received: from [76.13.13.26] by t1.bullet.mud.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 Received: from [76.13.10.165] by t3.bullet.mail.ac4.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 Received: from [127.0.0.1] by omp106.mail.ac4.yahoo.com with NNFMP; 07 Oct 2009 11:29:21 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 682497.15264.bm@omp106.mail.ac4.yahoo.com Received: (qmail 12248 invoked by uid 60001); 7 Oct 2009 11:29:21 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1254914961; bh=wVMRyV1DwKEjvuV95Et6/9edxRtGe9ZLqSVXV2BT0uI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=lnWrEd+DsDmFI/Rl1VusvKlOCuDK7YVU+eUm/IfyPM0Jbase4UowmrOGggDuS8DoY+nqEs30xuiSAC41Vydxx7a+wv00T2JemXZHeDkzpBqBxRyDN3hg8DTLfb2inGNPSaEYSDT5YL8s9ChKk+OHtPY7y4kPhY/WLn24fJITego= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=atpaeAe60i9LBYWqt782mhkoyS4ni9DPsQIe6q1qCm4iGYgGHV342zmnDU9m8QNqPX2z8d3hGhhDabHH8eBoHbdNFWs/3xsDRehRYzRJDbUx2/xIPekp8C5Z7l5AbE/CRPOO/MldPHhsvk/5b7xXDL8o00+wBI26Dm311ef/Npg=; Message-ID: <397638.11040.qm@web63902.mail.re1.yahoo.com> X-YMail-OSG: OFVnZWgVM1n4xe2MCOOH4KckpugU.meEAI3sNmiq2n2yt46q7YJxNxjEj4HzpKvzd.KPtBJNCnJQoSvLU34uaezpo6jm2_mamQuBDu4zX2Sa2nrDvUHHfLejqI0PqJJAv.TD_ZyrPiRihyiIdEqRQOLmbATBrFAsjzWCb7_6drgQo3yxDCY1yDSKAalh1MSlT.KExMaAxL82NaJ2.E8SMtogtgjuotpwNV50ZeUNAErfX2.pea67ahmYCGkOsnjL3nbLpM1hD64YqOQFAZCi.VzwB8v5wZKILGVXSCgHonGPD4QOUB7j56NT3YLXp5W5unEz3APdK3hx5hSGtxFuUg-- Received: from [98.203.21.152] by web63902.mail.re1.yahoo.com via HTTP; Wed, 07 Oct 2009 04:29:21 PDT X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.347.3 Date: Wed, 7 Oct 2009 04:29:21 -0700 (PDT) From: Barney Cordoba To: rihad In-Reply-To: <4ACC7A38.50809@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 11:29:23 -0000 =0A=0A--- On Wed, 10/7/09, rihad wrote:=0A=0A> From: rihad = =0A> Subject: Re: dummynet dropping too many packets=0A> To:= "Oleg Bulyzhin" =0A> Cc: freebsd-net@freebsd.org=0A> Dat= e: Wednesday, October 7, 2009, 7:23 AM=0A> rihad wrote:=0A> > Oleg Bulyzhin= wrote:=0A> >> On Wed, Oct 07, 2009 at 03:16:27PM +0500, rihad=0A> wrote:= =0A> >>> Oleg Bulyzhin wrote:=0A> >>>> On Wed, Oct 07, 2009 at 02:23:47PM += 0500,=0A> rihad wrote:=0A> >>>> =0A> >>>> Few questions:=0A> >>>> 1) why ar= e you not using fastforwarding?=0A> >>>> 2) search_steps/searches ratio is = not that=0A> good, are you using 'buckets'=0A> >>>>=A0 =A0 keyword in your = pipe=0A> configuration?=0A> >>>> 3) you have net.inet.ip.fw.one_pass =3D 0,= =0A> is it intended?=0A> >>>> =0A> >>> 1) and 3): the box does traffic acco= unting and=0A> shaping, so I need one_pass=3D0 to do both ngtee and pipes.= =0A> >> Still can not see any objection for not using=0A> fastforwarding, a= nd usually=0A> >> ipfw ruleset can be rearranged for using dummynet=0A> & n= etgraph with one_pass=3D1.=0A> > =0A> > You probably have some special sour= ces of=0A> documentation ;-) According to man ipfw, both=0A> "netgraph/ngte= e" and "pipe" decide the fate of the packet=0A> unless one_pass=3D0. Or do = you mean sprinkling smart skiptos=0A> here and there? ;-)=0A> > =0A> >> Cou= ld you show your 'ipfw show' output? (hide ip=0A> addresses if you wish but= =0A> >> keep counters please).=0A> >> =0A> > Here it is, in its whole glory= :=0A> > =0A> >=0A> 00100=A0=A0=A010434423=A0=A0=A01484891105=0A> allow ip f= rom any to any via lo0=0A> > 00200=A0 =A0 =A0 =A0 =A0 2=A0 =A0=0A> =A0 =A0 = =A0=A0=A014 deny ip from any to=0A> 127.0.0.0/8=0A> > 00300=A0 =A0 =A0 =A0 = =A0 1=A0 =A0=0A> =A0 =A0 =A0 =A0 4 deny ip from 127.0.0.0/8 to=0A> any=0A> = > 01000 3300039938 327603104711 allow ip from any to any=0A> in=0A> > 01010= =A0=A0=A026214900=A0 =A0 421138433=0A> allow ip from me to any out=0A> > 01= 020=A0 =A0 5453857=A0=0A> =A0=A0=A046806278 allow icmp from any to any out= =0A> > 01030 3268289053 327224694165 ngtee 1 ip from any to=0A> any out=0A>= >=0A> 01040=A0=A0=A018681181=A0=A0=A01089636054=0A> skipto 1100 ip from ta= ble(127) to any out recv bce0 xmit=0A> bce1=0A> > 01060=A0 777488848=A0 767= 43392754 pipe tablearg=0A> ip from any to table(0) out recv bce0 xmit bce1= =0A> > 01070=A0 776831109=A0 76682499457 allow ip from=0A> any to table(0) = out recv bce0 xmit bce1=0A> > 01100=A0=A0=A013102697=A0 =A0 808411842=0A> p= ipe tablearg ip from any to table(2) out=0A> > 65535=A0 662648946=A0 667114= 87830 allow ip from=0A> any to any=0A> > =0A> > table(127) is static in nat= ure and is under 100=0A> entries.=0A> > table(0) and table(2) have the same= IP clients'=0A> addresses but different pipe IDs.=0A> > =0A> >>> 2) Hm, I'= m not using "buckets", but rather=0A> net.inet.ip.dummynet.hash_size. It's = at default, 64. I've=0A> tried setting net.inet.ip.dummynet.hash_size=3D655= 36 in=0A> sysctl.conf but somehow it was still 64 after reboot, so I=0A> le= ft it at 64. Should I make it 128? 256? Does it matter=0A> that much? The l= oad is at approx. 70-120 consumers per pipe,=0A> so I thought 64 bucket siz= e was enough.=0A> >> It depends on traffic pattern, try to increase it=0A> = and watch=0A> >> search_steps/searches ratio (~1.001 is good=0A> enough)=0A= > >> =0A> After reconfiguring all pipes with hash_size=3D256 (4 times=0A> a= s much as before), the ratio has started decreasing slowly.=0A> Run by me e= very 5-100 seconds:=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.i= p.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | = bc -l=0A> 1.10639566354978963640=0A> [rihad@billing ~]$ echo "$(sysctl -n= =0A> net.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummyne= t.searches)" | bc -l=0A> 1.10638988711274017516=0A> [rihad@billing ~]$ echo= "$(sysctl -n=0A> net.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.in= et.ip.dummynet.searches)" | bc -l=0A> 1.10637649664889937145=0A> [rihad@bil= ling ~]$ echo "$(sysctl -n=0A> net.inet.ip.dummynet.search_steps)/$(sysctl = -n=0A> net.inet.ip.dummynet.searches)" | bc -l=0A> 1.10636898392044547569= =0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.ip.dummynet.search_s= teps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | bc -l=0A> 1.1063479= 8328730542254=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> net.inet.ip.dumm= ynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searches)" | bc -l= =0A> 1.10608591323771604268=0A> [rihad@billing ~]$ echo "$(sysctl -n=0A> ne= t.inet.ip.dummynet.search_steps)/$(sysctl -n=0A> net.inet.ip.dummynet.searc= hes)" | bc -l=0A> 1.10600110020578292697=0A> =0A> =0A> =0A> but the number = of drops is still there. Run every minute:=0A> Wed Oct=A0 7 11:00:44 UTC 20= 09=A0 =A0 34630 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:01:44 UTC 2009=A0 =A0 34630 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:02:44 UTC 2009=A0 =A0 34729 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:03:44 UTC 2009=A0 =A0 34= 729 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:04:= 44 UTC 2009=A0 =A0 34861 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:05:44 UTC 2009=A0 =A0 34932 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:06:44 UTC 2009=A0 =A0 35499 output= =0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:07:45 UTC 200= 9=A0 =A0 35780 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:08:45 UTC 2009=A0 =A0 35841 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:09:45 UTC 2009=A0 =A0 36348 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:10:45 UTC 2009=A0 =A0 36= 568 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:11:= 45 UTC 2009=A0 =A0 36673 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:12:45 UTC 2009=A0 =A0 36673 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:13:46 UTC 2009=A0 =A0 36673 output= =0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:14:46 UTC 200= 9=A0 =A0 36673 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct= =A0 7 11:15:46 UTC 2009=A0 =A0 36673 output=0A> packets dropped due to no b= ufs, etc.=0A> Wed Oct=A0 7 11:16:46 UTC 2009=A0 =A0 36849 output=0A> packet= s dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:17:46 UTC 2009=A0 =A0 37= 234 output=0A> packets dropped due to no bufs, etc.=0A> Wed Oct=A0 7 11:18:= 46 UTC 2009=A0 =A0 37949 output=0A> packets dropped due to no bufs, etc.=0A= > Wed Oct=A0 7 11:19:47 UTC 2009=A0 =A0 38043 output=0A> packets dropped du= e to no bufs, etc.=0A> Wed Oct=A0 7 11:20:47 UTC 2009=A0 =A0 38549 output= =0A> packets dropped due to no bufs, etc.=0A> =0A> 2200-2350 users online (= ipfw table load). I'll wait and see=0A> if the drop rate approaches 500-100= 0 per second as the=0A> number of online users comes close to 3-4K.=0A> =0A= > net.isr.direct=3D0=0A=0A=0AIts frightening to me that someone is managing= such a large network=0Awith dummynet. Talk about stealing your customer's = money.=0A=0ABC=0A=0A=0A From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 11:38:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC88F106568B for ; Wed, 7 Oct 2009 11:38:39 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 6A6A68FC0A for ; Wed, 7 Oct 2009 11:38:39 +0000 (UTC) Received: from [217.25.27.27] (port=16102 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MvUqr-000J71-00; Wed, 07 Oct 2009 15:38:37 +0400 Message-ID: <4ACC7DBD.8070007@mail.ru> Date: Wed, 07 Oct 2009 16:38:37 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Barney Cordoba References: <397638.11040.qm@web63902.mail.re1.yahoo.com> In-Reply-To: <397638.11040.qm@web63902.mail.re1.yahoo.com> Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 11:38:39 -0000 > Its frightening to me that someone is managing such a large network > with dummynet. Talk about stealing your customer's money. > We have no customers - we're a charity ISP. Any alternatives? ALTQ? From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 11:54:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B937106566B for ; Wed, 7 Oct 2009 11:54:26 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id 273708FC0A for ; Wed, 7 Oct 2009 11:54:25 +0000 (UTC) Received: by lath.rinet.ru (Postfix, from userid 222) id 18A7E704E; Wed, 7 Oct 2009 15:54:25 +0400 (MSD) Date: Wed, 7 Oct 2009 15:54:25 +0400 From: Oleg Bulyzhin To: rihad Message-ID: <20091007115425.GD88982@lath.rinet.ru> References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACC7308.6070301@mail.ru> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 11:54:26 -0000 On Wed, Oct 07, 2009 at 03:52:56PM +0500, rihad wrote: > You probably have some special sources of documentation ;-) According to > man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the packet > unless one_pass=0. Or do you mean sprinkling smart skiptos here and > there? ;-) you can 1) use ng_ether & ng_netflow. (so no need in 'ngtee' rule). 2) use 'tee' rule with ng_ksocket & ng_netflow > > > Could you show your 'ipfw show' output? (hide ip addresses if you wish but > > keep counters please). > > > Here it is, in its whole glory: > > 00100 10434423 1484891105 allow ip from any to any via lo0 > 00200 2 14 deny ip from any to 127.0.0.0/8 > 00300 1 4 deny ip from 127.0.0.0/8 to any > 01000 3300039938 327603104711 allow ip from any to any in > 01010 26214900 421138433 allow ip from me to any out > 01020 5453857 46806278 allow icmp from any to any out > 01030 3268289053 327224694165 ngtee 1 ip from any to any out > 01040 18681181 1089636054 skipto 1100 ip from table(127) to any out > recv bce0 xmit bce1 > 01060 777488848 76743392754 pipe tablearg ip from any to table(0) out > recv bce0 xmit bce1 > 01070 776831109 76682499457 allow ip from any to table(0) out recv > bce0 xmit bce1 > 01100 13102697 808411842 pipe tablearg ip from any to table(2) out > 65535 662648946 66711487830 allow ip from any to any I guess this one would be better(faster): 00050 allow ip from any to any in 00100 allow ip from any to any via lo0 01010 allow ip from me to any 01020 allow icmp from any to any 01030 ngtee 1 ip from any to any 01035 skipto 1040 ip from any to any recv bce0 xmit bce1 01036 allow ip from any to any 01040 skipto 1100 ip from table(127) to any 01060 pipe tablearg ip from any to table(0) 01070 allow ip from any to any 01100 pipe tablearg ip from any to table(2) 65535 allow ip from any to any P.S. have you tried net.inet.ip.fastforwarding=1? -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 12:10:33 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19272106566B; Wed, 7 Oct 2009 12:10:33 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx34.mail.ru (mx34.mail.ru [94.100.176.48]) by mx1.freebsd.org (Postfix) with ESMTP id C76E28FC1D; Wed, 7 Oct 2009 12:10:32 +0000 (UTC) Received: from [217.25.27.27] (port=15365 helo=[217.25.27.27]) by mx34.mail.ru with asmtp id 1MvVLi-000Q03-00; Wed, 07 Oct 2009 16:10:30 +0400 Message-ID: <4ACC8535.70403@mail.ru> Date: Wed, 07 Oct 2009 17:10:29 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <20091007115425.GD88982@lath.rinet.ru> In-Reply-To: <20091007115425.GD88982@lath.rinet.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 12:10:33 -0000 Oleg Bulyzhin wrote: > On Wed, Oct 07, 2009 at 03:52:56PM +0500, rihad wrote: > >> You probably have some special sources of documentation ;-) According to >> man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the packet >> unless one_pass=0. Or do you mean sprinkling smart skiptos here and >> there? ;-) > > you can > 1) use ng_ether & ng_netflow. (so no need in 'ngtee' rule). > 2) use 'tee' rule with ng_ksocket & ng_netflow > Thanks, I'll see into that if fast-forwarding doesn't help. >>> Could you show your 'ipfw show' output? (hide ip addresses if you wish but >>> keep counters please). >>> > >> Here it is, in its whole glory: >> >> 00100 10434423 1484891105 allow ip from any to any via lo0 >> 00200 2 14 deny ip from any to 127.0.0.0/8 >> 00300 1 4 deny ip from 127.0.0.0/8 to any >> 01000 3300039938 327603104711 allow ip from any to any in >> 01010 26214900 421138433 allow ip from me to any out >> 01020 5453857 46806278 allow icmp from any to any out >> 01030 3268289053 327224694165 ngtee 1 ip from any to any out >> 01040 18681181 1089636054 skipto 1100 ip from table(127) to any out >> recv bce0 xmit bce1 >> 01060 777488848 76743392754 pipe tablearg ip from any to table(0) out >> recv bce0 xmit bce1 >> 01070 776831109 76682499457 allow ip from any to table(0) out recv >> bce0 xmit bce1 >> 01100 13102697 808411842 pipe tablearg ip from any to table(2) out >> 65535 662648946 66711487830 allow ip from any to any > > I guess this one would be better(faster): > > 00050 allow ip from any to any in > 00100 allow ip from any to any via lo0 > 01010 allow ip from me to any > 01020 allow icmp from any to any > 01030 ngtee 1 ip from any to any > 01035 skipto 1040 ip from any to any recv bce0 xmit bce1 > 01036 allow ip from any to any > 01040 skipto 1100 ip from table(127) to any > 01060 pipe tablearg ip from any to table(0) > 01070 allow ip from any to any > 01100 pipe tablearg ip from any to table(2) > 65535 allow ip from any to any > Phew, were "out" that bad? I left them in as commentary. And the localhost anti-spoof check isn't such a bad security ring to get rid of in the name of performance ;-) Ok, got you, I'll take a note of it, thanks. > P.S. have you tried net.inet.ip.fastforwarding=1? > man 4 inet: IPCTL_FASTFORWARDING (ip.fastforwarding) Boolean: enable/disable the use of fast IP forwarding code. Defaults to off. When fast IP forwarding is enabled, IP packets are for- warded directly to the appropriate network inter- face with direct processing to completion, which greatly improves the throughput. All packets for local IP addresses, non-unicast, or with IP options are handled by the normal IP input processing path. All features of the normal (slow) IP forwarding path are supported including firewall (through pfil(9) hooks) checking, except ipsec(4) tunnel brokering. The IP fastforwarding path does not generate ICMP redirect or source quench messages. I'm afraid a bit that it will lock up the live remote system. Is it a drop in replacement given my ipfw rules? Why isn't it enabled by default? From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 12:13:08 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B4001065676; Wed, 7 Oct 2009 12:13:08 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx74.mail.ru (mx74.mail.ru [94.100.176.89]) by mx1.freebsd.org (Postfix) with ESMTP id 556458FC18; Wed, 7 Oct 2009 12:13:07 +0000 (UTC) Received: from [217.25.27.27] (port=46932 helo=[217.25.27.27]) by mx74.mail.ru with asmtp id 1MvVOE-0006TA-00; Wed, 07 Oct 2009 16:13:06 +0400 Message-ID: <4ACC85D1.5070908@mail.ru> Date: Wed, 07 Oct 2009 17:13:05 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <20091007115425.GD88982@lath.rinet.ru> <4ACC8535.70403@mail.ru> In-Reply-To: <4ACC8535.70403@mail.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 12:13:08 -0000 > Why isn't it enabled by default? > Answering myself: probably because of this: The IP fastforwarding path does not generate ICMP redirect or source quench messages. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 12:21:44 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 501DD106568F for ; Wed, 7 Oct 2009 12:21:44 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 10B5B8FC17 for ; Wed, 7 Oct 2009 12:21:44 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id 7620446B03; Wed, 7 Oct 2009 08:21:43 -0400 (EDT) Date: Wed, 7 Oct 2009 13:21:43 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: rihad In-Reply-To: <4ACC65A0.7030900@mail.ru> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 12:21:44 -0000 On Wed, 7 Oct 2009, rihad wrote: > Robert Watson wrote: >> >> On Wed, 7 Oct 2009, rihad wrote: >> >>>> snapshot of the top -SH output in the steady state? Let top run for a >>>> few minutes and then copy/paste the first 10-20 lines into an e-mail. >>>> >>> Sure. Mind you: now there's only 1800 entries in each of the two ipfw >>> tables, so any drops have stopped. But it only takes another 200-300 >>> entries to start dropping. >> >> Could you do the same in the net.isr.direct=1 configuration so we can >> compare? > > net.isr.direct=1: So it seems that CPU exhaustion is likely not the source of drops -- what I was looking for in both configurations were signs that any individual thread was approaching 80% utilization, which in a peak load situation might mean it hitting 100% and therefore leading to packet loss for that reason. The statistic you're monitoring has a couple of interpretations, but the most likely interpretation is overfilling the output queue on the network interface you're transmitting on. In turn there are various possible reasons for this happening, but the two most common would be: (1) Average load is exceeding the transmit capacity of the driver/hardware pipeline -- the pipe is just too small. (2) Peak capacity (burstiness) is exceeding the transmit capacity of the driver/hardware pipeline. The questions that Luigi and others have been asking about your dummynet configuration are to some extent oriented around determining whether the burstiness introduced by dummynet could be responsible for that. Suggestions like increasing timer resolution are intended to spread out the injection of packets by dummynet to attempt to reduce the peaks of burstiness that occur when multiple queues inject packets in a burst that exceeds the queue depth supported by combined hardware descriptor rings and software transmit queue. The two solutions, then are (a) to increase the timer resolution significantly so that packets are injected in smaller bursts and (b) increase the queue capacities. The hardware queue limits likely can't be raised w/o new hardware, but the ifnet transmit queue sizes can be increased. Timer resolution going up is almost certainly not a bad idea in your configuration, although does require a reboot as you have observed. On a side note: one other possible interpretation of that statistic is that you're seeing fragmentation problems. Usually in forwarding scenarios this is unlikely. However, it wouldn't hurt to make sure you have LRO turned off on the network interfaces you're using, assuming it's supported by the driver. Robert N M Watson Computer Laboratory University of Cambridge > > last pid: 92152; load averages: 0.99, 1.18, 1.15 > up 1+01:42:28 14:53:09 > 162 processes: 9 running, 136 sleeping, 17 waiting > CPU: 2.1% user, 0.0% nice, 5.4% system, 7.0% interrupt, 85.5% idle > Mem: 1693M Active, 1429M Inact, 447M Wired, 197M Cache, 214M Buf, 170M Free > Swap: 2048M Total, 12K Used, 2048M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root 171 ki31 0K 16K CPU6 6 24.3H 100.00% idle: cpu6 > 13 root 171 ki31 0K 16K CPU5 5 23.8H 95.95% idle: cpu5 > 14 root 171 ki31 0K 16K CPU4 4 23.4H 93.12% idle: cpu4 > 16 root 171 ki31 0K 16K CPU2 2 23.0H 90.19% idle: cpu2 > 11 root 171 ki31 0K 16K CPU7 7 24.2H 87.26% idle: cpu7 > 15 root 171 ki31 0K 16K CPU3 3 22.8H 86.18% idle: cpu3 > 18 root 171 ki31 0K 16K RUN 0 20.6H 84.96% idle: cpu0 > 17 root 171 ki31 0K 16K CPU1 1 933:23 47.85% idle: cpu1 > 29 root -68 - 0K 16K WAIT 1 522:02 46.88% irq256: bce0 > 465 root -68 - 0K 16K - 7 55:15 12.65% dummynet > 31 root -68 - 0K 16K WAIT 2 57:29 4.74% irq257: bce1 > 21 root -44 - 0K 16K WAIT 0 34:55 4.64% swi1: net > 19 root -32 - 0K 16K WAIT 4 51:41 3.96% swi4: clock > sio > 30 root -64 - 0K 16K WAIT 6 5:43 0.73% irq16: mfi0 > > > Almost 2000 entries in the table, traffic load= 420-430 mbps, drops haven't > yet started. > > Previous net.isr.direct=0: >> >>> >>> 155 processes: 10 running, 129 sleeping, 16 waiting >>> CPU: 2.4% user, 0.0% nice, 2.0% system, 9.3% interrupt, 86.2% idle >>> Mem: 1691M Active, 1491M Inact, 454M Wired, 130M Cache, 214M Buf, 170M >>> Free >>> Swap: 2048M Total, 12K Used, 2048M Free >>> >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>> 15 root 171 ki31 0K 16K CPU3 3 22.4H 97.85% idle: cpu3 >>> 14 root 171 ki31 0K 16K CPU4 4 23.0H 96.29% idle: cpu4 >>> 12 root 171 ki31 0K 16K CPU6 6 23.8H 94.58% idle: cpu6 >>> 16 root 171 ki31 0K 16K CPU2 2 22.5H 90.72% idle: cpu2 >>> 13 root 171 ki31 0K 16K CPU5 5 23.4H 90.58% idle: cpu5 >>> 18 root 171 ki31 0K 16K RUN 0 20.3H 85.60% idle: cpu0 >>> 17 root 171 ki31 0K 16K CPU1 1 910:03 78.37% idle: cpu1 >>> 11 root 171 ki31 0K 16K CPU7 7 23.8H 65.62% idle: cpu7 >>> 21 root -44 - 0K 16K CPU7 7 19:03 48.34% swi1: net >>> 29 root -68 - 0K 16K WAIT 1 515:49 19.63% irq256: bce0 >>> 31 root -68 - 0K 16K WAIT 2 56:05 5.52% irq257: bce1 >>> 19 root -32 - 0K 16K WAIT 5 50:05 3.86% swi4: clock >>> sio >>> 983 flowtools 44 0 12112K 6440K select 0 13:20 0.15% flow-capture >>> 465 root -68 - 0K 16K - 3 51:19 0.00% dummynet >>> 3 root -8 - 0K 16K - 1 7:41 0.00% g_up >>> 4 root -8 - 0K 16K - 2 7:14 0.00% g_down >>> 30 root -64 - 0K 16K WAIT 6 5:30 0.00% irq16: mfi0 >>> >>> >> >> > > From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 12:29:05 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26BB5106568B; Wed, 7 Oct 2009 12:29:05 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx76.mail.ru (mx76.mail.ru [94.100.176.91]) by mx1.freebsd.org (Postfix) with ESMTP id D52C78FC1E; Wed, 7 Oct 2009 12:29:04 +0000 (UTC) Received: from [217.25.27.27] (port=31374 helo=[217.25.27.27]) by mx76.mail.ru with asmtp id 1MvVdd-000PUM-00; Wed, 07 Oct 2009 16:29:01 +0400 Message-ID: <4ACC898C.8000807@mail.ru> Date: Wed, 07 Oct 2009 17:29:00 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <20091007115425.GD88982@lath.rinet.ru> In-Reply-To: <20091007115425.GD88982@lath.rinet.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 12:29:05 -0000 Oleg Bulyzhin wrote: > On Wed, Oct 07, 2009 at 03:52:56PM +0500, rihad wrote: > > P.S. have you tried net.inet.ip.fastforwarding=1? > Yup, it didn't help at all. Reverting it back to 0 for now. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 12:33:28 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5594D10656A4 for ; Wed, 7 Oct 2009 12:33:28 +0000 (UTC) (envelope-from rwatson@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 125A08FC12 for ; Wed, 7 Oct 2009 12:33:28 +0000 (UTC) Received: from [192.168.2.101] (host81-155-13-237.range81-155.btcentralplus.com [81.155.13.237]) by cyrus.watson.org (Postfix) with ESMTPSA id 431A546B03; Wed, 7 Oct 2009 08:33:26 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: "Robert N. M. Watson" In-Reply-To: Date: Wed, 7 Oct 2009 13:33:21 +0100 Content-Transfer-Encoding: 7bit Message-Id: <6693EC21-203E-4E3C-B93B-65C70E8BF128@freebsd.org> References: To: pluknet X-Mailer: Apple Mail (2.1076) Cc: FreeBSD Net Subject: Re: panic in soabort X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 12:33:28 -0000 On 1 Oct 2009, at 12:24, pluknet wrote: >>> If you run into this again, let me know. Also, are you using >>> accept filters >>> on the box? >>> >> >> Got it again (this time on 6.4-p5). > > P.S. > It's funny to say: I got it on two boxes nearly simultaneously. > Both from proftpd. See also my first mail (the same). I'll take a further look at this today sometime, but it is my belief (FYI) that the new socket life cycle in 7.x closes many possible races along these lines. As such, while we can try to track down and resolve this, you might find it simply "goes away" if you move to 7. If you do see it in 7 then I would be especially interested :-). There are two sockets involved here, and I'd like to get some debugging information on both. First, it looks like close() is being called on a listen socket -- could something be killing the proftpd instance on both your boxes at about the same time, perhaps due to shutdown or some admin event? For both sockets, the one passed to soclose and the one passed to soabort, could you provide structure dumps of: (1) *so (2) *(struct inpcb *)(so->so_pcb) (3) Depending on whether INP_TIMEWAIT is set in the inp_flags field, either *(struct tcpcb)((struct inpcb *)(so->so_pcb)->inp_ppcb) or the same cast but struct tcptw if it is in timewait. It seems likely we're dealing with a race condition in which close() on a listen socket aborts a pending connection at the same time as a network event, such as a received reset/fin/etc. Robert > >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 2; apic id = 02 >> fault virtual address = 0x104 >> fault code = supervisor read, page not present >> instruction pointer = 0x20:0xc06a3425 >> stack pointer = 0x28:0xef764bb0 >> frame pointer = 0x28:0xef764bbc >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, def32 1, gran 1 >> processor eflags = resume, IOPL = 0 >> current process = 74303 (proftpd) >> >> db> bt 74303 >> Tracing pid 74303 tid 101039 td 0xcaa08820 >> _mtx_lock_sleep(ccd50768,caa08820,0,0,0) at _mtx_lock_sleep+0x9d >> soabort(ccd506f4) at soabort+0x82 >> soclose(d1aa8b20) at soclose+0x21a >> soo_close(c9f50a20,caa08820) at soo_close+0x63 >> fdrop_locked(c9f50a20,caa08820,caf78a00,ef764ca8,c06875f3,...) at >> fdrop_locked+0xd0 >> fdrop(c9f50a20,caa08820,caa08820,ef764c64,c0689055,...) at fdrop+0x41 >> closef(c9f50a20,caa08820,0,ef764d38,cad8f648,...) at closef+0x42f >> kern_close(caa08820,a,ef764d30,c08e1d4b,caa08820,...) at kern_close >> +0x20d >> close(caa08820,ef764d04) at close+0x10 >> syscall(bfbf003b,3b,bfbf003b,8150034,811a434,...) at syscall+0x2bf >> Xint0x80_syscall() at Xint0x80_syscall+0x1f >> --- syscall (6, FreeBSD ELF32, close), eip = 0x2832230f, esp = >> 0xbfbfe6bc, ebp = 0xbfbfe6d8 --- >> db> show proc 74303 >> Process 74303 (proftpd) at 0xcad8f648: >> state: NORMAL >> uid: 36830 gids: 36830 >> parent: pid 95478 at 0xc8e60000 >> ABI: FreeBSD ELF32 >> arguments: proftpd: fatich_1 - 93.118.217.18: IDLE >> threads: 1 >> 101039 Run CPU 2 proftpd >> >> (gdb) list *(soabort+0x82) >> 0xc06ea2a6 is in soabort (/usr/src/sys/kern/uipc_socket.c:510). >> 505 int error; >> 506 >> 507 error = (*so->so_proto->pr_usrreqs->pru_abort)(so); >> 508 if (error) { >> 509 ACCEPT_LOCK(); >> 510 SOCK_LOCK(so); >> 511 sotryfree(so); /* note: does not decrement >> the ref count */ >> 512 return error; >> 513 } >> 514 return (0); >> >> -- >> wbr, >> pluknet >> > > > > -- > wbr, > pluknet From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 12:42:53 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0ADC1065676; Wed, 7 Oct 2009 12:42:53 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx40.mail.ru (mx40.mail.ru [94.100.176.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4B2998FC19; Wed, 7 Oct 2009 12:42:53 +0000 (UTC) Received: from [217.25.27.27] (port=43022 helo=[217.25.27.27]) by mx40.mail.ru with asmtp id 1MvVr1-0002HP-00; Wed, 07 Oct 2009 16:42:51 +0400 Message-ID: <4ACC8CC8.8050403@mail.ru> Date: Wed, 07 Oct 2009 17:42:48 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 12:42:53 -0000 Robert Watson wrote: > Suggestions like increasing timer resolution are intended to spread out > the injection of packets by dummynet to attempt to reduce the peaks of > burstiness that occur when multiple queues inject packets in a burst > that exceeds the queue depth supported by combined hardware descriptor > rings and software transmit queue. > Raising HZ from 1000 to 2000 has helped. There are now 200-300 global drops/s, as opposed to 300-1000 with HZ=1000. Or maybe net.isr.direct from 1 to 0 help. Or maybe hash_size from 64 to 256. Or maybe... > The two solutions, then are (a) to increase the timer resolution > significantly so that packets are injected in smaller bursts But isn't that bad that it can actually become worse? From /sys/conf/NOTES: # The granularity of operation is controlled by the kernel option HZ whose # default value (1000 on most architectures) means a granularity of 1ms # (1s/HZ). Historically, the default was 100, but finer granularity is # required for DUMMYNET and other systems on modern hardware. There are # reasonable arguments that HZ should, in fact, be 100 still; consider, # that reducing the granularity too much might cause excessive overhead in # clock interrupt processing, potentially causing ticks to be missed and thus # actually reducing the accuracy of operation. > and (b) increase the queue capacities. The hardware queue limits likely can't > be raised w/o new hardware, but the ifnet transmit queue sizes can be > increased. Can someone please say how to increase the "ifnet transmit queue sizes"? > Timer resolution going up is almost certainly not a bad idea in your configuration, although does require a reboot as you have observed. > OK, I'll try HZ=4000, but there are some required servers like flowtools/radius/mysql/perl app that are also running. > On a side note: one other possible interpretation of that statistic is > that you're seeing fragmentation problems. Usually in forwarding > scenarios this is unlikely. However, it wouldn't hurt to make sure you > have LRO turned off on the network interfaces you're using, assuming > it's supported by the driver. > I don't think fragments are the problem. The numbers are too small ;-) $ netstat -s|fgrep fragment 5318 fragments received 147 fragments dropped (dup or out of space) 5157 fragments dropped after timeout 4088 output datagrams fragmented 8180 fragments created 0 datagrams that can't be fragmented There's no such option as LRO shown, so I guess it's off: options=1bb From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 13:20:04 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85B80106568D for ; Wed, 7 Oct 2009 13:20:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5C5458FC0C for ; Wed, 7 Oct 2009 13:20:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n97DK4Jv070064 for ; Wed, 7 Oct 2009 13:20:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n97DK4Ca070063; Wed, 7 Oct 2009 13:20:04 GMT (envelope-from gnats) Date: Wed, 7 Oct 2009 13:20:04 GMT Message-Id: <200910071320.n97DK4Ca070063@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: kern/127587: commit references a PR X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 13:20:04 -0000 The following reply was made to PR kern/127587; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/127587: commit references a PR Date: Wed, 7 Oct 2009 13:12:58 +0000 (UTC) Author: stas Date: Wed Oct 7 13:12:43 2009 New Revision: 197832 URL: http://svn.freebsd.org/changeset/base/197832 Log: - Add support for new BGE chips (5761, 5784 and 57780). These chips uses new BGE_PCI_PRODID_ASICREV register to store the chip identifier and its revision. - Add new grouping macro for 7575+ chips (BGE_IS_5755_PLUS). - Add IDs for Fujitsu-branded Broadcom adapters. PR: kern/127587 Tested by: Thomas Quinot (BCM7561 A0) MFC after: 2 weeks Obtained from: OpenBSD Modified: head/sys/dev/bge/if_bge.c head/sys/dev/bge/if_bgereg.h Modified: head/sys/dev/bge/if_bge.c ============================================================================== --- head/sys/dev/bge/if_bge.c Wed Oct 7 12:38:19 2009 (r197831) +++ head/sys/dev/bge/if_bge.c Wed Oct 7 13:12:43 2009 (r197832) @@ -170,6 +170,7 @@ static const struct bge_type { { BCOM_VENDORID, BCOM_DEVICEID_BCM5720 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5721 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5722 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5723 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5750 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5750M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5751 }, @@ -184,12 +185,21 @@ static const struct bge_type { { BCOM_VENDORID, BCOM_DEVICEID_BCM5754M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5755 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5755M }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761E }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761S }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5761SE }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5764 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5780 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5780S }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5781 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5782 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5784 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5785F }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5785G }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5786 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5787 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM5787F }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5787M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5788 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5789 }, @@ -198,11 +208,19 @@ static const struct bge_type { { BCOM_VENDORID, BCOM_DEVICEID_BCM5903M }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5906 }, { BCOM_VENDORID, BCOM_DEVICEID_BCM5906M }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57760 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57780 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57788 }, + { BCOM_VENDORID, BCOM_DEVICEID_BCM57790 }, { SK_VENDORID, SK_DEVICEID_ALTIMA }, { TC_VENDORID, TC_DEVICEID_3C996 }, + { FJTSU_VENDORID, FJTSU_DEVICEID_PW008GE4 }, + { FJTSU_VENDORID, FJTSU_DEVICEID_PW008GE5 }, + { FJTSU_VENDORID, FJTSU_DEVICEID_PP250450 }, + { 0, 0 } }; @@ -216,6 +234,7 @@ static const struct bge_vendor { { BCOM_VENDORID, "Broadcom" }, { SK_VENDORID, "SysKonnect" }, { TC_VENDORID, "3Com" }, + { FJTSU_VENDORID, "Fujitsu" }, { 0, NULL } }; @@ -271,12 +290,18 @@ static const struct bge_revision { { BGE_CHIPID_BCM5755_A1, "BCM5755 A1" }, { BGE_CHIPID_BCM5755_A2, "BCM5755 A2" }, { BGE_CHIPID_BCM5722_A0, "BCM5722 A0" }, + { BGE_CHIPID_BCM5761_A0, "BCM5761 A0" }, + { BGE_CHIPID_BCM5761_A1, "BCM5761 A1" }, + { BGE_CHIPID_BCM5784_A0, "BCM5784 A0" }, + { BGE_CHIPID_BCM5784_A1, "BCM5784 A1" }, /* 5754 and 5787 share the same ASIC ID */ { BGE_CHIPID_BCM5787_A0, "BCM5754/5787 A0" }, { BGE_CHIPID_BCM5787_A1, "BCM5754/5787 A1" }, { BGE_CHIPID_BCM5787_A2, "BCM5754/5787 A2" }, { BGE_CHIPID_BCM5906_A1, "BCM5906 A1" }, { BGE_CHIPID_BCM5906_A2, "BCM5906 A2" }, + { BGE_CHIPID_BCM57780_A0, "BCM57780 A0" }, + { BGE_CHIPID_BCM57780_A1, "BCM57780 A1" }, { 0, NULL } }; @@ -297,9 +322,13 @@ static const struct bge_revision bge_maj { BGE_ASICREV_BCM5780, "unknown BCM5780" }, { BGE_ASICREV_BCM5714, "unknown BCM5714" }, { BGE_ASICREV_BCM5755, "unknown BCM5755" }, + { BGE_ASICREV_BCM5761, "unknown BCM5761" }, + { BGE_ASICREV_BCM5784, "unknown BCM5784" }, + { BGE_ASICREV_BCM5785, "unknown BCM5785" }, /* 5754 and 5787 share the same ASIC ID */ { BGE_ASICREV_BCM5787, "unknown BCM5754/5787" }, { BGE_ASICREV_BCM5906, "unknown BCM5906" }, + { BGE_ASICREV_BCM57780, "unknown BCM57780" }, { 0, NULL } }; @@ -309,6 +338,7 @@ static const struct bge_revision bge_maj #define BGE_IS_5705_PLUS(sc) ((sc)->bge_flags & BGE_FLAG_5705_PLUS) #define BGE_IS_5714_FAMILY(sc) ((sc)->bge_flags & BGE_FLAG_5714_FAMILY) #define BGE_IS_575X_PLUS(sc) ((sc)->bge_flags & BGE_FLAG_575X_PLUS) +#define BGE_IS_5755_PLUS(sc) ((sc)->bge_flags & BGE_FLAG_5755_PLUS) const struct bge_revision * bge_lookup_rev(uint32_t); const struct bge_vendor * bge_lookup_vendor(uint16_t); @@ -1758,8 +1788,7 @@ bge_blockinit(struct bge_softc *sc) val = BGE_WDMAMODE_ENABLE | BGE_WDMAMODE_ALL_ATTNS; /* Enable host coalescing bug fix. */ - if (sc->bge_asicrev == BGE_ASICREV_BCM5755 || - sc->bge_asicrev == BGE_ASICREV_BCM5787) + if (BGE_IS_5755_PLUS(sc)) val |= 1 << 29; /* Turn on write DMA state machine */ @@ -1768,6 +1797,12 @@ bge_blockinit(struct bge_softc *sc) /* Turn on read DMA state machine */ val = BGE_RDMAMODE_ENABLE | BGE_RDMAMODE_ALL_ATTNS; + if (sc->bge_asicrev == BGE_ASICREV_BCM5784 || + sc->bge_asicrev == BGE_ASICREV_BCM5785 || + sc->bge_asicrev == BGE_ASICREV_BCM57780) + val |= BGE_RDMAMODE_BD_SBD_CRPT_ATTN | + BGE_RDMAMODE_MBUF_RBD_CRPT_ATTN | + BGE_RDMAMODE_MBUF_SBD_CRPT_ATTN; if (sc->bge_flags & BGE_FLAG_PCIE) val |= BGE_RDMAMODE_FIFO_LONG_BURST; CSR_WRITE_4(sc, BGE_RDMA_MODE, val); @@ -1790,7 +1825,10 @@ bge_blockinit(struct bge_softc *sc) CSR_WRITE_4(sc, BGE_SBDC_MODE, BGE_SBDCMODE_ENABLE); /* Turn on send data completion state machine */ - CSR_WRITE_4(sc, BGE_SDC_MODE, BGE_SDCMODE_ENABLE); + val = BGE_SDCMODE_ENABLE; + if (sc->bge_asicrev == BGE_ASICREV_BCM5761) + val |= BGE_SDCMODE_CDELAY; + CSR_WRITE_4(sc, BGE_SDC_MODE, val); /* Turn on send data initiator state machine */ CSR_WRITE_4(sc, BGE_SDI_MODE, BGE_SDIMODE_ENABLE); @@ -1897,8 +1935,11 @@ bge_probe(device_t dev) const struct bge_vendor *v; uint32_t id; - id = pci_read_config(dev, BGE_PCI_MISC_CTL, 4) & - BGE_PCIMISCCTL_ASICREV; + id = pci_read_config(dev, BGE_PCI_MISC_CTL, 4) >> + BGE_PCIMISCCTL_ASICREV_SHIFT; + if (BGE_ASICREV(id) == BGE_ASICREV_USE_PRODID_REG) + id = pci_read_config(dev, + BGE_PCI_PRODID_ASICREV, 4); br = bge_lookup_rev(id); v = bge_lookup_vendor(vid); { @@ -1915,8 +1956,8 @@ bge_probe(device_t dev) br != NULL ? br->br_name : "NetXtreme Ethernet Controller"); } - snprintf(buf, 96, "%s, %sASIC rev. %#04x", model, - br != NULL ? "" : "unknown ", id >> 16); + snprintf(buf, 96, "%s, %sASIC rev. %#08x", model, + br != NULL ? "" : "unknown ", id); device_set_desc_copy(dev, buf); if (pci_get_subvendor(dev) == DELL_VENDORID) sc->bge_flags |= BGE_FLAG_NO_3LED; @@ -2411,8 +2452,11 @@ bge_attach(device_t dev) /* Save various chip information. */ sc->bge_chipid = - pci_read_config(dev, BGE_PCI_MISC_CTL, 4) & - BGE_PCIMISCCTL_ASICREV; + pci_read_config(dev, BGE_PCI_MISC_CTL, 4) >> + BGE_PCIMISCCTL_ASICREV_SHIFT; + if (BGE_ASICREV(sc->bge_chipid) == BGE_ASICREV_USE_PRODID_REG) + sc->bge_chipid = pci_read_config(dev, BGE_PCI_PRODID_ASICREV, + 4); sc->bge_asicrev = BGE_ASICREV(sc->bge_chipid); sc->bge_chiprev = BGE_CHIPREV(sc->bge_chipid); @@ -2431,6 +2475,15 @@ bge_attach(device_t dev) /* Save chipset family. */ switch (sc->bge_asicrev) { + case BGE_ASICREV_BCM5755: + case BGE_ASICREV_BCM5761: + case BGE_ASICREV_BCM5784: + case BGE_ASICREV_BCM5785: + case BGE_ASICREV_BCM5787: + case BGE_ASICREV_BCM57780: + sc->bge_flags |= BGE_FLAG_5755_PLUS | BGE_FLAG_575X_PLUS | + BGE_FLAG_5705_PLUS; + break; case BGE_ASICREV_BCM5700: case BGE_ASICREV_BCM5701: case BGE_ASICREV_BCM5703: @@ -2444,8 +2497,6 @@ bge_attach(device_t dev) /* FALLTHROUGH */ case BGE_ASICREV_BCM5750: case BGE_ASICREV_BCM5752: - case BGE_ASICREV_BCM5755: - case BGE_ASICREV_BCM5787: case BGE_ASICREV_BCM5906: sc->bge_flags |= BGE_FLAG_575X_PLUS; /* FALLTHROUGH */ @@ -2466,6 +2517,8 @@ bge_attach(device_t dev) if (BGE_IS_5705_PLUS(sc) && !(sc->bge_flags & BGE_FLAG_ADJUST_TRIM)) { if (sc->bge_asicrev == BGE_ASICREV_BCM5755 || + sc->bge_asicrev == BGE_ASICREV_BCM5761 || + sc->bge_asicrev == BGE_ASICREV_BCM5784 || sc->bge_asicrev == BGE_ASICREV_BCM5787) { if (sc->bge_chipid != BGE_CHIPID_BCM5722_A0) sc->bge_flags |= BGE_FLAG_JITTER_BUG; @@ -2873,8 +2926,7 @@ bge_reset(struct bge_softc *sc) /* Disable fastboot on controllers that support it. */ if (sc->bge_asicrev == BGE_ASICREV_BCM5752 || - sc->bge_asicrev == BGE_ASICREV_BCM5755 || - sc->bge_asicrev == BGE_ASICREV_BCM5787) { + BGE_IS_5755_PLUS(sc)) { if (bootverbose) device_printf(sc->bge_dev, "Disabling fastboot\n"); CSR_WRITE_4(sc, BGE_FASTBOOT_PC, 0x0); @@ -4689,6 +4741,8 @@ bge_sysctl_debug_info(SYSCTL_HANDLER_ARG } printf("Hardware Flags:\n"); + if (BGE_IS_5755_PLUS(sc)) + printf(" - 5755 Plus\n"); if (BGE_IS_575X_PLUS(sc)) printf(" - 575X Plus\n"); if (BGE_IS_5705_PLUS(sc)) Modified: head/sys/dev/bge/if_bgereg.h ============================================================================== --- head/sys/dev/bge/if_bgereg.h Wed Oct 7 12:38:19 2009 (r197831) +++ head/sys/dev/bge/if_bgereg.h Wed Oct 7 13:12:43 2009 (r197832) @@ -218,6 +218,7 @@ #define BGE_PCI_UNDI_TX_BD_PRODIDX_LO 0xAC #define BGE_PCI_ISR_MBX_HI 0xB0 #define BGE_PCI_ISR_MBX_LO 0xB4 +#define BGE_PCI_PRODID_ASICREV 0xBC /* PCI Misc. Host control register */ #define BGE_PCIMISCCTL_CLEAR_INTA 0x00000001 @@ -229,6 +230,7 @@ #define BGE_PCIMISCCTL_REG_WORDSWAP 0x00000040 #define BGE_PCIMISCCTL_INDIRECT_ACCESS 0x00000080 #define BGE_PCIMISCCTL_ASICREV 0xFFFF0000 +#define BGE_PCIMISCCTL_ASICREV_SHIFT 16 #define BGE_HIF_SWAP_OPTIONS (BGE_PCIMISCCTL_ENDIAN_WORDSWAP) #if BYTE_ORDER == LITTLE_ENDIAN @@ -245,66 +247,72 @@ (BGE_HIF_SWAP_OPTIONS|BGE_PCIMISCCTL_CLEAR_INTA| \ BGE_PCIMISCCTL_MASK_PCI_INTR|BGE_PCIMISCCTL_INDIRECT_ACCESS) -#define BGE_CHIPID_TIGON_I 0x40000000 -#define BGE_CHIPID_TIGON_II 0x60000000 -#define BGE_CHIPID_BCM5700_A0 0x70000000 -#define BGE_CHIPID_BCM5700_A1 0x70010000 -#define BGE_CHIPID_BCM5700_B0 0x71000000 -#define BGE_CHIPID_BCM5700_B1 0x71010000 -#define BGE_CHIPID_BCM5700_B2 0x71020000 -#define BGE_CHIPID_BCM5700_B3 0x71030000 -#define BGE_CHIPID_BCM5700_ALTIMA 0x71040000 -#define BGE_CHIPID_BCM5700_C0 0x72000000 -#define BGE_CHIPID_BCM5701_A0 0x00000000 /* grrrr */ -#define BGE_CHIPID_BCM5701_B0 0x01000000 -#define BGE_CHIPID_BCM5701_B2 0x01020000 -#define BGE_CHIPID_BCM5701_B5 0x01050000 -#define BGE_CHIPID_BCM5703_A0 0x10000000 -#define BGE_CHIPID_BCM5703_A1 0x10010000 -#define BGE_CHIPID_BCM5703_A2 0x10020000 -#define BGE_CHIPID_BCM5703_A3 0x10030000 -#define BGE_CHIPID_BCM5703_B0 0x11000000 -#define BGE_CHIPID_BCM5704_A0 0x20000000 -#define BGE_CHIPID_BCM5704_A1 0x20010000 -#define BGE_CHIPID_BCM5704_A2 0x20020000 -#define BGE_CHIPID_BCM5704_A3 0x20030000 -#define BGE_CHIPID_BCM5704_B0 0x21000000 -#define BGE_CHIPID_BCM5705_A0 0x30000000 -#define BGE_CHIPID_BCM5705_A1 0x30010000 -#define BGE_CHIPID_BCM5705_A2 0x30020000 -#define BGE_CHIPID_BCM5705_A3 0x30030000 -#define BGE_CHIPID_BCM5750_A0 0x40000000 -#define BGE_CHIPID_BCM5750_A1 0x40010000 -#define BGE_CHIPID_BCM5750_A3 0x40030000 -#define BGE_CHIPID_BCM5750_B0 0x41000000 -#define BGE_CHIPID_BCM5750_B1 0x41010000 -#define BGE_CHIPID_BCM5750_C0 0x42000000 -#define BGE_CHIPID_BCM5750_C1 0x42010000 -#define BGE_CHIPID_BCM5750_C2 0x42020000 -#define BGE_CHIPID_BCM5714_A0 0x50000000 -#define BGE_CHIPID_BCM5752_A0 0x60000000 -#define BGE_CHIPID_BCM5752_A1 0x60010000 -#define BGE_CHIPID_BCM5752_A2 0x60020000 -#define BGE_CHIPID_BCM5714_B0 0x80000000 -#define BGE_CHIPID_BCM5714_B3 0x80030000 -#define BGE_CHIPID_BCM5715_A0 0x90000000 -#define BGE_CHIPID_BCM5715_A1 0x90010000 -#define BGE_CHIPID_BCM5715_A3 0x90030000 -#define BGE_CHIPID_BCM5755_A0 0xa0000000 -#define BGE_CHIPID_BCM5755_A1 0xa0010000 -#define BGE_CHIPID_BCM5755_A2 0xa0020000 -#define BGE_CHIPID_BCM5722_A0 0xa2000000 -#define BGE_CHIPID_BCM5754_A0 0xb0000000 -#define BGE_CHIPID_BCM5754_A1 0xb0010000 -#define BGE_CHIPID_BCM5754_A2 0xb0020000 -#define BGE_CHIPID_BCM5787_A0 0xb0000000 -#define BGE_CHIPID_BCM5787_A1 0xb0010000 -#define BGE_CHIPID_BCM5787_A2 0xb0020000 -#define BGE_CHIPID_BCM5906_A1 0xc0010000 -#define BGE_CHIPID_BCM5906_A2 0xc0020000 +#define BGE_CHIPID_TIGON_I 0x4000 +#define BGE_CHIPID_TIGON_II 0x6000 +#define BGE_CHIPID_BCM5700_A0 0x7000 +#define BGE_CHIPID_BCM5700_A1 0x7001 +#define BGE_CHIPID_BCM5700_B0 0x7100 +#define BGE_CHIPID_BCM5700_B1 0x7101 +#define BGE_CHIPID_BCM5700_B2 0x7102 +#define BGE_CHIPID_BCM5700_B3 0x7103 +#define BGE_CHIPID_BCM5700_ALTIMA 0x7104 +#define BGE_CHIPID_BCM5700_C0 0x7200 +#define BGE_CHIPID_BCM5701_A0 0x0000 /* grrrr */ +#define BGE_CHIPID_BCM5701_B0 0x0100 +#define BGE_CHIPID_BCM5701_B2 0x0102 +#define BGE_CHIPID_BCM5701_B5 0x0105 +#define BGE_CHIPID_BCM5703_A0 0x1000 +#define BGE_CHIPID_BCM5703_A1 0x1001 +#define BGE_CHIPID_BCM5703_A2 0x1002 +#define BGE_CHIPID_BCM5703_A3 0x1003 +#define BGE_CHIPID_BCM5703_B0 0x1100 +#define BGE_CHIPID_BCM5704_A0 0x2000 +#define BGE_CHIPID_BCM5704_A1 0x2001 +#define BGE_CHIPID_BCM5704_A2 0x2002 +#define BGE_CHIPID_BCM5704_A3 0x2003 +#define BGE_CHIPID_BCM5704_B0 0x2100 +#define BGE_CHIPID_BCM5705_A0 0x3000 +#define BGE_CHIPID_BCM5705_A1 0x3001 +#define BGE_CHIPID_BCM5705_A2 0x3002 +#define BGE_CHIPID_BCM5705_A3 0x3003 +#define BGE_CHIPID_BCM5750_A0 0x4000 +#define BGE_CHIPID_BCM5750_A1 0x4001 +#define BGE_CHIPID_BCM5750_A3 0x4000 +#define BGE_CHIPID_BCM5750_B0 0x4100 +#define BGE_CHIPID_BCM5750_B1 0x4101 +#define BGE_CHIPID_BCM5750_C0 0x4200 +#define BGE_CHIPID_BCM5750_C1 0x4201 +#define BGE_CHIPID_BCM5750_C2 0x4202 +#define BGE_CHIPID_BCM5714_A0 0x5000 +#define BGE_CHIPID_BCM5752_A0 0x6000 +#define BGE_CHIPID_BCM5752_A1 0x6001 +#define BGE_CHIPID_BCM5752_A2 0x6002 +#define BGE_CHIPID_BCM5714_B0 0x8000 +#define BGE_CHIPID_BCM5714_B3 0x8003 +#define BGE_CHIPID_BCM5715_A0 0x9000 +#define BGE_CHIPID_BCM5715_A1 0x9001 +#define BGE_CHIPID_BCM5715_A3 0x9003 +#define BGE_CHIPID_BCM5755_A0 0xa000 +#define BGE_CHIPID_BCM5755_A1 0xa001 +#define BGE_CHIPID_BCM5755_A2 0xa002 +#define BGE_CHIPID_BCM5722_A0 0xa200 +#define BGE_CHIPID_BCM5754_A0 0xb000 +#define BGE_CHIPID_BCM5754_A1 0xb001 +#define BGE_CHIPID_BCM5754_A2 0xb002 +#define BGE_CHIPID_BCM5761_A0 0x5761000 +#define BGE_CHIPID_BCM5761_A1 0x5761100 +#define BGE_CHIPID_BCM5784_A0 0x5784000 +#define BGE_CHIPID_BCM5784_A1 0x5784100 +#define BGE_CHIPID_BCM5787_A0 0xb000 +#define BGE_CHIPID_BCM5787_A1 0xb001 +#define BGE_CHIPID_BCM5787_A2 0xb002 +#define BGE_CHIPID_BCM5906_A1 0xc001 +#define BGE_CHIPID_BCM5906_A2 0xc002 +#define BGE_CHIPID_BCM57780_A0 0x57780000 +#define BGE_CHIPID_BCM57780_A1 0x57780001 /* shorthand one */ -#define BGE_ASICREV(x) ((x) >> 28) +#define BGE_ASICREV(x) ((x) >> 12) #define BGE_ASICREV_BCM5701 0x00 #define BGE_ASICREV_BCM5703 0x01 #define BGE_ASICREV_BCM5704 0x02 @@ -319,9 +327,16 @@ #define BGE_ASICREV_BCM5754 0x0b #define BGE_ASICREV_BCM5787 0x0b #define BGE_ASICREV_BCM5906 0x0c +/* Should consult BGE_PCI_PRODID_ASICREV for ChipID */ +#define BGE_ASICREV_USE_PRODID_REG 0x0f +/* BGE_PCI_PRODID_ASICREV ASIC rev. identifiers. */ +#define BGE_ASICREV_BCM5761 0x5761 +#define BGE_ASICREV_BCM5784 0x5784 +#define BGE_ASICREV_BCM5785 0x5785 +#define BGE_ASICREV_BCM57780 0x57780 /* chip revisions */ -#define BGE_CHIPREV(x) ((x) >> 24) +#define BGE_CHIPREV(x) ((x) >> 8) #define BGE_CHIPREV_5700_AX 0x70 #define BGE_CHIPREV_5700_BX 0x71 #define BGE_CHIPREV_5700_CX 0x72 @@ -331,6 +346,9 @@ #define BGE_CHIPREV_5704_BX 0x21 #define BGE_CHIPREV_5750_AX 0x40 #define BGE_CHIPREV_5750_BX 0x41 +/* BGE_PCI_PRODID_ASICREV chip rev. identifiers. */ +#define BGE_CHIPREV_5761_AX 0x57611 +#define BGE_CHIPREV_5784_AX 0x57841 /* PCI DMA Read/Write Control register */ #define BGE_PCIDMARWCTL_MINDMA 0x000000FF @@ -861,6 +879,7 @@ #define BGE_SDCMODE_RESET 0x00000001 #define BGE_SDCMODE_ENABLE 0x00000002 #define BGE_SDCMODE_ATTN 0x00000004 +#define BGE_SDCMODE_CDELAY 0x00000010 /* Send Data completion status register */ #define BGE_SDCSTAT_ATTN 0x00000004 @@ -1378,6 +1397,9 @@ #define BGE_RDMAMODE_PCI_FIFOOREAD_ATTN 0x00000100 #define BGE_RDMAMODE_LOCWRITE_TOOBIG 0x00000200 #define BGE_RDMAMODE_ALL_ATTNS 0x000003FC +#define BGE_RDMAMODE_BD_SBD_CRPT_ATTN 0x00000800 +#define BGE_RDMAMODE_MBUF_RBD_CRPT_ATTN 0x00001000 +#define BGE_RDMAMODE_MBUF_SBD_CRPT_ATTN 0x00002000 #define BGE_RDMAMODE_FIFO_SIZE_128 0x00020000 #define BGE_RDMAMODE_FIFO_LONG_BURST 0x00030000 @@ -2101,6 +2123,7 @@ struct bge_status_block { #define BCOM_DEVICEID_BCM5720 0x1658 #define BCOM_DEVICEID_BCM5721 0x1659 #define BCOM_DEVICEID_BCM5722 0x165A +#define BCOM_DEVICEID_BCM5723 0x165B #define BCOM_DEVICEID_BCM5750 0x1676 #define BCOM_DEVICEID_BCM5750M 0x167C #define BCOM_DEVICEID_BCM5751 0x1677 @@ -2115,13 +2138,22 @@ struct bge_status_block { #define BCOM_DEVICEID_BCM5754M 0x1672 #define BCOM_DEVICEID_BCM5755 0x167B #define BCOM_DEVICEID_BCM5755M 0x1673 +#define BCOM_DEVICEID_BCM5761 0x1681 +#define BCOM_DEVICEID_BCM5761E 0x1680 +#define BCOM_DEVICEID_BCM5761S 0x1688 +#define BCOM_DEVICEID_BCM5761SE 0x1689 +#define BCOM_DEVICEID_BCM5764 0x1684 #define BCOM_DEVICEID_BCM5780 0x166A #define BCOM_DEVICEID_BCM5780S 0x166B #define BCOM_DEVICEID_BCM5781 0x16DD #define BCOM_DEVICEID_BCM5782 0x1696 +#define BCOM_DEVICEID_BCM5784 0x1698 +#define BCOM_DEVICEID_BCM5785F 0x16a0 +#define BCOM_DEVICEID_BCM5785G 0x1699 #define BCOM_DEVICEID_BCM5786 0x169A #define BCOM_DEVICEID_BCM5787 0x169B #define BCOM_DEVICEID_BCM5787M 0x1693 +#define BCOM_DEVICEID_BCM5787F 0x167f #define BCOM_DEVICEID_BCM5788 0x169C #define BCOM_DEVICEID_BCM5789 0x169D #define BCOM_DEVICEID_BCM5901 0x170D @@ -2129,6 +2161,10 @@ struct bge_status_block { #define BCOM_DEVICEID_BCM5903M 0x16FF #define BCOM_DEVICEID_BCM5906 0x1712 #define BCOM_DEVICEID_BCM5906M 0x1713 +#define BCOM_DEVICEID_BCM57760 0x1690 +#define BCOM_DEVICEID_BCM57780 0x1692 +#define BCOM_DEVICEID_BCM57788 0x1691 +#define BCOM_DEVICEID_BCM57790 0x1694 /* * Alteon AceNIC PCI vendor/device ID. @@ -2179,6 +2215,14 @@ struct bge_status_block { #define SUN_VENDORID 0x108e /* + * Fujitsu vendor/device IDs + */ +#define FJTSU_VENDORID 0x10cf +#define FJTSU_DEVICEID_PW008GE5 0x11a1 +#define FJTSU_DEVICEID_PW008GE4 0x11a2 +#define FJTSU_DEVICEID_PP250450 0x11cc /* PRIMEPOWER250/450 LAN */ + +/* * Offset of MAC address inside EEPROM. */ #define BGE_EE_MAC_OFFSET 0x7C @@ -2558,6 +2602,7 @@ struct bge_softc { #define BGE_FLAG_5705_PLUS 0x00002000 #define BGE_FLAG_5714_FAMILY 0x00004000 #define BGE_FLAG_575X_PLUS 0x00008000 +#define BGE_FLAG_5755_PLUS 0x00010000 #define BGE_FLAG_RX_ALIGNBUG 0x00100000 #define BGE_FLAG_NO_3LED 0x00200000 #define BGE_FLAG_ADC_BUG 0x00400000 @@ -2568,8 +2613,8 @@ struct bge_softc { #define BGE_FLAG_CRC_BUG 0x08000000 #define BGE_FLAG_5788 0x20000000 uint32_t bge_chipid; - uint8_t bge_asicrev; - uint8_t bge_chiprev; + uint32_t bge_asicrev; + uint32_t bge_chiprev; uint8_t bge_asf_mode; uint8_t bge_asf_count; struct bge_ring_data bge_ldata; /* rings */ _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 13:40:20 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C21E41065670 for ; Wed, 7 Oct 2009 13:40:20 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 8F5BB8FC17 for ; Wed, 7 Oct 2009 13:40:20 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id 1F49B46B2C; Wed, 7 Oct 2009 09:40:20 -0400 (EDT) Date: Wed, 7 Oct 2009 14:40:20 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: rihad In-Reply-To: <4ACC8CC8.8050403@mail.ru> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 13:40:20 -0000 On Wed, 7 Oct 2009, rihad wrote: >> Suggestions like increasing timer resolution are intended to spread out the >> injection of packets by dummynet to attempt to reduce the peaks of >> burstiness that occur when multiple queues inject packets in a burst that >> exceeds the queue depth supported by combined hardware descriptor rings and >> software transmit queue. > > Raising HZ from 1000 to 2000 has helped. There are now 200-300 global > drops/s, as opposed to 300-1000 with HZ=1000. Or maybe net.isr.direct from 1 > to 0 help. Or maybe hash_size from 64 to 256. Or maybe... Or maybe other random factors such as traffic load corresponding to major sports events, etc. :-) It's also possible that combining multiple changes cancels out the effect of one or another change. Given the rather large number of possible combinations of things to try, I'd suggest being fairly strategic in how you try them. Starting with just an original config + significant HZ increase is probably the best starting point. Changing hash_size is really about reducing CPU use, so if in the whole you're not getting close to the capacity of a core for any given thread involved in the work, it may not make much difference (tuning these data structures is a bit of a black art). >> The two solutions, then are (a) to increase the timer resolution >> significantly so that packets are injected in smaller bursts > > But isn't that bad that it can actually become worse? From /sys/conf/NOTES: > > # The granularity of operation is controlled by the kernel option HZ whose > # default value (1000 on most architectures) means a granularity of 1ms > # (1s/HZ). Historically, the default was 100, but finer granularity is > # required for DUMMYNET and other systems on modern hardware. There are > # reasonable arguments that HZ should, in fact, be 100 still; consider, > # that reducing the granularity too much might cause excessive overhead in > # clock interrupt processing, potentially causing ticks to be missed and thus > # actually reducing the accuracy of operation. Right: we fire the timer on every CPU at 1/HZ seconds, which means quite a lot of work being done. On systems where timers are proportionally more expensive -- especially when using hardware virtualization, for example, we do recommend tuning the timers down. And our boot loader will actually do it for you: we auto-detect vmware, parallels, kqemu, virtualbox, etc, and adjust the timer rate from from 1000 to 100 during the boot. That said, in your configuration I see little argument for a lower timer rate: you need to burst packets at frequent intervals or risk overfilling queues, and the overheads of additional timer tickets on your system shouldn't be too bad as you have both very fast hardware and a lot of idle time. I would suggest making just the HZ -> 4000 change for now and see how it goes. >> and (b) increase the queue capacities. The hardware queue limits likely >> can't be raised w/o new hardware, but the ifnet transmit queue sizes can be >> increased. > > Can someone please say how to increase the "ifnet transmit queue sizes"? Unfortunately, I fear that this is driver-specific, and in the case of bce requires a recompile. In the driver init code in if_bce, the following code appears: ifp->if_snd.ifq_drv_maxlen = USABLE_TX_BD; IFQ_SET_MAXLEN(&ifp->if_snd, ifp->if_snd.ifq_drv_maxlen); IFQ_SET_READY(&ifp->if_snd); Which evaluates to a architecture-specific value due to varying pagesize. You might just try forcing it to 1024. >> Timer resolution going up is almost certainly not a bad idea in your >> configuration, although does require a reboot as you have observed. > > OK, I'll try HZ=4000, but there are some required servers like > flowtools/radius/mysql/perl app that are also running. That should be fine. >> On a side note: one other possible interpretation of that statistic is that >> you're seeing fragmentation problems. Usually in forwarding scenarios this >> is unlikely. However, it wouldn't hurt to make sure you have LRO turned >> off on the network interfaces you're using, assuming it's supported by the >> driver. >> > I don't think fragments are the problem. The numbers are too small ;-) > $ netstat -s|fgrep fragment > 5318 fragments received > 147 fragments dropped (dup or out of space) > 5157 fragments dropped after timeout > 4088 output datagrams fragmented > 8180 fragments created > 0 datagrams that can't be fragmented > > There's no such option as LRO shown, so I guess it's off: > options=1bb That probably rules that out as a source of problems then. Robert From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 14:43:59 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63103106568D; Wed, 7 Oct 2009 14:43:59 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id 13D068FC1D; Wed, 7 Oct 2009 14:43:58 +0000 (UTC) Received: from [217.25.27.27] (port=44527 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1MvXkD-0006zZ-00; Wed, 07 Oct 2009 18:43:57 +0400 Message-ID: <4ACCA92A.9070803@mail.ru> Date: Wed, 07 Oct 2009 19:43:54 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 14:43:59 -0000 Robert Watson wrote: > I would suggest making just the HZ -> 4000 change for now and see how it > goes. > OK, I will try testing HZ=4000 tomorrow morning, although I'm pretty sure there still will be some drops. >> Can someone please say how to increase the "ifnet transmit queue sizes"? > > Unfortunately, I fear that this is driver-specific, and in the case of > bce requires a recompile. In the driver init code in if_bce, the > following code appears: > > ifp->if_snd.ifq_drv_maxlen = USABLE_TX_BD; > IFQ_SET_MAXLEN(&ifp->if_snd, ifp->if_snd.ifq_drv_maxlen); > IFQ_SET_READY(&ifp->if_snd); > > Which evaluates to a architecture-specific value due to varying > pagesize. You might just try forcing it to 1024. > I think I'll try this too, if HZ=4000 doesn't help, thanks a lot. In the long run we'll switch to some high-quality 10 GiGE cards. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 14:49:25 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7C38106568F for ; Wed, 7 Oct 2009 14:49:24 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id E1CB98FC08 for ; Wed, 7 Oct 2009 14:49:23 +0000 (UTC) Received: (qmail 7973 invoked from network); 7 Oct 2009 16:49:21 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 7 Oct 2009 16:49:21 +0200 Date: Wed, 7 Oct 2009 16:49:21 +0200 (CEST) From: Ingo Flaschberger To: rihad In-Reply-To: <4ACCA92A.9070803@mail.ru> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> User-Agent: Alpine 2.00 (LFD 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 14:49:25 -0000 Hi, can you send me the dmesg ouput from your networkcards when they are detected at booting? can you also send me a lspci and lspci -v ? Kind regards, ingo flaschberger From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 14:58:21 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06D031065676; Wed, 7 Oct 2009 14:58:21 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx40.mail.ru (mx40.mail.ru [94.100.176.54]) by mx1.freebsd.org (Postfix) with ESMTP id AD12B8FC1A; Wed, 7 Oct 2009 14:58:20 +0000 (UTC) Received: from [217.25.27.27] (port=57275 helo=[217.25.27.27]) by mx40.mail.ru with asmtp id 1MvXy7-0004gZ-00; Wed, 07 Oct 2009 18:58:19 +0400 Message-ID: <4ACCAC89.1040805@mail.ru> Date: Wed, 07 Oct 2009 19:58:17 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 14:58:21 -0000 Robert Watson wrote: > In the driver init code in if_bce, the following code appears: > > ifp->if_snd.ifq_drv_maxlen = USABLE_TX_BD; > IFQ_SET_MAXLEN(&ifp->if_snd, ifp->if_snd.ifq_drv_maxlen); > IFQ_SET_READY(&ifp->if_snd); > > Which evaluates to a architecture-specific value due to varying > pagesize. You might just try forcing it to 1024. > In dev/bce/if_bcereg.h: #define TX_PAGES 2 #define TOTAL_TX_BD_PER_PAGE (BCM_PAGE_SIZE / sizeof(struct tx_bd)) #define USABLE_TX_BD_PER_PAGE (TOTAL_TX_BD_PER_PAGE - 1) #define TOTAL_TX_BD (TOTAL_TX_BD_PER_PAGE * TX_PAGES) #define USABLE_TX_BD (USABLE_TX_BD_PER_PAGE * TX_PAGES) #define MAX_TX_BD (TOTAL_TX_BD - 1) meaning that ifq_drv_maxlen is expected to end up smaller than MAX_TX_BD. What if MAX_TX_BD is itself way smaller than 1024? From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 15:02:24 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 924591065676 for ; Wed, 7 Oct 2009 15:02:24 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id 449E98FC08 for ; Wed, 7 Oct 2009 15:02:24 +0000 (UTC) Received: from [217.25.27.27] (port=61051 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1MvY22-000ApD-00; Wed, 07 Oct 2009 19:02:22 +0400 Message-ID: <4ACCAD7D.9080207@mail.ru> Date: Wed, 07 Oct 2009 20:02:21 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Ingo Flaschberger References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 15:02:24 -0000 Ingo Flaschberger wrote: > Hi, > > can you send me the dmesg ouput from your networkcards when they are > detected at booting? > Hello, bce0: mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7 bce0: Ethernet address: 00:1d:09:xx:xx:xx bce0: [ITHREAD] bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x03050C05); Flags( MFW MSI ) bce1: mem 0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3 bce1: Ethernet address: 00:1d:09:xx:xx:xx bce1: [ITHREAD] bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x03050C05); Flags( MFW MSI ) > can you also send me a lspci and lspci -v ? > Sorry, this is FreeBSD, not Linux ;-) # pciconf -l ... bce0@pci0:7:0:0: class=0x020000 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 bce1@pci0:3:0:0: class=0x020000 card=0x01b21028 chip=0x164c14e4 rev=0x12 hdr=0x00 From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 15:28:18 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8043106566B for ; Wed, 7 Oct 2009 15:28:18 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 21AC78FC15 for ; Wed, 7 Oct 2009 15:28:17 +0000 (UTC) Received: (qmail 1498 invoked from network); 7 Oct 2009 17:28:16 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 7 Oct 2009 17:28:16 +0200 Date: Wed, 7 Oct 2009 17:28:15 +0200 (CEST) From: Ingo Flaschberger To: rihad In-Reply-To: <4ACCAD7D.9080207@mail.ru> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> <4ACCAD7D.9080207@mail.ru> User-Agent: Alpine 2.00 (LFD 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 15:28:18 -0000 Dear Rihad, >> can you also send me a lspci and lspci -v ? >> > Sorry, this is FreeBSD, not Linux ;-) you find a lspci in ports. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 15:41:52 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A1911065692 for ; Wed, 7 Oct 2009 15:41:52 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 577458FC2C for ; Wed, 7 Oct 2009 15:41:50 +0000 (UTC) Received: (qmail 22405 invoked from network); 7 Oct 2009 17:41:49 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 7 Oct 2009 17:41:49 +0200 Date: Wed, 7 Oct 2009 17:41:48 +0200 (CEST) From: Ingo Flaschberger To: rihad In-Reply-To: <4ACCAD7D.9080207@mail.ru> Message-ID: References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> <4ACCAD7D.9080207@mail.ru> User-Agent: Alpine 2.00 (LFD 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 15:41:52 -0000 Dear Rihad, bge network card seems to have small tx/rx rings? If I understood the src/sys/dev/bge/if_bgereg.h correct, the ring size is 512 descriptors, while intel based cards (em) have up to 4096 descriptors. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 15:50:58 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F218C106566B for ; Wed, 7 Oct 2009 15:50:58 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id A303C8FC1F for ; Wed, 7 Oct 2009 15:50:58 +0000 (UTC) Received: from [217.25.27.27] (port=4583 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1MvYn3-000Nd7-00; Wed, 07 Oct 2009 19:50:57 +0400 Message-ID: <4ACCB8DD.5040205@mail.ru> Date: Wed, 07 Oct 2009 20:50:53 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Ingo Flaschberger References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> <4ACCAD7D.9080207@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 15:50:59 -0000 Ingo Flaschberger wrote: > Dear Rihad, > > bge network card seems to have small tx/rx rings? > If I understood the src/sys/dev/bge/if_bgereg.h correct, the > ring size is 512 descriptors, while intel based cards > (em) have up to 4096 descriptors. > We have bce, not bge. I'm gonna try HZ=4000 tomorrow and see if that helps. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 16:34:20 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 011971065672 for ; Wed, 7 Oct 2009 16:34:20 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outL.internet-mail-service.net (outl.internet-mail-service.net [216.240.47.235]) by mx1.freebsd.org (Postfix) with ESMTP id DB4808FC18 for ; Wed, 7 Oct 2009 16:34:19 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id F3A8BB98A9; Wed, 7 Oct 2009 09:34:19 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 4D46C2D601C; Wed, 7 Oct 2009 09:34:19 -0700 (PDT) Message-ID: <4ACCC30E.7080504@elischer.org> Date: Wed, 07 Oct 2009 09:34:22 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> In-Reply-To: <4ACC7308.6070301@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Oleg Bulyzhin Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 16:34:20 -0000 rihad wrote: > Oleg Bulyzhin wrote: > You probably have some special sources of documentation ;-) According to > man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the packet > unless one_pass=0. Or do you mean sprinkling smart skiptos here and > there? ;-) > ngtee should not have any affect on the packet.. it takes a copy.. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 16:42:33 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B92E10656A5; Wed, 7 Oct 2009 16:42:33 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 15B0F8FC17; Wed, 7 Oct 2009 16:42:32 +0000 (UTC) Received: from [217.25.27.27] (port=42561 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MvZax-0008MC-00; Wed, 07 Oct 2009 20:42:31 +0400 Message-ID: <4ACCC4F3.3030302@mail.ru> Date: Wed, 07 Oct 2009 21:42:27 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <4ACCC30E.7080504@elischer.org> In-Reply-To: <4ACCC30E.7080504@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Oleg Bulyzhin Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 16:42:33 -0000 Julian Elischer wrote: > rihad wrote: >> Oleg Bulyzhin wrote: > >> You probably have some special sources of documentation ;-) According >> to man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the >> packet unless one_pass=0. Or do you mean sprinkling smart skiptos here >> and there? ;-) >> > > ngtee should not have any affect on the packet.. it takes a copy.. > That's a logical conclusion, although I prefer trusting the man at hand (pun intended) if I haven't tested it myself to see how it works: ngtee cookie A copy of packet is diverted into netgraph, original packet is either accepted or continues with the next rule, depending on net.inet.ip.fw.one_pass sysctl variable. See ng_ipfw(4) for more information on netgraph and ngtee actions. Although... I've a question to Mr. Oleg: > 2) use 'tee' rule with ng_ksocket & ng_netflow tee port Send a copy of packets matching this rule to the divert(4) socket bound to port port. The search continues with the next rule. how is it different from one_pass=0? Both tee and ngtee w/ one_pass=0 continue with the next rule. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 16:49:54 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86B8F1065696 for ; Wed, 7 Oct 2009 16:49:54 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outI.internet-mail-service.net (outi.internet-mail-service.net [216.240.47.232]) by mx1.freebsd.org (Postfix) with ESMTP id 5F9638FC15 for ; Wed, 7 Oct 2009 16:49:54 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id B9C06B98A1; Wed, 7 Oct 2009 09:49:54 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 324F82D601E; Wed, 7 Oct 2009 09:49:53 -0700 (PDT) Message-ID: <4ACCC6B3.3020903@elischer.org> Date: Wed, 07 Oct 2009 09:49:55 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> In-Reply-To: <4ACCA92A.9070803@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Eugene Grosbein , Robert Watson , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 16:49:54 -0000 rihad wrote: > Robert Watson wrote: > >> I would suggest making just the HZ -> 4000 change for now and see how >> it goes. >> > OK, I will try testing HZ=4000 tomorrow morning, although I'm pretty > sure there still will be some drops. > >>> Can someone please say how to increase the "ifnet transmit queue sizes"? >> >> Unfortunately, I fear that this is driver-specific, and in the case of >> bce requires a recompile. In the driver init code in if_bce, the >> following code appears: >> >> ifp->if_snd.ifq_drv_maxlen = USABLE_TX_BD; >> IFQ_SET_MAXLEN(&ifp->if_snd, ifp->if_snd.ifq_drv_maxlen); >> IFQ_SET_READY(&ifp->if_snd); >> >> Which evaluates to a architecture-specific value due to varying >> pagesize. You might just try forcing it to 1024. >> > I think I'll try this too, if HZ=4000 doesn't help, thanks a lot. > In the long run we'll switch to some high-quality 10 GiGE cards. for output.. It may still be possible for the output queues to be overrun as teh bursts from dummynet can be much faster than even 10G hardware can dispose of the data. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 17:50:04 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90D291065676 for ; Wed, 7 Oct 2009 17:50:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6563A8FC18 for ; Wed, 7 Oct 2009 17:50:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n97Ho4qP007467 for ; Wed, 7 Oct 2009 17:50:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n97Ho3Ja007466; Wed, 7 Oct 2009 17:50:04 GMT (envelope-from gnats) Date: Wed, 7 Oct 2009 17:50:04 GMT Message-Id: <200910071750.n97Ho3Ja007466@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Gleb Kurtsou Cc: Subject: Re: bin/139346: [patch] arp(8) add option to remove static entries listed in file X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Gleb Kurtsou List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 17:50:04 -0000 The following reply was made to PR bin/139346; it has been noted by GNATS. From: Gleb Kurtsou To: gavin@FreeBSD.org, bug-followup@FreeBSD.org Cc: Subject: Re: bin/139346: [patch] arp(8) add option to remove static entries listed in file Date: Wed, 7 Oct 2009 20:40:53 +0300 --zhXaljGHf11kAtnf Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Updated patch attached: includes man page update. --zhXaljGHf11kAtnf Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="arp-df-2.path.txt" diff --git a/usr.sbin/arp/arp.8 b/usr.sbin/arp/arp.8 index d7306d3..683ec42 100644 --- a/usr.sbin/arp/arp.8 +++ b/usr.sbin/arp/arp.8 @@ -51,6 +51,8 @@ .Op Fl i Ar interface .Fl a .Nm +.Fl d Fl f Ar filename +.Nm .Fl s Ar hostname ether_addr .Op Cm temp .Op Cm blackhole No \&| Cm reject @@ -99,7 +101,9 @@ Alternatively, the .Fl d flag may be combined with the .Fl a -flag to delete all entries. +flag to delete all entries or with +.Fl f +flag to delete all entries listed in file. .It Fl i Ar interface Limit the operation scope to the .Tn ARP @@ -174,6 +178,11 @@ Cause the file to be read and multiple entries to be set in the .Tn ARP tables. +Combining with +.Fl d +flag may be used to delete entries from the +.Tn ARP +tables. Entries in the file should be of the form .Pp diff --git a/usr.sbin/arp/arp.c b/usr.sbin/arp/arp.c index 8a3410f..11d1df6 100644 --- a/usr.sbin/arp/arp.c +++ b/usr.sbin/arp/arp.c @@ -101,7 +101,7 @@ static int valid_type(int type); static int nflag; /* no reverse dns lookups */ static char *rifname; -static int expire_time, flags, doing_proxy, proxy_only; +static int expire_time, flags, func, doing_proxy, proxy_only; /* which function we're supposed to do */ #define F_GET 1 @@ -109,23 +109,28 @@ static int expire_time, flags, doing_proxy, proxy_only; #define F_FILESET 3 #define F_REPLACE 4 #define F_DELETE 5 +#define F_FILEDELETE 6 #define SETFUNC(f) { if (func) usage(); func = (f); } int main(int argc, char *argv[]) { - int ch, func = 0; + int ch; int rtn = 0; int aflag = 0; /* do it for all entries */ + func = 0; while ((ch = getopt(argc, argv, "andfsSi:")) != -1) switch(ch) { case 'a': aflag = 1; break; case 'd': - SETFUNC(F_DELETE); + if (func == F_FILESET) + func = F_FILEDELETE; + else + SETFUNC(F_DELETE); break; case 'n': nflag = 1; @@ -137,7 +142,10 @@ main(int argc, char *argv[]) SETFUNC(F_SET); break; case 'f' : - SETFUNC(F_FILESET); + if (func == F_DELETE) + func = F_FILEDELETE; + else + SETFUNC(F_FILESET); break; case 'i': rifname = optarg; @@ -197,6 +205,7 @@ main(int argc, char *argv[]) } break; case F_FILESET: + case F_FILEDELETE: if (argc != 1) usage(); rtn = file(argv[0]); @@ -213,7 +222,7 @@ static int file(char *name) { FILE *fp; - int i, retval; + int i, j, retval; char line[100], arg[5][50], *args[5], *p; if ((fp = fopen(name, "r")) == NULL) @@ -237,8 +246,23 @@ file(char *name) retval = 1; continue; } - if (set(i, args)) - retval = 1; + switch (func) { + case F_FILESET: + if (set(i, args)) + retval = 1; + break; + case F_FILEDELETE: + for (j = 2; j < i; j++) + if (strncmp(args[j], "pub", 3) == 0) { + j = 0; + break; + } + if (delete(args[0], j == 0 ? SIN_PROXY : 0)) + retval = 1; + break; + default: + usage(); + } } fclose(fp); return (retval); @@ -650,11 +674,12 @@ nuke_entry(struct sockaddr_dl *sdl __unused, static void usage(void) { - fprintf(stderr, "%s\n%s\n%s\n%s\n%s\n%s\n%s\n", + fprintf(stderr, "%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n", "usage: arp [-n] [-i interface] hostname", " arp [-n] [-i interface] -a", " arp -d hostname [pub]", " arp -d [-i interface] -a", + " arp -d -f filename", " arp -s hostname ether_addr [temp] [reject | blackhole] [pub [only]]", " arp -S hostname ether_addr [temp] [reject | blackhole] [pub [only]]", " arp -f filename"); --zhXaljGHf11kAtnf-- From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 18:01:56 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B3DE106566B; Wed, 7 Oct 2009 18:01:56 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 52A278FC15; Wed, 7 Oct 2009 18:01:56 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n97I1ucb023879; Wed, 7 Oct 2009 18:01:56 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n97I1u5B023875; Wed, 7 Oct 2009 18:01:56 GMT (envelope-from linimon) Date: Wed, 7 Oct 2009 18:01:56 GMT Message-Id: <200910071801.n97I1u5B023875@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/139387: [ipsec] Wrong lenth of PF_KEY messages in promiscuous mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 18:01:56 -0000 Old Synopsis: Wrong lenth of PF_KEY messages in promiscuous mode New Synopsis: [ipsec] Wrong lenth of PF_KEY messages in promiscuous mode Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Wed Oct 7 18:00:49 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=139387 From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 18:55:20 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECB7B106568F for ; Wed, 7 Oct 2009 18:55:20 +0000 (UTC) (envelope-from j@joostm.nl) Received: from gmp-eb-inf-2.sun.com (gmp-eb-inf-2.sun.com [192.18.6.24]) by mx1.freebsd.org (Postfix) with ESMTP id 22D9D8FC1E for ; Wed, 7 Oct 2009 18:55:19 +0000 (UTC) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n97ItITR018801 for ; Wed, 7 Oct 2009 18:55:19 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KR500200QWTXM00@fe-emea-09.sun.com> for freebsd-net@freebsd.org; Wed, 07 Oct 2009 19:55:16 +0100 (BST) Received: from [172.17.114.60] ([unknown] [82.204.20.246]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KR500HUSR83QTA0@fe-emea-09.sun.com> for freebsd-net@freebsd.org; Wed, 07 Oct 2009 19:55:16 +0100 (BST) Date: Sun, 04 Oct 2009 21:58:13 +0200 From: Joost Mulders Sender: Joost.Mulders@Sun.COM To: freebsd-net@freebsd.org Message-id: <91A98EB1-FFB3-4462-BE8C-BFEB2A215BF0@joostm.nl> X-Mailer: Apple Mail (2.1076) Subject: recommended PCI for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 18:55:21 -0000 Hi all, I'm preparing the build of my "residential gateway", a home storage server and router. I could use recommendations for a PCI wlan adapter that works well in AP mode (a/b/g). Thanks much! Joost From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 18:56:37 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD0AE1065694 for ; Wed, 7 Oct 2009 18:56:37 +0000 (UTC) (envelope-from j@joostm.nl) Received: from gmp-eb-inf-1.sun.com (gmp-eb-inf-1.sun.com [192.18.6.21]) by mx1.freebsd.org (Postfix) with ESMTP id 1F0DA8FC26 for ; Wed, 7 Oct 2009 18:56:36 +0000 (UTC) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n97Iuagv000920 for ; Wed, 7 Oct 2009 18:56:36 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KR500800R91G700@fe-emea-09.sun.com> for freebsd-net@freebsd.org; Wed, 07 Oct 2009 19:56:15 +0100 (BST) Received: from [172.17.114.60] ([unknown] [82.204.20.246]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KR500HUSR83QTA0@fe-emea-09.sun.com> for freebsd-net@freebsd.org; Wed, 07 Oct 2009 19:56:15 +0100 (BST) Date: Wed, 07 Oct 2009 20:55:11 +0200 From: Joost Mulders Sender: Joost.Mulders@Sun.COM To: freebsd-net@freebsd.org Message-id: <768BD7F8-7B04-4111-8A69-0D1118361E96@joostm.nl> X-Mailer: Apple Mail (2.1076) Subject: PCI card for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 18:56:37 -0000 Howdy, Can you recommend a WLAN PCI card for AP use in my FreeBSD "residential gateway". I'm looking for a 802.11 a/b/g card, no need for "n" yet. Recommendations and "works for me's" are highly appreciated. Thank you, Joost From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 19:06:10 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6439B106568B for ; Wed, 7 Oct 2009 19:06:10 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id 03C9B8FC13 for ; Wed, 7 Oct 2009 19:06:09 +0000 (UTC) Received: (qmail 91954 invoked from network); 7 Oct 2009 19:06:09 -0000 Received: from unknown (HELO ?192.168.0.163?) (spawk@66.206.120.2) by acm.poly.edu with AES256-SHA encrypted SMTP; 7 Oct 2009 19:06:09 -0000 Message-ID: <4ACCE651.1060607@acm.poly.edu> Date: Wed, 07 Oct 2009 15:04:49 -0400 From: Boris Kochergin User-Agent: Thunderbird 2.0.0.23 (X11/20090910) MIME-Version: 1.0 To: Joost Mulders References: <768BD7F8-7B04-4111-8A69-0D1118361E96@joostm.nl> In-Reply-To: <768BD7F8-7B04-4111-8A69-0D1118361E96@joostm.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: PCI card for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 19:06:10 -0000 Joost Mulders wrote: > Howdy, > > Can you recommend a WLAN PCI card for AP use in my FreeBSD > "residential gateway". I'm looking for a 802.11 a/b/g card, no need > for "n" yet. > > Recommendations and "works for me's" are highly appreciated. > > Thank you, Joost > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" I have a ton of these that I'm very happy with: http://www.newegg.com/Product/Product.aspx?Item=N82E16833127075 The heavily-used ones exhibit the problem described at: http://lists.freebsd.org/pipermail/freebsd-net/2009-September/022894.html ...but that's a driver, not a hardware, issue. The one in the router at my house doesn't exhibit the problem, the two differences there being that the machine is 7.0-RELEASE, and there are only a couple of clients connected to it, as opposed to the ~30 connected to my heavily-used ones. -Boris From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 19:24:45 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F7881065670 for ; Wed, 7 Oct 2009 19:24:45 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mx1.freebsd.org (Postfix) with ESMTP id 2F05B8FC0C for ; Wed, 7 Oct 2009 19:24:44 +0000 (UTC) Received: by ewy18 with SMTP id 18so8662525ewy.43 for ; Wed, 07 Oct 2009 12:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=+1xBeT/QEMSYuJ7FL/qdi8rsHjJmNWJxBTQocjP2xjg=; b=Mz6Mkzw9VrCVYu1YHmwrSREGgE9qN/E9L+cpkK8MyOaUuVjW+GQ4BehP96mk0h+f1w XcvbYyJhK/zgvf44LTP2eu5Ff+sA3wJs+NInLrXUu3dpqjbC0OGlryuyihz5Af00wJa0 0cFIDlumKk5LWG9nqaJ0QpnR/LxbMB267AVYo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=hy8O4hhQ1bXnHDnKpAWJMzIXls5aPHjEzw96c7BMJl7E8YurR9Jzzy3EB+6XS0ey6+ Ho2v2ujVCURUe5A9Ktnk6H41TA4RQtzPqWJXB4G8pyJUVPjpCzSTQgsb9IG5MjKzMFHQ f5xdCeyYcElz4+GxHTyqD+d7nVQgdvWs6YMdE= Received: by 10.210.7.17 with SMTP id 17mr346573ebg.50.1254943484301; Wed, 07 Oct 2009 12:24:44 -0700 (PDT) Received: from mac-mini.lan (bl5-224-203.dsl.telepac.pt [82.154.224.203]) by mx.google.com with ESMTPS id 28sm1235359eyg.36.2009.10.07.12.24.43 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 07 Oct 2009 12:24:43 -0700 (PDT) Sender: Rui Paulo Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Rui Paulo In-Reply-To: <91A98EB1-FFB3-4462-BE8C-BFEB2A215BF0@joostm.nl> Date: Wed, 7 Oct 2009 20:24:41 +0100 Content-Transfer-Encoding: 7bit Message-Id: <5F574815-77C8-43B3-9737-9CB5FD7CF288@freebsd.org> References: <91A98EB1-FFB3-4462-BE8C-BFEB2A215BF0@joostm.nl> To: Joost Mulders X-Mailer: Apple Mail (2.1076) Cc: freebsd-net@freebsd.org Subject: Re: recommended PCI for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 19:24:45 -0000 On 4 Oct 2009, at 20:58, Joost Mulders wrote: > Hi all, > > I'm preparing the build of my "residential gateway", a home storage > server and router. I could use recommendations for a PCI wlan > adapter that works well in AP mode (a/b/g). Wistron cards work well for me. -- Rui Paulo From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 19:27:15 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D726410656A3 for ; Wed, 7 Oct 2009 19:27:15 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mx1.freebsd.org (Postfix) with ESMTP id 64E7D8FC16 for ; Wed, 7 Oct 2009 19:27:15 +0000 (UTC) Received: by ewy18 with SMTP id 18so8664611ewy.43 for ; Wed, 07 Oct 2009 12:27:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=5AyQPS75dj9x7zwZlGwK0dWeMAgTHOnf8Hjx3imFbfk=; b=j+M5GYZDJic8d1ZuEja63a+Z7RjsZqxQuBKfsqnKOT0b9FjDwt0gd5NIBrqUfB3C0o jjIGoDk5RxmmF1eTHMZ/HTRTEr7QD7cZLlv9TmmpD0olJrGkicshsFHtUuP+BmBu/3CG o8wNhZuFLHzINh1JWooZ+vFF54zUZ+lhOI128= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=VHtqO7FR5wpn7Rb7HM5XPcikOvilmhSACgYCeb+0QVueLjgL+oU5i+i6SABsu/xDTk fll5kWFkQf5mLXbtgjOrOSKTmtIyYOlTNy0xG8osgvBDu6LxuTFmeSdKk7s0PmQnDrW+ zMbfknalIaE8EtN/EyIlsAUKLvF+xuXCaM2D0= Received: by 10.211.154.10 with SMTP id g10mr334447ebo.49.1254943634421; Wed, 07 Oct 2009 12:27:14 -0700 (PDT) Received: from mac-mini.lan (bl5-224-203.dsl.telepac.pt [82.154.224.203]) by mx.google.com with ESMTPS id 28sm1379628eyg.44.2009.10.07.12.27.13 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 07 Oct 2009 12:27:13 -0700 (PDT) Sender: Rui Paulo Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Rui Paulo In-Reply-To: <4ACCE651.1060607@acm.poly.edu> Date: Wed, 7 Oct 2009 20:27:12 +0100 Content-Transfer-Encoding: 7bit Message-Id: <824A1142-42E3-4B2E-AE72-489AD5AB7D49@freebsd.org> References: <768BD7F8-7B04-4111-8A69-0D1118361E96@joostm.nl> <4ACCE651.1060607@acm.poly.edu> To: Boris Kochergin X-Mailer: Apple Mail (2.1076) Cc: freebsd-net@freebsd.org, Joost Mulders Subject: Re: PCI card for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 19:27:15 -0000 On 7 Oct 2009, at 20:04, Boris Kochergin wrote: > > The heavily-used ones exhibit the problem described at: > > http://lists.freebsd.org/pipermail/freebsd-net/2009-September/022894.html > > ...but that's a driver, not a hardware, issue. The one in the router > at my house doesn't exhibit the problem, the two differences there > being that the machine is 7.0-RELEASE, and there are only a couple > of clients connected to it, as opposed to the ~30 connected to my > heavily-used ones. Are you sure you have sufficient RAM to hold 30 users? What you seem to be saying is that the ath driver is leaking mbufs and hence you get a panic, right? -- Rui Paulo From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 19:52:50 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8318C106566B; Wed, 7 Oct 2009 19:52:50 +0000 (UTC) (envelope-from nlandys@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.24]) by mx1.freebsd.org (Postfix) with ESMTP id 229938FC19; Wed, 7 Oct 2009 19:52:49 +0000 (UTC) Received: by qw-out-2122.google.com with SMTP id 5so1771691qwi.7 for ; Wed, 07 Oct 2009 12:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=2mYbbIySDNE6rWq0SQzDHW/8Y9MoUSvV7S5/ApoPtxo=; b=Oz69svHdUbvAtptP0HWXzPSIF+BAGCURWHGeox+IYvIVBsxOwE3O4bZuvMXiq2hT7R uVNxWqWTOJIcTFb1PLQ9F0V2RkvTiXnmTAt+raWsdQ2b88yQbF2HkHz1OILG/fnoCMsJ zl8nIX1DufN7Ym5D7eXjkSOyj+FE0DixBOFaI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Fgbc73QJX06IrcQlsGDHa93+UH+SJ0Ex+Ym0gp76r3ZmUoLD1PoQ92DNZLLBKZRCv3 RFfy03UsHTM6KWPInia4JByFUbfbDQFe1dQ/Br92fBONg19Qvqj3Urg5CDeXl9SxLa/v +yaCQwhX964iHTn4m2ygEJi6ttNhvHsrbZEKs= MIME-Version: 1.0 Received: by 10.229.43.79 with SMTP id v15mr379910qce.40.1254943732231; Wed, 07 Oct 2009 12:28:52 -0700 (PDT) In-Reply-To: <5F574815-77C8-43B3-9737-9CB5FD7CF288@freebsd.org> References: <91A98EB1-FFB3-4462-BE8C-BFEB2A215BF0@joostm.nl> <5F574815-77C8-43B3-9737-9CB5FD7CF288@freebsd.org> Date: Wed, 7 Oct 2009 12:28:52 -0700 Message-ID: <560f92640910071228n3fa8f408j665a9275ba937c53@mail.gmail.com> From: Nerius Landys To: Rui Paulo Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, Joost Mulders Subject: Re: recommended PCI for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 19:52:50 -0000 >> I'm preparing the build of my "residential gateway", a home storage server >> and router. I could use recommendations for a PCI wlan adapter that works >> well in AP mode (a/b/g). I have my FreeBSD box configured in AP mode. EnGenius Technologies EPI-3601S (600MW 802.11G PCI Card with Superg 108MBPS-Atheros) http://www.provantage.com/engenius-technologies-epi-3601s~7ENGN00W.htm This card works quite well for me. From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 20:33:50 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA4C8106566B for ; Wed, 7 Oct 2009 20:33:50 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id 7799E8FC15 for ; Wed, 7 Oct 2009 20:33:49 +0000 (UTC) Received: (qmail 94239 invoked from network); 7 Oct 2009 20:33:48 -0000 Received: from unknown (HELO ?192.168.0.184?) (spawk@66.206.120.2) by acm.poly.edu with AES256-SHA encrypted SMTP; 7 Oct 2009 20:33:48 -0000 Message-ID: <4ACCFADF.6090603@acm.poly.edu> Date: Wed, 07 Oct 2009 16:32:31 -0400 From: Boris Kochergin User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: Rui Paulo References: <768BD7F8-7B04-4111-8A69-0D1118361E96@joostm.nl> <4ACCE651.1060607@acm.poly.edu> <824A1142-42E3-4B2E-AE72-489AD5AB7D49@freebsd.org> In-Reply-To: <824A1142-42E3-4B2E-AE72-489AD5AB7D49@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Joost Mulders Subject: Re: PCI card for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 20:33:50 -0000 Rui Paulo wrote: > On 7 Oct 2009, at 20:04, Boris Kochergin wrote: >> >> The heavily-used ones exhibit the problem described at: >> >> http://lists.freebsd.org/pipermail/freebsd-net/2009-September/022894.html >> >> >> ...but that's a driver, not a hardware, issue. The one in the router >> at my house doesn't exhibit the problem, the two differences there >> being that the machine is 7.0-RELEASE, and there are only a couple of >> clients connected to it, as opposed to the ~30 connected to my >> heavily-used ones. > > Are you sure you have sufficient RAM to hold 30 users? > > What you seem to be saying is that the ath driver is leaking mbufs and > hence you get a panic, right? > > -- > Rui Paulo > > > Pretty sure. The Soekris has 256 MB of RAM. I suspect that it's a leak because things are usually great, even with ~30 users, with the "80211node" portion from "vmstat -m" only taking a few hundred KiB of RAM. Every few days, it spikes to over 100 MiB, and the "ath0: ath_rx_proc: no mbuf!" kernel messages appear, often followed by a panic. I haven't had time to try another rate-control algorithm, as per the responses to my post, but I'll report back when I do get around to it. -Boris From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 21:35:49 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 031D21065670; Wed, 7 Oct 2009 21:35:49 +0000 (UTC) (envelope-from rpaulo@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CF74E8FC13; Wed, 7 Oct 2009 21:35:48 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n97LZmmp005066; Wed, 7 Oct 2009 21:35:48 GMT (envelope-from rpaulo@freefall.freebsd.org) Received: (from rpaulo@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n97LZmtK005062; Wed, 7 Oct 2009 21:35:48 GMT (envelope-from rpaulo) Date: Wed, 7 Oct 2009 21:35:48 GMT Message-Id: <200910072135.n97LZmtK005062@freefall.freebsd.org> To: rpaulo@FreeBSD.org, freebsd-net@FreeBSD.org, rpaulo@FreeBSD.org From: rpaulo@FreeBSD.org Cc: Subject: Re: kern/137170: [ath] atheros AR9285 not recognised X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 21:35:49 -0000 Synopsis: [ath] atheros AR9285 not recognised Responsible-Changed-From-To: freebsd-net->rpaulo Responsible-Changed-By: rpaulo Responsible-Changed-When: Wed Oct 7 21:35:31 UTC 2009 Responsible-Changed-Why: I'm working on this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=137170 From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 22:02:35 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61950106568B for ; Wed, 7 Oct 2009 22:02:35 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id F01FA8FC16 for ; Wed, 7 Oct 2009 22:02:34 +0000 (UTC) Received: from lstewart-laptop.caia.swin.edu.au (216-239-45-4.google.com [216.239.45.4]) (authenticated bits=0) by lauren.room52.net (8.14.3/8.14.3) with ESMTP id n97M2Q3I064702 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 8 Oct 2009 09:02:27 +1100 (EST) (envelope-from lstewart@freebsd.org) Message-ID: <4ACD0FEE.2040902@freebsd.org> Date: Wed, 07 Oct 2009 15:02:22 -0700 From: Lawrence Stewart User-Agent: Thunderbird 2.0.0.23 (X11/20090909) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.6 required=5.0 tests=AWL,BAYES_00,RDNS_DYNAMIC, SPF_SOFTFAIL autolearn=disabled version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on lauren.room52.net Subject: Reproducible if_wpi panic "not enough data, data_len 2 space 1" on FreeBSD 9-CURRENT with WPA X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 22:02:35 -0000 Hi All, I have an Intel 3945ABG in my Toshiba R600 laptop which fairly reliably panics a while after I associate it with a WPA AP. It will happily associate and pass packets for some time (10's of minutes) and then panic. I have managed to trigger it both times I've tried to associate with this particular AP (some sort of NetGear 802.11n capable unit). Plenty of details here: http://people.freebsd.org/~lstewart/misc/wpi_debug/core.txt http://people.freebsd.org/~lstewart/misc/wpi_debug/kgdb_output.txt My /etc/wpa_supplicant.conf: ########## ctrl_interface=/var/run/wpa_supplicant ctrl_interface_group=wheel network={ ssid="X" proto=WPA psk="Y" } ########## Suggestions most welcome. In particular, the comment above the KASSERT that is firing in frame 10 seems to indicate that some prodding at the logic in the michael_mic() function may be required. Cheers, Lawrence From owner-freebsd-net@FreeBSD.ORG Wed Oct 7 23:17:18 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C60581065672; Wed, 7 Oct 2009 23:17:18 +0000 (UTC) (envelope-from onemda@gmail.com) Received: from mail-bw0-f227.google.com (mail-bw0-f227.google.com [209.85.218.227]) by mx1.freebsd.org (Postfix) with ESMTP id 270508FC12; Wed, 7 Oct 2009 23:17:17 +0000 (UTC) Received: by bwz27 with SMTP id 27so4320381bwz.43 for ; Wed, 07 Oct 2009 16:17:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=WhY1nq0jBRNJ9FN0hSfebSCD6TZugKHX9kNohO1P6cE=; b=USeIQ02UNG2kNTeWV8s0xAZEwF0FbZKf94KWCYDoCy/aMDK32gFS3geuwx25YH32ek k3Cl8NqovqSGwyl/g5UUy2rxApOiv3UhlB8PvBx3hvul13rGhhLDlQZfiTMQIsdsTQXs y0T50WdzCg+pq5DdVIWeKBtF4x4upf4EpdZnY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=OkhK3Unt29zFiKogKtVkS84Jigqn9R7zO5EEmybHFjN0MmOK7rvqQdYNfrFI6fPZLS hLQimrm4P5sL3sbmrpErxiO67j1nSDU7baVkaDuCLUgok+pNXndHL0ZbtV0sahkQo29u 41+lvhO4cwyvi1zS8xO8m73eYkbdHtuvpQWGw= MIME-Version: 1.0 Received: by 10.103.125.19 with SMTP id c19mr209252mun.59.1254957436889; Wed, 07 Oct 2009 16:17:16 -0700 (PDT) In-Reply-To: <4ACCFADF.6090603@acm.poly.edu> References: <768BD7F8-7B04-4111-8A69-0D1118361E96@joostm.nl> <4ACCE651.1060607@acm.poly.edu> <824A1142-42E3-4B2E-AE72-489AD5AB7D49@freebsd.org> <4ACCFADF.6090603@acm.poly.edu> Date: Thu, 8 Oct 2009 01:17:16 +0200 Message-ID: <3a142e750910071617y2bc3b9e1r9af2d3ce54a818cd@mail.gmail.com> From: Paul B Mahol To: Boris Kochergin Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, Rui Paulo , Joost Mulders Subject: Re: PCI card for wlan AP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2009 23:17:18 -0000 On 10/7/09, Boris Kochergin wrote: > Rui Paulo wrote: >> On 7 Oct 2009, at 20:04, Boris Kochergin wrote: >>> >>> The heavily-used ones exhibit the problem described at: >>> >>> http://lists.freebsd.org/pipermail/freebsd-net/2009-September/022894.html >>> >>> >>> >>> ...but that's a driver, not a hardware, issue. The one in the router >>> at my house doesn't exhibit the problem, the two differences there >>> being that the machine is 7.0-RELEASE, and there are only a couple of >>> clients connected to it, as opposed to the ~30 connected to my >>> heavily-used ones. >> >> Are you sure you have sufficient RAM to hold 30 users? >> >> What you seem to be saying is that the ath driver is leaking mbufs and >> hence you get a panic, right? >> >> -- >> Rui Paulo >> >> >> > Pretty sure. The Soekris has 256 MB of RAM. I suspect that it's a leak > because things are usually great, even with ~30 users, with the > "80211node" portion from "vmstat -m" only taking a few hundred KiB of > RAM. Every few days, it spikes to over 100 MiB, and the "ath0: Interesting, is it actually memory leak? (I try to hunt one down when using if_ndis(4) but without success) > ath_rx_proc: no mbuf!" kernel messages appear, often followed by a > panic. I haven't had time to try another rate-control algorithm, as per > the responses to my post, but I'll report back when I do get around to it. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 00:54:03 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C9D91065696 for ; Thu, 8 Oct 2009 00:54:03 +0000 (UTC) (envelope-from mdounin@mdounin.ru) Received: from mdounin.cust.ramtel.ru (mdounin.cust.ramtel.ru [81.19.69.81]) by mx1.freebsd.org (Postfix) with ESMTP id 22D318FC1E for ; Thu, 8 Oct 2009 00:54:02 +0000 (UTC) Received: from mdounin.ru (mdounin.cust.ramtel.ru [81.19.69.81]) by mdounin.cust.ramtel.ru (Postfix) with ESMTP id 1A4D31700D for ; Thu, 8 Oct 2009 04:37:23 +0400 (MSD) Date: Thu, 8 Oct 2009 04:37:23 +0400 From: Maxim Dounin To: freebsd-net@freebsd.org Message-ID: <20091008003723.GD1229@mdounin.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Subject: deadlock with pf uid rules + syncache X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 00:54:03 -0000 Hello! We with ru@ investigated lockups our colleagues observing on RELENG_7 machines, and was able to track [at least some of] them down to the following deadlock: 1. syncache does syn+ack retransmit on timeout. it holds tcp_sc_head mutex, and output packet processed by pf which in turn waits for tcp lock to lookup uid. 2. em0 dispatches new ack packet to listen socket, tcp_input() holds tcp lock, syncache lookup requires tcp_sc_head. Here is data from ddb: db> bt 19 Tracing pid 19 tid 100011 td 0xc6ca0000 sched_switch(c6ca0000,0,1,c06388b0,a38b7a66,...) at sched_switch+0x320 mi_switch(1,0,c08455c6,2e3,c091d634,...) at mi_switch+0x143 turnstile_wait(c7881140,c6d76d80,1,bd5,c6d76d86,...) at turnstile_wait+0x234 _rw_rlock(c0966b8c,c73619f6,bd5,c736392c,ce3,...) at _rw_rlock+0x166 pf_socket_lookup(2,c6bc6a80,0,ce3,c6bc6a1c,...) at pf_socket_lookup+0xc8 pf_test_tcp(c6bc6b18,c6bc6b14,2,c7236400,c73c7800,...) at pf_test_tcp+0x86f pf_test(2,c704e800,c6bc6b74,0,0,...) at pf_test+0xde1 pf_check_out(0,c6bc6b74,c704e800,2,0,...) at pf_check_out+0x43 pfil_run_hooks(c0965e40,c6bc6c04,c704e800,2,0,...) at pfil_run_hooks+0x78 ip_output(c73c7800,0,c6bc6bd8,0,0,...) at ip_output+0x7c2 syncache_respond(c8929418,0,0,0,c0842ffa,...) at syncache_respond+0x255 syncache_timer(c6fd7ac0,0,c0842ffa,f1,1,...) at syncache_timer+0x127 softclock(0,0,c083f980,4cb,c6ce5070,...) at softclock+0x21f ithread_loop(c6c95330,c6bc6d38,c083f823,323,c6c97840,...) at ithread_loop+0x145 fork_exit(c05e7520,c6c95330,c6bc6d38) at fork_exit+0x94 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6bc6d70, ebp = 0 --- db> bt 31 Tracing pid 31 tid 100031 td 0xc6d76d80 sched_switch(c6d76d80,0,1,c06388ee,a38981fa,...) at sched_switch+0x320 mi_switch(1,0,c08455c6,2e3,c091d7d8,...) at mi_switch+0x143 turnstile_wait(c7881be0,c6ca0000,0,c6fd7ac0,1f0,...) at turnstile_wait+0x234 _mtx_lock_sleep(c6fd7ac0,c6d76d80,0,c0850acb,1f0,...) at _mtx_lock_sleep+0x10e _mtx_lock_flags(c6fd7ac0,0,c0850acb,1f0,e6f87b14,...) at _mtx_lock_flags+0x67 syncache_lookup(e6f87aec,e6f87a8c,e6f8799c,c07db700,c091d8b8,...) at syncache_lookup+0x5e syncache_expand(e6f87aec,e6f87b14,c7593024,e6f87b2c,c753d600,...) at syncache_expand+0x1e tcp_input(c753d600,14,c704e800,1,0,...) at tcp_input+0x459 ip_input(c753d600,c057d932,800,c704e800,800,...) at ip_input+0x620 netisr_dispatch(2,c753d600,0,c7257537,3ac,...) at netisr_dispatch+0x55 ether_demux(c704e800,c753d600,3,0,3,...) at ether_demux+0x1aa ether_input(c704e800,c753d600,c6d76e18,c08baa84,c082d68e,...) at ether_input+0x343 ether_demux(c6da6000,c753d600,3,0,3,...) at ether_demux+0x107 ether_input(c6da6000,c753d600,c082d68e,11ba,c084572e,...) at ether_input+0x343 em_rxeof(e6f87ca4,c05f860d,c6da159c,8,c6dc832c,...) at em_rxeof+0x499 em_handle_rxtx(c6dc4000,1,c0845360,52,c6da159c,...) at em_handle_rxtx+0x2e taskqueue_run(c6da1580,c6da159c,c0836cc3,0,c083f823,...) at taskqueue_run+0x10b taskqueue_thread_loop(c6dc835c,e6f87d38,c083f823,323,c6dcbb00,...) at taskqueue_thread_loop+0x68 fork_exit(c0636ee0,c6dc835c,e6f87d38) at fork_exit+0x94 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe6f87d70, ebp = 0 --- db> show turnstile 0xc7881be0 Lock: 0xc6fd7ac0 - (sleep mutex) tcp_sc_head Lock Owner: 0xc6ca0000 (tid 100011, pid 19, "swi4: clock sio") Shared Waiters: empty Exclusive Waiters: 0xc6d76d80 (tid 100031, pid 31, "em0 taskq") Pending Threads: empty db> show turnstile 0xc7881140 Lock: 0xc0966b8c - (rw) tcp Lock Owner: 0xc6d76d80 (tid 100031, pid 31, "em0 taskq") Shared Waiters: 0xc6ca0000 (tid 100011, pid 19, "swi4: clock sio") 0xc7994d80 (tid 100192, pid 1833, "httpd") 0xc7883b40 (tid 100282, pid 1831, "httpd") 0xc7b1a000 (tid 100252, pid 1832, "httpd") Exclusive Waiters: 0xc6c9ab40 (tid 100013, pid 21, "swi1: net") 0xc7971000 (tid 100202, pid 1120, "nginx") 0xc7a04480 (tid 100204, pid 1122, "nginx") 0xc79a9000 (tid 100278, pid 1830, "httpd") Pending Threads: empty Unfortunately "show alllocks" output wasn't properly captured by textdump (as witness_list_lock() uses printf() for output, not db_printf() which is captured), but here is some reconstruction (lines from textdump's ddb.txt and msgbuf.txt, my memory about their relevant order): db> show alllocks Process 1319 (postgres) thread 0xc804a480 (100326) exclusive sx so_rcv_sx r = 0 (0xc7b088ac) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 1117 (sshd) thread 0xc708b240 (100057) exclusive sx so_rcv_sx r = 0 (0xc7b06a4c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 956 (sshd) thread 0xc72476c0 (100119) exclusive sx so_rcv_sx r = 0 (0xc7d1bd8c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 55 (g_mirror home) thread 0xc70c2d80 (100055) exclusive sx gmirror:lock r = 0 (0xc705442c) locked @ /usr/src/sys/modules/geom/geom_mirror/../../../geom/mirror/g_mirror.c:1858 Process 31 (em0 taskq) thread 0xc6d76d80 (100031) exclusive rw tcpinp r = 0 (0xc751d36c) locked @ /usr/src/sys/netinet/tcp_input.c:480 exclusive rw tcp r = 0 (0xc0966b8c) locked @ /usr/src/sys/netinet/tcp_input.c:400 Process 27 (swi6: Giant taskq) thread 0xc6ca06c0 (100027) exclusive sleep mutex Giant r = 0 (0xc0910630) locked @ /usr/src/sys/kern/kern_intr.c:1125 Process 19 (swi4: clock sio) thread 0xc6ca0000 (100011) shared rw PFil hook read/write mutex r = 0 (0xc0965e58) locked @ /usr/src/sys/net/pfil.c:73 exclusive sleep mutex tcp_sc_head r = 0 (0xc6fd7ac0) locked @ /usr/src/sys/kern/kern_timeout.c:241 The machine in question runs RELENG_7 from something about 2009-10-05. Please let me know if you need some more info. Maxim Dounin From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 05:23:42 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 503FD1065672 for ; Thu, 8 Oct 2009 05:23:42 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx30.mail.ru (mx30.mail.ru [94.100.176.44]) by mx1.freebsd.org (Postfix) with ESMTP id 10E148FC16 for ; Thu, 8 Oct 2009 05:23:41 +0000 (UTC) Received: from [217.25.27.27] (port=27591 helo=[217.25.27.27]) by mx30.mail.ru with asmtp id 1MvlTY-000E3r-00 for freebsd-net@freebsd.org; Thu, 08 Oct 2009 09:23:40 +0400 Message-ID: <4ACD775B.90901@mail.ru> Date: Thu, 08 Oct 2009 10:23:39 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Subject: Choosing two 10GiGE cards X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rihad List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 05:23:42 -0000 Hi there, We think it's time to switch from two GiGE bce cards to 10GiGE. According to http://www.freebsd.org/releases/7.2R/hardware.html the 10GiGE cards listed below are supported on amd64. Anyone has personal experience using any of them them? Should I prefer drivers that support DEVICE_POLLING (actually, only ixgb as per "man polling")? I'll try to find out from the manufacturer how their own models differ from each other (all this LR/SR/CX4 stuff). Thanks. [i386,amd64] The ixgb(4) driver supports the following cards: * Intel PRO/10GbE LR Server Adapter * Intel PRO/10GbE SR Server Adapter [i386,amd64] The cxgb(4) driver supports 10 Gigabit and 1 Gigabit Ethernet adapters based on the T3 and T3B chipset: * Chelsio 10GBase-CX4 * Chelsio 10GBase-LR * Chelsio 10GBase-SR [i386,amd64] The mxge(4) driver supports 10 Gigabit Ethernet adapters based on the Myricom LANai Z8E chips: * Myricom 10GBase-CX4 (10G-PCIE-8A-C, 10G-PCIE-8AL-C) * Myricom 10GBase-R (10G-PCIE-8A-R, 10G-PCIE-8AL-R) * Myricom 10G XAUI over ribbon fiber (10G-PCIE-8A-Q, 10G-PCIE-8AL-Q) [i386,amd64] The nxge(4) driver supports Neterion Xframe 10 Gigabit Ethernet adapters listed in http://www.neterion.com/how/pricing.html. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 05:29:07 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F4A2106566B for ; Thu, 8 Oct 2009 05:29:07 +0000 (UTC) (envelope-from andrew@modulus.org) Received: from email.octopus.com.au (email.octopus.com.au [122.100.2.232]) by mx1.freebsd.org (Postfix) with ESMTP id D7A3B8FC13 for ; Thu, 8 Oct 2009 05:29:06 +0000 (UTC) Received: by email.octopus.com.au (Postfix, from userid 1002) id ACF065CBAF3; Thu, 8 Oct 2009 16:28:57 +1100 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on email.octopus.com.au X-Spam-Level: X-Spam-Status: No, score=-1.4 required=10.0 tests=ALL_TRUSTED autolearn=failed version=3.2.3 Received: from [10.1.50.60] (ppp121-44-86-247.lns10.syd6.internode.on.net [121.44.86.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: admin@email.octopus.com.au) by email.octopus.com.au (Postfix) with ESMTP id B932B5CBAE5; Thu, 8 Oct 2009 16:28:53 +1100 (EST) Message-ID: <4ACD781A.1030301@modulus.org> Date: Thu, 08 Oct 2009 16:26:50 +1100 From: Andrew Snow User-Agent: Thunderbird 2.0.0.14 (X11/20080523) MIME-Version: 1.0 To: rihad References: <4ACD775B.90901@mail.ru> In-Reply-To: <4ACD775B.90901@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Choosing two 10GiGE cards X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 05:29:07 -0000 The only one worth getting IMO is Intel EXPX9502CX4 (INTEL 10 GIGABIT CX4 DUAL PORT SERVER ADAPTER) It is low power and very fast, and works under FreeBSD. Like all Intel NICs It supports interrupt modulation so polling support isn't really needed. - Andrew From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 05:35:51 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D8591065698 for ; Thu, 8 Oct 2009 05:35:51 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id F01178FC16 for ; Thu, 8 Oct 2009 05:35:50 +0000 (UTC) Received: from [217.25.27.27] (port=38668 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1MvlfJ-0008sv-00; Thu, 08 Oct 2009 09:35:49 +0400 Message-ID: <4ACD7A33.90308@mail.ru> Date: Thu, 08 Oct 2009 10:35:47 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Andrew Snow References: <4ACD775B.90901@mail.ru> <4ACD781A.1030301@modulus.org> In-Reply-To: <4ACD781A.1030301@modulus.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: Choosing two 10GiGE cards X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 05:35:51 -0000 Andrew Snow wrote: > > The only one worth getting IMO is Intel EXPX9502CX4 > (INTEL 10 GIGABIT CX4 DUAL PORT SERVER ADAPTER) > > It is low power and very fast, and works under FreeBSD. Like all Intel > NICs It supports interrupt modulation so polling support isn't really > needed. > > Thanks. What does DUAL PORT mean? It has two jacks? I think one such adapter will be more than enough to replace our two 1000 mbps cards, whether two jacks or not? From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 05:45:04 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECFFE106566B for ; Thu, 8 Oct 2009 05:45:04 +0000 (UTC) (envelope-from andrew@modulus.org) Received: from email.octopus.com.au (email.octopus.com.au [122.100.2.232]) by mx1.freebsd.org (Postfix) with ESMTP id B06028FC0A for ; Thu, 8 Oct 2009 05:45:04 +0000 (UTC) Received: by email.octopus.com.au (Postfix, from userid 1002) id 7DB2D5CBAE6; Thu, 8 Oct 2009 16:44:56 +1100 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on email.octopus.com.au X-Spam-Level: X-Spam-Status: No, score=-1.4 required=10.0 tests=ALL_TRUSTED autolearn=failed version=3.2.3 Received: from [10.1.50.60] (ppp121-44-86-247.lns10.syd6.internode.on.net [121.44.86.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: admin@email.octopus.com.au) by email.octopus.com.au (Postfix) with ESMTP id 8D5515CBAF5; Thu, 8 Oct 2009 16:44:52 +1100 (EST) Message-ID: <4ACD7BD9.7000901@modulus.org> Date: Thu, 08 Oct 2009 16:42:49 +1100 From: Andrew Snow User-Agent: Thunderbird 2.0.0.14 (X11/20080523) MIME-Version: 1.0 To: rihad References: <4ACD775B.90901@mail.ru> <4ACD781A.1030301@modulus.org> <4ACD7A33.90308@mail.ru> In-Reply-To: <4ACD7A33.90308@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Choosing two 10GiGE cards X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 05:45:05 -0000 rihad wrote: >> The only one worth getting IMO is Intel EXPX9502CX4 >> (INTEL 10 GIGABIT CX4 DUAL PORT SERVER ADAPTER) > Thanks. What does DUAL PORT mean? It has two jacks? I think one such > adapter will be more than enough to replace our two 1000 mbps cards, > whether two jacks or not? Correct, it has two ports on the one card. It uses a PCIe x8 slot so plenty of bandwidth to serve two ports. - Andrew From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 05:50:03 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4F4A106566B for ; Thu, 8 Oct 2009 05:50:03 +0000 (UTC) (envelope-from siquijorphilips@gmail.com) Received: from mail-vw0-f180.google.com (mail-vw0-f180.google.com [209.85.212.180]) by mx1.freebsd.org (Postfix) with ESMTP id 827C78FC19 for ; Thu, 8 Oct 2009 05:50:03 +0000 (UTC) Received: by vws10 with SMTP id 10so3154942vws.7 for ; Wed, 07 Oct 2009 22:50:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=SMx1ivcblEwh4l9pmWXcGyKtzoMGLKjwz70B929onKM=; b=nxa1p3oqgN6QOdOSZHN2zp2gtrqVVZ6Bpvt6scHl2bBGpsOc39rPt4bQ0D6Ly9/6Gm Rr/OzHFlkfKhthEg10uVQfJn4tNqaGa7E2CQdikf6g+AwODkK14vVqxQPKDATOokFcUs RZR/ktk+fUBy1QX14JbMsSeRx0/9q4tlgMqIQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=fmV7V71fodOpqIJzf4C8oIitdhECgZ5Prrp5S6Gmv0c/V6yoKwTAgXt1viobdh1vNP 0XnrItgYsy1bSKDOrroIRn0ZnVnvafOcmFWno0B9Y6xcglMJDF4kHW85OO7uCbdAtjIN 2buPSoUsuvA3Z4Uq03emjtFz8g290kiu/ZPM0= MIME-Version: 1.0 Received: by 10.220.68.226 with SMTP id w34mr1219534vci.116.1254979460191; Wed, 07 Oct 2009 22:24:20 -0700 (PDT) Date: Thu, 8 Oct 2009 13:24:20 +0800 Message-ID: From: Siquijor Philips To: freebsd-net@freebsd.org, freebsd-drivers-request@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 05:50:03 -0000 Hi, I got a dual-port Intel Gigabit NIC with 82576 (ET) chipset http://www.intel.com/Assets/PDF/prodbrief/320116.pdf. It has a feature on IPsec offloading but it only mentioned Microsoft Windows 2008 and Vista servers. I wonder if FreeBSD have also support on this feature? Thanks, Siquijor From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 06:06:10 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C22271065692 for ; Thu, 8 Oct 2009 06:06:10 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id 7CE868FC14 for ; Thu, 8 Oct 2009 06:06:10 +0000 (UTC) Received: by lath.rinet.ru (Postfix, from userid 222) id 05BE5704C; Thu, 8 Oct 2009 10:06:09 +0400 (MSD) Date: Thu, 8 Oct 2009 10:06:09 +0400 From: Oleg Bulyzhin To: rihad Message-ID: <20091008060608.GA23793@lath.rinet.ru> References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <4ACCC30E.7080504@elischer.org> <4ACCC4F3.3030302@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACCC4F3.3030302@mail.ru> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@freebsd.org, Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 06:06:10 -0000 On Wed, Oct 07, 2009 at 09:42:27PM +0500, rihad wrote: > Julian Elischer wrote: > > rihad wrote: > >> Oleg Bulyzhin wrote: > > > >> You probably have some special sources of documentation ;-) According > >> to man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the > >> packet unless one_pass=0. Or do you mean sprinkling smart skiptos here > >> and there? ;-) > >> > > > > ngtee should not have any affect on the packet.. it takes a copy.. > > > That's a logical conclusion, although I prefer trusting the man at hand > (pun intended) if I haven't tested it myself to see how it works: > > ngtee cookie > A copy of packet is diverted into netgraph, original packet is > either accepted or continues with the next rule, depending on > net.inet.ip.fw.one_pass sysctl variable. See ng_ipfw(4) > for more > information on netgraph and ngtee actions. > > > Although... I've a question to Mr. Oleg: > > > 2) use 'tee' rule with ng_ksocket & ng_netflow > > tee port > Send a copy of packets matching this rule to the divert(4) > socket > bound to port port. The search continues with the next rule. > > how is it different from one_pass=0? Both tee and ngtee w/ one_pass=0 > continue with the next rule. tee & ngtee are similar with one_pass=0 and different with one_pass=1 -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 07:18:24 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 77018106566B for ; Thu, 8 Oct 2009 07:18:24 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yw0-f186.google.com (mail-yw0-f186.google.com [209.85.211.186]) by mx1.freebsd.org (Postfix) with ESMTP id 308508FC0A for ; Thu, 8 Oct 2009 07:18:23 +0000 (UTC) Received: by ywh16 with SMTP id 16so5109815ywh.13 for ; Thu, 08 Oct 2009 00:18:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=xD2Z1lALcHW//Ly4gwDYAR/L+L5YqHPAG4/CijWOqCE=; b=IzxQTHpnYEG7N9pu1aA+jW9ArjuJFdC9vD4zRXBXb9wq3ApA6TjDLQ0KV5KdgQQH51 e3SoRJkaCRGfD36Pi0xyV6O0cIraHJ0mLs+BVuFFfACiiixeBpO7OlMojMyy8RHNwUCY 8bGr7CY6u0HVeKr3Rj8febRhTwkgFWEPgtwwM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=aWXXyqJ/qFPMeMgDdjY36O+mvqXKxPM8GfzuW9ropviHV8yLnKZVz4S1WrKyBu15TL YF+j6Ma42j9eV4QjEYTV3/JzjtLBU2143aY6vlbL9XKmS9fcIA+CnAWAi0PsnvyFOjfV t7vXb05g5egDzJzeseZnTTy+xcJiSo+GaDkoI= MIME-Version: 1.0 Received: by 10.101.212.14 with SMTP id o14mr852246anq.161.1254986303356; Thu, 08 Oct 2009 00:18:23 -0700 (PDT) In-Reply-To: <4ACD7BD9.7000901@modulus.org> References: <4ACD775B.90901@mail.ru> <4ACD781A.1030301@modulus.org> <4ACD7A33.90308@mail.ru> <4ACD7BD9.7000901@modulus.org> Date: Thu, 8 Oct 2009 00:18:23 -0700 Message-ID: <2a41acea0910080018w234261a7ud92a289298e5ad75@mail.gmail.com> From: Jack Vogel To: Andrew Snow Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: rihad , freebsd-net@freebsd.org Subject: Re: Choosing two 10GiGE cards X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 07:18:24 -0000 If you really want to make the switch to 10G I would also seriously consider moving to FreeBSD 8, the changes that have been made in the stack will help you get the most out of the hardware. When it comes to Intel hardware the model Andrew cites is the CX4 version of the 82598, you can also get it in fiber. The next gen mac, 82599, is out, but I am not sure how easily they are obtainable in the channels yet. My ixgbe driver supports both, the 599 will have some advantages, I am just about ready to release the latest driver that will now have Hardware LRO among other enhancements. I believe Dell and Supermicro will, or already have i7 servers with a couple of these as LOMs. Thanks for the plug btw Andrew! Jack On Wed, Oct 7, 2009 at 10:42 PM, Andrew Snow wrote: > rihad wrote: > >> The only one worth getting IMO is Intel EXPX9502CX4 >>> (INTEL 10 GIGABIT CX4 DUAL PORT SERVER ADAPTER) >>> >> > Thanks. What does DUAL PORT mean? It has two jacks? I think one such >> adapter will be more than enough to replace our two 1000 mbps cards, whether >> two jacks or not? >> > > Correct, it has two ports on the one card. It uses a PCIe x8 slot so plenty > of bandwidth to serve two ports. > > - Andrew > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 11:39:49 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E1D6106566B; Thu, 8 Oct 2009 11:39:49 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 815708FC0A; Thu, 8 Oct 2009 11:39:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id n98BJmvT073179; Thu, 8 Oct 2009 22:19:48 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Thu, 8 Oct 2009 22:19:48 +1100 (EST) From: Ian Smith To: rihad In-Reply-To: <4ACCA92A.9070803@mail.ru> Message-ID: <20091008212623.D88524@sola.nimnet.asn.au> References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-net@freebsd.org, Eugene Grosbein , Robert Watson , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 11:39:49 -0000 On Wed, 7 Oct 2009, rihad wrote: > Robert Watson wrote: > > > I would suggest making just the HZ -> 4000 change for now and see how it > > goes. > > > OK, I will try testing HZ=4000 tomorrow morning, although I'm pretty sure > there still will be some drops. Even if there are, I'd like to know what (rough) percentage in increased interrupt load you experience with HZ=4000 vs 1000 on that beast in your application, or of any discernable effects on other running processes? [Thanks guys for this interesting thread] > > > Can someone please say how to increase the "ifnet transmit queue sizes"? > > > > Unfortunately, I fear that this is driver-specific, and in the case of bce > > requires a recompile. In the driver init code in if_bce, the following > > code appears: > > > > ifp->if_snd.ifq_drv_maxlen = USABLE_TX_BD; > > IFQ_SET_MAXLEN(&ifp->if_snd, ifp->if_snd.ifq_drv_maxlen); > > IFQ_SET_READY(&ifp->if_snd); > > > > Which evaluates to a architecture-specific value due to varying pagesize. > > You might just try forcing it to 1024. > > > I think I'll try this too, if HZ=4000 doesn't help, thanks a lot. > In the long run we'll switch to some high-quality 10 GiGE cards. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 12:14:08 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64C941065692; Thu, 8 Oct 2009 12:14:08 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx40.mail.ru (mx40.mail.ru [94.100.176.54]) by mx1.freebsd.org (Postfix) with ESMTP id 155D98FC1A; Thu, 8 Oct 2009 12:14:07 +0000 (UTC) Received: from [217.25.27.27] (port=64599 helo=[217.25.27.27]) by mx40.mail.ru with asmtp id 1Mvrsk-0007OC-00; Thu, 08 Oct 2009 16:14:06 +0400 Message-ID: <4ACDD789.8030804@mail.ru> Date: Thu, 08 Oct 2009 17:14:01 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 12:14:08 -0000 Robert Watson wrote: > I would suggest making just the HZ -> 4000 change for now and see how it > goes. > Been running for a few hours under these changed sysctls: kern.clockrate: { hz = 4000, tick = 250, profhz = 4000, stathz = 129 } net.inet.ip.dummynet.io_fast: 1 net.inet.ip.dummynet.hash_size: 512 net.isr.direct: 0 net.inet.ip.intr_queue_maxlen: 5000 kern.ipc.nmbclusters=111111 Current stats: net.inet.ip.dummynet.search_steps: 190347891 net.inet.ip.dummynet.searches: 188979871 net.inet.ip.dummynet.io_pkt_drop: 0 net.inet.ip.intr_queue_drops: 0 Around 1800 entries in each of the two ipfw tables. About 420 mbps of traffic passing through. ipfw pipe's created with GRED queuing algorithm, if it matters: ipfw pipe 64 config bw 64kbit/s mask dst-ip 0xffffffff gred 0.002/1800/2000/0.1 queue 2000 etc. 0 global output packet drops per second. top -SH: last pid: 59120; load averages: 0.87, 1.19, 1.27 up 0+04:31:38 17:11:59 141 processes: 9 running, 115 sleeping, 17 waiting CPU: 3.8% user, 0.0% nice, 3.0% system, 8.8% interrupt, 84.4% idle Mem: 1302M Active, 1482M Inact, 350M Wired, 40K Cache, 214M Buf, 801M Free Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 16 root 171 ki31 0K 16K CPU2 2 237:10 97.02% idle: cpu2 14 root 171 ki31 0K 16K CPU4 4 225:33 96.48% idle: cpu4 12 root 171 ki31 0K 16K CPU6 6 249:31 88.96% idle: cpu6 15 root 171 ki31 0K 16K CPU3 3 224:18 88.92% idle: cpu3 13 root 171 ki31 0K 16K CPU5 5 228:39 87.74% idle: cpu5 11 root 171 ki31 0K 16K CPU7 7 235:02 87.35% idle: cpu7 18 root 171 ki31 0K 16K RUN 0 198:11 81.98% idle: cpu0 17 root 171 ki31 0K 16K CPU1 1 209:31 71.78% idle: cpu1 21 root -44 - 0K 16K WAIT 0 88:24 47.31% swi1: net 29 root -68 - 0K 16K WAIT 1 34:06 18.95% irq256: bce0 470 root -68 - 0K 16K - 7 21:24 6.01% dummynet 19 root -32 - 0K 16K WAIT 5 13:37 5.86% swi4: clock sio 31 root -68 - 0K 16K WAIT 2 7:45 4.44% irq257: bce1 I'll wait for the number of entries to reach 2000-2200 or so, and post back again. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 12:58:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4DEA106566B; Thu, 8 Oct 2009 12:58:39 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx34.mail.ru (mx34.mail.ru [94.100.176.48]) by mx1.freebsd.org (Postfix) with ESMTP id 464BD8FC12; Thu, 8 Oct 2009 12:58:39 +0000 (UTC) Received: from [217.25.27.27] (port=11791 helo=[217.25.27.27]) by mx34.mail.ru with asmtp id 1MvsZp-000Hsi-00; Thu, 08 Oct 2009 16:58:37 +0400 Message-ID: <4ACDE1FB.6020602@mail.ru> Date: Thu, 08 Oct 2009 17:58:35 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 12:58:39 -0000 Robert Watson wrote: > I would suggest making just the HZ -> 4000 change for now and see how > it goes. > 2018 users online, 73 drops have just occurred. p.s.: already 123 drops. It will only get worse after some time. Traffic load: 440-450 mbps. top -HS: last pid: 68314; load averages: 1.35, 1.22, 1.25 up 0+05:13:28 17:53:49 145 processes: 11 running, 118 sleeping, 16 waiting CPU: 1.4% user, 0.0% nice, 2.8% system, 10.3% interrupt, 85.5% idle Mem: 1337M Active, 1683M Inact, 355M Wired, 40K Cache, 214M Buf, 560M Free Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 14 root 171 ki31 0K 16K CPU4 4 257:35 99.41% idle: cpu4 12 root 171 ki31 0K 16K RUN 6 286:39 98.14% idle: cpu6 18 root 171 ki31 0K 16K RUN 0 225:16 92.72% idle: cpu0 15 root 171 ki31 0K 16K RUN 3 255:35 90.04% idle: cpu3 16 root 171 ki31 0K 16K CPU2 2 272:04 87.40% idle: cpu2 13 root 171 ki31 0K 16K CPU5 5 260:52 81.69% idle: cpu5 17 root 171 ki31 0K 16K CPU1 1 239:06 75.29% idle: cpu1 21 root -44 - 0K 16K CPU7 7 108:49 57.37% swi1: net 11 root 171 ki31 0K 16K CPU7 7 267:48 49.02% idle: cpu7 29 root -68 - 0K 16K WAIT 1 41:45 20.90% irq256: bce0 470 root -68 - 0K 16K - 5 27:01 9.18% dummynet 19 root -32 - 0K 16K WAIT 1 16:13 6.59% swi4: clock sio 31 root -68 - 0K 16K WAIT 2 9:35 4.35% irq257: bce1 > Robert Watson wrote: > Suggestions like increasing timer resolution are intended to spread > out the injection of packets by dummynet to attempt to reduce the > peaks of burstiness that occur when multiple queues inject packets in > a burst that exceeds the queue depth supported by combined hardware > descriptor rings and software transmit queue. How to tweak the software transmit queue? P.S.: still only 123 drops (while io_pkt_drop: 0, intr_queue_drops: 0), but it was a warning. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 15:32:53 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E53901065679; Thu, 8 Oct 2009 15:32:53 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id 7E13F8FC1D; Thu, 8 Oct 2009 15:32:53 +0000 (UTC) Received: from [217.25.27.27] (port=32520 helo=[217.25.27.27]) by mx38.mail.ru with asmtp id 1Mvuz4-0006T5-00; Thu, 08 Oct 2009 19:32:51 +0400 Message-ID: <4ACE061E.4010807@mail.ru> Date: Thu, 08 Oct 2009 20:32:46 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Robert Watson References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 15:32:54 -0000 Robert Watson wrote: > I would suggest making just the HZ -> 4000 change for now and see how > it goes. > ~4000 online users, ~450-470 mbps traffic, 300-600 global drops per second. Same ole. Not funny at all. net.inet.ip.dummynet.io_pkt_drop: 0 net.inet.ip.intr_queue_drops: 0 net.inet.ip.fastforwarding: 1 > Robert Watson wrote: > Suggestions like increasing timer resolution are intended to spread > out the injection of packets by dummynet to attempt to reduce the > peaks of burstiness that occur when multiple queues inject packets in > a burst that exceeds the queue depth supported by combined hardware > descriptor rings and software transmit queue. My last chance is to tweak the software transmit queue. Just how can I alter its size? P.S.: We're definitely going to buy a 10GigE card. Like this one: http://cgi.ebay.com/EXPX9502CX4-10-CX4-DualPort-Svr-Adap-Intel-Corp_W0QQitemZ150358746205QQcmdZViewItemQQptZLH_DefaultDomain_0?hash=item230214645d From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 15:42:42 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63BCE1065692; Thu, 8 Oct 2009 15:42:42 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx71.mail.ru (mx71.mail.ru [94.100.176.85]) by mx1.freebsd.org (Postfix) with ESMTP id EF8388FC15; Thu, 8 Oct 2009 15:42:41 +0000 (UTC) Received: from [217.25.27.27] (port=5346 helo=[217.25.27.27]) by mx71.mail.ru with asmtp id 1Mvv8Z-00095g-00; Thu, 08 Oct 2009 19:42:39 +0400 Message-ID: <4ACE086C.2020308@mail.ru> Date: Thu, 08 Oct 2009 20:42:36 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Ian Smith References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACCA92A.9070803@mail.ru> <20091008212623.D88524@sola.nimnet.asn.au> In-Reply-To: <20091008212623.D88524@sola.nimnet.asn.au> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Eugene Grosbein , Robert Watson , Luigi Rizzo , Julian Elischer Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 15:42:42 -0000 Ian Smith wrote: > On Wed, 7 Oct 2009, rihad wrote: > > > Robert Watson wrote: > > > > > I would suggest making just the HZ -> 4000 change for now and see how it > > > goes. > > > > > OK, I will try testing HZ=4000 tomorrow morning, although I'm pretty sure > > there still will be some drops. > > Even if there are, I'd like to know what (rough) percentage in increased > interrupt load you experience with HZ=4000 vs 1000 on that beast in your > application, or of any discernable effects on other running processes? > Besides having little (if any) positive effect on the output packet drop rate, it runs pretty well, there's no apparent difference, no drop in performance etc. Current interrupt load snapshot as per systat -vmstat: Interrupts 59606 total atkbd0 1 ata0 irq14 931 mfi0 irq16 uhci0 uhci 4001 cpu0: time 23549 bce0 256 3118 bce1 257 3999 cpu3: time 3999 cpu2: time 4001 cpu1: time 4003 cpu4: time 4003 cpu5: time 4001 cpu6: time 4001 cpu7: time From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 16:45:02 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F9F61065670 for ; Thu, 8 Oct 2009 16:45:02 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outQ.internet-mail-service.net (outq.internet-mail-service.net [216.240.47.240]) by mx1.freebsd.org (Postfix) with ESMTP id 019878FC19 for ; Thu, 8 Oct 2009 16:45:01 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 82FB2D61EE; Thu, 8 Oct 2009 09:45:01 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id ACA942D62FD; Thu, 8 Oct 2009 09:18:20 -0700 (PDT) Message-ID: <4ACE10CF.2030806@elischer.org> Date: Thu, 08 Oct 2009 09:18:23 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <4ACCC30E.7080504@elischer.org> <4ACCC4F3.3030302@mail.ru> <20091008060608.GA23793@lath.rinet.ru> In-Reply-To: <20091008060608.GA23793@lath.rinet.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: rihad , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 16:45:02 -0000 Oleg Bulyzhin wrote: > On Wed, Oct 07, 2009 at 09:42:27PM +0500, rihad wrote: >> Julian Elischer wrote: >>> rihad wrote: >>>> Oleg Bulyzhin wrote: >>>> You probably have some special sources of documentation ;-) According >>>> to man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the >>>> packet unless one_pass=0. Or do you mean sprinkling smart skiptos here >>>> and there? ;-) >>>> >>> ngtee should not have any affect on the packet.. it takes a copy.. >>> >> That's a logical conclusion, although I prefer trusting the man at hand >> (pun intended) if I haven't tested it myself to see how it works: >> >> ngtee cookie >> A copy of packet is diverted into netgraph, original packet is >> either accepted or continues with the next rule, depending on >> net.inet.ip.fw.one_pass sysctl variable. See ng_ipfw(4) >> for more >> information on netgraph and ngtee actions. >> >> >> Although... I've a question to Mr. Oleg: >> >>> 2) use 'tee' rule with ng_ksocket & ng_netflow >> tee port >> Send a copy of packets matching this rule to the divert(4) >> socket >> bound to port port. The search continues with the next rule. >> >> how is it different from one_pass=0? Both tee and ngtee w/ one_pass=0 >> continue with the next rule. > > tee & ngtee are similar with one_pass=0 and different with one_pass=1 that seems like a bug to me.. neither tee should ever terminate a search. if you want to terminate it, add a specific rule to do so. Unfortunately I wasn't involved in writing it. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 16:45:17 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A88A1065672 for ; Thu, 8 Oct 2009 16:45:17 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outA.internet-mail-service.net (outa.internet-mail-service.net [216.240.47.224]) by mx1.freebsd.org (Postfix) with ESMTP id 0C8E88FC14 for ; Thu, 8 Oct 2009 16:45:16 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id A0BB0ACB80; Thu, 8 Oct 2009 09:45:16 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 9A5182D60BE; Thu, 8 Oct 2009 09:45:14 -0700 (PDT) Message-ID: <4ACE171D.9080707@elischer.org> Date: Thu, 08 Oct 2009 09:45:17 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: rihad References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACDE1FB.6020602@mail.ru> In-Reply-To: <4ACDE1FB.6020602@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Eugene Grosbein , Robert Watson , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 16:45:17 -0000 rihad wrote: > Robert Watson wrote: >> I would suggest making just the HZ -> 4000 change for now and see how >> it goes. >> > 2018 users online, 73 drops have just occurred. > p.s.: already 123 drops. > It will only get worse after some time. > > Traffic load: 440-450 mbps. > > top -HS: > last pid: 68314; load averages: 1.35, 1.22, 1.25 > up 0+05:13:28 17:53:49 > 145 processes: 11 running, 118 sleeping, 16 waiting > CPU: 1.4% user, 0.0% nice, 2.8% system, 10.3% interrupt, 85.5% idle > Mem: 1337M Active, 1683M Inact, 355M Wired, 40K Cache, 214M Buf, 560M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 14 root 171 ki31 0K 16K CPU4 4 257:35 99.41% idle: cpu4 > 12 root 171 ki31 0K 16K RUN 6 286:39 98.14% idle: cpu6 > 18 root 171 ki31 0K 16K RUN 0 225:16 92.72% idle: cpu0 > 15 root 171 ki31 0K 16K RUN 3 255:35 90.04% idle: cpu3 > 16 root 171 ki31 0K 16K CPU2 2 272:04 87.40% idle: cpu2 > 13 root 171 ki31 0K 16K CPU5 5 260:52 81.69% idle: cpu5 > 17 root 171 ki31 0K 16K CPU1 1 239:06 75.29% idle: cpu1 > 21 root -44 - 0K 16K CPU7 7 108:49 57.37% swi1: net > 11 root 171 ki31 0K 16K CPU7 7 267:48 49.02% idle: cpu7 > 29 root -68 - 0K 16K WAIT 1 41:45 20.90% irq256: bce0 > 470 root -68 - 0K 16K - 5 27:01 9.18% dummynet > 19 root -32 - 0K 16K WAIT 1 16:13 6.59% swi4: > clock sio > 31 root -68 - 0K 16K WAIT 2 9:35 4.35% irq257: bce1 > >> Robert Watson wrote: >> Suggestions like increasing timer resolution are intended to spread >> out the injection of packets by dummynet to attempt to reduce the >> peaks of burstiness that occur when multiple queues inject packets in >> a burst that exceeds the queue depth supported by combined hardware >> descriptor rings and software transmit queue. > > How to tweak the software transmit queue? > > > P.S.: still only 123 drops (while io_pkt_drop: 0, intr_queue_drops: 0), > but it was a warning. you can not do anything about it if one of the custommers sends a burst of 3000 udp packets at their maximum speed(or maybe some combination of custommers to something which results in an aggreagate burst rate like that. In other words you may always continue to get moments when the pipe releases a bunch of stuff that has a potential to over-run something down stream. Think of it as a dam in a stream... If you have no dam, teh water level goes up and down gradually and by small amounts, but if you have a dam, you can release water in such a way that the stream is flooded higher than it would normally ever get. work out how large a burst of data the pipe will release in 1/4000th of a second and using a small packet size, work out how many packets that is. Then make sure that the driver software input queue is capable of holding that many packets. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 16:54:47 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4D522106568D; Thu, 8 Oct 2009 16:54:47 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx39.mail.ru (mx39.mail.ru [94.100.176.53]) by mx1.freebsd.org (Postfix) with ESMTP id E3A648FC08; Thu, 8 Oct 2009 16:54:46 +0000 (UTC) Received: from [217.25.27.27] (port=15706 helo=[217.25.27.27]) by mx39.mail.ru with asmtp id 1MvwGK-000N12-00; Thu, 08 Oct 2009 20:54:44 +0400 Message-ID: <4ACE194F.6030601@mail.ru> Date: Thu, 08 Oct 2009 21:54:39 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <4ACCC30E.7080504@elischer.org> <4ACCC4F3.3030302@mail.ru> <20091008060608.GA23793@lath.rinet.ru> <4ACE10CF.2030806@elischer.org> In-Reply-To: <4ACE10CF.2030806@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org, Oleg Bulyzhin Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 16:54:47 -0000 Julian Elischer wrote: >> tee & ngtee are similar with one_pass=0 and different with one_pass=1 > > that seems like a bug to me.. > neither tee should ever terminate a search. > > if you want to terminate it, add a specific rule to do so. > Unfortunately I wasn't involved in writing it. > > +1 ngtee shouldn't terminate the search, i.e. behave exactly that way tee does. From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 17:05:37 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DE201065670; Thu, 8 Oct 2009 17:05:37 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx74.mail.ru (mx74.mail.ru [94.100.176.89]) by mx1.freebsd.org (Postfix) with ESMTP id AAA878FC27; Thu, 8 Oct 2009 17:05:36 +0000 (UTC) Received: from [217.25.27.27] (port=52432 helo=[217.25.27.27]) by mx74.mail.ru with asmtp id 1MvwQo-0004p8-00; Thu, 08 Oct 2009 21:05:34 +0400 Message-ID: <4ACE1B95.2090508@mail.ru> Date: Thu, 08 Oct 2009 22:04:21 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Julian Elischer References: <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su> <4ACB6223.1000709@mail.ru> <20091006161240.GA49940@svzserv.kemerovo.su> <4ACC5563.602@mail.ru> <4ACC56A6.1030808@mail.ru> <4ACC5DEC.1010006@mail.ru> <4ACC65A0.7030900@mail.ru> <4ACC8CC8.8050403@mail.ru> <4ACDE1FB.6020602@mail.ru> <4ACE171D.9080707@elischer.org> In-Reply-To: <4ACE171D.9080707@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Eugene Grosbein , Robert Watson , Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 17:05:37 -0000 Julian Elischer wrote: > you can not do anything about it if one of the custommers sends a burst > of 3000 udp packets at their maximum speed(or maybe some combination of > custommers to something which results in an aggreagate burst rate like > that. > In other words you may always continue to get moments when the pipe > releases a bunch of stuff that has a potential to over-run something > down stream. > I've said this before, but: the PC is only dealing with the downstream traffic, i.e. traffic arriving from the net. I won't say anything against dummynet bursts overfilling hardware buffers, but they're _only_taking place when the number of entries in the ipfw tables reaches 2000 or so, and the traffic load is at about 430-450 mbps. That is, it _never_ happens before both conditions are true. Although raising HZ from 1000 up to 4000 hasn't helped, which is somewhat contrary to the idea of bursts. My idea is to switch to an Intel 10GigE card, which is unavoidable sooner or later anyway, given our natural growth. Probably an upgrade to 8.0 will do some good, too. > Think of it as a dam in a stream... > > If you have no dam, teh water level goes up and down gradually and by > small amounts, but if you have a dam, you can release water in such a > way that the stream is flooded higher than it would normally ever get. > > > work out how large a burst of data the pipe will release in 1/4000th of > a second and using a small packet size, work out how many packets that > is. Then make sure that the driver software input queue is > capable of holding that many packets. > > From owner-freebsd-net@FreeBSD.ORG Thu Oct 8 17:46:40 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8322106566B for ; Thu, 8 Oct 2009 17:46:40 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.157]) by mx1.freebsd.org (Postfix) with ESMTP id BCAD28FC0C for ; Thu, 8 Oct 2009 17:46:39 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id e21so52815fga.13 for ; Thu, 08 Oct 2009 10:46:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=dN95889aefycihrXOwxWlBBp5vt1sLguaz8oB3Jzcq4=; b=WaqtRIlo0V7zHLB+Lde6sPpVJ+SklF3MS0ghddVdILWezTuyLQ/4pg3OXc3jfAcbJr KxpuY8N+TtF1LJ30A0cqnHDrZYApAeCO1VDFHlFy3Igq5sgviFgr0oiNgV7AFnXGcdyq UrsKPLItR7UeChddYrA98+yE+R8OVcF0GyBl4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=OwwCxI/0gBTnL448ebTEvPW/pIOs28FI0WSANPhda3ZTy/rmrTGTFLZinMyUFH4t52 jpNgBHI2cibHw6ySSesnA6QdbsWr/fotKiI1srKj67y+hgeubfNGx7T8vygga+vZ7mb5 6topd40dYgU3OPDuPdEU8U3voV+lQkgqIY6I4= Received: by 10.86.227.13 with SMTP id z13mr1428182fgg.72.1255023998068; Thu, 08 Oct 2009 10:46:38 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id e11sm470618fga.4.2009.10.08.10.46.32 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 08 Oct 2009 10:46:35 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Thu, 8 Oct 2009 10:45:21 -0700 From: Pyun YongHyeon Date: Thu, 8 Oct 2009 10:45:21 -0700 To: Siquijor Philips Message-ID: <20091008174521.GE3843@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, freebsd-drivers-request@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2009 17:46:40 -0000 On Thu, Oct 08, 2009 at 01:24:20PM +0800, Siquijor Philips wrote: > Hi, > > I got a dual-port Intel Gigabit NIC with 82576 (ET) chipset > http://www.intel.com/Assets/PDF/prodbrief/320116.pdf. It has a feature > on IPsec offloading but it only mentioned Microsoft Windows 2008 and > Vista servers. I wonder if FreeBSD have also support on this feature? > AFAIK it's not yet, not sure whether Jack has plan to implement the offloading. I know old Intel i82550 also supported IPSec offloading but Intel didn't release required information to implement it. 3Com also supported IPSec offloading in their 3XP hardwares(txp(4)) but the offloading was not implemented. > Thanks, > Siquijor From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 01:35:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82C2F106566B for ; Fri, 9 Oct 2009 01:35:39 +0000 (UTC) (envelope-from stef-list@memberwebs.com) Received: from memberwebs.com (memberwebs.com [94.75.203.95]) by mx1.freebsd.org (Postfix) with ESMTP id 8AD9E8FC19 for ; Fri, 9 Oct 2009 01:35:38 +0000 (UTC) Received: from [172.27.5.159] (unknown [172.27.5.159]) by memberwebs.com (Postfix) with ESMTP id A7FEA83E4B5 for ; Fri, 9 Oct 2009 01:35:34 +0000 (UTC) Message-ID: <4ACE9367.2050909@memberwebs.com> Date: Thu, 08 Oct 2009 20:35:35 -0500 From: Stef Walter User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: "freebsd-net@FreeBSD.org" Content-Type: multipart/mixed; boundary="------------000806010602050803090805" Subject: Unbreak setfib + routing daemon [patch roundup] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: stef@memberwebs.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 01:35:39 -0000 This is a multi-part message in MIME format. --------------000806010602050803090805 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit I've been running the attached patch for a month on routers in production. To summarize: * Routing daemons listen to routing messages. * Without the patch routing messages all FIBs are sent to all listeners. * Routing daemons get confused. This patch makes routing daemons listening to routing messages only receive those that affect their FIB. Some routing messages (such as interface address announcements etc..) are still sent to all listeners. Patches attached for CURRENT, latest 8.0, and 7.2. Previous discussion: http://www.mail-archive.com/freebsd-net@freebsd.org/msg30815.html PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/134931 Cheers, Stef --------------000806010602050803090805 Content-Type: text/x-diff; name="freebsd8-route-messages-respect-fib-2.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="freebsd8-route-messages-respect-fib-2.patch" --- ./sys/net/rtsock.c.orig 2009-08-31 15:26:03.000000000 +0000 +++ ./sys/net/rtsock.c 2009-08-31 19:06:53.000000000 +0000 @@ -93,4 +93,9 @@ SYSCTL_NODE(_net, OID_AUTO, route, CTLFLAG_RD, 0, ""); +struct rt_dispatch_ctx { + unsigned short family; /* Socket family */ + int fibnum; /* FIB for message or -1 for all */ +}; + struct walkarg { int w_tmemsize; @@ -114,5 +119,5 @@ static void rt_getmetrics(const struct rt_metrics_lite *in, struct rt_metrics *out); -static void rt_dispatch(struct mbuf *, const struct sockaddr *); +static void rt_dispatch(struct mbuf *, const struct sockaddr *, int); static struct netisr_handler rtsock_nh = { @@ -155,17 +160,19 @@ { struct sockproto route_proto; - unsigned short *family; + struct rt_dispatch_ctx *ctx; struct m_tag *tag; + int fibnum = -1; route_proto.sp_family = PF_ROUTE; - tag = m_tag_find(m, PACKET_TAG_RTSOCKFAM, NULL); + tag = m_tag_find(m, PACKET_TAG_RTSOCK, NULL); if (tag != NULL) { - family = (unsigned short *)(tag + 1); - route_proto.sp_protocol = *family; + ctx = (struct rt_dispatch_ctx*)(tag + 1); + route_proto.sp_protocol = ctx->family; + fibnum = ctx->fibnum; m_tag_delete(m, tag); } else route_proto.sp_protocol = 0; - raw_input(m, &route_proto, &route_src); + raw_input(m, &route_proto, &route_src, fibnum); } @@ -784,8 +791,8 @@ unsigned short family = rp->rcb_proto.sp_family; rp->rcb_proto.sp_family = 0; - rt_dispatch(m, info.rti_info[RTAX_DST]); + rt_dispatch(m, info.rti_info[RTAX_DST], so->so_fibnum); rp->rcb_proto.sp_family = family; } else - rt_dispatch(m, info.rti_info[RTAX_DST]); + rt_dispatch(m, info.rti_info[RTAX_DST], so->so_fibnum); } } @@ -1010,5 +1017,5 @@ */ void -rt_missmsg(int type, struct rt_addrinfo *rtinfo, int flags, int error) +rt_missmsg(int type, struct rt_addrinfo *rtinfo, int flags, int error, int fibnum) { struct rt_msghdr *rtm; @@ -1025,5 +1032,5 @@ rtm->rtm_errno = error; rtm->rtm_addrs = rtinfo->rti_addrs; - rt_dispatch(m, sa); + rt_dispatch(m, sa, fibnum); } @@ -1050,5 +1057,5 @@ ifm->ifm_data = ifp->if_data; ifm->ifm_addrs = 0; - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } @@ -1062,5 +1069,5 @@ */ void -rt_newaddrmsg(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt) +rt_newaddrmsg(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt, int fibnum) { struct rt_addrinfo info; @@ -1120,5 +1127,5 @@ rtm->rtm_addrs = info.rti_addrs; } - rt_dispatch(m, sa); + rt_dispatch(m, sa, fibnum); } } @@ -1156,5 +1163,5 @@ ifmam->ifmam_index = ifp->if_index; ifmam->ifmam_addrs = info.rti_addrs; - rt_dispatch(m, ifma->ifma_addr); + rt_dispatch(m, ifma->ifma_addr, -1); } @@ -1216,5 +1223,5 @@ m->m_pkthdr.len += data_len; mtod(m, struct if_announcemsghdr *)->ifan_msglen += data_len; - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } } @@ -1232,10 +1239,11 @@ m = rt_makeifannouncemsg(ifp, RTM_IFANNOUNCE, what, &info); if (m != NULL) - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } static void -rt_dispatch(struct mbuf *m, const struct sockaddr *sa) +rt_dispatch(struct mbuf *m, const struct sockaddr *sa, int fibnum) { + struct rt_dispatch_ctx *ctx; struct m_tag *tag; @@ -1243,14 +1251,16 @@ * Preserve the family from the sockaddr, if any, in an m_tag for * use when injecting the mbuf into the routing socket buffer from - * the netisr. + * the netisr. Additionally save the fibnum if needed. */ - if (sa != NULL) { - tag = m_tag_get(PACKET_TAG_RTSOCKFAM, sizeof(unsigned short), - M_NOWAIT); + if (sa != NULL || fibnum >= 0) { + tag = m_tag_get(PACKET_TAG_RTSOCK, + sizeof(struct rt_dispatch_ctx*), M_NOWAIT); if (tag == NULL) { m_freem(m); return; } - *(unsigned short *)(tag + 1) = sa->sa_family; + ctx = (struct rt_dispatch_ctx*)(tag + 1); + ctx->family = sa->sa_family; + ctx->fibnum = fibnum; m_tag_prepend(m, tag); } --- ./sys/net/raw_usrreq.c.orig 2009-08-31 16:04:58.000000000 +0000 +++ ./sys/net/raw_usrreq.c 2009-08-31 18:34:38.000000000 +0000 @@ -70,5 +70,5 @@ */ void -raw_input(struct mbuf *m0, struct sockproto *proto, struct sockaddr *src) +raw_input(struct mbuf *m0, struct sockproto *proto, struct sockaddr *src, int fibnum) { struct rawcb *rp; @@ -84,4 +84,7 @@ rp->rcb_proto.sp_protocol != proto->sp_protocol) continue; + if (fibnum >= 0 && rp->rcb_socket && + fibnum != rp->rcb_socket->so_fibnum) + continue; if (last) { struct mbuf *n; --- ./sys/net/raw_cb.h.orig 2009-08-31 18:34:56.000000000 +0000 +++ ./sys/net/raw_cb.h 2009-08-31 18:35:05.000000000 +0000 @@ -73,5 +73,5 @@ int raw_attach(struct socket *, int); void raw_detach(struct rawcb *); -void raw_input(struct mbuf *, struct sockproto *, struct sockaddr *); +void raw_input(struct mbuf *, struct sockproto *, struct sockaddr *, int); /* --- ./sys/net/route.c.orig 2009-08-31 18:18:30.000000000 +0000 +++ ./sys/net/route.c 2009-08-31 18:59:20.000000000 +0000 @@ -384,5 +384,5 @@ bzero(&info, sizeof(info)); info.rti_info[RTAX_DST] = dst; - rt_missmsg(msgtype, &info, 0, err); + rt_missmsg(msgtype, &info, 0, err, fibnum); } done: @@ -609,5 +609,5 @@ info.rti_info[RTAX_NETMASK] = netmask; info.rti_info[RTAX_AUTHOR] = src; - rt_missmsg(RTM_REDIRECT, &info, flags, error); + rt_missmsg(RTM_REDIRECT, &info, flags, error, fibnum); if (ifa != NULL) ifa_free(ifa); @@ -1433,5 +1433,5 @@ rt->rt_ifp->if_index; } - rt_newaddrmsg(cmd, ifa, error, rt); + rt_newaddrmsg(cmd, ifa, error, rt, fibnum); if (cmd == RTM_DELETE) { /* --- ./sys/net/route.h.orig 2009-08-31 18:56:05.000000000 +0000 +++ ./sys/net/route.h 2009-08-31 18:59:32.000000000 +0000 @@ -381,6 +381,6 @@ void rt_ifannouncemsg(struct ifnet *, int); void rt_ifmsg(struct ifnet *); -void rt_missmsg(int, struct rt_addrinfo *, int, int); -void rt_newaddrmsg(int, struct ifaddr *, int, struct rtentry *); +void rt_missmsg(int, struct rt_addrinfo *, int, int, int); +void rt_newaddrmsg(int, struct ifaddr *, int, struct rtentry *, int); void rt_newmaddrmsg(int, struct ifmultiaddr *); int rt_setgate(struct rtentry *, struct sockaddr *, struct sockaddr *); --- ./sys/netinet6/nd6_rtr.c.orig 2009-08-31 18:19:54.000000000 +0000 +++ ./sys/netinet6/nd6_rtr.c 2009-08-31 19:09:27.000000000 +0000 @@ -449,5 +449,5 @@ ifa = NULL; - rt_missmsg(cmd, &info, rt->rt_flags, 0); + rt_missmsg(cmd, &info, rt->rt_flags, 0, -1); if (ifa != NULL) ifa_free(ifa); --- ./sys/netinet6/in6.c.orig 2009-08-31 19:00:43.000000000 +0000 +++ ./sys/netinet6/in6.c 2009-08-31 19:01:00.000000000 +0000 @@ -1238,5 +1238,5 @@ rt_key(&rt0) = (struct sockaddr *)&addr; rt0.rt_flags = RTF_HOST | RTF_STATIC; - rt_newaddrmsg(RTM_DELETE, ifa, 0, &rt0); + rt_newaddrmsg(RTM_DELETE, ifa, 0, &rt0, -1); /* @@ -1831,5 +1831,5 @@ rt_key(&rt) = (struct sockaddr *)&addr; rt.rt_flags = RTF_UP | RTF_HOST | RTF_STATIC; - rt_newaddrmsg(RTM_ADD, &ia->ia_ifa, 0, &rt); + rt_newaddrmsg(RTM_ADD, &ia->ia_ifa, 0, &rt, -1); } --- ./sys/sys/mbuf.h.orig 2009-08-31 18:26:12.000000000 +0000 +++ ./sys/sys/mbuf.h 2009-08-31 18:26:24.000000000 +0000 @@ -897,5 +897,5 @@ #define PACKET_TAG_MACLABEL (19 | MTAG_PERSISTENT) /* MAC label */ #define PACKET_TAG_PF 21 /* PF + ALTQ information */ -#define PACKET_TAG_RTSOCKFAM 25 /* rtsock sa family */ +#define PACKET_TAG_RTSOCK 25 /* rtsock extra info */ #define PACKET_TAG_IPOPTIONS 27 /* Saved IP options */ #define PACKET_TAG_CARP 28 /* CARP info */ --- sys/netinet/in.c.orig 2009-09-28 23:45:49.000000000 +0000 +++ sys/netinet/in.c 2009-09-28 23:46:35.000000000 +0000 @@ -1012,5 +1012,5 @@ rt_newaddrmsg(RTM_ADD, (struct ifaddr *)target, - 0, &msg_rt); + 0, &msg_rt, -1); RTFREE(pfx_ro.ro_rt); } @@ -1073,5 +1073,5 @@ rt_newaddrmsg(RTM_DELETE, (struct ifaddr *)target, - 0, &rt); + 0, &rt, -1); return (0); } --------------000806010602050803090805 Content-Type: text/x-diff; name="freebsd9-route-messages-respect-fib-2.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="freebsd9-route-messages-respect-fib-2.patch" --- ./sys/net/rtsock.c.orig 2009-08-31 15:26:03.000000000 +0000 +++ ./sys/net/rtsock.c 2009-08-31 19:06:53.000000000 +0000 @@ -93,4 +93,9 @@ SYSCTL_NODE(_net, OID_AUTO, route, CTLFLAG_RD, 0, ""); +struct rt_dispatch_ctx { + unsigned short family; /* Socket family */ + int fibnum; /* FIB for message or -1 for all */ +}; + struct walkarg { int w_tmemsize; @@ -114,5 +119,5 @@ static void rt_getmetrics(const struct rt_metrics_lite *in, struct rt_metrics *out); -static void rt_dispatch(struct mbuf *, const struct sockaddr *); +static void rt_dispatch(struct mbuf *, const struct sockaddr *, int); static struct netisr_handler rtsock_nh = { @@ -155,17 +160,19 @@ { struct sockproto route_proto; - unsigned short *family; + struct rt_dispatch_ctx *ctx; struct m_tag *tag; + int fibnum = -1; route_proto.sp_family = PF_ROUTE; - tag = m_tag_find(m, PACKET_TAG_RTSOCKFAM, NULL); + tag = m_tag_find(m, PACKET_TAG_RTSOCK, NULL); if (tag != NULL) { - family = (unsigned short *)(tag + 1); - route_proto.sp_protocol = *family; + ctx = (struct rt_dispatch_ctx*)(tag + 1); + route_proto.sp_protocol = ctx->family; + fibnum = ctx->fibnum; m_tag_delete(m, tag); } else route_proto.sp_protocol = 0; - raw_input(m, &route_proto, &route_src); + raw_input(m, &route_proto, &route_src, fibnum); } @@ -784,8 +791,8 @@ unsigned short family = rp->rcb_proto.sp_family; rp->rcb_proto.sp_family = 0; - rt_dispatch(m, info.rti_info[RTAX_DST]); + rt_dispatch(m, info.rti_info[RTAX_DST], so->so_fibnum); rp->rcb_proto.sp_family = family; } else - rt_dispatch(m, info.rti_info[RTAX_DST]); + rt_dispatch(m, info.rti_info[RTAX_DST], so->so_fibnum); } } @@ -1010,5 +1017,5 @@ */ void -rt_missmsg(int type, struct rt_addrinfo *rtinfo, int flags, int error) +rt_missmsg(int type, struct rt_addrinfo *rtinfo, int flags, int error, int fibnum) { struct rt_msghdr *rtm; @@ -1025,5 +1032,5 @@ rtm->rtm_errno = error; rtm->rtm_addrs = rtinfo->rti_addrs; - rt_dispatch(m, sa); + rt_dispatch(m, sa, fibnum); } @@ -1050,5 +1057,5 @@ ifm->ifm_data = ifp->if_data; ifm->ifm_addrs = 0; - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } @@ -1062,5 +1069,5 @@ */ void -rt_newaddrmsg(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt) +rt_newaddrmsg(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt, int fibnum) { struct rt_addrinfo info; @@ -1120,5 +1127,5 @@ rtm->rtm_addrs = info.rti_addrs; } - rt_dispatch(m, sa); + rt_dispatch(m, sa, fibnum); } } @@ -1156,5 +1163,5 @@ ifmam->ifmam_index = ifp->if_index; ifmam->ifmam_addrs = info.rti_addrs; - rt_dispatch(m, ifma->ifma_addr); + rt_dispatch(m, ifma->ifma_addr, -1); } @@ -1216,5 +1223,5 @@ m->m_pkthdr.len += data_len; mtod(m, struct if_announcemsghdr *)->ifan_msglen += data_len; - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } } @@ -1232,10 +1239,11 @@ m = rt_makeifannouncemsg(ifp, RTM_IFANNOUNCE, what, &info); if (m != NULL) - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } static void -rt_dispatch(struct mbuf *m, const struct sockaddr *sa) +rt_dispatch(struct mbuf *m, const struct sockaddr *sa, int fibnum) { + struct rt_dispatch_ctx *ctx; struct m_tag *tag; @@ -1243,14 +1251,16 @@ * Preserve the family from the sockaddr, if any, in an m_tag for * use when injecting the mbuf into the routing socket buffer from - * the netisr. + * the netisr. Additionally save the fibnum if needed. */ - if (sa != NULL) { - tag = m_tag_get(PACKET_TAG_RTSOCKFAM, sizeof(unsigned short), - M_NOWAIT); + if (sa != NULL || fibnum >= 0) { + tag = m_tag_get(PACKET_TAG_RTSOCK, + sizeof(struct rt_dispatch_ctx*), M_NOWAIT); if (tag == NULL) { m_freem(m); return; } - *(unsigned short *)(tag + 1) = sa->sa_family; + ctx = (struct rt_dispatch_ctx*)(tag + 1); + ctx->family = sa->sa_family; + ctx->fibnum = fibnum; m_tag_prepend(m, tag); } --- ./sys/net/raw_usrreq.c.orig 2009-08-31 16:04:58.000000000 +0000 +++ ./sys/net/raw_usrreq.c 2009-08-31 18:34:38.000000000 +0000 @@ -70,5 +70,5 @@ */ void -raw_input(struct mbuf *m0, struct sockproto *proto, struct sockaddr *src) +raw_input(struct mbuf *m0, struct sockproto *proto, struct sockaddr *src, int fibnum) { struct rawcb *rp; @@ -84,4 +84,7 @@ rp->rcb_proto.sp_protocol != proto->sp_protocol) continue; + if (fibnum >= 0 && rp->rcb_socket && + fibnum != rp->rcb_socket->so_fibnum) + continue; if (last) { struct mbuf *n; --- ./sys/net/raw_cb.h.orig 2009-08-31 18:34:56.000000000 +0000 +++ ./sys/net/raw_cb.h 2009-08-31 18:35:05.000000000 +0000 @@ -73,5 +73,5 @@ int raw_attach(struct socket *, int); void raw_detach(struct rawcb *); -void raw_input(struct mbuf *, struct sockproto *, struct sockaddr *); +void raw_input(struct mbuf *, struct sockproto *, struct sockaddr *, int); /* --- ./sys/net/route.c.orig 2009-08-31 18:18:30.000000000 +0000 +++ ./sys/net/route.c 2009-08-31 18:59:20.000000000 +0000 @@ -384,5 +384,5 @@ bzero(&info, sizeof(info)); info.rti_info[RTAX_DST] = dst; - rt_missmsg(msgtype, &info, 0, err); + rt_missmsg(msgtype, &info, 0, err, fibnum); } done: @@ -609,5 +609,5 @@ info.rti_info[RTAX_NETMASK] = netmask; info.rti_info[RTAX_AUTHOR] = src; - rt_missmsg(RTM_REDIRECT, &info, flags, error); + rt_missmsg(RTM_REDIRECT, &info, flags, error, fibnum); if (ifa != NULL) ifa_free(ifa); @@ -1433,5 +1433,5 @@ rt->rt_ifp->if_index; } - rt_newaddrmsg(cmd, ifa, error, rt); + rt_newaddrmsg(cmd, ifa, error, rt, fibnum); if (cmd == RTM_DELETE) { /* --- ./sys/net/route.h.orig 2009-08-31 18:56:05.000000000 +0000 +++ ./sys/net/route.h 2009-08-31 18:59:32.000000000 +0000 @@ -381,6 +381,6 @@ void rt_ifannouncemsg(struct ifnet *, int); void rt_ifmsg(struct ifnet *); -void rt_missmsg(int, struct rt_addrinfo *, int, int); -void rt_newaddrmsg(int, struct ifaddr *, int, struct rtentry *); +void rt_missmsg(int, struct rt_addrinfo *, int, int, int); +void rt_newaddrmsg(int, struct ifaddr *, int, struct rtentry *, int); void rt_newmaddrmsg(int, struct ifmultiaddr *); int rt_setgate(struct rtentry *, struct sockaddr *, struct sockaddr *); --- ./sys/netinet6/nd6_rtr.c.orig 2009-08-31 18:19:54.000000000 +0000 +++ ./sys/netinet6/nd6_rtr.c 2009-08-31 19:09:27.000000000 +0000 @@ -449,5 +449,5 @@ ifa = NULL; - rt_missmsg(cmd, &info, rt->rt_flags, 0); + rt_missmsg(cmd, &info, rt->rt_flags, 0, -1); if (ifa != NULL) ifa_free(ifa); --- ./sys/netinet6/in6.c.orig 2009-08-31 19:00:43.000000000 +0000 +++ ./sys/netinet6/in6.c 2009-08-31 19:01:00.000000000 +0000 @@ -1238,5 +1238,5 @@ rt_key(&rt0) = (struct sockaddr *)&addr; rt0.rt_flags = RTF_HOST | RTF_STATIC; - rt_newaddrmsg(RTM_DELETE, ifa, 0, &rt0); + rt_newaddrmsg(RTM_DELETE, ifa, 0, &rt0, -1); /* @@ -1831,5 +1831,5 @@ rt_key(&rt) = (struct sockaddr *)&addr; rt.rt_flags = RTF_UP | RTF_HOST | RTF_STATIC; - rt_newaddrmsg(RTM_ADD, &ia->ia_ifa, 0, &rt); + rt_newaddrmsg(RTM_ADD, &ia->ia_ifa, 0, &rt, -1); } --- ./sys/sys/mbuf.h.orig 2009-08-31 18:26:12.000000000 +0000 +++ ./sys/sys/mbuf.h 2009-08-31 18:26:24.000000000 +0000 @@ -897,5 +897,5 @@ #define PACKET_TAG_MACLABEL (19 | MTAG_PERSISTENT) /* MAC label */ #define PACKET_TAG_PF 21 /* PF + ALTQ information */ -#define PACKET_TAG_RTSOCKFAM 25 /* rtsock sa family */ +#define PACKET_TAG_RTSOCK 25 /* rtsock extra info */ #define PACKET_TAG_IPOPTIONS 27 /* Saved IP options */ #define PACKET_TAG_CARP 28 /* CARP info */ --- sys/netinet/in.c.orig 2009-09-28 23:45:49.000000000 +0000 +++ sys/netinet/in.c 2009-09-28 23:46:35.000000000 +0000 @@ -1012,5 +1012,5 @@ rt_newaddrmsg(RTM_ADD, (struct ifaddr *)target, - 0, &msg_rt); + 0, &msg_rt, -1); RTFREE(pfx_ro.ro_rt); } @@ -1073,5 +1073,5 @@ rt_newaddrmsg(RTM_DELETE, (struct ifaddr *)target, - 0, &rt); + 0, &rt, -1); return (0); } --------------000806010602050803090805 Content-Type: text/x-diff; name="freebsd72-route-messages-respect-fib-2.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="freebsd72-route-messages-respect-fib-2.patch" --- ./sys/net/rtsock.c.orig 2008-11-25 02:59:29.000000000 +0000 +++ ./sys/net/rtsock.c 2009-09-11 00:07:13.000000000 +0000 @@ -83,4 +83,9 @@ &rtsintrq.ifq_maxlen, 0, "maximum routing socket dispatch queue length"); +struct rt_dispatch_ctx { + unsigned short family; /* Socket family */ + int fibnum; /* FIB for message or -1 for all */ +}; + struct walkarg { int w_tmemsize; @@ -104,5 +109,5 @@ static void rt_getmetrics(const struct rt_metrics_lite *in, struct rt_metrics *out); -static void rt_dispatch(struct mbuf *, const struct sockaddr *); +static void rt_dispatch(struct mbuf *, const struct sockaddr *, int); static void @@ -123,17 +128,19 @@ { struct sockproto route_proto; - unsigned short *family; + struct rt_dispatch_ctx *ctx; struct m_tag *tag; + int fibnum = -1; route_proto.sp_family = PF_ROUTE; - tag = m_tag_find(m, PACKET_TAG_RTSOCKFAM, NULL); + tag = m_tag_find(m, PACKET_TAG_RTSOCK, NULL); if (tag != NULL) { - family = (unsigned short *)(tag + 1); - route_proto.sp_protocol = *family; + ctx = (struct rt_dispatch_ctx*)(tag + 1); + route_proto.sp_protocol = ctx->family; + fibnum = ctx->fibnum; m_tag_delete(m, tag); } else route_proto.sp_protocol = 0; - raw_input(m, &route_proto, &route_src); + raw_input(m, &route_proto, &route_src, fibnum); } @@ -605,8 +612,8 @@ unsigned short family = rp->rcb_proto.sp_family; rp->rcb_proto.sp_family = 0; - rt_dispatch(m, info.rti_info[RTAX_DST]); + rt_dispatch(m, info.rti_info[RTAX_DST], so->so_fibnum); rp->rcb_proto.sp_family = family; } else - rt_dispatch(m, info.rti_info[RTAX_DST]); + rt_dispatch(m, info.rti_info[RTAX_DST], so->so_fibnum); } } @@ -829,5 +836,5 @@ */ void -rt_missmsg(int type, struct rt_addrinfo *rtinfo, int flags, int error) +rt_missmsg(int type, struct rt_addrinfo *rtinfo, int flags, int error, int fibnum) { struct rt_msghdr *rtm; @@ -844,5 +851,5 @@ rtm->rtm_errno = error; rtm->rtm_addrs = rtinfo->rti_addrs; - rt_dispatch(m, sa); + rt_dispatch(m, sa, fibnum); } @@ -869,5 +876,5 @@ ifm->ifm_data = ifp->if_data; ifm->ifm_addrs = 0; - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } @@ -881,5 +888,5 @@ */ void -rt_newaddrmsg(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt) +rt_newaddrmsg(int cmd, struct ifaddr *ifa, int error, struct rtentry *rt, int fibnum) { struct rt_addrinfo info; @@ -937,5 +944,5 @@ rtm->rtm_addrs = info.rti_addrs; } - rt_dispatch(m, sa); + rt_dispatch(m, sa, fibnum); } } @@ -973,5 +980,5 @@ ifmam->ifmam_index = ifp->if_index; ifmam->ifmam_addrs = info.rti_addrs; - rt_dispatch(m, ifma->ifma_addr); + rt_dispatch(m, ifma->ifma_addr, -1); } @@ -1033,5 +1040,5 @@ m->m_pkthdr.len += data_len; mtod(m, struct if_announcemsghdr *)->ifan_msglen += data_len; - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } } @@ -1049,10 +1056,11 @@ m = rt_makeifannouncemsg(ifp, RTM_IFANNOUNCE, what, &info); if (m != NULL) - rt_dispatch(m, NULL); + rt_dispatch(m, NULL, -1); } static void -rt_dispatch(struct mbuf *m, const struct sockaddr *sa) +rt_dispatch(struct mbuf *m, const struct sockaddr *sa, int fibnum) { + struct rt_dispatch_ctx *ctx; struct m_tag *tag; @@ -1060,14 +1068,16 @@ * Preserve the family from the sockaddr, if any, in an m_tag for * use when injecting the mbuf into the routing socket buffer from - * the netisr. + * the netisr. Additionally save the fibnum if needed. */ - if (sa != NULL) { - tag = m_tag_get(PACKET_TAG_RTSOCKFAM, sizeof(unsigned short), - M_NOWAIT); + if (sa != NULL || fibnum >= 0) { + tag = m_tag_get(PACKET_TAG_RTSOCK, + sizeof(struct rt_dispatch_ctx*), M_NOWAIT); if (tag == NULL) { m_freem(m); return; } - *(unsigned short *)(tag + 1) = sa->sa_family; + ctx = (struct rt_dispatch_ctx*)(tag + 1); + ctx->family = sa->sa_family; + ctx->fibnum = fibnum; m_tag_prepend(m, tag); } --- ./sys/net/raw_usrreq.c.orig 2008-11-25 02:59:29.000000000 +0000 +++ ./sys/net/raw_usrreq.c 2009-09-11 00:07:13.000000000 +0000 @@ -68,5 +68,5 @@ */ void -raw_input(struct mbuf *m0, struct sockproto *proto, struct sockaddr *src) +raw_input(struct mbuf *m0, struct sockproto *proto, struct sockaddr *src, int fibnum) { struct rawcb *rp; @@ -82,4 +82,7 @@ rp->rcb_proto.sp_protocol != proto->sp_protocol) continue; + if (fibnum >= 0 && rp->rcb_socket && + fibnum != rp->rcb_socket->so_fibnum) + continue; if (last) { struct mbuf *n; --- ./sys/net/raw_cb.h.orig 2008-11-25 02:59:29.000000000 +0000 +++ ./sys/net/raw_cb.h 2009-09-11 00:07:13.000000000 +0000 @@ -71,5 +71,5 @@ int raw_attach(struct socket *, int); void raw_detach(struct rawcb *); -void raw_input(struct mbuf *, struct sockproto *, struct sockaddr *); +void raw_input(struct mbuf *, struct sockproto *, struct sockaddr *, int); /* --- ./sys/net/route.c.orig 2008-11-25 02:59:29.000000000 +0000 +++ ./sys/net/route.c 2009-09-11 00:08:28.000000000 +0000 @@ -338,5 +338,5 @@ info.rti_info[RTAX_IFA] = newrt->rt_ifa->ifa_addr; } - rt_missmsg(RTM_ADD, &info, newrt->rt_flags, 0); + rt_missmsg(RTM_ADD, &info, newrt->rt_flags, 0, fibnum); } else { KASSERT(rt == newrt, ("locking wrong route")); @@ -362,5 +362,5 @@ bzero(&info, sizeof(info)); info.rti_info[RTAX_DST] = dst; - rt_missmsg(msgtype, &info, 0, err); + rt_missmsg(msgtype, &info, 0, err, fibnum); } } @@ -575,5 +575,5 @@ info.rti_info[RTAX_NETMASK] = netmask; info.rti_info[RTAX_AUTHOR] = src; - rt_missmsg(RTM_REDIRECT, &info, flags, error); + rt_missmsg(RTM_REDIRECT, &info, flags, error, fibnum); } @@ -1464,5 +1464,5 @@ */ RT_LOCK(rt); - rt_newaddrmsg(cmd, ifa, error, rt); + rt_newaddrmsg(cmd, ifa, error, rt, fibnum); if (cmd == RTM_DELETE) { /* --- ./sys/net/route.h.orig 2008-11-25 02:59:29.000000000 +0000 +++ ./sys/net/route.h 2009-09-11 00:07:13.000000000 +0000 @@ -350,6 +350,6 @@ void rt_ifannouncemsg(struct ifnet *, int); void rt_ifmsg(struct ifnet *); -void rt_missmsg(int, struct rt_addrinfo *, int, int); -void rt_newaddrmsg(int, struct ifaddr *, int, struct rtentry *); +void rt_missmsg(int, struct rt_addrinfo *, int, int, int); +void rt_newaddrmsg(int, struct ifaddr *, int, struct rtentry *, int); void rt_newmaddrmsg(int, struct ifmultiaddr *); int rt_setgate(struct rtentry *, struct sockaddr *, struct sockaddr *); --- ./sys/netinet6/nd6_rtr.c.orig 2008-11-25 02:59:29.000000000 +0000 +++ ./sys/netinet6/nd6_rtr.c 2009-09-11 00:07:13.000000000 +0000 @@ -444,5 +444,5 @@ } - rt_missmsg(cmd, &info, rt->rt_flags, 0); + rt_missmsg(cmd, &info, rt->rt_flags, 0, -1); } --- ./sys/netinet6/in6.c.orig 2009-06-10 10:31:11.000000000 +0000 +++ ./sys/netinet6/in6.c 2009-09-11 00:09:37.000000000 +0000 @@ -190,5 +190,5 @@ } - rt_newaddrmsg(cmd, ifa, e, nrt); + rt_newaddrmsg(cmd, ifa, e, nrt, -1); if (cmd == RTM_DELETE) RTFREE_LOCKED(nrt); --- ./sys/sys/mbuf.h.orig 2008-11-25 02:59:29.000000000 +0000 +++ ./sys/sys/mbuf.h 2009-09-11 00:10:16.000000000 +0000 @@ -852,5 +852,5 @@ #define PACKET_TAG_MACLABEL (19 | MTAG_PERSISTENT) /* MAC label */ #define PACKET_TAG_PF 21 /* PF + ALTQ information */ -#define PACKET_TAG_RTSOCKFAM 25 /* rtsock sa family */ +#define PACKET_TAG_RTSOCK 25 /* rtsock extra info */ #define PACKET_TAG_IPOPTIONS 27 /* Saved IP options */ #define PACKET_TAG_CARP 28 /* CARP info */ --------------000806010602050803090805-- From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 04:14:15 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2524A1065695 for ; Fri, 9 Oct 2009 04:14:15 +0000 (UTC) (envelope-from siquijorphilips@gmail.com) Received: from mail-vw0-f180.google.com (mail-vw0-f180.google.com [209.85.212.180]) by mx1.freebsd.org (Postfix) with ESMTP id 4F4558FC18 for ; Fri, 9 Oct 2009 04:14:13 +0000 (UTC) Received: by vws10 with SMTP id 10so3776717vws.7 for ; Thu, 08 Oct 2009 21:14:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=i2wWBDmy28/4deRAgWSkSahHovUSXJ5VbKyCwRlC0Aw=; b=jF1+2rNuRHigULuZfNhM5CTwGKrVc3o5+tDal3ItalO6AsjoRr+g/VUqk0nZRf6mEZ MsdtxNn46a9Guh1pnSjuk7vFn4uZjUG8iWEWiTbbmMHa7LGIedVuJKOyV8+xRqb1+LId l6iYrw32cNRgaH+OeqVwfe1G7ovbGo7rvxkmg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=EYSd3fhLfbryMmngODcCRd1N2Lijyi4y0FGLt+XNUs8RtcoPoPGl+84ryl3ZgjtLB+ SfQAgkiAlZtrSv8HrAn+t9ApsihUPQ5dF0kRUV5CYDemivQ8ydeoKPV/jAZyx9DK74yk H0m3bykbYDhbEKYo5mWviuLNB/3x3Ob1REed0= MIME-Version: 1.0 Received: by 10.220.111.195 with SMTP id t3mr3369433vcp.8.1255061653283; Thu, 08 Oct 2009 21:14:13 -0700 (PDT) In-Reply-To: <20091008174521.GE3843@michelle.cdnetworks.com> References: <20091008174521.GE3843@michelle.cdnetworks.com> Date: Fri, 9 Oct 2009 12:14:13 +0800 Message-ID: From: Siquijor Philips To: pyunyh@gmail.com Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, freebsd-drivers@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 04:14:15 -0000 On Fri, Oct 9, 2009 at 1:45 AM, Pyun YongHyeon wrote: > On Thu, Oct 08, 2009 at 01:24:20PM +0800, Siquijor Philips wrote: >> Hi, >> >> I got a dual-port Intel Gigabit NIC with 82576 (ET) chipset >> http://www.intel.com/Assets/PDF/prodbrief/320116.pdf. It has a feature >> on IPsec offloading but it only mentioned Microsoft Windows 2008 and >> Vista servers. I wonder if FreeBSD have also support on this feature? >> > > AFAIK it's not yet, not sure whether Jack has plan to implement the > offloading. I know old Intel i82550 also supported IPSec offloading > but Intel didn't release required information to implement it. 3Com > also supported IPSec offloading in their 3XP hardwares(txp(4)) but > the offloading was not implemented. > Hi Pyun, Thanks for your info! By the way, who is Jack? Is he the author of this driver? I really need to have this feature usable on the driver. I bought these NICs with 82576 chipset for the purpose of implementing IPsec in my network and my current FreeBSD servers could benefit it. Just really thought it has support because FreeBSD was already part of the supported operating system. I was alerted when I've re-read the product info document again that it only support Windows 2008 and Vista platforms. Now, I've confirmed that with the current existing driver. I hope this guy has the plan of implementing it sooner because I really need it. Thanks, Siquijor From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 07:21:38 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 040871065695 for ; Fri, 9 Oct 2009 07:21:38 +0000 (UTC) (envelope-from bennett@cs.niu.edu) Received: from mp.cs.niu.edu (mp.cs.niu.edu [131.156.145.41]) by mx1.freebsd.org (Postfix) with ESMTP id CAA8D8FC1B for ; Fri, 9 Oct 2009 07:21:37 +0000 (UTC) Received: from mp.cs.niu.edu (bennett@localhost [127.0.0.1]) by mp.cs.niu.edu (8.14.3/8.14.3) with ESMTP id n996MwYJ015693; Fri, 9 Oct 2009 01:22:58 -0500 (CDT) Date: Fri, 9 Oct 2009 01:22:58 -0500 (CDT) From: Scott Bennett Message-Id: <200910090622.n996MwJV015692@mp.cs.niu.edu> To: freebsd-net@freebsd.org, rihad Cc: Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 07:21:38 -0000 On Wed, 07 Oct 2009 20:02:21 +0500 rihad wrote: >Ingo Flaschberger wrote: >> Hi, >> >> can you send me the dmesg ouput from your networkcards when they are >> detected at booting? >> >Hello, > >bce0: mem >0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7 >bce0: Ethernet address: 00:1d:09:xx:xx:xx >bce0: [ITHREAD] >bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W >(0x03050C05); Flags( MFW MSI ) >bce1: mem >0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3 >bce1: Ethernet address: 00:1d:09:xx:xx:xx >bce1: [ITHREAD] >bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W >(0x03050C05); Flags( MFW MSI ) > > >> can you also send me a lspci and lspci -v ? >> >Sorry, this is FreeBSD, not Linux ;-) > Try sysutils/pciutils in ports. Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at cs.niu.edu * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * ********************************************************************** From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 08:27:44 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D797A1065670 for ; Fri, 9 Oct 2009 08:27:44 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id 91D578FC08 for ; Fri, 9 Oct 2009 08:27:44 +0000 (UTC) Received: by lath.rinet.ru (Postfix, from userid 222) id 5EB55704E; Fri, 9 Oct 2009 12:27:43 +0400 (MSD) Date: Fri, 9 Oct 2009 12:27:43 +0400 From: Oleg Bulyzhin To: Julian Elischer Message-ID: <20091009082743.GB70940@lath.rinet.ru> References: <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <4ACCC30E.7080504@elischer.org> <4ACCC4F3.3030302@mail.ru> <20091008060608.GA23793@lath.rinet.ru> <4ACE10CF.2030806@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4ACE10CF.2030806@elischer.org> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: rihad , freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 08:27:44 -0000 On Thu, Oct 08, 2009 at 09:18:23AM -0700, Julian Elischer wrote: > that seems like a bug to me.. > neither tee should ever terminate a search. agree. But documented bug is a feature ;) and i'm not sure if we fix this wouldnt it break POLA? > > if you want to terminate it, add a specific rule to do so. > Unfortunately I wasn't involved in writing it. ip_fw_pfil.c: revision 1.17 date: 2005/02/05 12:06:33; author: glebius; state: Exp; lines: +47 -0 Add a ng_ipfw node, implementing a quick and simple interface between ipfw(4) and netgraph(4) facilities. Reviewed by: andre, brooks, julian ^^^^^^ -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================ From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 09:15:53 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60ADE106566B for ; Fri, 9 Oct 2009 09:15:53 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: from mail-fx0-f222.google.com (mail-fx0-f222.google.com [209.85.220.222]) by mx1.freebsd.org (Postfix) with ESMTP id EB7278FC08 for ; Fri, 9 Oct 2009 09:15:52 +0000 (UTC) Received: by fxm22 with SMTP id 22so6372878fxm.36 for ; Fri, 09 Oct 2009 02:15:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=NbeAIQdv549XNcR1IiKUOK5nRTQnsabXy5e1/AZvJTo=; b=S47GPD5mEDKF5z/AYp9jJ7WjNy7kNo4BNQUaWPFpYS8TAgQNt5YVULvXx+XlLLyWOr JFFz5NMqoBgmpdT/+4L4MUxbEhtM1/yLX2BGCiKbZAzBvhrwZRX8i61CCYSPqmKpVWys I79xw3mkHx+Mh/zHPK3IPndKcyYFHqcF9a1mg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=BLUx7+z8JONg1S3U2vjUUh4BLZa2B+vfinOl0ONAzYnrBN0U/Zk4hmae9YLyRCeDAR uhUPJ0Eo4ej8WiDiFUnQ6eKasrxG3iao8nByftX+M7a4IGg2hYA88oSYuQZgjj6BBi5j NP73MHu3ded0oPsUA2fqeWjX69+8OvVE8xgVw= MIME-Version: 1.0 Received: by 10.86.232.5 with SMTP id e5mr2113751fgh.27.1255077962237; Fri, 09 Oct 2009 01:46:02 -0700 (PDT) Date: Fri, 9 Oct 2009 10:46:02 +0200 Message-ID: From: Jacques Fourie To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Route re-calculation in ip_output() X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 09:15:53 -0000 Hi, I've noticed what I believe to be a bug in ip_output(). The piece of code in question is when the firewall changes the destination address of an outgoing packet and the subsequent re-calculation of the route. The issue should be clear from the attached diff - basically what happens is that for the second route lookup dst can point to ro->ro_rt->rt_gateway instead of &ro->ro_dst. It seems as if this issue is present on 7,8 and 9? --- ip_output.c 2009-10-09 10:37:40.537408240 +0200 +++ /home/jacques/ip_output.c 2009-10-09 10:43:46.232819440 +0200 @@ -521,8 +521,10 @@ #endif error = netisr_queue(NETISR_IP, m); goto done; - } else + } else { + dst = (struct sockaddr_in *)&ro->ro_dst; goto again; /* Redo the routing table lookup. */ + } Regards, Jacques From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 14:35:12 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D7991065672; Fri, 9 Oct 2009 14:35:12 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from mx75.mail.ru (mx75.mail.ru [94.100.176.90]) by mx1.freebsd.org (Postfix) with ESMTP id 724B88FC22; Fri, 9 Oct 2009 14:35:11 +0000 (UTC) Received: from [217.25.27.27] (port=23773 helo=[217.25.27.27]) by mx75.mail.ru with asmtp id 1MwGYn-000IAF-00; Fri, 09 Oct 2009 18:35:09 +0400 Message-ID: <4ACF4A15.1010203@mail.ru> Date: Fri, 09 Oct 2009 19:35:01 +0500 From: rihad User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Oleg Bulyzhin References: <4AC8A76B.3050502@mail.ru> <20091007085902.GA88982@lath.rinet.ru> <4ACC5E23.8090405@mail.ru> <20091007100503.GB88982@lath.rinet.ru> <4ACC6A7B.5050808@mail.ru> <20091007104525.GC88982@lath.rinet.ru> <4ACC7308.6070301@mail.ru> <20091007115425.GD88982@lath.rinet.ru> In-Reply-To: <20091007115425.GD88982@lath.rinet.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: dummynet dropping too many packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 14:35:12 -0000 Oleg Bulyzhin wrote: > On Wed, Oct 07, 2009 at 03:52:56PM +0500, rihad wrote: > >> You probably have some special sources of documentation ;-) According to >> man ipfw, both "netgraph/ngtee" and "pipe" decide the fate of the packet >> unless one_pass=0. Or do you mean sprinkling smart skiptos here and >> there? ;-) > > you can > 1) use ng_ether & ng_netflow. (so no need in 'ngtee' rule). > 2) use 'tee' rule with ng_ksocket & ng_netflow > >>> Could you show your 'ipfw show' output? (hide ip addresses if you wish but >>> keep counters please). >>> > >> Here it is, in its whole glory: >> >> 00100 10434423 1484891105 allow ip from any to any via lo0 >> 00200 2 14 deny ip from any to 127.0.0.0/8 >> 00300 1 4 deny ip from 127.0.0.0/8 to any >> 01000 3300039938 327603104711 allow ip from any to any in >> 01010 26214900 421138433 allow ip from me to any out >> 01020 5453857 46806278 allow icmp from any to any out >> 01030 3268289053 327224694165 ngtee 1 ip from any to any out >> 01040 18681181 1089636054 skipto 1100 ip from table(127) to any out >> recv bce0 xmit bce1 >> 01060 777488848 76743392754 pipe tablearg ip from any to table(0) out >> recv bce0 xmit bce1 >> 01070 776831109 76682499457 allow ip from any to table(0) out recv >> bce0 xmit bce1 >> 01100 13102697 808411842 pipe tablearg ip from any to table(2) out >> 65535 662648946 66711487830 allow ip from any to any > > I guess this one would be better(faster): > > 00050 allow ip from any to any in > 00100 allow ip from any to any via lo0 > 01010 allow ip from me to any > 01020 allow icmp from any to any > 01030 ngtee 1 ip from any to any > 01035 skipto 1040 ip from any to any recv bce0 xmit bce1 > 01036 allow ip from any to any > 01040 skipto 1100 ip from table(127) to any > 01060 pipe tablearg ip from any to table(0) > 01070 allow ip from any to any > 01100 pipe tablearg ip from any to table(2) > 65535 allow ip from any to any > Tried it just now - no visible effect. 400-700 packet drops per second which is around 5-7 mbps dropped on output. So I don't think getting rid of one_pass=0 would help at all. From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 15:58:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E974E106566B for ; Fri, 9 Oct 2009 15:58:26 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outD.internet-mail-service.net (outd.internet-mail-service.net [216.240.47.227]) by mx1.freebsd.org (Postfix) with ESMTP id D220A8FC14 for ; Fri, 9 Oct 2009 15:58:26 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id E1604E3133; Fri, 9 Oct 2009 08:58:26 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 491F82D601A; Fri, 9 Oct 2009 08:58:26 -0700 (PDT) Message-ID: <4ACF5DA5.6060806@elischer.org> Date: Fri, 09 Oct 2009 08:58:29 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: Jacques Fourie References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Route re-calculation in ip_output() X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 15:58:27 -0000 Jacques Fourie wrote: > Hi, > > I've noticed what I believe to be a bug in ip_output(). The piece of > code in question is when the firewall changes the destination address > of an outgoing packet and the subsequent re-calculation of the route. > The issue should be clear from the attached diff - basically what > happens is that for the second route lookup dst can point to > ro->ro_rt->rt_gateway instead of &ro->ro_dst. It seems as if this > issue is present on 7,8 and 9? Is this a problem? generally, the aim of a fwd firewall rule is to set the next hop (gateway). so this may be what is required.. > > --- ip_output.c 2009-10-09 10:37:40.537408240 +0200 > +++ /home/jacques/ip_output.c 2009-10-09 10:43:46.232819440 +0200 > @@ -521,8 +521,10 @@ > #endif > error = netisr_queue(NETISR_IP, m); > goto done; > - } else > + } else { > + dst = (struct sockaddr_in *)&ro->ro_dst; > goto again; /* Redo the routing table lookup. */ > + } > > > Regards, > Jacques > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 16:24:34 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F41B41065676 for ; Fri, 9 Oct 2009 16:24:33 +0000 (UTC) (envelope-from freebsd@optiksecurite.com) Received: from relais.videotron.ca (relais.videotron.ca [24.201.245.36]) by mx1.freebsd.org (Postfix) with ESMTP id CE95B8FC17 for ; Fri, 9 Oct 2009 16:24:33 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; charset=ISO-8859-1; format=flowed Received: from [69.69.69.193] ([24.201.201.211]) by VL-MH-MR002.ip.videotron.ca (Sun Java(tm) System Messaging Server 6.3-4.01 (built Aug 3 2007; 32bit)) with ESMTP id <0KR900HZE6S5LIF0@VL-MH-MR002.ip.videotron.ca> for freebsd-net@freebsd.org; Fri, 09 Oct 2009 11:24:05 -0400 (EDT) Message-id: <4ACF5597.5000107@optiksecurite.com> Date: Fri, 09 Oct 2009 11:24:07 -0400 From: Martin Turgeon User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) To: freebsd-net@freebsd.org Subject: How can I get >100 connections in FIN_WAIT_2 state from the same IP? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 16:24:34 -0000 Hi everyone, I would like to know if anyone knows the reason why I get a lot of connections (more than 100) from the same IP in FIN_WAIT_2 state. Refering to this diagram (http://www.jxos.org/Projects/TCP/tcpstate.html), the connection enter in FIN_WAIT_1 when the server closes the connection and in FIN_WAIT_2 when the client ACK the FIN from the server. For the connection to stay in FIN_WAIT_2, the client must never send his FIN, right? In this case the connections are on port 80. Is it a problem with the client's browser or OS? Is it possible that some mobile devices doesn't close their connections correctly to save bandwidth and battery? I know this isn't specific to FreeBSD, but thanks for your answer anyway Martin From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 17:35:22 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA25B1065672 for ; Fri, 9 Oct 2009 17:35:22 +0000 (UTC) (envelope-from Lutz.Bichler@gmx.de) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.freebsd.org (Postfix) with SMTP id 0258D8FC0C for ; Fri, 9 Oct 2009 17:35:21 +0000 (UTC) Received: (qmail 13955 invoked by uid 0); 9 Oct 2009 17:08:41 -0000 Received: from 79.207.244.133 by www124.gmx.net with HTTP; Fri, 09 Oct 2009 19:08:38 +0200 (CEST) Content-Type: text/plain; charset="iso-8859-1" Date: Fri, 09 Oct 2009 19:08:39 +0200 From: "Lutz Bichler" Message-ID: <20091009170839.142800@gmx.net> MIME-Version: 1.0 To: freebsd-net@freebsd.org X-Authenticated: #11523194 X-Flags: 0001 X-Mailer: WWW-Mail 6100 (Global Message Exchange) X-Priority: 3 X-Provags-ID: V01U2FsdGVkX19DVQQObSVDoQPy3s7FHe5eGit6yob3GnyosRJVfu mqETM1Kn7BclGOe0DKa0U2AImt/hkPiK45Rg== Content-Transfer-Encoding: 8bit X-GMX-UID: JRT6IqNdZDIrFxTn1WY2YJN5emhmYwGV X-FuHaFi: 0.75 Subject: Intel WiFi 5100/5300 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 17:35:22 -0000 Hi, does anybody know what happened to the attempts to support Intel WiFi 5100/5300 interfaces in the iwn-driver? Are any patches available which could be used to start working on support for these interfaces? Lutz -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 18:17:01 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0A9D106568B for ; Fri, 9 Oct 2009 18:17:01 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yw0-f197.google.com (mail-yw0-f197.google.com [209.85.211.197]) by mx1.freebsd.org (Postfix) with ESMTP id 878858FC1C for ; Fri, 9 Oct 2009 18:17:01 +0000 (UTC) Received: by ywh35 with SMTP id 35so19824796ywh.7 for ; Fri, 09 Oct 2009 11:17:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=a4ck6qdlwt8aGERWP5oBbFFGf5MbYprBPZf5FlZi5yU=; b=j9YjA5tdqWNaSrT6QhMbepvJyHK+7f1dllRK2BUj2kdyBj2bMgwRFml6h5ZFU/4CnC 1Agi2W1SKcW7T0+OetUnDFCQKLIYyWDMdHSY9DlvPSvC2Irdk6h4s1pABxH8iTH9qKiF RVITgDCWIF3hAvoMnqlk2Xx2z0YeDdei6iLqw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=rPcRKfmG8YE2Ia/mwht0pakl+ugBxlVY8mdLOD8dZYidRwzZMlSRw/YNQOEpUZMjLU KPWtITQiN9TpuLaPs87VdeVvf2dm2cCWTl3RCys5gRMwb88KSrZg7WChQkzB6zBIrRGR TYh+22sh/MRX7SK0b9c6sZmtAmTAG6KVb7vxc= MIME-Version: 1.0 Received: by 10.101.64.16 with SMTP id r16mr3401661ank.195.1255112220558; Fri, 09 Oct 2009 11:17:00 -0700 (PDT) In-Reply-To: References: <20091008174521.GE3843@michelle.cdnetworks.com> Date: Fri, 9 Oct 2009 11:17:00 -0700 Message-ID: <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> From: Jack Vogel To: Siquijor Philips Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pyunyh@gmail.com, freebsd-net@freebsd.org, freebsd-drivers@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 18:17:02 -0000 I am Jack, the network engineer at Intel responsible for all FreeBSD wired lan drivers. This is the first I've seen about this. Our understanding was that the infrastructure needed to do IPSec was not available for either Linux or FreeBSD, can you please explain things? If everything is there except the support in the driver then I might be able to add that to my queue. Cheers, Jack On Thu, Oct 8, 2009 at 9:14 PM, Siquijor Philips wrote: > On Fri, Oct 9, 2009 at 1:45 AM, Pyun YongHyeon wrote: > > On Thu, Oct 08, 2009 at 01:24:20PM +0800, Siquijor Philips wrote: > >> Hi, > >> > >> I got a dual-port Intel Gigabit NIC with 82576 (ET) chipset > >> http://www.intel.com/Assets/PDF/prodbrief/320116.pdf. It has a feature > >> on IPsec offloading but it only mentioned Microsoft Windows 2008 and > >> Vista servers. I wonder if FreeBSD have also support on this feature? > >> > > > > AFAIK it's not yet, not sure whether Jack has plan to implement the > > offloading. I know old Intel i82550 also supported IPSec offloading > > but Intel didn't release required information to implement it. 3Com > > also supported IPSec offloading in their 3XP hardwares(txp(4)) but > > the offloading was not implemented. > > > > Hi Pyun, > > Thanks for your info! By the way, who is Jack? Is he the author of > this driver? I really need to have this feature usable on the driver. > I bought these NICs with 82576 chipset for the purpose of implementing > IPsec in my network and my current FreeBSD servers could benefit it. > Just really thought it has support because FreeBSD was already part of > the supported operating system. I was alerted when I've re-read the > product info document again that it only support Windows 2008 and > Vista platforms. Now, I've confirmed that with the current existing > driver. I hope this guy has the plan of implementing it sooner because > I really need it. > > Thanks, > Siquijor > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 18:28:12 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 809271065676 for ; Fri, 9 Oct 2009 18:28:12 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mx01.netsrc.de (mx01.netsrc.de [89.107.71.100]) by mx1.freebsd.org (Postfix) with ESMTP id 45AE18FC0A for ; Fri, 9 Oct 2009 18:28:11 +0000 (UTC) Received: from jessie.localnet (dslb-088-067-212-236.pools.arcor-ip.net [88.67.212.236]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx01.netsrc.de (Postfix) with ESMTP id 990F2192FD9; Fri, 9 Oct 2009 20:04:19 +0200 (CEST) From: Bernhard Schmidt To: freebsd-net@freebsd.org Date: Fri, 9 Oct 2009 20:03:57 +0200 User-Agent: KMail/1.12.1 (Linux/2.6.30-2-686; KDE/4.3.1; i686; ; ) References: <20091009170839.142800@gmx.net> In-Reply-To: <20091009170839.142800@gmx.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200910092003.57367.bschmidt@techwires.net> Cc: Lutz Bichler Subject: Re: Intel WiFi 5100/5300 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 18:28:12 -0000 On Friday 09 October 2009 19:08:39 Lutz Bichler wrote: > Hi, > > does anybody know what happened to the attempts to support Intel WiFi > 5100/5300 interfaces in the iwn-driver? Are any patches available which > could be used to start working on support for these interfaces? I'm curious too, as I'm playing with idea to start porting the latest changes to if_iwn from OpenBSD. Already started with adding the 5000 series firmware to iwnfw... -- Bernhard From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 18:47:34 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9777D1065672 for ; Fri, 9 Oct 2009 18:47:34 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outK.internet-mail-service.net (outk.internet-mail-service.net [216.240.47.234]) by mx1.freebsd.org (Postfix) with ESMTP id 78B968FC14 for ; Fri, 9 Oct 2009 18:47:34 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 86195E4A37; Fri, 9 Oct 2009 11:47:34 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id C3E2F2D601C; Fri, 9 Oct 2009 11:47:33 -0700 (PDT) Message-ID: <4ACF8548.8090309@elischer.org> Date: Fri, 09 Oct 2009 11:47:36 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: Jack Vogel References: <20091008174521.GE3843@michelle.cdnetworks.com> <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> In-Reply-To: <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pyunyh@gmail.com, Siquijor Philips , freebsd-drivers@freebsd.org, freebsd-net@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 18:47:34 -0000 Jack Vogel wrote: > I am Jack, the network engineer at Intel responsible for all FreeBSD wired > lan drivers. > This is the first I've seen about this. Our understanding was that the > infrastructure needed > to do IPSec was not available for either Linux or FreeBSD, can you please > explain things? > > If everything is there except the support in the driver then I might be able > to add that to > my queue. > If we knew what the driver would need then that might help too :-) (ha gotcha) :-) > Cheers, > > Jack > > > On Thu, Oct 8, 2009 at 9:14 PM, Siquijor Philips > wrote: > >> On Fri, Oct 9, 2009 at 1:45 AM, Pyun YongHyeon wrote: >>> On Thu, Oct 08, 2009 at 01:24:20PM +0800, Siquijor Philips wrote: >>>> Hi, >>>> >>>> I got a dual-port Intel Gigabit NIC with 82576 (ET) chipset >>>> http://www.intel.com/Assets/PDF/prodbrief/320116.pdf. It has a feature >>>> on IPsec offloading but it only mentioned Microsoft Windows 2008 and >>>> Vista servers. I wonder if FreeBSD have also support on this feature? >>>> >>> AFAIK it's not yet, not sure whether Jack has plan to implement the >>> offloading. I know old Intel i82550 also supported IPSec offloading >>> but Intel didn't release required information to implement it. 3Com >>> also supported IPSec offloading in their 3XP hardwares(txp(4)) but >>> the offloading was not implemented. >>> >> Hi Pyun, >> >> Thanks for your info! By the way, who is Jack? Is he the author of >> this driver? I really need to have this feature usable on the driver. >> I bought these NICs with 82576 chipset for the purpose of implementing >> IPsec in my network and my current FreeBSD servers could benefit it. >> Just really thought it has support because FreeBSD was already part of >> the supported operating system. I was alerted when I've re-read the >> product info document again that it only support Windows 2008 and >> Vista platforms. Now, I've confirmed that with the current existing >> driver. I hope this guy has the plan of implementing it sooner because >> I really need it. >> >> Thanks, >> Siquijor >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 18:49:47 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44CBF106568F for ; Fri, 9 Oct 2009 18:49:47 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-fx0-f222.google.com (mail-fx0-f222.google.com [209.85.220.222]) by mx1.freebsd.org (Postfix) with ESMTP id 716B48FC0C for ; Fri, 9 Oct 2009 18:49:46 +0000 (UTC) Received: by fxm22 with SMTP id 22so6789160fxm.36 for ; Fri, 09 Oct 2009 11:49:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=QZD+uaYq5jWfXIEBpVuh/yoNES22e8d0fl8Jt0Uk5CI=; b=hLq9Cp6Qx1DYAX8yujonAupH9POdSixBDB8+hREsCI9/hDJHpiMnDt6E/IPfGZb0bD WFZlZP5r5E+ffzhJxJdO/mqsLYBil5D13mdHif6t4OEg10nRMOJ9OhfUdKcif7aaBpG+ N9UhBF0A91Mt7giEeM7mbqzQt6AZBI/j9OgK0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=Un76pzgvwLJJhuXldKPqlQylORezFuSTf3n3UuA/luTQCaLQUB7TOwZfrosvCXJWBp eL9qOoKbgYJuCnMHAEcU0Y9ha5qjPsqAtb7rIS0UMy/buN/l5fwFqNo9KPj0TkmFuOu7 fXif2TyJvd0b3ikvWRmNxtvWATTLgWk4Gl/Z4= Received: by 10.86.222.5 with SMTP id u5mr2728816fgg.67.1255114185469; Fri, 09 Oct 2009 11:49:45 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 4sm585275fge.22.2009.10.09.11.49.40 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 09 Oct 2009 11:49:42 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 9 Oct 2009 11:48:31 -0700 From: Pyun YongHyeon Date: Fri, 9 Oct 2009 11:48:31 -0700 To: Jack Vogel Message-ID: <20091009184831.GH3843@michelle.cdnetworks.com> References: <20091008174521.GE3843@michelle.cdnetworks.com> <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: Siquijor Philips , sam@FreeBSD.org, freebsd-drivers@freebsd.org, freebsd-net@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 18:49:47 -0000 On Fri, Oct 09, 2009 at 11:17:00AM -0700, Jack Vogel wrote: > I am Jack, the network engineer at Intel responsible for all FreeBSD wired > lan drivers. > This is the first I've seen about this. Our understanding was that the > infrastructure needed > to do IPSec was not available for either Linux or FreeBSD, can you please > explain things? > I guess we already have crypto(9) infrastructure to support IPSec in kernel. CCed to sam who may know what is required to implement IPSec offloading in ethernet driver. > If everything is there except the support in the driver then I might be able > to add that to > my queue. > > Cheers, > > Jack > > > On Thu, Oct 8, 2009 at 9:14 PM, Siquijor Philips > wrote: > > > On Fri, Oct 9, 2009 at 1:45 AM, Pyun YongHyeon wrote: > > > On Thu, Oct 08, 2009 at 01:24:20PM +0800, Siquijor Philips wrote: > > >> Hi, > > >> > > >> I got a dual-port Intel Gigabit NIC with 82576 (ET) chipset > > >> http://www.intel.com/Assets/PDF/prodbrief/320116.pdf. It has a feature > > >> on IPsec offloading but it only mentioned Microsoft Windows 2008 and > > >> Vista servers. I wonder if FreeBSD have also support on this feature? > > >> > > > > > > AFAIK it's not yet, not sure whether Jack has plan to implement the > > > offloading. I know old Intel i82550 also supported IPSec offloading > > > but Intel didn't release required information to implement it. 3Com > > > also supported IPSec offloading in their 3XP hardwares(txp(4)) but > > > the offloading was not implemented. > > > > > > > Hi Pyun, > > > > Thanks for your info! By the way, who is Jack? Is he the author of > > this driver? I really need to have this feature usable on the driver. > > I bought these NICs with 82576 chipset for the purpose of implementing > > IPsec in my network and my current FreeBSD servers could benefit it. > > Just really thought it has support because FreeBSD was already part of > > the supported operating system. I was alerted when I've re-read the > > product info document again that it only support Windows 2008 and > > Vista platforms. Now, I've confirmed that with the current existing > > driver. I hope this guy has the plan of implementing it sooner because > > I really need it. > > > > Thanks, > > Siquijor From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 19:03:33 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC5AF1065672 for ; Fri, 9 Oct 2009 19:03:33 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outI.internet-mail-service.net (outi.internet-mail-service.net [216.240.47.232]) by mx1.freebsd.org (Postfix) with ESMTP id 9F6C78FC1C for ; Fri, 9 Oct 2009 19:03:33 +0000 (UTC) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id A107DDB4C9; Fri, 9 Oct 2009 12:03:33 -0700 (PDT) X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id D60AC2D601B; Fri, 9 Oct 2009 12:03:32 -0700 (PDT) Message-ID: <4ACF8907.9000905@elischer.org> Date: Fri, 09 Oct 2009 12:03:35 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: pyunyh@gmail.com References: <20091008174521.GE3843@michelle.cdnetworks.com> <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> <20091009184831.GH3843@michelle.cdnetworks.com> In-Reply-To: <20091009184831.GH3843@michelle.cdnetworks.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Siquijor Philips , freebsd-net@freebsd.org, sam@FreeBSD.org, Jack Vogel , freebsd-drivers@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 19:03:33 -0000 Pyun YongHyeon wrote: > On Fri, Oct 09, 2009 at 11:17:00AM -0700, Jack Vogel wrote: >> I am Jack, the network engineer at Intel responsible for all FreeBSD wired >> lan drivers. >> This is the first I've seen about this. Our understanding was that the >> infrastructure needed >> to do IPSec was not available for either Linux or FreeBSD, can you please >> explain things? >> > > I guess we already have crypto(9) infrastructure to support IPSec > in kernel. CCed to sam who may know what is required to implement > IPSec offloading in ethernet driver. > I guess what is required is dependent on whether it's just crypto support, or whether the card is expected to track all the security associations, or whether it expects to track just a subset of them. I'm guessing that the latter may be the case. From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 19:13:36 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57505106568D; Fri, 9 Oct 2009 19:13:36 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yx0-f171.google.com (mail-yx0-f171.google.com [209.85.210.171]) by mx1.freebsd.org (Postfix) with ESMTP id CB9758FC16; Fri, 9 Oct 2009 19:13:35 +0000 (UTC) Received: by yxe1 with SMTP id 1so453740yxe.3 for ; Fri, 09 Oct 2009 12:13:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=9XF/kXD3MJmxIHxiiqZnkdgIgC+kRCFAwwCR44zkx6c=; b=j3ZFdmFrfG9ul7xSZXSvSHCU0ca7UtrupdpI/au+aFBH+a9Ln+WsNTuuMh6hw3UDoG QiNAxHuUj7Rg0F+IdNj6eK+aAMgeHPyj2X8k7yBGTXzeLsfIofnAMbKPiK2zS7Gq6gxW nE7N34woCdstYLSz//WhgqXeyv/+Mwt66E9XY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=pTiU9+SItSysriiF2EJq9XH6VLQO+wOCI7MdtDJpw3BFhXPX3IH8yCShQlLNMr9cDk +iRMGsDE4aTf8whFuCKSq7TjlH6joj+/q0npST5QL0w5qsjgpxzSTeAfzs9+poI4CIFR LU7c9FiZiQqm+cikPA5J7/naJSVxuMQ9GJS1o= MIME-Version: 1.0 Received: by 10.101.80.14 with SMTP id h14mr3556266anl.129.1255115614881; Fri, 09 Oct 2009 12:13:34 -0700 (PDT) In-Reply-To: <4ACF8907.9000905@elischer.org> References: <20091008174521.GE3843@michelle.cdnetworks.com> <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> <20091009184831.GH3843@michelle.cdnetworks.com> <4ACF8907.9000905@elischer.org> Date: Fri, 9 Oct 2009 12:13:34 -0700 Message-ID: <2a41acea0910091213p3d8ccbbq7c29b407ec0b4c79@mail.gmail.com> From: Jack Vogel To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pyunyh@gmail.com, Siquijor Philips , sam@freebsd.org, freebsd-drivers@freebsd.org, freebsd-net@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 19:13:36 -0000 I am out sick today, but I will see what I can find out early next week. Jack On Fri, Oct 9, 2009 at 12:03 PM, Julian Elischer wrote: > Pyun YongHyeon wrote: > >> On Fri, Oct 09, 2009 at 11:17:00AM -0700, Jack Vogel wrote: >> >>> I am Jack, the network engineer at Intel responsible for all FreeBSD >>> wired >>> lan drivers. >>> This is the first I've seen about this. Our understanding was that the >>> infrastructure needed >>> to do IPSec was not available for either Linux or FreeBSD, can you please >>> explain things? >>> >>> >> I guess we already have crypto(9) infrastructure to support IPSec >> in kernel. CCed to sam who may know what is required to implement >> IPSec offloading in ethernet driver. >> >> > I guess what is required is dependent on whether it's just crypto > support, or whether the card is expected to track all the security > associations, or whether it expects to track just a subset of them. > > I'm guessing that the latter may be the case. > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 19:37:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2DA031065698 for ; Fri, 9 Oct 2009 19:37:26 +0000 (UTC) (envelope-from jacques.fourie@gmail.com) Received: from mail-fx0-f222.google.com (mail-fx0-f222.google.com [209.85.220.222]) by mx1.freebsd.org (Postfix) with ESMTP id B32738FC28 for ; Fri, 9 Oct 2009 19:37:25 +0000 (UTC) Received: by fxm22 with SMTP id 22so6822975fxm.36 for ; Fri, 09 Oct 2009 12:37:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=PvubUvJ/tGCHG34yr21jzV/JBhP+r3zLAa4x8ks2H5M=; b=KpTPDw4mvYsIqz3135UZcv+FZRViANf9Jf9Qw98KoTD5DnWrZ7Ufsp3EfPt7MX1gtm C38u/w1vRoqmHI9QRF62xeVlHtA2n+pNombNzw+ACpXITt/1pw93WI5jEUWZQfzEthak mYhg/qC8Fh5FWGEVKxxlCtKjqQ34qDXAZaXmI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=JQ2Hdcp5cKYs65D4lMGTfgtLCCFTkfaURVjnUYzcrYhmpcbHh2W3xDPfy5ABqfWw2N 5dBvn4ehh59rlIyt5xlCkpxCH97YhdwrBhNPegqjPEohoiXTLo3decXPWA7hBmeuLVsR SYi3dh/5ZHAMUwUrM8OwPLgOp7YFxE+PNtymk= MIME-Version: 1.0 Received: by 10.86.229.18 with SMTP id b18mr2776308fgh.34.1255117043224; Fri, 09 Oct 2009 12:37:23 -0700 (PDT) In-Reply-To: <4ACF5DA5.6060806@elischer.org> References: <4ACF5DA5.6060806@elischer.org> Date: Fri, 9 Oct 2009 21:37:22 +0200 Message-ID: From: Jacques Fourie To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Route re-calculation in ip_output() X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 19:37:26 -0000 On Fri, Oct 9, 2009 at 5:58 PM, Julian Elischer wrote= : > Jacques Fourie wrote: >> >> Hi, >> >> I've noticed what I believe to be a bug in ip_output(). The piece of >> code in question is when the firewall changes the destination address >> of an outgoing packet and the subsequent re-calculation of the route. >> The issue should be clear from the attached diff - basically what >> happens is that for the second route lookup dst can point to >> ro->ro_rt->rt_gateway instead of &ro->ro_dst. It seems as if this >> issue is present on 7,8 and 9? > > Is this a problem? > generally, the aim of a fwd firewall rule is to set the next hop > (gateway). so this may be what is required.. > > >> >> --- ip_output.c 2009-10-09 10:37:40.537408240 +0200 >> +++ /home/jacques/ip_output.c =A0 2009-10-09 10:43:46.232819440 +0200 >> @@ -521,8 +521,10 @@ >> =A0#endif >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0error =3D netisr_queue(NE= TISR_IP, m); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0goto done; >> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 } else >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } else { >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 dst =3D (struct sockaddr_i= n *)&ro->ro_dst; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0goto again; =A0 =A0 /* Re= do the routing table lookup. >> */ >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } >> >> >> Regards, >> Jacques >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > If I understand everything correctly the handling of fwd rules seem to do exactly what I propose in the patch. See the code starting with 'if (fwd_tag) {' in ip_output.c? As far as I understand it fwd rules do not change the destination IP address in the mbuf so the patch will not affect the handling of fwd rules. Jacques From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 20:33:25 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5398E1065672 for ; Fri, 9 Oct 2009 20:33:25 +0000 (UTC) (envelope-from glen.j.barber@gmail.com) Received: from mail-fx0-f222.google.com (mail-fx0-f222.google.com [209.85.220.222]) by mx1.freebsd.org (Postfix) with ESMTP id D54D38FC16 for ; Fri, 9 Oct 2009 20:33:24 +0000 (UTC) Received: by fxm22 with SMTP id 22so6861050fxm.36 for ; Fri, 09 Oct 2009 13:33:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=4DE8BNSOC4fsCgZEciwkX2TDA965avjlLzLYwg0Tyn8=; b=kSdPkqtE3pRr2dEindm0PtAjIW43eRCp4YeWVaEegKAKKX001DpfBywsyJiAb90/Iy oEElhwlLYnCgf+bYZGw87vDCh/XtQZcGbMnaVoMU7H2uNmVpD2GYiSUd0v1RJCbt7ZQY 8WPf7w2H7MTxQbFHE6/Cq5DUmrzxCKS0E8gtM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=ZDgvwfohdX7t4Qfup49fSTFCL2Mfstc+WFUydsJYsAJ5kz48RDHw0q1jCUSLTeM2yw g7cOI8RQWGOpHMrLI3n5lsWDW4MSIWH672tMEFh+4jPH46l/pGjDZwBbbQf8lHk3zJvc ivGxRZ9i27+bbvhv6qyBE+I+5W5pikXXXLUhc= MIME-Version: 1.0 Received: by 10.223.144.207 with SMTP id a15mr1020145fav.63.1255120403855; Fri, 09 Oct 2009 13:33:23 -0700 (PDT) In-Reply-To: <200910092003.57367.bschmidt@techwires.net> References: <20091009170839.142800@gmx.net> <200910092003.57367.bschmidt@techwires.net> Date: Fri, 9 Oct 2009 20:33:23 +0000 Message-ID: <4ad871310910091333u29b5520aycf13971e163cbe5d@mail.gmail.com> From: Glen Barber To: Bernhard Schmidt Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Intel WiFi 5100/5300 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 20:33:25 -0000 Hi On Fri, Oct 9, 2009 at 6:03 PM, Bernhard Schmidt w= rote: > On Friday 09 October 2009 19:08:39 Lutz Bichler wrote: >> Hi, >> >> does anybody know what happened to the attempts to support Intel WiFi >> =A05100/5300 interfaces in the iwn-driver? Are any patches available whi= ch >> =A0could be used to start working on support for these interfaces? > > I'm curious too, as I'm playing with idea to start porting the latest cha= nges > to if_iwn from OpenBSD. Already started with adding the 5000 series firmw= are to > iwnfw... > If you need patches tested for 8-STABLE for the 5100 AGN, I am more than happy to test. --=20 Glen Barber From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 21:36:12 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AF8C106568B for ; Fri, 9 Oct 2009 21:36:12 +0000 (UTC) (envelope-from ml@netfence.it) Received: from cp-out8.libero.it (cp-out8.libero.it [212.52.84.108]) by mx1.freebsd.org (Postfix) with ESMTP id 97B578FC0C for ; Fri, 9 Oct 2009 21:36:11 +0000 (UTC) Received: from soth.ventu (151.51.27.177) by cp-out8.libero.it (8.5.107) id 4ACB3E120078657A for freebsd-net@freebsd.org; Fri, 9 Oct 2009 23:36:10 +0200 Received: from alamar.ventu (alamar.ventu [10.1.2.18]) by soth.ventu (8.14.3/8.14.2) with ESMTP id n99La9Gk070585 for ; Fri, 9 Oct 2009 23:36:09 +0200 (CEST) (envelope-from ml@netfence.it) Message-ID: <4ACFACC9.5010605@netfence.it> Date: Fri, 09 Oct 2009 23:36:09 +0200 From: Andrea Venturoli User-Agent: Thunderbird 2.0.0.23 (X11/20090828) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: FreeBSD + Samba + Active Directory X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 21:36:12 -0000 Hello. I have a setup with two FreeBSD 6.3 domain controllers using samba + openldap + nss_ldap. The company might be switching to Active Directory soon (not my choice, before you ask :-), so I might need to reconfigure the two FreeBSD boxes to become AD members (with winbindd, nss, whatever). I see there's a lot of documentation around and I'm going to read that; here I just want to ask if everything works as advertised, if there are some differences between theory and practice, bugs to watch for, gotchas, etc... Thanks in advance to anyone who cares to share it's experience. bye & Thanks av. From owner-freebsd-net@FreeBSD.ORG Fri Oct 9 21:54:45 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50309106566B for ; Fri, 9 Oct 2009 21:54:45 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from tomjudge.vm.bytemark.co.uk (tomjudge.vm.bytemark.co.uk [80.68.91.100]) by mx1.freebsd.org (Postfix) with ESMTP id 137308FC19 for ; Fri, 9 Oct 2009 21:54:44 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by tomjudge.vm.bytemark.co.uk (Postfix) with ESMTP id A9E91489D2; Fri, 9 Oct 2009 22:54:43 +0100 (BST) X-Virus-Scanned: Debian amavisd-new at tomjudge.vm.bytemark.co.uk Received: from tomjudge.vm.bytemark.co.uk ([127.0.0.1]) by localhost (tomjudge.vm.bytemark.co.uk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YIsRKFjMi9ft; Fri, 9 Oct 2009 22:54:40 +0100 (BST) Received: from rita.nodomain (unknown [192.168.205.6]) by tomjudge.vm.bytemark.co.uk (Postfix) with ESMTP id DC479489D0; Fri, 9 Oct 2009 22:54:39 +0100 (BST) Message-ID: <4ACFB0FB.8070501@tomjudge.com> Date: Fri, 09 Oct 2009 21:54:03 +0000 From: Tom Judge User-Agent: Thunderbird 2.0.0.23 (X11/20090822) MIME-Version: 1.0 To: Andrea Venturoli References: <4ACFACC9.5010605@netfence.it> In-Reply-To: <4ACFACC9.5010605@netfence.it> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: FreeBSD + Samba + Active Directory X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Oct 2009 21:54:45 -0000 Andrea Venturoli wrote: > Hello. > > I have a setup with two FreeBSD 6.3 domain controllers using samba + > openldap + nss_ldap. > The company might be switching to Active Directory soon (not my > choice, before you ask :-), so I might need to reconfigure the two > FreeBSD boxes to become AD members (with winbindd, nss, whatever). > > I see there's a lot of documentation around and I'm going to read > that; here I just want to ask if everything works as advertised, if > there are some differences between theory and practice, bugs to watch > for, gotchas, etc... > > Thanks in advance to anyone who cares to share it's experience. Here is our recipe: 1) Install security/krb5 2) Install net/samba3 with ADS support and set KRB5_HOME=/usr/local 3) Setup /etc/krb5.conf and smb.conf 4) Link /usr/local/etc/krb5.conf to /etc/krb5.conf 5) kinit administrator 6) net ads join 7) net ads testjoin Hope this helps Tom From owner-freebsd-net@FreeBSD.ORG Sat Oct 10 09:50:02 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8D73106568B for ; Sat, 10 Oct 2009 09:50:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9D3E38FC1F for ; Sat, 10 Oct 2009 09:50:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9A9o2dD088286 for ; Sat, 10 Oct 2009 09:50:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9A9o2wM088285; Sat, 10 Oct 2009 09:50:02 GMT (envelope-from gnats) Date: Sat, 10 Oct 2009 09:50:02 GMT Message-Id: <200910100950.n9A9o2wM088285@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Nikolai Saoukh Cc: Subject: Re: kern/139204: [arp] DHCP server replies rejected, ARP entry lost before max_age X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Nikolai Saoukh List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2009 09:50:02 -0000 The following reply was made to PR kern/139204; it has been noted by GNATS. From: Nikolai Saoukh To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/139204: [arp] DHCP server replies rejected, ARP entry lost before max_age Date: Sat, 10 Oct 2009 13:25:46 +0400 > I'm going to make a guess that this is ARP-related, based on a comment > in the PR. After some inspection I think the problem is in /sbin/dhclient. Patch in PR bin/96018 introduced unicast REQUESTs. That requests are sent from random port (not from bootpc (68)), thus defeating bpf filter in dhclient for replies. Moreover when there are two or more interfaces on the same network, that requests can be transmitted from another interface (with different ethernet address). IP packet with the same IP number (but changed ethernet address) confuse ARP machinery. See also PR ports/139405. IMHO, category, severity, ... of this PR should be edited. From owner-freebsd-net@FreeBSD.ORG Sat Oct 10 15:03:16 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D37D01065694; Sat, 10 Oct 2009 15:03:16 +0000 (UTC) (envelope-from siquijorphilips@gmail.com) Received: from mail-vw0-f180.google.com (mail-vw0-f180.google.com [209.85.212.180]) by mx1.freebsd.org (Postfix) with ESMTP id 6DC2C8FC19; Sat, 10 Oct 2009 15:03:16 +0000 (UTC) Received: by vws10 with SMTP id 10so4720260vws.7 for ; Sat, 10 Oct 2009 08:03:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=rrq1Z3KUL7E2pCJn6vG70I4KoAG0pL093uszUHryrF4=; b=wau17Yexmrrc4oHJAgR2LxXYYsjkjFdCrS5LL8hbLXZXkGVkd2xBU79Oe4qnaJ0Lgz 6ukypVu3cgMWbZxlni6cUFU8f0ZYJxBw238KaFLOV3i9B2MVaBEnYbBolIKG8A6AihEU tNS9hZwJjZjkn3sr9u2Yfih6xzmdDP4Kjb5NY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=a+ods/EWpD7K/A54E56yzb961joiuG8U7RdD1udQi42uSA3C+uEFEEsg4sNGdIStms qmAKxT3+jfWP1WsAqf7i7GaNpaiRvCPyREq6aSvl9/PZoixvr2z4pOM7q9p2+hImApU+ t/7/7I4gF+KDAnYGYwYgr3HoRtz37H3ue0+/Y= MIME-Version: 1.0 Received: by 10.220.79.140 with SMTP id p12mr5904553vck.57.1255186995672; Sat, 10 Oct 2009 08:03:15 -0700 (PDT) In-Reply-To: <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> References: <20091008174521.GE3843@michelle.cdnetworks.com> <2a41acea0910091117q6cfab252sa8d5dfcf0182b660@mail.gmail.com> Date: Sat, 10 Oct 2009 23:03:15 +0800 Message-ID: From: Siquijor Philips To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 Cc: pyunyh@gmail.com, freebsd-net@freebsd.org, freebsd-drivers@freebsd.org Subject: Re: intel 82576 ipsec offload? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2009 15:03:16 -0000 On Sat, Oct 10, 2009 at 2:17 AM, Jack Vogel wrote: > I am Jack, the network engineer at Intel responsible for all FreeBSD wired > lan drivers. Hi Jack! Thanks for introducing yourself since I am new to this mailing list :-) > This is the first I've seen about this. Our understanding was that the > infrastructure needed > to do IPSec was not available for either Linux or FreeBSD, can you please > explain things? > Yes, basically I am implementing an IPsec infrastructure to our network. Our network is composed of a main office and 3 branch offices to be linked over VPN. I'm using both FreeBSD and Windows platforms. Our FreeBSD (7.1-Release) platforms comprises the 4 firewall/gateways (which should be also our VPN concentrators) as well as our mail server (7.1-Release). The Windows platforms comprises the local/remote clients (FreeBSD/Windows XP/Vista) and the Active Directory server (Windows Server 2008). Since our 4 FreeBSD perimeter firewall/gateways are currently processing big amount of traffic, so I have decided to buy these Intel NICs with IPsec offloading just to make sure it can carry out the current traffic processing. Aside from that, our local and remote FreeBSD clients will also be configured on transport-mode IPsec sooner because these clients are also network intensive hosts. So, from here I really wanted to have the IPsec offloading be available to my NICs since I intend to buy these as its primary purpose. > If everything is there except the support in the driver then I might be able > to add that to > my queue. > Yes, please because I really need to have my IPsec infra working sooner. Thank you so much! Siquijor > Cheers, > > Jack > > > On Thu, Oct 8, 2009 at 9:14 PM, Siquijor Philips > wrote: >> >> On Fri, Oct 9, 2009 at 1:45 AM, Pyun YongHyeon wrote: >> > On Thu, Oct 08, 2009 at 01:24:20PM +0800, Siquijor Philips wrote: >> >> Hi, >> >> >> >> I got a dual-port Intel Gigabit NIC with 82576 (ET) chipset >> >> http://www.intel.com/Assets/PDF/prodbrief/320116.pdf. It has a feature >> >> on IPsec offloading but it only mentioned Microsoft Windows 2008 and >> >> Vista servers. I wonder if FreeBSD have also support on this feature? >> >> >> > >> > AFAIK it's not yet, not sure whether Jack has plan to implement the >> > offloading. I know old Intel i82550 also supported IPSec offloading >> > but Intel didn't release required information to implement it. 3Com >> > also supported IPSec offloading in their 3XP hardwares(txp(4)) but >> > the offloading was not implemented. >> > >> >> Hi Pyun, >> >> Thanks for your info! By the way, who is Jack? Is he the author of >> this driver? I really need to have this feature usable on the driver. >> I bought these NICs with 82576 chipset for the purpose of implementing >> IPsec in my network and my current FreeBSD servers could benefit it. >> Just really thought it has support because FreeBSD was already part of >> the supported operating system. I was alerted when I've re-read the >> product info document again that it only support Windows 2008 and >> Vista platforms. Now, I've confirmed that with the current existing >> driver. I hope this guy has the plan of implementing it sooner because >> I really need it. >> >> Thanks, >> Siquijor >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >