From owner-freebsd-stable@FreeBSD.ORG Fri Nov 25 04:20:41 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3694B1065670 for ; Fri, 25 Nov 2011 04:20:41 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id E97508FC0C for ; Fri, 25 Nov 2011 04:20:40 +0000 (UTC) Received: from lstewart.caia.swin.edu.au (lstewart.caia.swin.edu.au [136.186.229.95]) by lauren.room52.net (Postfix) with ESMTPSA id DE8BE7E820; Fri, 25 Nov 2011 15:20:38 +1100 (EST) Message-ID: <4ECF1796.2060107@freebsd.org> Date: Fri, 25 Nov 2011 15:20:38 +1100 From: Lawrence Stewart User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111006 Thunderbird/7.0.1 MIME-Version: 1.0 To: Kris Bauer References: <4ECE9914.6020502@turing.b2n.org> <1A5B3A48-7DF3-4018-A244-152BDE96299A@lassitu.de> <20111125020004.GA36109@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net Cc: "freebsd-stable@freebsd.org List" , Jeremy Chadwick Subject: Re: TCP Reassembly Issues X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Nov 2011 04:20:41 -0000 On 11/25/11 14:19, Kris Bauer wrote: > On Thu, Nov 24, 2011 at 8:00 PM, Jeremy Chadwick > wrote: > >> On Thu, Nov 24, 2011 at 07:13:39PM -0600, Kris Bauer wrote: >>> On Thu, Nov 24, 2011 at 5:35 PM, Adrian Chadd >> wrote: >>> >>>> Have you tried disabling the tcp offload features of your NIC? >>>> >>>> >>>> Adrian >>>> >>> >>> To test this, I added net.inet.tcp.tso=0 to sysctl.conf and restarted the >>> box; it didn't work. net.inet.tcp.reass.cursegments immediately started >>> climbing up and were exhausted within an hour. >> >> I think Adrian was referring to RXCSUM and TXCSUM on your NIC; TSO is >> another offloading feature. >> >> See ifconfig(8) for how to disable those. >> >> Be aware that disabling them in real-time (e.g. ifconfig xxx -rxcsum >> -txcsum) may cause problems; there are some NIC drivers on FreeBSD which >> do not like you doing this once the NIC has established link (meaning >> "reloading the driver" (for lack of better term) results in wonky >> behaviour). So you may instead want to add those hyphen-options to your >> ifconfig_XXX lines in /etc/rc.conf and reboot the box. >> >> If none of this solves the problem, then I consider this a priority 0 >> blocker (read: "all hands on deck") issue with the IP stack in FreeBSD >> 9.x and will need immediate attention. >> >> I would strongly recommend a developer or clueful end-user begin >> tracking down who committed all of these bits and CC them into the >> thread. I would start by looking who implemented the >> net.inet.tcp.reass.cursegments sysctl, because that isn't in RELENG_8 at >> all. >> > > I have added -rxcsum -txcsum -tso to rc.conf and rebooted the box. This > has not solved the problem. After a half-hour usage, I'm already up to > reass.cursegments=2182 and it keeps climbing. This is pretty much guaranteed to be an accounting problem in the TCP reassembly code (netinet/tcp_reass.c), not a driver related issue. I would not expect any amount of tweaking, tuning or driver option twiddling to change the outcome (but if you do find something which alleviates it, do let us know). Kris, are you in a position to test kernel patches on the machine which is experiencing this problem? Cheers, Lawrence