From owner-freebsd-current@FreeBSD.ORG Wed Jul 22 23:07:38 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 403E2106566B for ; Wed, 22 Jul 2009 23:07:38 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 1560E8FC1C for ; Wed, 22 Jul 2009 23:07:37 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.2/8.14.1) with ESMTP id n6MN7YCt012789; Wed, 22 Jul 2009 16:07:34 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.2/8.13.4/Submit) id n6MN7YhU012788; Wed, 22 Jul 2009 16:07:34 -0700 (PDT) Date: Wed, 22 Jul 2009 16:07:34 -0700 (PDT) From: Matthew Dillon Message-Id: <200907222307.n6MN7YhU012788@apollo.backplane.com> To: Chris Hedley References: Cc: freebsd-current@freebsd.org Subject: Re: Linux NFS ate my bge X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2009 23:07:38 -0000 :requests. : :I've (re)discovered this now that I've recently got my assorted config :problems largely ironed out and have my FreeBSD box up to date again, I'm :reminded of an unresolved problem from way back, which is that my bge card :collapses after being subjected to a large amount of NFS traffic coming :from my Linux box, e.g. recompiling KDE on a discless workstation, which :has been responsible for three embuggerances so far today. If you are using a NFS UDP mount, try using a NFS TCP mount instead. This could very well instantly fix your issues even if it does not solve the underlying bugs. With a UDP mount the server can get a barrage of IP fragmented UDP packets, particularly from linux clients. While I don't know any specific with regards to bugs in the bge driver itself, I do know that for a UDP mount to operate adequately over a GigE network the NFS server needs about a 2 Megabyte socket buffer to receive the bursts. This is not something that would typically be seen between two FreeBSD boxes as FreeBSD's asynchronous NFS client traffic (mainly read-ahead) is limited by available synchronous nfsiod threads, and further limited by default mount options, internal #defined concurrency limits, and by the out-of-order transmission of the read ahead rpcs caused by the way the nfsiod threads operate (which results in out-of-order processing on the server side and stalls the linear reads done by the client). Linux clients, on the other-hand can generate an enormous number of concurrent RPCs. Use of a TCP mount instead of a UDP mount solves the sockbuf and IP fragmentation issues. The TCP connection will not use fragmented IP packets, will not blow away the server's receive-side sockbuf, and does a much better job dealing with any packet loss, to boot. -Matt