From owner-freebsd-stable Tue Jul 16 6:55:13 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C96A737B407 for ; Tue, 16 Jul 2002 06:54:56 -0700 (PDT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9D7A8441ED for ; Tue, 16 Jul 2002 06:41:52 -0700 (PDT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id JAA11170; Tue, 16 Jul 2002 09:41:50 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id g6GDfKp51673; Tue, 16 Jul 2002 09:41:20 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15668.8831.992051.789196@grasshopper.cs.duke.edu> Date: Tue, 16 Jul 2002 09:41:19 -0400 (EDT) To: Alexander Maret Cc: "'freebsd-stable@freebsd.org'" , naoyuki_tai@mac.com Subject: Re: AW: mbuf clusters behavior (NMBCLUSTERS) In-Reply-To: <58A002A02C5ED311812E0050044517F00D2860@erlangen01.atrada.de> References: <58A002A02C5ED311812E0050044517F00D2860@erlangen01.atrada.de> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Alexander Maret writes: > > -----Ursprungliche Nachricht----- > > Von: naoyuki_tai@mac.com [mailto:naoyuki_tai@mac.com] > > > > When I copy a 500M file (like iso image) from the workstation to the > > server, the server starts to emit: > > > > Jul 12 09:28:54 nile /kernel: All mbuf clusters exhausted, > > please see tuning(7). > > I'm getting the same error here with 4.6-R. > > > So, I bumped up the nmbclusters to kern.ipc.nmbclusters=65536 > > I allocated 128Mbytes to the mbuf clusters, hoping that it is > > big enough. > > But, it still shows that the same > > > > All mbuf clusters exhausted, please see tuning(7). > > The same for me. > > > > How can I prevent this "mbuf clusters exhaustion"? > > I would be interested in an answer, too. RH Linux has a (really dumb) default of using UDP mounts with a 16K read and write size. A small percentage of packet loss will cause many fragments to build up on the IP reassembly queues and run you out of mbuf clusters. There are 3 things you could do to fix this: a) Use more reasonable NFS mount options on the linux boxes. 8K UDP should help quite a bit, 32K TCP should totally eliminate the problem. b) Drastically reduce net.inet.ip.maxfragpackets (make sure that (maxfragpackets * (16384/1472)) < (maxmbufclusters - some_constant). This will limit the amount of packets (and therefor mbufs) on the IP reassembly queues. This may could destroy your observed NFS write performance from the linux side if you do not also implement (a) above. c) Eliminate all packet loss in your network between the client and server. Not likely, as I suspect some of the packet loss might even be inside the linux or freebsd boxes. I suggest you do both (a) and (b). > > > Relating to this, I do not understand why that the mbuf clusters > > are not freed fast enough. I watched "top" and it does seem to be IP reassembly queues are timed-out out after IPFRAGTTL, which is quite a long time. Drew To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message