Date: Tue, 16 Jul 2002 09:41:19 -0400 (EDT) From: Andrew Gallatin <gallatin@cs.duke.edu> To: Alexander Maret <maret@atrada.net> Cc: "'freebsd-stable@freebsd.org'" <freebsd-stable@freebsd.org>, naoyuki_tai@mac.com Subject: Re: AW: mbuf clusters behavior (NMBCLUSTERS) Message-ID: <15668.8831.992051.789196@grasshopper.cs.duke.edu> In-Reply-To: <58A002A02C5ED311812E0050044517F00D2860@erlangen01.atrada.de> References: <58A002A02C5ED311812E0050044517F00D2860@erlangen01.atrada.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Alexander Maret writes: > > -----Ursprungliche Nachricht----- > > Von: naoyuki_tai@mac.com [mailto:naoyuki_tai@mac.com] > > > > When I copy a 500M file (like iso image) from the workstation to the > > server, the server starts to emit: > > > > Jul 12 09:28:54 nile /kernel: All mbuf clusters exhausted, > > please see tuning(7). > > I'm getting the same error here with 4.6-R. > > > So, I bumped up the nmbclusters to kern.ipc.nmbclusters=65536 > > I allocated 128Mbytes to the mbuf clusters, hoping that it is > > big enough. > > But, it still shows that the same > > > > All mbuf clusters exhausted, please see tuning(7). > > The same for me. > > > > How can I prevent this "mbuf clusters exhaustion"? > > I would be interested in an answer, too. RH Linux has a (really dumb) default of using UDP mounts with a 16K read and write size. A small percentage of packet loss will cause many fragments to build up on the IP reassembly queues and run you out of mbuf clusters. There are 3 things you could do to fix this: a) Use more reasonable NFS mount options on the linux boxes. 8K UDP should help quite a bit, 32K TCP should totally eliminate the problem. b) Drastically reduce net.inet.ip.maxfragpackets (make sure that (maxfragpackets * (16384/1472)) < (maxmbufclusters - some_constant). This will limit the amount of packets (and therefor mbufs) on the IP reassembly queues. This may could destroy your observed NFS write performance from the linux side if you do not also implement (a) above. c) Eliminate all packet loss in your network between the client and server. Not likely, as I suspect some of the packet loss might even be inside the linux or freebsd boxes. I suggest you do both (a) and (b). > > > Relating to this, I do not understand why that the mbuf clusters > > are not freed fast enough. I watched "top" and it does seem to be IP reassembly queues are timed-out out after IPFRAGTTL, which is quite a long time. Drew To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?15668.8831.992051.789196>