From owner-freebsd-stable Sun Jun 30 15:13:15 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9F4A037B400 for ; Sun, 30 Jun 2002 15:13:10 -0700 (PDT) Received: from mail.sandvine.com (sandvine.com [199.243.201.138]) by mx1.FreeBSD.org (Postfix) with ESMTP id D313543E0A for ; Sun, 30 Jun 2002 15:13:09 -0700 (PDT) (envelope-from don@sandvine.com) Received: by mail.sandvine.com with Internet Mail Service (5.5.2653.19) id ; Sun, 30 Jun 2002 18:13:09 -0400 Message-ID: From: Don Bowman To: 'Doug White' , Don Bowman Cc: "'freebsd-stable@freebsd.org'" Subject: RE: number mbufs / cluster Date: Sun, 30 Jun 2002 18:13:08 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Doug White wrote: > On Sat, 29 Jun 2002, Don Bowman wrote: > > > > > I have an application where there is lots of connections, > > but packet sizes aren't that large. I'm running into the > > problem where I need an enormous amount of clusters, > > but the number of mbufs needed isn't as big: > > > > $ netstat -m > > 141164/141200/512000 mbufs in use (current/peak/max): > > 141163 mbufs allocated to data > > 1 mbufs allocated to fragment reassembly queue headers > > 127997/128000/128000 mbuf clusters in use (current/peak/max) > > 291300 Kbytes allocated to network (10% of mb_map in use) > > 121192 requests for memory denied > > holy cow. How much memory does this thing have and what did you do to > abuse it soo much? :-) I didn't think you could jack those values that > high and not get panics. > I tried changing MCLSHIFT to 10. Although this would seem like the right thing to do, some trouble ensues... Some things like nfs don't seem to work right (just UDP traffic as far as I could see). (10 would yield a 1K cluster). Are there any assumptions somewhere that a cluster is >= MTU size? Also, how does one change the number of mbufs down? The sysctl value is read-only. # Mbufs seems to be a constant 4x # clusters. My socket buffer sizes are set to 8k (xmit and rx). I can't really make them any smaller for this application. I'm trying to manage about ~40K sockets. I have 1GB of physical memory. I'm running into the limits shown above at about 4K-5K sockets (but each doing 10x the work I expect in the 40K case). Also, I'm not really sure why the number of clusters I'm using is so large. There shouldn't be any packet loss on this GE link (at least not due to congestion), I'm only running it at about 2-7 hundred Mbps (depending on the input I feed it). Data flow is fairly bidirectional. O yeah, as to the comment about panics... Sure, I'm getting them, which is why I'm doing the tuning :) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message