From owner-freebsd-net@FreeBSD.ORG Fri Aug 1 12:48:01 2003 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 445F537B401 for ; Fri, 1 Aug 2003 12:48:01 -0700 (PDT) Received: from csmail.commserv.ucsb.edu (cspdc.commserv.ucsb.edu [128.111.251.12]) by mx1.FreeBSD.org (Postfix) with ESMTP id 723C343F85 for ; Fri, 1 Aug 2003 12:48:00 -0700 (PDT) (envelope-from steve@expertcity.com) Received: from expertcity.com ([68.111.37.3]) by csmail.commserv.ucsb.edu (Netscape Messaging Server 3.62) with ESMTP id 471 for ; Fri, 1 Aug 2003 12:47:59 -0700 Message-ID: <3F2AC3F5.3010804@expertcity.com> Date: Fri, 01 Aug 2003 12:48:05 -0700 From: Steve Francis User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030612 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: mbuf clusters exhausted w/o reaching max? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Aug 2003 19:48:01 -0000 I have a FreeBSD 4.8-RELEASE #5 system that reported: Aug 1 11:50:39 rack2-101 /kernel: All mbuf clusters exhausted, please see tuning(7). Aug 1 11:50:39 rack2-101 /kernel: All mbufs exhausted, please see tuning(7). Yet its not close to the max allowed for clusters. rack2-101.nyc# netstat -m 1338/4240/131072 mbufs in use (current/peak/max): 1338 mbufs allocated to data 709/3366/32768 mbuf clusters in use (current/peak/max) 7792 Kbytes allocated to network (7% of mb_map in use) 50 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines rack2-101.nyc# I assume the problem is related to "50 requests for memory denied". However, this system has 1G RAM, and about half of it free, as far as I can tell. (ps -axlm ouput below, as well as systat and top.) So from all three tools, I see over 400M of inactive pages. I.e. pages that could be freed for use, and presumably allocated as mbuf clusters, but need to be flushed first. So is this an error in the mbuf cluster allocation code, the vm system not freeing pages fast enough to keep up with the demand for the mbufs, or something else again? Any ideas appreciated. Thanks ps -axlm UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND 1002 87815 87814 75 2 0 390252 386472 - R ?? 668:11.73 ./bin/cCom_f 0 87829 1 0 2 0 4420 3764 select S p0- 9:50.28 /usr/bin/per 0 87823 1 1 2 0 4408 3752 select S p0- 13:29.65 /usr/bin/per 1002 87817 1 1 2 0 3344 1516 poll S ?? 2:31.43 BrokerLogger 0 87813 1 13 10 0 3256 2596 wait Is ?? 0:00.00 /usr/bin/per 0 62540 62538 0 18 0 1328 840 pause Ss p0 0:00.04 -csh (csh) 0 388 257 0 3 0 1312 712 ttyin I+ d1 0:00.01 -csh (csh) 0 115 1 0 2 0 3048 1756 select Ss ?? 0:05.88 sendmail: ac 0 155 1 83 2 0 3932 2748 select I d1- 13:28.21 /usr/local/s 25 118 1 0 18 0 2920 1576 pause Is ?? 0:00.47 sendmail: Qu 0 87814 87813 12 10 0 632 204 wait I ?? 0:00.00 sh -c ./bin/ 0 1 0 0 10 0 552 112 wait ILs ?? 0:00.01 /sbin/init - 0 101 1 0 2 0 1312 784 select Ss ?? 0:06.99 /usr/sbin/nt 0 142 1 177 10 0 2076 1532 nanslp S d1- 1:09.31 /usr/bin/per 0 62538 110 0 2 0 5708 2180 select S ?? 0:00.21 sshd: root@t 0 64191 62540 0 28 0 428 248 - R+ p0 0:00.00 ps -axlm 0 110 1 0 2 0 3008 1568 select Is ?? 0:03.09 /usr/sbin/ss 0 108 1 0 10 0 1024 660 nanslp Ss ?? 0:00.47 /usr/sbin/cr 1000 168 1 0 10 0 968 528 nanslp S d1- 0:06.88 /usr/local/b 0 257 1 0 10 0 1268 824 wait Is d1 0:00.03 login [pam] 0 23 1 71 18 0 212 44 pause Is ?? 0:00.00 adjkerntz -i 0 97 1 0 2 0 940 580 select Ss ?? 0:00.41 /usr/sbin/sy 0 112 1 0 2 0 924 468 select Ss ?? 0:00.24 /usr/sbin/us 0 249 1 0 3 0 952 544 ttyin Is+ v1 0:00.00 /usr/libexec 0 248 1 0 3 0 952 544 ttyin Is+ v0 0:00.00 /usr/libexec 0 250 1 0 3 0 952 544 ttyin Is+ v2 0:00.00 /usr/libexec 0 251 1 0 3 0 952 544 ttyin Is+ v3 0:00.00 /usr/libexec 0 252 1 0 3 0 952 544 ttyin Is+ v4 0:00.00 /usr/libexec 0 253 1 0 3 0 952 544 ttyin Is+ v5 0:00.00 /usr/libexec 0 254 1 0 3 0 952 544 ttyin Is+ v6 0:00.00 /usr/libexec 0 255 1 0 3 0 952 544 ttyin Is+ v7 0:00.00 /usr/libexec 0 256 1 0 3 0 948 516 siodcd I ?? 0:00.00 /usr/libexec 0 0 0 3 -18 0 0 0 sched DLs ?? 0:00.00 (swapper) 0 6 0 0 -2 0 0 0 vlruwt DL ?? 0:00.61 (vnlru) 0 4 0 0 -18 0 0 0 psleep DL ?? 0:00.63 (bufdaemon) 0 3 0 3 18 0 0 0 psleep DL ?? 0:00.00 (vmdaemon) 0 2 0 0 -18 0 0 0 psleep DL ?? 0:00.24 (pagedaemon 0 5 0 0 18 0 0 0 syncer DL ?? 0:45.58 (syncer) rack2-101.nyc# systat -vmstat: 2 users Load 0.65 0.65 0.61 Aug 1 12:43 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 402424 6116 413732 5916 55372 count All 1026416 6792 3985912 7120 pages Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt 1 cow 8151 total 2 10 6222 617086499 8151 110 11 155608 wire stray irq7 400952 act 7921 mux irq9 16.4%Sys 19.0%Intr 26.3%User 0.0%Nice 38.3%Idl 416452 inact 2 ata0 irq14 | | | | | | | | | | 53404 cache sio0 irq4 ========++++++++++>>>>>>>>>>>>> 1968 free sio1 irq3 daefr 100 clk irq0 Namei Name-cache Dir-cache 6 prcfr 128 rtc irq8 Calls hits % hits % react 756 756 100 pdwake 8 zfod pdpgs Disks ad0 md0 3 ofod intrn KB/t 16.00 0.00 35 %slo-z 114496 buf tps 2 0 15 tfree 17 dirtybuf MB/s 0.03 0.00 70438 desiredvnodes % busy 0 0 17609 numvnodes 3657 freevnodes top: last pid: 64296; load averages: 0.64, 0.65, 0.61 up 2+19:12:20 12:43:41 31 processes: 3 running, 28 sleeping CPU states: 22.1% user, 0.0% nice, 18.0% system, 22.8% interrupt, 37.1% idle Mem: 391M Active, 407M Inact, 152M Wired, 52M Cache, 112M Buf, 1984K Free Swap: 1000M Total, 1000M Free