From owner-freebsd-stable@FreeBSD.ORG Mon Dec 11 18:30:48 2006 Return-Path: X-Original-To: freebsd-stable@FreeBSD.org Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6941C16A509 for ; Mon, 11 Dec 2006 18:30:48 +0000 (UTC) (envelope-from quetzal@zone3000.net) Received: from mx1.sitevalley.com (sitevalley.com [209.67.60.43]) by mx1.FreeBSD.org (Postfix) with SMTP id A78CE440AB for ; Mon, 11 Dec 2006 18:07:54 +0000 (GMT) (envelope-from quetzal@zone3000.net) Received: from unknown (HELO localhost) (217.144.69.37) by 209.67.61.254 with SMTP; 11 Dec 2006 18:08:45 -0000 Date: Mon, 11 Dec 2006 20:08:17 +0200 From: Nikolay Pavlov To: LI Xin Message-ID: <20061211180817.GA4557@zone3000.net> Mail-Followup-To: Nikolay Pavlov , LI Xin , freebsd-stable@FreeBSD.org, Robert Watson References: <20061122195549.GA57018@zone3000.net> <338b359d969e9c68deaf49096aa91995@mail.geekcn.org> <20061123160208.GA62732@zone3000.net> <456662F4.6000306@delphij.net> <20061125103755.GA78288@zone3000.net> <456887CD.50606@delphij.net> <20061125212004.GA22786@zone3000.net> <457D0FC7.6010808@delphij.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <457D0FC7.6010808@delphij.net> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 6.1-RELEASE-p10 Cc: Robert Watson , freebsd-stable@FreeBSD.org Subject: Re: deadlock in "zoneli" state on 6.2-PRERELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Dec 2006 18:30:48 -0000 On Monday, 11 December 2006 at 15:59:03 +0800, LI Xin wrote: > Hi, > > Would you please give the following patch a try? > > http://people.freebsd.org/~delphij/misc/patch-zonelim-drain > > Note: Please revert my previous patch against sys/kern/kern_mbuf.c. > > This patch should be applied against sys/vm/ [RELENG_6 or RELENG_6_2], > it schedules a drain of uma zones when they are low on memory. This time things worked out a bit better, there was no Kernel panic and my server managed to overcome the "magic" number 65550 mbufs. But very soon the server reached another limit - 131072 mbuf clusters (This is my limit for kern.ipc.nmbclusters). And server started to drop the packets. After I've removed the overload I found my server responding but when I actually accessed it I found out that although the number of connections has reduces considerably, the memory allocated for the net did not become free. So I believe that there is still a mbufs leak somewhere. root@accel1:~# sockstat -4 | wc -l 17 root@accel1:~# netstat -m 1082/131578/132660 mbufs in use (current/cache/total) 1080/129992/131072/131072 mbuf clusters in use (current/cache/total/max) ^^^^^^^^^^^^^^^^^^^^^^^^^^ 1080/128712 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 2430K/292878K/295309K bytes allocated to network (current/cache/total) ^^^^^^^^^^^^^^^^^^^^^ 0/1058420/529208 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/4/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 156 calls to protocol drain routines root@accel1:~# vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS UMA Kegs: 140, 0, 84, 12, 84 UMA Zones: 120, 0, 84, 6, 84 UMA Slabs: 64, 0, 904, 40, 31603 UMA RCntSlabs: 104, 0, 65536, 28, 65536 UMA Hash: 128, 0, 4, 26, 6 16 Bucket: 76, 0, 23, 27, 32 32 Bucket: 140, 0, 20, 8, 29 64 Bucket: 268, 0, 23, 33, 46 128 Bucket: 524, 0, 1423, 5, 1590808 VM OBJECT: 132, 0, 18094, 147, 43780 MAP: 192, 0, 7, 33, 7 KMAP ENTRY: 68, 57456, 17, 151, 151954 MAP ENTRY: 68, 0, 718, 290, 182847 PV ENTRY: 24, 2228360, 124232, 4528, 861505 DP fakepg: 72, 0, 0, 0, 0 mt_zone: 64, 0, 182, 54, 182 16: 16, 0, 2828, 1435, 273492 32: 32, 0, 1769, 491, 116114 64: 64, 0, 3199, 2701, 181154 128: 128, 0, 1813, 1517, 273347 256: 256, 0, 365, 415, 13230 512: 512, 0, 1006, 10, 35578 1024: 1024, 0, 49, 83, 72385 2048: 2048, 0, 170, 38, 40942 4096: 4096, 0, 129, 22, 89367 Files: 72, 0, 100, 5624, 227230 MAC labels: 20, 0, 20726, 15778, 368518 PROC: 536, 0, 71, 27, 1441 THREAD: 376, 0, 98, 22, 98 KSEGRP: 88, 0, 98, 62, 98 UPCALL: 44, 0, 0, 0, 0 VMSPACE: 296, 0, 28, 24, 1398 mbuf_packet: 256, 0, 1080, 128712, 6714669 mbuf: 256, 0, 2, 2866, 7912768 mbuf_cluster: 2048, 131072, 129792, 1280, 166272 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0 g_bio: 132, 0, 0, 783, 194850 ata_request: 204, 0, 0, 0, 0 ata_composite: 196, 0, 0, 0, 0 VNODE: 348, 0, 20116, 3, 22965 VNODEPOLL: 76, 0, 0, 0, 0 S VFS Cache: 68, 0, 16387, 77, 18383 L VFS Cache: 291, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 12, 166126 DIRHASH: 1024, 0, 1688, 128, 3307 NFSMOUNT: 480, 0, 0, 0, 0 NFSNODE: 536, 0, 0, 0, 0 PIPE: 408, 0, 6, 21, 595 KNOTE: 68, 0, 0, 112, 64 socket: 356, 131076, 31, 5766, 111627 unpcb: 140, 131096, 12, 44, 171 ipq: 32, 4181, 0, 0, 0 udpcb: 180, 131076, 6, 38, 164 inpcb: 180, 131076, 13, 6081, 111291 tcpcb: 464, 131072, 13, 5555, 111291 tcptw: 48, 8268, 0, 858, 68707 syncache: 100, 15366, 0, 312, 92731 hostcache: 76, 15400, 4989, 61, 4989 tcpreass: 20, 8281, 0, 338, 3819 sackhole: 20, 0, 0, 338, 191981 ripcb: 180, 131076, 0, 0, 0 rtentry: 132, 0, 19, 39, 29 pfsrctrpl: 100, 15015, 0, 0, 0 pfrulepl: 604, 0, 9, 9, 9 pfstatepl: 260, 15000, 7, 10568, 92462 pfaltqpl: 128, 0, 0, 0, 0 pfpooladdrpl: 68, 0, 2, 110, 2 pfrktable: 1240, 0, 4, 5, 8 pfrkentry: 156, 0, 5, 45, 5 pfrkentry2: 156, 0, 0, 0, 0 pffrent: 16, 203, 0, 0, 0 pffrag: 48, 0, 0, 0, 0 pffrcache: 48, 10062, 0, 0, 0 pffrcent: 12, 50141, 0, 0, 0 pfstatescrub: 28, 0, 0, 0, 0 pfiaddrpl: 92, 0, 0, 0, 0 pfospfen: 108, 0, 345, 51, 345 pfosfp: 28, 0, 188, 193, 188 SWAPMETA: 276, 121576, 0, 0, 0 Mountpoints: 740, 0, 6, 4, 6 FFS inode: 132, 0, 20074, 52, 22922 FFS1 dinode: 128, 0, 0, 0, 0 FFS2 dinode: 256, 0, 20074, 41, 22922 Should i add this to my PR? http://www.freebsd.org/cgi/query-pr.cgi?pr=106317 > > Cheers, > -- > Xin LI http://www.delphij.net/ > FreeBSD - The Power to Serve! > -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ======================================================================