From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 2 20:45:29 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2A3F1065692; Fri, 2 Oct 2009 20:45:29 +0000 (UTC) (envelope-from anders@FreeBSD.org) Received: from fupp.net (totem.fix.no [80.91.36.20]) by mx1.freebsd.org (Postfix) with ESMTP id 8DB608FC19; Fri, 2 Oct 2009 20:45:29 +0000 (UTC) Received: from localhost (totem.fix.no [80.91.36.20]) by fupp.net (Postfix) with ESMTP id 109BC37BEB; Fri, 2 Oct 2009 22:29:37 +0200 (CEST) Received: from fupp.net ([80.91.36.20]) by localhost (totem.fix.no [80.91.36.20]) (amavisd-new, port 10024) with LMTP id Ce2yl1x2cgur; Fri, 2 Oct 2009 22:29:36 +0200 (CEST) Received: by fupp.net (Postfix, from userid 1000) id A66CB37BEA; Fri, 2 Oct 2009 22:29:36 +0200 (CEST) Date: Fri, 2 Oct 2009 22:29:36 +0200 From: Anders Nordby To: freebsd-hackers@freebsd.org Message-ID: <20091002202936.GA836@fupp.net> Mail-Followup-To: Anders Nordby , freebsd-hackers@freebsd.org, alc@FreeBSD.org, phk@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-PGP-Key: http://anders.fix.no/pgp/ X-PGP-Key-FingerPrint: 1E0F C53C D8DF 6A8F EAAD 19C5 D12A BC9F 0083 5956 Cc: alc@FreeBSD.org, phk@FreeBSD.org Subject: No end to swap zone exhausted problems? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2009 20:45:29 -0000 Hi, Better post about my swap zone problems here than tear all my hair out. This concerns me: 1) How can SWAPMETA usage continue increasing, while swap space used is not? Example: http://anders.fupp.net/test/swapmeta.png. In the swap meta graph, used space is USED and free space is LIMIT-USED (FREE numbers are a bit odd, it starts with 0) from the SWAPMETA line when running vmstat -z. In the week graph, you can see that after 3 days swap space usage settles at around 90 GB, while swap meta keeps increasing. Is there a leak? 2) By default, kern.maxswzone is set to 32 MB in FreeBSD: kern.maxswzone: 33554432 This leads to vmstat -z showing: ITEM SIZE LIMIT USED FREE REQUESTS FAILURES SWAPMETA: 288, 116519, 0, 0, 0, 0 According to vmstat man page, memory numbers are printed in kilobytes. Where is the relation between maxswzone and SWAPMETA? By booting with different settings for maxswzone and checking what SWAPMETA LIMIT turns out to be, I get this table: kern.maxswzone SWAPMETA LIMIT ============== ============== 32 MB 116519 64 MB 233025 128 MB 466037 256 MB 932074 512 MB 995410 1 GB 995410 2 GB 995410 How come increasing maxswzone beyond 256 MB does not yield a larger SWAPMETA than 995410? Does it stop around 260 MB maxswzone? How can I increase SWAPMETA further to be able to swap more? When maxswzone is set to 2 GB, the number goes negative: kern.maxswzone: -2147483648 I guess my luck is out? Some additional info: - When LIMIT-USED from SWAPMETA gets to 0, the system freezes with the dreaded "swap zone exhausted, increase kern.maxswzone" message. I monitor this. It's nice to know when your system will freeze. - KMEM settings are: vm.kmem_size_max: 3865468109 vm.kmem_size_min: 0 vm.kmem_size: 2718302208 Had no luck by tuning and increasing kmem_size and kmem_size_max, it mostly leads to crashes. - I am running FreeBSD/amd64 7.2-RELEASE, and I have 8 GB physical RAM. - By lowering kern.maxusers from 386 to 16 and trimming my kernel config down to a bare minimum I got SWAPMETA LIMIT increased to a whopping 995475, an increase of ~64 KB. I need more than this. - Why do I use so much swap space? I am running Varnish with malloc for storage, and I have a rather large dataset (6 million objects for this set of servers) with mostly 1 week expire time. Performance wise it works well, better than mmapping a large file which is/was the most common method for Varnish (this gives occasional ~2 minute freezes). Regards, -- Anders.