Date: Fri, 30 Mar 2012 21:02:03 +0000 From: "Philip M. Gollucci" <pgollucci@gmail.com> To: freebsd-stable@freebsd.org Subject: Re: 9-STABLE, ZFS, NFS, ggatec - suspected memory leak Message-ID: <4F761F4B.90409@p6m7g8.com> In-Reply-To: <20120330083830.GB65313@e-Gitt.NET> References: <20120330083830.GB65313@e-Gitt.NET>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigF63D4A7FF0DE228FE25981AE Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239643.ht= ml Same issue different thread. Different software. Its not NFS, its ZFS. I don't really have a place to try it on 8.2, but my hunch from things I've done rather similarly which don't cause tell me its a new issue in 9.0, though I won't swear by that. On 03/30/12 08:38, Oliver Brandmueller wrote: > Hi all, >=20 > Setup: >=20 > I'm running 2 machines (amd64, 16GB) with FreeBSD 9-STABLE (Mar 14 so=20 > far) acting as NFS servers. They each serve 3 zpools (holding a single = > zfs, hourly snapshots). The zpools each are 3-way mirrors of ggate=20 > devices, each 2 TB, so 2 TB per zpool. Compression is "on" (to save=20 > bandwith to the backend, compressratio around 1.05 to 1.15), atime is=20 > off. >=20 > There is no special tuning in loader.conf (except I tried to limit ZFS = > ARC to 8GB lately, which doesn't change a lot). sysctl.conf has: >=20 > kern.ipc.maxsockbuf=3D33554432 > net.inet.tcp.sendspace=3D8388608 > net.inet.tcp.recvspace=3D8388608 > kern.maxfiles=3D64000 > vfs.nfsd.maxthreads=3D254 >=20 > Without the first three zfs+ggate goes bad after a short time (checksum= =20 > errors, stall), the latter are mainly for NFS and some regular local=20 > cleanup stuff. >=20 > The machines have 4 em and 2 igb network interfaces. 3 of the are=20 > dedicated links (with no switches) to the ggate servers, one is a=20 > dedicated link to a third machine which gets feeded with incremental=20 > snapshots by ZFS send (as backup and fallaback of last resort), one=20 > interface for management tasks and one to an internal network with the = > NFS clients. >=20 > The NFS clients are mostly FreeBSD 6, 7 and 9 STABLE machines (migratio= n=20 > to 9 is running), no NFSv4 (by now), all tcp NFS links, merely no=20 > locking. >=20 > Data consists of a lot of files, this is mainly mailboxes (IMAP:=20 > dovecot, incoming mail with exim, some simple web stuff with apache), s= o=20 > lots of small files, only few bigger ones. Directory structures to a=20 > reasonable depth. >=20 > System is on a SSD (ufs, trim), additionally there are 3 (4 actually, 1= =20 > unused by now) 120GB SSDs serving as cache devices for the zpools. I=20 > first starting using the whole device, but in my hopes to change=20 > something limited cache to partitions of 32GB without change in=20 > behaviour. >=20 >=20 > Problem: >=20 > After about a week of runtime in normal workload the systems starts to = > swap (with about 300 to 500 MB of RAM free). Lots of swapping in and=20 > out, but only very few swap space used (30 to 50 MB). ZFS ARC at that=20 > point reaches it's minimum (while using up to it's configured maximum=20 > before). Most of the RAM is wired. L2ARC headers, accourding to=20 > zfs-stats eat about 1GB, ARC is at 1.8GB at this time. No userland=20 > processes using lots of RAM. >=20 > After some time the system becomes unresponsive, the only way to fix=20 > this I had found by now is to reboot the machine (which of course gives= =20 > a service interruption). >=20 >>From the start of swapping to unresponsiveness I have about 2 to 4 hour= s=20 > to check several things (if I just knew what!). >=20 > Workload distribution is not even over the day, from my munin graphs I = > can see, that wired grows at time of higher workload. At night with=20 > lower workload (but far from nothing, let's say about 1/3 to 1/2 in=20 > writes, but probably <1/4 in reads from weekday workload) I can barely = > see any groth of the wired graph. >=20 > So where is my memory going, any ideas what to change? >=20 > Kernel is stripped down from GENERIC and then everything I need loaded = > as modules. >=20 > Kernel config: http://sysadm.in/zprob/ZSTOR > loader.conf : http://sysadm.in/zprob/loader.conf > dmesg.boot : http://sysadm.in/zprob/dmesg.boot >=20 >=20 --=20 ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Director Operations, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. --------------enigF63D4A7FF0DE228FE25981AE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFPdh9NdbiP+9ubjBwRAkZEAJ4rZGmt83oUHdifobQuNVGdUI7GLgCeMJhJ PA65Y/FdyUzcnuqvoSD3Dqg= =7TSj -----END PGP SIGNATURE----- --------------enigF63D4A7FF0DE228FE25981AE--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F761F4B.90409>