Date: Thu, 8 Nov 2012 08:56:38 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Xavier Beaudouin <kiwi@oav.net> Cc: freebsd-fs@freebsd.org Subject: Re: 9.0-RELEASE-p4 + NFS + ZFS = issues... :/ (probably a memory leak) Message-ID: <1066915051.120042.1352382998672.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <7791bd21138de89e5c47cca5513d0757@oav.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Xavier Beaudouin wrote: > Hello there, > > I try to make a FreeBSD ZFS NFS server with the following > configuration > : > > - Supermicro X7DB8 > - 2 L5420 > - 12G RAM > - 8 x 300G SCSI drives using the 2 U320 scsi channel. > - lagg (LACP) of 2 e1000 > - zpool : > > # zpool status > pool: vol > state: ONLINE > scan: scrub repaired 0 in 2h11m with 0 errors on Thu Nov 1 07:42:05 > 2012 > config: > > NAME STATE READ WRITE CKSUM > vol ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > da0 ONLINE 0 0 0 > da4 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > da5 ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > raidz1-3 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > > > I have tried this patch : http://people.freebsd.org/~namei-leak.patch > > And issue still here. > > Also tried to move from newnfs to oldnfs server and client, same > issue. > > Issue is comming when copying from a netapp to this machine a 70G data > with bunch of big files (eg 600M ~ 8G) and web files. > > All NFS clients use NFSv3 and UDP. > There is this patch for NFSv3 over UDP: http://people.freebsd.org/~rmacklem/udp-timer.patch (It applies to both NFS clients, since it is krpc related.) I can't remember what the implications of not having this patch is, but if the client retries NFS RPCs too agressively, it could use a lot of DRC storage (mbuf clusters mainly) in the NFS server (if the NFS server is a FreeBSD box). OR Try the mounts using TCP and see if the problem goes away. If it is slow leak, doing "vmstat -z" and "vmstat -m" while the system is getting constipated might show what is leaking. > Tried : > > nfs client netapp and freebsd on the same machine : 1srt rsync from > netapp to freebsd ok : no issue, second rsync : memory leak, swap is > used (about less than 100Mb, but this trigger the bug. > > nfs client of the netapp is the "filer" machine, and the destination > is > local zfs spool. The rsync to make up to date data trigger as well the > issue. > I'll admit you lost me w.r.t. what is the NFS client and what is the NFS server. (Does "netapp" refer to a Netapp Filer or a FreeBSD box?) I'm not sure that matters, except that it isn't obvious to me if the leak you are referring to is in the NFS client or NFS server? > /boot/loader.conf > zfs_load="YES" > if_lagg_load="YES" > autoboot_delay="10" # Delay in seconds before autobooting, > start > # http://icesquare.com/wordpress/how-to-improve-zfs-performance/ > #I have 8G of Ram > #vfs.zfs.prefetch_disable=0 > > #If Ram = 4GB, set the value to 512M > #If Ram = 8GB, set the value to 1024M > #vfs.zfs.arc_min="1024M" > > #Ram x 0.5 - 512 MB > #vfs.zfs.arc_max="3584M" > > #Ram x 2 > vm.kmem_size_max="24G" > > #Ram x 1.5 > #vm.kmem_size="12G" > > > /etc/sysctl.conf > # $FreeBSD: release/9.0.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z > mux $ > # > # This file is read when going to multi-user and its contents piped > thru > # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for > details. > # > > # Uncomment this to prevent users from seeing information about > processes that > # are being run under another UID. > #security.bsd.see_other_uids=0 > kern.maxfiles=64000 > vfs.nfsd.maxthreads=16 > vfs.zfs.prefetch_disable=1 > > > The latter (vfs.zfs.prefetch_disable=1) make the bug a bit strange. > Inactive memory keeps growing, but active and free is still very low > during the copy. Then machine use all of his memory, and NFS server > not > responsive. > > /etc/rc.conf > hostname="filernew" > keymap="fr.iso.acc.kbd" > cloned_interfaces="lagg0" > ifconfig_em0="up" > ifconfig_em1="up" > ifconfig_lagg0="inet 172.31.x.x netmask 255.255.0.0 laggproto lacp > laggport em0 laggport em1" If you can take lagg out of the mix, I'd try that. > defaultrouter="172.31.x.x" > sshd_enable="YES" > ntpd_enable="YES" > powerd_enable="YES" > # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable > dumpdev="NO" > zfs_enable="YES" > puppet_enable="YES" > # NFS > rpcbind_enable="YES" > nfs_server_enable="YES" > nfs_server_flags="-u -t -n 4" > mountd_flags="-r" > mountd_enable="YES" > rpc_lockd_enable="YES" > rpc_statd_enable="YES" > > # Added by Puppet > bsnmpd_enable="YES" > > Kind regards... > > Xavier > Good luck with it, rick > -- > Xavier Beaudouin > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1066915051.120042.1352382998672.JavaMail.root>