From owner-freebsd-fs@FreeBSD.ORG Thu Nov 8 13:56:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DCB382A2 for ; Thu, 8 Nov 2012 13:56:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 8C7878FC16 for ; Thu, 8 Nov 2012 13:56:40 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAG+5m1CDaFvO/2dsb2JhbABEFoYCvlqCHgEBAQMBAQEBICsgCwUWDgoCAg0ZAikBCSYGCAcEARwEh2MGC6kYkw6BIIpyGgaFFIETA4hainSCLYEcjyeDDYE+CBce X-IronPort-AV: E=Sophos;i="4.80,738,1344225600"; d="scan'208";a="187155294" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 08 Nov 2012 08:56:38 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A8131B3EEF; Thu, 8 Nov 2012 08:56:38 -0500 (EST) Date: Thu, 8 Nov 2012 08:56:38 -0500 (EST) From: Rick Macklem To: Xavier Beaudouin Message-ID: <1066915051.120042.1352382998672.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <7791bd21138de89e5c47cca5513d0757@oav.net> Subject: Re: 9.0-RELEASE-p4 + NFS + ZFS = issues... :/ (probably a memory leak) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 13:56:40 -0000 Xavier Beaudouin wrote: > Hello there, > > I try to make a FreeBSD ZFS NFS server with the following > configuration > : > > - Supermicro X7DB8 > - 2 L5420 > - 12G RAM > - 8 x 300G SCSI drives using the 2 U320 scsi channel. > - lagg (LACP) of 2 e1000 > - zpool : > > # zpool status > pool: vol > state: ONLINE > scan: scrub repaired 0 in 2h11m with 0 errors on Thu Nov 1 07:42:05 > 2012 > config: > > NAME STATE READ WRITE CKSUM > vol ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > da0 ONLINE 0 0 0 > da4 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > da5 ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > raidz1-3 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > > > I have tried this patch : http://people.freebsd.org/~namei-leak.patch > > And issue still here. > > Also tried to move from newnfs to oldnfs server and client, same > issue. > > Issue is comming when copying from a netapp to this machine a 70G data > with bunch of big files (eg 600M ~ 8G) and web files. > > All NFS clients use NFSv3 and UDP. > There is this patch for NFSv3 over UDP: http://people.freebsd.org/~rmacklem/udp-timer.patch (It applies to both NFS clients, since it is krpc related.) I can't remember what the implications of not having this patch is, but if the client retries NFS RPCs too agressively, it could use a lot of DRC storage (mbuf clusters mainly) in the NFS server (if the NFS server is a FreeBSD box). OR Try the mounts using TCP and see if the problem goes away. If it is slow leak, doing "vmstat -z" and "vmstat -m" while the system is getting constipated might show what is leaking. > Tried : > > nfs client netapp and freebsd on the same machine : 1srt rsync from > netapp to freebsd ok : no issue, second rsync : memory leak, swap is > used (about less than 100Mb, but this trigger the bug. > > nfs client of the netapp is the "filer" machine, and the destination > is > local zfs spool. The rsync to make up to date data trigger as well the > issue. > I'll admit you lost me w.r.t. what is the NFS client and what is the NFS server. (Does "netapp" refer to a Netapp Filer or a FreeBSD box?) I'm not sure that matters, except that it isn't obvious to me if the leak you are referring to is in the NFS client or NFS server? > /boot/loader.conf > zfs_load="YES" > if_lagg_load="YES" > autoboot_delay="10" # Delay in seconds before autobooting, > start > # http://icesquare.com/wordpress/how-to-improve-zfs-performance/ > #I have 8G of Ram > #vfs.zfs.prefetch_disable=0 > > #If Ram = 4GB, set the value to 512M > #If Ram = 8GB, set the value to 1024M > #vfs.zfs.arc_min="1024M" > > #Ram x 0.5 - 512 MB > #vfs.zfs.arc_max="3584M" > > #Ram x 2 > vm.kmem_size_max="24G" > > #Ram x 1.5 > #vm.kmem_size="12G" > > > /etc/sysctl.conf > # $FreeBSD: release/9.0.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z > mux $ > # > # This file is read when going to multi-user and its contents piped > thru > # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for > details. > # > > # Uncomment this to prevent users from seeing information about > processes that > # are being run under another UID. > #security.bsd.see_other_uids=0 > kern.maxfiles=64000 > vfs.nfsd.maxthreads=16 > vfs.zfs.prefetch_disable=1 > > > The latter (vfs.zfs.prefetch_disable=1) make the bug a bit strange. > Inactive memory keeps growing, but active and free is still very low > during the copy. Then machine use all of his memory, and NFS server > not > responsive. > > /etc/rc.conf > hostname="filernew" > keymap="fr.iso.acc.kbd" > cloned_interfaces="lagg0" > ifconfig_em0="up" > ifconfig_em1="up" > ifconfig_lagg0="inet 172.31.x.x netmask 255.255.0.0 laggproto lacp > laggport em0 laggport em1" If you can take lagg out of the mix, I'd try that. > defaultrouter="172.31.x.x" > sshd_enable="YES" > ntpd_enable="YES" > powerd_enable="YES" > # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable > dumpdev="NO" > zfs_enable="YES" > puppet_enable="YES" > # NFS > rpcbind_enable="YES" > nfs_server_enable="YES" > nfs_server_flags="-u -t -n 4" > mountd_flags="-r" > mountd_enable="YES" > rpc_lockd_enable="YES" > rpc_statd_enable="YES" > > # Added by Puppet > bsnmpd_enable="YES" > > Kind regards... > > Xavier > Good luck with it, rick > -- > Xavier Beaudouin > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"