From owner-freebsd-fs@FreeBSD.ORG Thu May 30 01:19:07 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2D9DE9E4 for ; Thu, 30 May 2013 01:19:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id ECD329D0 for ; Thu, 30 May 2013 01:19:06 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAMinplGDaFvO/2dsb2JhbABahnS+RIEadIIjAQEEASNWBRYYAgINGQJZBogaBqgGkhCBJow7fjQHgkOBFAOXO5FAgysggTU2 X-IronPort-AV: E=Sophos;i="4.87,767,1363147200"; d="scan'208";a="30524839" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu.net.uoguelph.ca with ESMTP; 29 May 2013 21:18:59 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 918D5B4046; Wed, 29 May 2013 21:18:59 -0400 (EDT) Date: Wed, 29 May 2013 21:18:59 -0400 (EDT) From: Rick Macklem To: Bryan Drewery Message-ID: <316743389.56838.1369876739579.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <51A6A096.8030103@shatow.net> Subject: Re: NFS+ZFS+nullfs on Server random Permission Denied errors on client (nfsv4) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 01:19:07 -0000 Bryan Drewery wrote: > Server: > 9.1-RELEASE > mount: > > tank/distfiles/freebsd on /tank/distfiles/freebsd (zfs, NFS > > exported, local, noatime, noexec, nfsv4acls) > /etc/zfs/exports: > > /tank/distfiles/freebsd -maproot=root -network 10.10.0.0/16 > > Clients: > > 8.3-RELEASE > fstab: > > tank:/tank/distfiles/freebsd /mnt/distfiles nfs > > rw,bg,noatime,intr,rsize=65536,wsize=65536,readahead=8,nfsv4 0 0 > > HEAD r250991: > > tank:/tank/distfiles/freebsd /mnt/distfiles nfs > > rw,bg,noatime,intr,rsize=65536,wsize=65536,readahead=8,nfsv4 0 0 > > 9.1-STABLE r247421: > > tank:/tank/distfiles/freebsd /mnt/distfiles nfs > > rw,bg,noatime,intr,rsize=65536,wsize=65536,readahead=8,nfsv4 0 0 > > Problem: > > For months I constantly get random Permission Denied errors on the > client side. Just trying the read again can fix the problem. The > client > will usually show this in dmesg as the same time: > > nfsv4 client/server protocol prob err=10020 > This error means "no file handle" and I believe it will happen when the client tries to access the root of the exported tree. (The root specified by the "V4: ..." line in /etc/exports.) > Example errors: > > root@c1100_1:/usr/local/poudriere/data/logs/bulk/91amd64-default/2013-05-29_19:17:45/logs/errors > > # grep "Permission denied" * > > dconf-0.12.1_1.log:cannot open gnome3/dconf-0.12.1.tar.xz: > > Permission denied > > p5-Pod-Tests-1.19_1.log:pax: Unable to open > > /mnt/distfiles//Pod-Tests-1.19.tar.gz to read > > sofia-sip-1.12.11.log:pax: Unable to open > > /mnt/distfiles//sofia-sip-1.12.11.tar.gz to read > > tex-ptex-3.3_1.log:fetch: texlive-20120701-texmf.tar.xz: Permission > > denied > > tkman-2.2_2.log:pax: Unable to open /mnt/distfiles//tkman-2.2.tar.gz > > to read > > > I have realized that this problem manifests when I nullfs mount the > /tank/distfiles/freebsd directory on the server and start doing reads > from it, while also reading from the clients. > > It became so bad that I had to move my mail off of NFS and back onto > the > mail server itself since I had questions of where mail had disappeared > to! > > Has anyone else experienced this? Is is fixed in STABLE already? > > > Bryan Drewery There have been assorted commits to nullfs recently, but I have no idea if they will have fixed this. If you want the NFS server to work reliably, I'd recommend you not use nullfs. You should be able to access the ZFS volumes locally in the server safely, just don't put any nullfs mount in the mix. rick