From owner-freebsd-fs@FreeBSD.ORG Tue Dec 24 22:39:34 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D1A4777B for ; Tue, 24 Dec 2013 22:39:34 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 824211633 for ; Tue, 24 Dec 2013 22:39:34 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: X-IronPort-AV: E=Sophos;i="4.95,545,1384318800"; d="scan'208";a="83072814" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 24 Dec 2013 17:39:33 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id ED422B4055; Tue, 24 Dec 2013 17:39:32 -0500 (EST) Date: Tue, 24 Dec 2013 17:39:32 -0500 (EST) From: Rick Macklem To: Jason Keltz Message-ID: <1566997692.1186754.1387924772956.JavaMail.root@uoguelph.ca> In-Reply-To: <52A7E53D.8000002@cse.yorku.ca> Subject: Re: mount ZFS snapshot on Linux system MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: FreeBSD Filesystems , Steve Dickson X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Dec 2013 22:39:34 -0000 Jason Keltz wrote: > On 10/12/2013 7:21 PM, Rick Macklem wrote: > > Jason Keltz wrote: > >> I'm running FreeBSD 9.2 with various ZFS datasets. > >> I export a dataset to a Linux system (RHEL64), and mount it. It > >> works > >> fine... > >> When I try to access the ZFS snapshot directory on the Linux NFS > >> client, > >> things go weird. > >> Thanks to Jason's testing, I have just committed the following patch http://people.freebsd.org/~rmacklem/nfsv4-zfs-snapshot.patch to head as r259845, which I think fixes the problem. If things go smoothly, I'll MFC it to 9/stable and 10/stable in 1 week. Hopefully this is now resolved, rick > >> With NFSv4: > >> > >> [jas@archive /]# cd /mnt/.zfs/snapshot > >> [jas@archive snapshot]# ls > >> 20131203 20131205 20131206 20131207 20131208 20131209 > >> 20131210 > >> [jas@archive snapshot]# cd 20131210 > >> 20131210: Not a directory. > >> > >> huh? > >> > >> [jas@archive snapshot]# ls -al > >> total 77 > >> dr-xr-xr-x 9 root root 9 Dec 10 11:20 . > >> dr-xr-xr-x 4 root root 4 Nov 28 15:42 .. > >> drwxr-xr-x 380 root root 380 Dec 2 15:56 20131203 > >> drwxr-xr-x 381 root root 381 Dec 3 11:24 20131205 > >> drwxr-xr-x 381 root root 381 Dec 3 11:24 20131206 > >> drwxr-xr-x 381 root root 381 Dec 3 11:24 20131207 > >> drwxr-xr-x 381 root root 381 Dec 3 11:24 20131208 > >> drwxr-xr-x 381 root root 381 Dec 3 11:24 20131209 > >> drwxr-xr-x 381 root root 381 Dec 3 11:24 20131210 > >> [jas@archive snapshot]# stat * > >> [jas@archive snapshot]# ls -al > >> total 292 > >> dr-xr-xr-x 9 root root 9 Dec 10 11:20 . > >> dr-xr-xr-x 4 root root 4 Nov 28 15:42 .. > >> -rw-r--r-- 1 uax guest 137647 Mar 17 2010 20131203 > >> -rw-r--r-- 1 uax guest 865 Jul 31 2009 20131205 > >> -rw-r--r-- 1 uax guest 137647 Mar 17 2010 20131206 > >> -rw-r--r-- 1 uax guest 771 Jul 31 2009 20131207 > >> -rw-r--r-- 1 uax guest 778 Jul 31 2009 20131208 > >> -rw-r--r-- 1 uax guest 5281 Jul 31 2009 20131209 > >> -rw------- 1 btx faculty 893 Jul 13 20:21 20131210 > >> > >> But it gets even more fun.. > >> > >> # ls -ali > >> total 205 > >> 2 dr-xr-xr-x 9 root root 9 Dec 10 11:20 . > >> 1 dr-xr-xr-x 4 root root 4 Nov 28 15:42 .. > >> 863 -rw-r--r-- 1 uax guest 137647 Mar 17 2010 20131203 > >> 4 drwxr-xr-x 381 root root 381 Dec 3 11:24 20131205 > >> 4 drwxr-xr-x 381 root root 381 Dec 3 11:24 20131206 > >> 4 drwxr-xr-x 381 root root 381 Dec 3 11:24 20131207 > >> 4 drwxr-xr-x 381 root root 381 Dec 3 11:24 20131208 > >> 4 drwxr-xr-x 381 root root 381 Dec 3 11:24 20131209 > >> 4 drwxr-xr-x 381 root root 381 Dec 3 11:24 20131210 > >> > >> This is not a user id mapping issue because all the files in /mnt > >> have > >> the proper owner/groups, and I can access them there fine. > >> > >> I also tried explicitly exporting .zfs/snapshot. The result isn't > >> any > >> different. > >> > >> If I use nfs v3 it "works", but I'm seeing a whole lot of errors > >> like > >> these in syslog: > >> > >> Dec 10 12:32:28 jungle mountd[49579]: can't delete exports for > >> /local/backup/home9/.zfs/snapshot/20131203: Invalid argument > >> Dec 10 12:32:28 jungle mountd[49579]: can't delete exports for > >> /local/backup/home9/.zfs/snapshot/20131209: Invalid argument > >> Dec 10 12:32:28 jungle mountd[49579]: can't delete exports for > >> /local/backup/home9/.zfs/snapshot/20131210: Invalid argument > >> Dec 10 12:32:28 jungle mountd[49579]: can't delete exports for > >> /local/backup/home9/.zfs/snapshot/20131207: Invalid argument > >> > >> It's not clear to me why this doesn't just "work". > >> > >> Can anyone provide any advice on debugging this? > >> > > As I think you already know, I know nothing about ZFS and never > > use it. > Yup! :) > > Having said that, I suspect that there are filenos (i-node #s) > > that are the same in the snapshot as in the parent file system > > tree. > > > > The basic assumptions are: > > - within a file system, all i-node# are unique (represent one file > > object only) and all file objects have the same fsid > > - when the fsid changes, that indicates a file system boundary and > > fileno (i-node#s) can be reused in the subtree with a different > > fsid > > > > For NFSv3, the server should export single volumes only (all > > objects > > have the same fsid and the filenos are unique). This is indicated > > to > > the VFS by the use of the NOCROSSMOUNT flag on VOP_LOOKUP() and > > friends. > > > > For NFSv4, the server does export multiple volumes and the boundary > > is indicated by a change in fsid value. > > > > I suspect ZFS snaphots don't obey the above in some way, but that > > is > > just a hunch. > > > > Now, how to narrow this down... > > - Do the above tests (both NFSv4 and NFSv3) and capture the > > packets, > > then look at them in wireshark. In particular, look at the > > fileid numbers > > and fsid values for the various directories under .zfs. > > I gave this a shot, but I haven't used wireshark to capture NFS > traffic > before, so if I need to provide additional details, let me know.. > > NFSv4: > > For /mnt/.zfs/snapshot/20131203: > fileid=4 > fsid4.major=1446349656 > fsid4.minor=222 > > For /mnt/.zfs/snapshot/20131205: > fileid=4 > fsid4.major=1845998066 > fsid4.minor=222 > > For /mnt/jas: > fileid=144 > fsid4.major=597946950 > fsid4.minor=222 > > For /mnt/jas1: > fileid=338 > fsid4.major=597946950 > fsid4.minor=222 > > So fsid is the same for all the different "data" directories, which > is > what I would expect given what you said. I guess each snapshot is > seen > as a unique filesystem... but then a repeating inode in different > filesystems shouldn't be a problem... > > NFSv3: > > For /mnt/.zfs/snapshot/20131203: > fileid=4 > fsid=0x0000000056358b58 > > For /mnt/.zfs/snapshot/20131205: > fileid=4 > fsid=0x000000006e07b1f2 > > For /mnt/jas > fileid=144 > fsid=0x0000000023a3f246 > > For /mnt/jas1: > fileid=338 > fsid=0x0000000023a3f246 > > Here, it seems it's the same, even though it's NFSv3... hmm. > > > > - Try mounting the individual snapshot directory, like > > .zfs/snapshot/20131209 and see if that works (for both NFSv3 > > and NFSv4). > > Hmm .. I tried this: > > /local/backup/home9/.zfs/snapshot/20131203 -ro > archive-mrpriv.cs.yorku.ca > V4: / > > ... but syslog reports: > > Dec 10 22:28:22 jungle mountd[85405]: can't export > /local/backup/home9/.zfs/snapshot/20131203 > > ... and of course I can't mount from either v3/v4. > > On the other hand, I kept it as: > > /local/backup/home9 -ro archive-mrpriv.cs.yorku.ca > V4:/ > > ... and was able to NFSv4 mount > /local/backup/home9/.zfs/snapshot/20131203, and this does indeed > work. > > > - Try doing the mounts with a FreeBSD client and see if you get the > > same > > behaviour? > I found this: > http://forums.freenas.org/threads/mounting-snapshot-directory-using-nfs-from-linux-broken.6060/ > .. implies it will work from FreeBSD/Nexenta, just not Linux. > Found this as well: > https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/lKyfYsjPMNM > > Jason. > >