From owner-freebsd-fs@FreeBSD.ORG Tue Jan 15 23:32:07 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E6F1B3F1 for ; Tue, 15 Jan 2013 23:32:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id AF2B527C for ; Tue, 15 Jan 2013 23:32:07 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAG/l9VCDaFvO/2dsb2JhbABFhjq3YnOCHgEBAQQBAQEgKyALGw4KAgINGQIjBgEJJgYIBwQBHASHZgMPDKV/gkCGZQ2HfoEjimWBCIMVgRMDiGGKfViBVoEcihuFEoMTgVE1 X-IronPort-AV: E=Sophos;i="4.84,475,1355115600"; d="scan'208";a="12075217" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 15 Jan 2013 18:32:05 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 3C2F0B3F18; Tue, 15 Jan 2013 18:32:05 -0500 (EST) Date: Tue, 15 Jan 2013 18:32:05 -0500 (EST) From: Rick Macklem To: Sergey Kandaurov Message-ID: <2118820107.2024400.1358292725230.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: getcwd lies on/under nfs4-mounted zfs dataset MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2013 23:32:08 -0000 pluknet@gmail.com wrote: > Hi. > > We stuck with the problem getting wrong current directory path > when sitting on/under zfs dataset filesystem mounted over NFSv4. > Both nfs server and client are 10.0-CURRENT from December or so. > > The component path "user3" unexpectedly appears to be "." (dot). > nfs-client:/home/user3 # pwd > /home/. > nfs-client:/home/user3/var/run # pwd > /home/./var/run > > nfs-client:~ # procstat -f 3225 > PID COMM FD T V FLAGS REF OFFSET PRO NAME > 3225 a.out text v r r-------- - - - /home/./var/a.out > 3225 a.out ctty v c rw------- - - - /dev/pts/2 > 3225 a.out cwd v d r-------- - - - /home/./var > 3225 a.out root v d r-------- - - - / > > The used setup follows. > > 1. NFS Server with local ZFS: > # cat /etc/exports > V4: / -sec=sys > > # zfs list > pool1 10.4M 122G 580K /pool1 > pool1/user3 on /pool1/user3 (zfs, NFS exported, local, nfsv4acls) > > Exports list on localhost: > /pool1/user3 109.70.28.0 > /pool1 109.70.28.0 > > # zfs get sharenfs pool1/user3 > NAME PROPERTY VALUE SOURCE > pool1/user3 sharenfs -alldirs -maproot=root -network=109.70.28.0/24 > local > > 2. pool1 is mounted on NFSv4 client: > nfs-server:/pool1 on /home (nfs, noatime, nfsv4acls) > > So that on NFS client the "pool1/user3" dataset comes at /home/user3. > / - ufs > /home - zpool-over-nfsv4 > /home/user3 - zfs dataset "pool1/user3" > > At the same time it works as expected when we're not on zfs dataset, > but directly on its parent zfs pool (also over NFSv4), e.g. > nfs-client:/home/non_dataset_dir # pwd > /home/non_dataset_dir > > The ls command works as expected: > nfs-client:/# ls -dl /home/user3/var/ > drwxrwxrwt+ 6 root wheel 6 Jan 10 16:19 /home/user3/var/ > Well, if you are just looking for a work around, you could try mounting /home/user3 separately. Otherwise, here's roughly what needs to happen for it to work. (There may be some additional trick(s) I am not aware of.) On the server, ZFS must report: - different fsids for /home vs /home/user3 - fileno (A) must be the same value for "." and ".." for the zfs dataset root (and set VV_ROOT on the vnode) - fileno (B) for "user3" reported by readdir() on /home must be different than what "." and ".." report. Then the NFS server will report a different value (B) for Mounted_on_fileno than it does for Fileno (A), when the client gets attributes for the directory /home/user3. When the client sees Mounted_on_fileno != Fileno, it knows it is at a server mount point boundary and should report the correct stuff to stat() and readdir(). I haven't tested this for a while, so it might be broken for UFS as well. If that's the case, I can probably try and track down the problem here. If not, you can capture packets when you do the getcwd() and then look at them in wireshark, so you can see what the server is returning for Fileno and Mounted_on_fileno. They should be different for "/home/user3" and the latter one should be the value returned by Readdir of "/home" for the "user3" entry. I won't be in a position to look at a wireshark trace until April, so I can't help with that at this time. Since I've never used ZFS, I have no idea what it considers a "mount point"? (Generically, within a mount point there needs to be "same fsid" and a unique set of "fileno" values for all objects. When crossing the mount point, VV_ROOT needs to be set and the mounted_on_vp (or whatever it's called) must refer to the parent. "/home" for this case.) I don't know if this helps, rick ps: Solaris10 clients don't get this to work, so you always need to mount each server file system separately, which is the "work around" I suggested at the beginning of this post. > -- > wbr, > pluknet > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"