From owner-freebsd-fs@freebsd.org Fri Mar 25 23:53:46 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB7EEADDCEC for ; Fri, 25 Mar 2016 23:53:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 8E3581185 for ; Fri, 25 Mar 2016 23:53:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:6vCp7BT7ZKXdvOcK7L9MOfBe2dpsv+yvbD5Q0YIujvd0So/mwa64YxKN2/xhgRfzUJnB7Loc0qyN4/CmAzVLuMfJmUtBWaIPfidNsd8RkQ0kDZzNImzAB9muURYHGt9fXkRu5XCxPBsdMs//Y1rPvi/6tmZKSV3BPAZ4bt74BpTVx5zukbviq9uJOk4R1XKUWvBbElaflU3prM4YgI9veO4a6yDihT92QdlQ3n5iPlmJnhzxtY+a9Z9n9DlM6bp6r5YTGY2zRakzTKRZATI6KCh1oZSz7ViQBTeIs1sdSGINkhsAJQne7BD2V9+ltCrhtutx0QGTJtDtRPYyVCj0vIlxTxq9siYMNHYc+WrUjsF1xPZBpRuqpBhyxqbJZ46IOf5mfuXWdIVJFiJ6Qs9NWnkZUcuHZIwVAr9EZL4Aog== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DQAQChzvVW/61jaINehAF9BrpYAQ2BcBcKhSJKAoFjFAEBAQEBAQEBYyeCLYIUAQEBAwEBAQEgBCcgCwULAgEIGAICDRkCAicBCSYCBAgHBAEcBId+CA6wWpA8AQEBAQEBBAEBAQEBARYEfIUigXuCSYQKDAcBAQeDFoJWBZdghXGCcoI3hFCHdIUxjwgCHgEBQoQBIDAHh1kBCBcffgEBAQ X-IronPort-AV: E=Sophos;i="5.24,392,1454994000"; d="scan'208";a="274535784" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 25 Mar 2016 19:53:36 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 665BB15F579; Fri, 25 Mar 2016 19:53:36 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 2uw-qyATx3hD; Fri, 25 Mar 2016 19:53:35 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 10DFA15F57B; Fri, 25 Mar 2016 19:53:35 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id u7k54cHRoeIF; Fri, 25 Mar 2016 19:53:34 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id E1CE415F579; Fri, 25 Mar 2016 19:53:34 -0400 (EDT) Date: Fri, 25 Mar 2016 19:53:34 -0400 (EDT) From: Rick Macklem To: Hiroshi Nishida Cc: freebsd-fs@freebsd.org Message-ID: <1294209833.31699182.1458950014610.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <56F42EF4.5000505@asusa.net> References: <56F42EF4.5000505@asusa.net> Subject: Re: Problem with FUSE + fts MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - IE7 (Win)/8.0.9_GA_6191) Thread-Topic: Problem with FUSE + fts Thread-Index: JVjynZEdPTCikLadX0Ej/KgjQZyEmg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Mar 2016 23:53:46 -0000 Hiroshi Nishida wrote: > Hi, > > I found a weird error with FUSE + fts_read(). > Every time a command like find, rm -r that calls fts_read() is used for a > FUSE mounted filesystem, it outputs "XXX: No such file or directory" errors > for some (not all) files/directories. > I think I see the same thing when doing an "rm -r" on a fuse/GlusterFS volume. To be honest, I just add a "-f" to the command to shut it up and then it deleted the tree. I think, in general, what readdir() returns after an entry is unlink'd is undefined behaviour. As such, the safe way to delete all of a directory is something like: - in a loop until readdir() returns EOF - opendir() - readdir() the first entry - unlink() that entry - closedir() --> So that all you ever do is readdir() the first entry after an opendir(). However few, if any, apps do this and loop on readdir(), unlink() instead. >From my limited experience with fuse, the directory offset is an index (which isn't monotonically increasing for GlusterFS) and those indexes only work while the directory is open. I think that in Linux, an opendir(), readdir() reads an entire directory into storage maintained by the libc functions and then readdir() returns subsequent entries without doing a kernel syscall. As such, deletion of entries that, in turn, change directory offsets in the file system, don't affect this. In FreeBSD, this (reading the entire directory when opened) is done for "unionfs", but not otherwise. --> This causes grief for NFS, due to directory offsets (called cookies in NFS) change when entries are deleted. --> It is also the case that many (including Linux, I think) return these "cookies" in their "struct dirent", but FreeBSD does not. I have been tempted to enable "read the entire directory on opendir()" for other file system types, but the downside is that apps. will use memory for this (possibly lots for large directories). Since most use UFS or ZFS and don't see problems, I haven't tried to make a case for this change. (I may try if I find that this change fixes fuse.) If you don't mind rebuilding libc from sources, take a look at opendir.c and readdir.c (I can't remember exactly how it is done, but you should find where it chooses to read the entire directory upon opendir or the first readdir) and you could try enabling that for fuse. The directory offset problem is a thorny one, but will at least be partially fixed by a new "struct dirent" with a d_off field in it. This has to be done someday to make ino_t 64bits anyhow. *** If you see this for cases other than "rm -r", such as "ls -lR", then all of the above is bogus. Another issue (which I doubt is causing this) is that FreeBSD has a 32bit ino_t but fuse uses a 64bit inode#, so the FreeBSD fuse interface ends up truncating the high order 32bits off. (GlusterFS uses more than 32bits, but the high order bits are invariant, so it doesn't break things.) Hope this somehow helps, rick > In /usr/src/lib/libc/gen/fts.c, there is fts_safe_changedir(FTS *, FTSENT *, > int, char *) and the error seems to occur there in the following way: > > FTS *sp; > FTSENT *p = sp->fts_cur; // Current node > DIR *dirp = opendir2(p->fts_accpath, oflag); // Open dir > int fd = _dirfd(dirp); // File descriptor of dirp > struct stat sb; _fstat(fd, &sb); // fstat current node through fd > > p->fts_ino != sb.st_ino // This happens for some reason...... and sets errno > = ENOENT > > When the error happens, p->fts_ino always has a small number and sb.st_ino > has a great number like: > p->fts_ino = 13, sb.st_ino = 54136 > > So, a new inode number seems to be allocated to sb.st_ino though the node > already has an inode number. > > I would appreciate hearing any feedback on this, though I already posted > fuse-devel ML and haven't received any helpful responses yet. > The problem seems to be particular to FreeBSD because I don't get any errors > with Ubuntu. > > The sample FUSE program is located at > https://github.com/scopedog/FUSE-Test > > and fts.c is also located under freebsd and ubuntu dirs (interestingly, fts.c > of FreeBSD and Ubuntu are almost identical). > > Thank you. > > -- > Hiroshi Nishida > nishida@asusa.net > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >