From owner-freebsd-fs@FreeBSD.ORG Fri Nov 16 17:02:29 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C862EE9E for ; Fri, 16 Nov 2012 17:02:29 +0000 (UTC) (envelope-from bgold@simons-rock.edu) Received: from hedwig.simons-rock.edu (hedwig.simons-rock.edu [208.81.88.14]) by mx1.freebsd.org (Postfix) with ESMTP id 803F18FC08 for ; Fri, 16 Nov 2012 17:02:29 +0000 (UTC) Received: from behemoth (behemoth.simons-rock.edu [10.30.2.44]) by hedwig.simons-rock.edu (Postfix) with ESMTP id DE39714B for ; Fri, 16 Nov 2012 12:02:28 -0500 (EST) From: "Brian Gold" To: Subject: odd phantom directory (cross posted from freebsd-general) Date: Fri, 16 Nov 2012 12:02:29 -0500 Message-ID: <072001cdc41c$291283a0$7b378ae0$@simons-rock.edu> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: Ac3EHCkI7Iry60q+S6efPxFPyEP22A== Content-Language: en-us X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 17:02:29 -0000 Hi all, My problem appears to be resolved for now, but I would definitely like to understand what the root cause of the issue was. I'm crossposting this to freebsd-fs based on some a suggestion from a freebsd-general user. -------------------------------------------- First post -------------------------------------------- I ran into a rather odd issue this morning with my FreeBSD 9.0-Release system running ZFS v28. This system serves as an RSYNC host which all of our other systems back up to each night. Last night, I started getting the following error: file has vanished: "/backup/ldap1/etc/pki" Now, usually when I get a file has vanished error during an RSYNC run, it indicates that the source file/directory on the system that is sending the rsync backup has been deleted or moved before rsync got a chance to actually send it. That doesn't appear to be the case here. "/backup/ldap1/etc/pki" is the destination directory on my Freebsd/ZFS server. I take a look in "/backup/ldap1/etc" on my Freebsd server and the "pki" subdirectory is no longer listed. Ok, so I run "mkdir /backup/ldap1/etc/pki" and get the following error: "mkdir: /backup/ldap1/etc/pki: File exists". Odd Just to double check, I run "ls -la /backup/ldap1/etc/pki" and get the following: "ls: /backup/ldap1/etc/pki: No such file or directory" Alright, how about a simple touch? "touch: /backup/ldap1/etc/pki: No such file or directory" Fine. Maybe there is something funky about the "/backup/ldap1/etc" directory that is preventing me from doing any of this. "mkdir /backup/ldap1/etc/pki2". That works just fine. What the heck? Looking at the output of my daily security run, I see the following: Checking setuid files and devices: find: /backup/ldap1/etc/fonts/conf.avail: No such file or directory find: /backup/ldap1/etc/fonts/conf.d/30-metric-aliases.conf: No such file or directory find: /backup/ldap1/etc/pki: No such file or directory So, it looks like there are a few files/directories in /backup/ldap1/etc that were affected. Looking through dmesg and /var/log/messages, I don't see anything out of the ordinary. I'm running a zpool scrub now just to be on the safe side, but I haven't seen any checksum or other errors so far. Any thoughts as to what might be causing this? -------------------------------------------- second post -------------------------------------------- It looks like this may be the same issue as reported here: http://lists.freebsd.org/pipermail/freebsd-current/2011-October/027902.html but that thread seems to have just died off about a year ago. Zfs scrub is still running, but not reported errors so far. I'm going to run a "zdb -ccv backup" once that is done. >From looking over this other thread, I tried just a simple "ls /backup/ldap1/etc" and "/backup/ldap1/etc/pki" does show up if I do "ls" without any arguments. If I do an "ls -l" then it doesn't show up. -------------------------------------------- third post -------------------------------------------- Ok, really confused now. I just ran an "rm -rf /backup/ldap1", which errored out when trying to rm "/backup/ldap1/etc/pki", "/backup/ldap1/etc/fonts/conf.d/30-metric-aliases.conf", and "/backup/ldap1/etc/fonts/conf.avail". Everything else got purged correctly, except for those phantom files. I then reran my rsync script, which DIDN'T error this time, shipped all the files over, and I can now read those phantom files/folders just fine.