From owner-freebsd-fs@FreeBSD.ORG Fri Mar 14 23:57:02 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AE9B3A20 for ; Fri, 14 Mar 2014 23:57:02 +0000 (UTC) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 24479E1B for ; Fri, 14 Mar 2014 23:57:01 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id n15so2258818lbi.13 for ; Fri, 14 Mar 2014 16:57:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=OR2RClCsUJ4xIm7QzcchzvG/eis/GWvTffSX2rI44/A=; b=f3W+pMK+47vEynr5isNEPPPLDdJrSGuCRmCQrwi1xjIbS7CjqX1kI1L5qLTHHGKo5b DFZK08iQdtS3FV1Q2Ba9OaBF9avhFZecTzthLA8QMbFtLHupf7PLPJfIr12pSyKtFOCN tWSVDZ+hCEvKhBGT90Zqt9QHi/d3GIdD3mygKYvWQuykPKpc0rJ6krm+QRuU2pJ1oUMW J7K7IwBKO5I5FpDz6FJaYy5zo1QNtTxFIAtHCR+b+xT9D+FMD0cKXQA6v6R+B2xbgw+8 R7NpEGP/46lUBIhX6dBE+D0cIfP140DIiljx8TgYBeWQkV5shyUneK9bQ/JLdF/n9Wvo Qeeg== MIME-Version: 1.0 X-Received: by 10.112.52.104 with SMTP id s8mr7106620lbo.7.1394841420340; Fri, 14 Mar 2014 16:57:00 -0700 (PDT) Received: by 10.114.230.65 with HTTP; Fri, 14 Mar 2014 16:57:00 -0700 (PDT) Date: Fri, 14 Mar 2014 16:57:00 -0700 Message-ID: Subject: rsync w/ fake-super -> crashes zfs From: javocado To: FreeBSD Filesystems Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2014 23:57:02 -0000 System specifics: ZFS version 28 FreeBSD 8.3-RELEASE We're seeing a repeatable outcome where a remote rsync command like: rsync -axzHAXS --rsync-path="rsync --fake-super" --exclude '*/rsync.%stat' backing up to our zfs filesystem (with 15M inodes) will lead to a panic with output like: Fatal trap 12: page fault while in kernel modecpuid = 4; apic id = 04fault virtual address = 0x160fault code = supervisor read data, page not presentinstruction pointer = 0x20:0xffffffff80abb546stack pointer = 0x28:0xffffff976c62b910frame pointer = 0x28:0xffffff976c62b9d0code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1processor eflags = interrupt enabled, resume, IOPL = 0current process = 7295 (rsync)[thread pid 7295 tid 101008 ]Stopped at zfs_freebsd_remove+0x426: movq 0x160(%rax),%rsi On the sending side (RHEL, ext3), rsync reports errors like: rsync: failed to read xattr rsync.%statrsync: failed to write xattr rsync.%statrsync: get_xattr_names: llistxattr which we've seen occasionally with other systems when running rsync with fake-super, but it usually doesn't lead to a crash.* On the receiving side, other than the crashes, we are seeing a few new files (that don't exist on the source) named: rsync.%stat which correspond to and contain the owner and permission attributes that should have been stored in the extattr's for the file or directory. Not sure if they are a red herring, but they're usually not something we see (perhaps that's related to the --exclude '*/rsync.%stat' and rsync not being able to cleanup properly). We are still testing to see if any options in the rsync command (above) may be contributing to the crash, since fake-super in and of itself runs fine under basic (rsync -av --rsync-path="rsync --fake-super" /src /dst) circumstances. But we suspect that the problem is related to fake-super and its reliance upon extattr's. What we really need is a solution to the crashing - some way to make ZFS stop choking on whatever --fake-super produces and/or how it's interacting with extattr's on ZFS. Thanks! * we sometimes also see on the sending side w/ fake-super: rsync: failed to write xattr rsync.%stat for "xxxxxx/file" : No such file or directory (2) when (1) the file exists, (2) it's a symlink but that isn't happening in this instance. We only mention it here as another oddity of fake-super + ZFS + extattr