From owner-freebsd-fs@FreeBSD.ORG Fri Mar 21 00:33:37 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B6F29702 for ; Fri, 21 Mar 2014 00:33:37 +0000 (UTC) Received: from mail-la0-x22a.google.com (mail-la0-x22a.google.com [IPv6:2a00:1450:4010:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2C258CA5 for ; Fri, 21 Mar 2014 00:33:36 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id ec20so1186031lab.15 for ; Thu, 20 Mar 2014 17:33:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=6t7wtov8CIhcshUpaCaXOUupFdtYBkAXXr6GCWPFwKo=; b=BmK30U3KRzn493H38JsNBpHMtC6bkhxn25u0HF4Zr+LPKi1VJNTCW8uRbjxDeWvYFQ +irCVxczXGm5SCLJRFKKsPIdR+c4LvF/Sn1h4GnAytYiIN+jKTolyFaTlhehXHFM6bxK 17y+T05+4IDWmBPdzOMYOrLppgo8J3oozOkW3niFv45Rkj7zquhFqQbwi24cLsAVIzWX yD7n39fs8Y33XzGb0gk4yUPUsjuPJY+I+Ar2dV/X4JBPG2ABmeBSnqPpe7V2vHNUul4W FL8J7LYAA13kNm7T05HH5Ktxs/AsI2Nn4MCzgCg5Qg83vTX4djRWLV77NXwpirZeU8tv LKfw== MIME-Version: 1.0 X-Received: by 10.112.35.130 with SMTP id h2mr30018603lbj.15.1395362014062; Thu, 20 Mar 2014 17:33:34 -0700 (PDT) Received: by 10.114.76.81 with HTTP; Thu, 20 Mar 2014 17:33:34 -0700 (PDT) In-Reply-To: References: Date: Thu, 20 Mar 2014 17:33:34 -0700 Message-ID: Subject: Re: rsync w/ fake-super -> crashes zfs From: javocado To: FreeBSD Filesystems Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Mar 2014 00:33:37 -0000 Gathering more info here... When re-running the rsync command, stripped down: rsync -av --rsync-path=3D"rsync --fake-super" /src /dst we're not seeing any system crashes, yet, but rsync is choking/quitting quite a bit saying a file here or there has a corrupt extattr. Does this point to a problem with ZFS or more with rsync. Any thoughts on the fake-super impact and how it leads to crashing? On Fri, Mar 14, 2014 at 4:57 PM, javocado wrote: > System specifics: > ZFS version 28 > FreeBSD 8.3-RELEASE > > We're seeing a repeatable outcome where a remote rsync command like: > > rsync -axzHAXS --rsync-path=3D"rsync --fake-super" --exclude '*/rsync.%st= at' > > backing up to our zfs filesystem (with 15M inodes) will lead to a panic > with output like: > > Fatal trap 12: page fault while in kernel modecpuid =3D 4; apic id =3D 04= fault virtual address =3D 0x160fault code =3D supervisor rea= d data, page not presentinstruction pointer =3D 0x20:0xffffffff80abb546= stack pointer =3D 0x28:0xffffff976c62b910frame pointer = =3D 0x28:0xffffff976c62b9d0code segment =3D base 0x0, limit 0xff= fff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, g= ran 1processor eflags =3D interrupt enabled, resume, IOPL =3D 0curre= nt process =3D 7295 (rsync)[thread pid 7295 tid 101008 ]Stopped at = zfs_freebsd_remove+0x426: movq 0x160(%rax),%rsi > > > On the sending side (RHEL, ext3), rsync reports errors like: > > rsync: failed to read xattr rsync.%statrsync: failed to write xattr rsync= .%statrsync: get_xattr_names: llistxattr > > which we've seen occasionally with other systems when running rsync with = fake-super, but it usually doesn't lead to a crash.* > > On the receiving side, other than the crashes, we are seeing a few new fi= les (that don't exist on the source) named: > > rsync.%stat > > which correspond to and contain the owner and permission attributes that = should have been stored in the extattr's for the file or directory. Not sur= e if they are a red herring, but they're usually not something we see (perh= aps that's related to the --exclude '*/rsync.%stat' and rsync not being abl= e to cleanup properly). > > We are still testing to see if any options in the rsync command (above) m= ay be contributing to the crash, since fake-super in and of itself runs fin= e under basic (rsync -av --rsync-path=3D"rsync --fake-super" /src /dst) cir= cumstances. But we suspect that the problem is related to fake-super and it= s reliance upon extattr's. > > What we really need is a solution to the crashing - some way to make ZFS = stop choking on whatever --fake-super produces and/or how it's interacting = with extattr's on ZFS. > > > Thanks! > > * we sometimes also see on the sending side w/ fake-super: > rsync: failed to write xattr rsync.%stat for "xxxxxx/file" : No such file > or directory (2) > > when (1) the file exists, (2) it's a symlink > but that isn't happening in this instance. We only mention it here as > another oddity of fake-super + ZFS + extattr > >