From owner-freebsd-fs@FreeBSD.ORG Mon Mar 24 20:12:53 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3725C759 for ; Mon, 24 Mar 2014 20:12:53 +0000 (UTC) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9FC673E3 for ; Mon, 24 Mar 2014 20:12:52 +0000 (UTC) Received: by mail-lb0-f170.google.com with SMTP id s7so4049714lbd.15 for ; Mon, 24 Mar 2014 13:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=K25Q2XzqBzlzpOomKOegLayCzqOoJJGVhQr/T08ROnw=; b=Lny8sMh2Aak5NK/9AQWh9JD770Q5nBRDg9dTmiEfQSSE2QRKDQlYTdoWMgctz2k7bO ihiyBVYcyB1jNxO8irWT93f7baGCsT+DBzKPdte3enF7Q3ZXWg5VLHiJjrwLrhoC4Re6 zlDtQWu3O1tINBheZo+tVC3NIE94YaGo5RZYHW3M9YgwVErM6wt7+jOo1c2+IiZ/4fc3 CQ6GFZ/sDd/unuuvcnINdRnXg+l7huE2xx7M2++cr9kFyX1vwwTrGVWEI6uCbBI60cFK oGNa9Y4aCLhoiFNzsyWXf/6Sh4BPOe5tSuhd3eI+alPjtfZTg87UVQJiCetkJirphrj7 6qtg== MIME-Version: 1.0 X-Received: by 10.112.24.172 with SMTP id v12mr51225lbf.74.1395691970802; Mon, 24 Mar 2014 13:12:50 -0700 (PDT) Received: by 10.114.76.81 with HTTP; Mon, 24 Mar 2014 13:12:50 -0700 (PDT) In-Reply-To: References: Date: Mon, 24 Mar 2014 13:12:50 -0700 Message-ID: Subject: Re: rsync w/ fake-super -> crashes zfs From: javocado To: FreeBSD Filesystems Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 20:12:53 -0000 A bit more info: We've been able to reproduce the crash with panic output: Fatal trap 12: page fault while in kernel mode cpuid =3D 11; apic id =3D 15 fault virtual address =3D 0x160 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80abb546 stack pointer =3D 0x28:0xffffffa59b580910 frame pointer =3D 0x28:0xffffffa59b5809d0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 93937 (rsync) [thread pid 93937 tid 102776 ] Stopped at zfs_freebsd_remove+0x426: movq 0x160(%rax),%rsi db> by simply adding --delete to this command: rsync -av --rsync-path=3D"rsync --fake-super" /src /dst the file it was deleting at the time of the crash had valid (readable) extattrs (rsync.%stat - for the fake-super info) and nothing else special for acl owner@:rw-p--aARWcCos:------:allow group@:r-----a-R-c--s:------:allow everyone@:r-----a-R-c--s:------:allow Question: should it be safe to remove that file traditionally, as in: rm /file/that/zfs+rsync+fake_super/croaked/on How's would that rm differ from what rsync was trying to do? On Thu, Mar 20, 2014 at 5:33 PM, javocado wrote: > Gathering more info here... > > When re-running the rsync command, stripped down: > > rsync -av --rsync-path=3D"rsync --fake-super" /src /dst > > we're not seeing any system crashes, yet, but rsync is choking/quitting > quite a bit saying a file here or there has a corrupt extattr. Does this > point to a problem with ZFS or more with rsync. > > Any thoughts on the fake-super impact and how it leads to crashing? > > > On Fri, Mar 14, 2014 at 4:57 PM, javocado wrote: > >> System specifics: >> ZFS version 28 >> FreeBSD 8.3-RELEASE >> >> We're seeing a repeatable outcome where a remote rsync command like: >> >> rsync -axzHAXS --rsync-path=3D"rsync --fake-super" --exclude '*/rsync.%s= tat' >> >> backing up to our zfs filesystem (with 15M inodes) will lead to a panic >> with output like: >> >> Fatal trap 12: page fault while in kernel modecpuid =3D 4; apic id =3D 0= 4fault virtual address =3D 0x160fault code =3D supervisor re= ad data, page not presentinstruction pointer =3D 0x20:0xffffffff80abb54= 6stack pointer =3D 0x28:0xffffff976c62b910frame pointer = =3D 0x28:0xffffff976c62b9d0code segment =3D base 0x0, limit 0xf= ffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, = gran 1processor eflags =3D interrupt enabled, resume, IOPL =3D 0curr= ent process =3D 7295 (rsync)[thread pid 7295 tid 101008 ]Stopped at= zfs_freebsd_remove+0x426: movq 0x160(%rax),%rsi >> >> >> On the sending side (RHEL, ext3), rsync reports errors like: >> >> rsync: failed to read xattr rsync.%statrsync: failed to write xattr rsyn= c.%statrsync: get_xattr_names: llistxattr >> >> which we've seen occasionally with other systems when running rsync with= fake-super, but it usually doesn't lead to a crash.* >> >> On the receiving side, other than the crashes, we are seeing a few new f= iles (that don't exist on the source) named: >> >> rsync.%stat >> >> which correspond to and contain the owner and permission attributes that= should have been stored in the extattr's for the file or directory. Not su= re if they are a red herring, but they're usually not something we see (per= haps that's related to the --exclude '*/rsync.%stat' and rsync not being ab= le to cleanup properly). >> >> We are still testing to see if any options in the rsync command (above) = may be contributing to the crash, since fake-super in and of itself runs fi= ne under basic (rsync -av --rsync-path=3D"rsync --fake-super" /src /dst) ci= rcumstances. But we suspect that the problem is related to fake-super and i= ts reliance upon extattr's. >> >> What we really need is a solution to the crashing - some way to make ZFS= stop choking on whatever --fake-super produces and/or how it's interacting= with extattr's on ZFS. >> >> >> Thanks! >> >> * we sometimes also see on the sending side w/ fake-super: >> rsync: failed to write xattr rsync.%stat for "xxxxxx/file" : No such fil= e >> or directory (2) >> >> when (1) the file exists, (2) it's a symlink >> but that isn't happening in this instance. We only mention it here as >> another oddity of fake-super + ZFS + extattr >> >> >