From owner-freebsd-fs@FreeBSD.ORG Fri May 3 22:50:10 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B1C0DB88 for ; Fri, 3 May 2013 22:50:10 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: from platinum.linux.pl (platinum.edu.pl [81.161.192.4]) by mx1.freebsd.org (Postfix) with ESMTP id 59CDA193A for ; Fri, 3 May 2013 22:50:09 +0000 (UTC) Received: by platinum.linux.pl (Postfix, from userid 87) id 5D79E47E1A; Sat, 4 May 2013 00:41:24 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on platinum.linux.pl X-Spam-Level: X-Spam-Status: No, score=-1.3 required=3.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.3.2 Received: from [10.255.0.2] (unknown [83.151.38.73]) by platinum.linux.pl (Postfix) with ESMTPA id A5C9747E0F; Sat, 4 May 2013 00:41:22 +0200 (CEST) Message-ID: <51843D0E.2020907@platinum.linux.pl> Date: Sat, 04 May 2013 00:41:18 +0200 From: Adam Nowacki User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: mike@bayphoto.com Subject: Re: zfs issue - disappearing data References: <5183F739.2040908@bayphoto.com> In-Reply-To: <5183F739.2040908@bayphoto.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 May 2013 22:50:10 -0000 Looks like we have a leak with extended attributes: # zfs create -o mountpoint=/test root/test # touch /test/file1 # setextattr user test abc /test/file1 # zdb root/test Object lvl iblk dblk dsize lsize %full type 8 1 16K 512 0 512 0.00 ZFS plain file 9 1 16K 512 1K 512 100.00 ZFS directory 10 1 16K 512 512 512 100.00 ZFS plain file object 8 - the file, object 9 - extended attributes directory, object 10 - value of the 'test' extended attribute # rm /test/file1 # zdb root/test Object lvl iblk dblk dsize lsize %full type 10 1 16K 512 512 512 100.00 ZFS plain file objects 8 and 9 are deleted, object 10 is still there (leaked). On 2013-05-03 19:43, Mike Carlson wrote: > We had a critical issue with a zfs server that exports shares via samba > (3.5) last night > > system info: > uname -a > > FreeBSD zfs-1.discdrive.bayphoto.com 9.1-RELEASE FreeBSD 9.1-RELEASE > #0 r243825: Tue Dec 4 09:23:10 UTC 2012 > root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > > zpool history: > > History for 'data': > 2013-02-25.17:11:37 zpool create data raidz /dev/gpt/disk1.nop > /dev/gpt/disk2.nop /dev/gpt/disk3.nop /dev/gpt/disk4.nop > 2013-02-25.17:11:41 zpool add data raidz /dev/gpt/disk5.nop > /dev/gpt/disk6.nop /dev/gpt/disk7.nop /dev/gpt/disk8.nop > 2013-02-25.17:11:47 zpool add data raidz /dev/gpt/disk9.nop > /dev/gpt/disk10.nop /dev/gpt/disk11.nop /dev/gpt/disk12.nop > 2013-02-25.17:11:53 zpool add data raidz /dev/gpt/disk13.nop > /dev/gpt/disk14.nop /dev/gpt/disk15.nop /dev/gpt/disk16.nop > 2013-02-25.17:11:57 zpool add data raidz /dev/gpt/disk17.nop > /dev/gpt/disk18.nop /dev/gpt/disk19.nop /dev/gpt/disk20.nop > 2013-02-25.17:12:02 zpool add data raidz /dev/gpt/disk21.nop > /dev/gpt/disk22.nop /dev/gpt/disk23.nop /dev/gpt/disk24.nop > 2013-02-25.17:12:08 zpool add data spare /dev/gpt/disk25.nop > /dev/gpt/disk26.nop > 2013-02-25.17:12:15 zpool add data log /dev/gpt/log.nop > 2013-02-25.17:12:19 zfs set checksum=fletcher4 data > 2013-02-25.17:12:22 zfs set compression=lzjb data > 2013-02-25.17:12:25 zfs set aclmode=passthrough data > 2013-02-25.17:12:30 zfs set aclinherit=passthrough data > 2013-02-25.17:13:25 zpool export data > 2013-02-25.17:15:33 zpool import -d /dev/gpt data > 2013-03-01.12:31:58 zpool add data cache /dev/gpt/cache.nop > 2013-03-15.12:22:22 zfs create data/XML_WORKFLOW > 2013-03-27.12:05:42 zfs create data/IMAGEQUIX > 2013-03-27.13:32:54 zfs create data/ROES_ORDERS > 2013-03-27.13:32:59 zfs create data/ROES_PRINTABLES > 2013-03-27.13:33:21 zfs destroy data/ROES_PRINTABLES > 2013-03-27.13:33:26 zfs create data/ROES_PRINTABLE > > We had a file structure drop off: > > /data/XML_WORKFLOW/XML_ORDERS/ > > around 5/2/2012 @ 17:00 > > In that directory, there were a few thousand directories (containing > images and a couple metadata text/xml files) > > What is odd, is doing a du -h in the parent XML_WORKFLOW directory, only > reports ~150MB: > > # find . -type f |wc -l > 86 > # du -sh . > 130M . > > > however, df reports 1.5GB: > > # df -h . > Filesystem Size Used Avail Capacity Mounted on > data/XML_WORKFLOW 28T 1.5G 28T 0% /data/XML_WORKFLOW > > zdb -d shows: > > # zdb -d data/XML_WORKFLOW > Dataset data/XML_WORKFLOW [ZPL], ID 139, cr_txg 339633, 1.53G, > 212812 objects > > Digging further into zdb, the path is missing for most of those objects: > > # zdb -ddddd data/XML_WORKFLOW 635248 > Dataset data/XML_WORKFLOW [ZPL], ID 139, cr_txg 339633, 1.53G, > 212812 objects, rootbp DVA[0]=<5:b274264000:2000> > DVA[1]=<0:b4d81a8000:2000> [L0 DMU objset] fletcher4 lzjb LE > contiguous unique double size=800L/200P birth=1202311L/1202311P > fill=212812 cksum=16d24fb5aa:6c2e0aff6bc:129af90fe2eff:2612f938c5292b > > Object lvl iblk dblk dsize lsize %full type > 635248 1 16K 512 6.00K 512 100.00 ZFS plain file > 168 bonus System attributes > dnode flags: USED_BYTES USERUSED_ACCOUNTED > dnode maxblkid: 0 > path ??? > uid 11258 > gid 10513 > atime Thu May 2 17:31:26 2013 > mtime Thu May 2 17:31:26 2013 > ctime Thu May 2 17:31:26 2013 > crtime Thu May 2 17:13:58 2013 > gen 1197180 > mode 100600 > size 52 > parent 635247 > links 1 > pflags 40800000005 > Indirect blocks: > 0 L0 3:a9da05a000:2000 200L/200P F=1 B=1197391/1197391 > > segment [0000000000000000, 0000000000000200) size 512 > > The application that writes to this volume runs on a windows client, so > far, it has exhibited identical behavior across two zfs servers, but not > on a generic windows server 2003 network share. > > The question is, what is happening to the data. Is it a samba issue? Is > it ZFS? I've enabled the samba full_audit module to track file > deletions, so I should have more information on that side. > > If anyone has seen similar behavior please let me know > > Mike C