From owner-freebsd-fs@FreeBSD.ORG Wed Feb 19 11:39:46 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CAAA7CF for ; Wed, 19 Feb 2014 11:39:46 +0000 (UTC) Received: from mail.myphotobook.de (mail.myphotobook.de [85.237.68.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7951D1B4F for ; Wed, 19 Feb 2014 11:39:45 +0000 (UTC) Received: (qmail 78495 invoked by uid 89); 19 Feb 2014 11:39:44 -0000 Received: from unknown (HELO umbrella.core) (k.kockro@myphotobook.de@87.234.224.68) by mail.myphotobook.de with AES128-SHA encrypted SMTP; 19 Feb 2014 11:39:44 -0000 Message-ID: <5304985D.7030701@myphotobook.de> Date: Wed, 19 Feb 2014 12:41:17 +0100 From: Kai Kockro User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Andriy Gapon , freebsd-fs@FreeBSD.org Subject: Re: ZFS directory not accessable References: <530487FD.10209@myphotobook.de> <53048D43.1010201@FreeBSD.org> <530490A8.70209@myphotobook.de> <53049354.3040700@FreeBSD.org> In-Reply-To: <53049354.3040700@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Feb 2014 11:39:46 -0000 zdb -ddddddd umbrella/usr/home 1027 Dataset umbrella/usr/home [ZPL], ID 52, cr_txg 48, 100G, 114028 objects, rootbp DVA[0]=<0:6c023c400:200> DVA[1]=<0:c003b8a00:200> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=2947973L/2947973P fill=114028 cksum=1d607c8fa9:939a0909e55:192c77959d738:310b5a95037557 Object lvl iblk dblk dsize lsize %full type 1027 1 16K 3.00K 2K 3.00K 100.00 ZFS directory (K=inherit) (Z=inherit) 168 bonus System attributes dnode flags: USED_BYTES USERUSED_ACCOUNTED dnode maxblkid: 0 path /user/.SkypeOLD/user_name uid 1001 gid 0 atime Tue Feb 11 12:45:46 2014 mtime Fri Feb 14 07:36:26 2014 ctime Fri Feb 14 13:43:50 2014 crtime Wed Feb 29 21:32:14 2012 gen 15787 mode 40755 size 41 parent 978 links 5 pflags 40800000144 Assertion failed: (uintptr_t)&((uint64_t *)(zap)->zap_u.zap_fat.zap_phys) [(1<<(((zap)->zap_u.zap_fat.zap_block_shift) - 3 - 1)) + (1<<(((zap)->zap_u.zap_fat.zap_block_shift) - 3 - 1))] - (uintptr_t)zap->zap_u.zap_fat.zap_phys == zap->zap_dbuf->db_size (0x800 == 0xc00), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c, line 450. Abbruch(core dumped) I did a bt on this core dump: (gdb) bt #0 0x0000000801c8f45c in thr_kill () from /lib/libc.so.7 #1 0x0000000801d338db in abort () from /lib/libc.so.7 #2 0x0000000801916ede in zap_lockdir () from /lib/libzpool.so.2 #3 0x0000000801917214 in zap_get_stats () from /lib/libzpool.so.2 #4 0x00000000004072e0 in ?? () #5 0x000000000040758a in ?? () #6 0x0000000000409dd6 in ?? () #7 0x000000000040a424 in ?? () #8 0x000000000040c0f8 in ?? () #9 0x0000000000404f11 in ?? () #10 0x000000080062d000 in ?? () #11 0x0000000000000000 in ?? () kgdb GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/modules/nvidia.ko...done. Loaded symbols for /boot/modules/nvidia.ko Reading symbols from /boot/modules/vboxdrv.ko...done. Loaded symbols for /boot/modules/vboxdrv.ko Reading symbols from /boot/kernel/umodem.ko...Reading symbols from /boot/kernel/umodem.ko.symbols...done. done. Loaded symbols for /boot/kernel/umodem.ko Reading symbols from /boot/modules/vboxnetflt.ko...done. Loaded symbols for /boot/modules/vboxnetflt.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ether.ko Reading symbols from /boot/modules/vboxnetadp.ko...done. Loaded symbols for /boot/modules/vboxnetadp.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko #0 fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:603 603 movq %r12,%rdi /* function */ (kgdb) tid 100558 [Switching to thread 426 (Thread 100558)]#0 sched_switch (td=0xfffffe000c00c920, newtd=0xfffffe0007212920, flags=) at /usr/src/sys/kern/sched_ule.c:1904 1904 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xfffffe000c00c920, newtd=0xfffffe0007212920, flags=) at /usr/src/sys/kern/sched_ule.c:1904 #1 0xffffffff808d6ee4 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:485 #2 0xffffffff809130a2 in sleepq_wait (wchan=0xfffffe00c0af0e00, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:621 #3 0xffffffff808d5d80 in _sx_slock_hard (sx=0xfffffe00c0af0e00, opts=, file=, line=) at /usr/src/sys/kern/kern_sx.c:926 #4 0xffffffff808d679d in _sx_slock (sx=, opts=, file=, line=) at sx.h:188 #5 0xffffffff818659ed in zap_get_leaf_byblk (zap=, blkid=9223372036854777145, tx=0x0, lt=RW_READER, lp=0xffffff824d2ce6e0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:524 #6 0xffffffff81865e28 in zap_deref_leaf (zap=0xfffffe00c0af0e00, h=, tx=0x0, lt=RW_READER, lp=0xffffff824d2ce6e0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:585 #7 0xffffffff818660f7 in fzap_cursor_retrieve (zap=0xfffffe00c0af0e00, zc=0xffffff824d2ce6d0, za=0xffffff824d2ce720) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:1181 #8 0xffffffff8186abe5 in zap_cursor_retrieve (zc=0xffffff824d2ce6d0, za=0xffffff824d2ce720) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1290 #9 0xffffffff8189bbe8 in zfs_freebsd_readdir (ap=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2550 #10 0xffffffff80db2e28 in VOP_READDIR_APV (vop=0xffffffff8191a6c0, a=0xffffff824d2ce8e0) at vnode_if.c:1793 #11 0xffffffff809789f2 in kern_getdirentries (td=0xfffffe000c00c920, fd=, buf=0x80108e000 , count=4096, basep=0xffffff824d2ce9b8) at vnode_if.h:758 #12 0xffffffff80978cf3 in sys_getdirentries (td=, uap=0xffffff824d2cea70) at /usr/src/sys/kern/vfs_syscalls.c:4145 #13 0xffffffff80cb39aa in amd64_syscall (td=0xfffffe000c00c920, traced=0) at subr_syscall.c:135 #14 0xffffffff80c9e107 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 #15 0x0000000800cffe7c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) Thanks, Kai Am 19.02.2014 12:19, schrieb Andriy Gapon: > on 19/02/2014 13:08 Kai Kockro said the following: >> Hello Andriy, >> >> >> ps -p 4883 >> PID TT STAT TIME COMMAND >> 4883 1- D+ 0:00,00 ls -G -f >> >> procstat -kk 4883 >> PID TID COMM TDNAME KSTACK >> 4883 100558 ls - mi_switch+0x194 >> sleepq_wait+0x42 _sx_slock_hard+0x3b0 _sx_slock+0x3d >> zap_get_leaf_byblk+0xbd zap_deref_leaf+0x68 fzap_cursor_retrieve+0xe7 >> zap_cursor_retrieve+0x155 zfs_freebsd_readdir+0x2d8 VOP_READDIR_APV+0x78 >> kern_getdirentries+0x212 sys_getdirentries+0x23 amd64_syscall+0x5ea >> Xfast_syscall+0xf7 > Can you please also obtain the following info? > 1. Find out an inode number of the affected directory and its underlying ZFS > filesystem, e.g. by using stat(1). Then execute `zdb -ddddddd `. > > 2. Run 'kgdb', issue 'tid 100558' command and 'bt' command. > Examine the stack trace and try to get as much information as possible from > frames with _sx_slock and zap_get_leaf_byblk calls. For pointers please print > objects that they point to (e.g. 'print *ptr'). > In particular I am interested in *sx, *zap, *l. > >> ps -p 5236 >> PID TT STAT TIME COMMAND >> 5236 3- D+ 0:00,00 ls -G -la >> >> procstat -kk 5236 >> PID TID COMM TDNAME KSTACK >> 5236 101104 ls - mi_switch+0x194 >> sleepq_wait+0x42 __lockmgr_args+0x9a0 vop_stdlock+0x39 >> VOP_LOCK1_APV+0x84 _vn_lock+0x47 vacl_get_acl+0x55 >> sys___acl_get_link+0x10d amd64_syscall+0x5ea Xfast_syscall+0xf7 >> >> >> >> Best, >> Kai >> >> Am 19.02.2014 11:53, schrieb Andriy Gapon: >>> on 19/02/2014 12:31 Kai Kockro said the following: >>>> Hello, >>>> >>>> i have an 9.2-STABLE r262153 installation on a T420s laptop. Its a ZROOT >>>> on a INTEL SSDSA2BW160G3L drive. >>>> >>>> I have a folder ( skype profil ), which is not accessable anymore. I can >>>> do ls, du and so on, but the states are in D+ after this. >>>> >>>> STRG + T shows: >>>> >>>> [user@host 11:01am] ~/.SkypeOLD/user_profil/>ls -f >>>> load: 0.13 cmd: ls 4883 [] 2.76r 0.00u 0.00s 0% 2124k >>>> load: 0.13 cmd: ls 4883 [] 3.50r 0.00u 0.00s 0% 2124k >>>> load: 0.13 cmd: ls 4883 [] 3.92r 0.00u 0.00s 0% 2124k >>>> load: 0.20 cmd: ls 4883 [] 4.13r 0.00u 0.00s 0% 2124k >>>> load: 0.20 cmd: ls 4883 [] 4.31r 0.00u 0.00s 0% 2124k >>>> >>>> What can i do? Any debug infos needed? >>> You can start with getting procstat -kk output for a stuck process. >>> > -- Kai Kockro Leitung Operations myphotobook gmbh Oranienstrasse 183 10999 Berlin Tel.:+49 (0)30 616 508 100 Fax: +49 (0)30 616 508 200 Geschäftsführer: Vanessa Dill, Martin Lux Amtsgericht Charlottenburg; HRB 94377