Date: Thu, 10 Nov 2016 16:20:40 +0100 From: Henri Hennebert <hlh@restart.be> To: Andriy Gapon <avg@FreeBSD.org>, freebsd-stable@FreeBSD.org Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock Message-ID: <6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be> In-Reply-To: <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <aaf2df40-b0df-2141-9ed8-5b947d8d5a33@FreeBSD.org> <43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org> <a14d508d-351f-71f4-c7cc-ac73dbcde357@restart.be> <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/10/2016 15:00, Andriy Gapon wrote:
> On 10/11/2016 12:30, Henri Hennebert wrote:
>> On 11/10/2016 11:21, Andriy Gapon wrote:
>>> On 09/11/2016 15:58, Eric van Gyzen wrote:
>>>> On 11/09/2016 07:48, Henri Hennebert wrote:
>>>>> I encounter a strange deadlock on
>>>>>
>>>>> FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260:
>>>>> Fri Nov 4 02:51:33 CET 2016
>>>>> root@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ amd64
>>>>>
>>>>> This system is exclusively running on zfs.
>>>>>
>>>>> After 3 or 4 days, `periodic daily` is locked in the directory
>>>>> /usr/local/news/bin
>>>>>
>>>>> [root@avoriaz ~]# ps xa|grep find
>>>>> 85656 - D 0:01.13 find / ( ! -fstype local -o -fstype rdonly ) -prune
>>>>> -o ( -name [#,]* -o -name .#* -o -name a.out -o -nam
>>>>> 462 1 S+ 0:00.00 grep find
>>>>> [root@avoriaz ~]# procstat -f 85656
>>>>> PID COMM FD T V FLAGS REF OFFSET PRO NAME
>>>>> 85656 find text v r r------- - - - /usr/bin/find
>>>>> 85656 find cwd v d r------- - - - /usr/local/news/bin
>>>>> 85656 find root v d r------- - - - /
>>>>> 85656 find 0 v c r------- 3 0 - /dev/null
>>>>> 85656 find 1 p - rw------ 1 0 - -
>>>>> 85656 find 2 v r -w------ 7 17 - -
>>>>> 85656 find 3 v d r------- 1 0 - /home/root
>>>>> 85656 find 4 v d r------- 1 0 - /home/root
>>>>> 85656 find 5 v d r----n-- 1 533545184 - /usr/local/news/bin
>>>>> [root@avoriaz ~]#
>>>>>
>>>>> If I try `ls /usr/local/news/bin` it is also locked.
>>>>>
>>>>> After `shutdown -r now` the system remain locked after the line '0 0 0 0 0 0'
>>>>>
>>>>> After a reset and reboot I can access /usr/local/news/bin.
>>>>>
>>>>> I delete this directory and reinstall the package `portupgrade -fu news/inn`
>>>>>
>>>>> 5 days later `periodic daily`is locked on the same directory :-o
>>>>>
>>>>> Any idea?
>>>>
>>>> I can't help with the deadlock, but someone who _can_ help will probably ask for
>>>> the output of "procstat -kk PID" with the PID of the "find" process.
>>>
>>> In fact, it's procstat -kk -a. With just one thread we would see that a thread
>>> is blocked on something, but we won't see why that something can not be acquired.
>>>
>>>
>> I attach the result,
>
> Interesting. I can not spot any suspicious thread that would hold the vnode
> lock. Could you please run kgdb (just like that, no arguments), then execute
> 'bt' command and then select a frame when _vn_lock is called with 'fr N'
> command. Then please 'print *vp' and share the result.
>
I Think I miss something in your request:
[root@avoriaz ~]# kgdb
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
--- clip ---
Loaded symbols for /boot/kernel/accf_data.ko
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
/usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500,
flags=<value optimized out>)
at /usr/src/sys/kern/sched_ule.c:1973
1973 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500,
flags=<value optimized out>)
at /usr/src/sys/kern/sched_ule.c:1973
#1 0xffffffff80566b15 in tc_fill_vdso_timehands32 (vdso_th32=0x0) at
/usr/src/sys/kern/kern_tc.c:2121
#2 0xffffffff80555227 in timekeep_push_vdso () at
/usr/src/sys/kern/kern_sharedpage.c:174
#3 0xffffffff80566226 in tc_windup () at /usr/src/sys/kern/kern_tc.c:1426
#4 0xffffffff804eaa41 in hardclock_cnt (cnt=1, usermode=<value
optimized out>) at /usr/src/sys/kern/kern_clock.c:589
#5 0xffffffff808fac74 in handleevents (now=<value optimized out>,
fake=0) at /usr/src/sys/kern/kern_clocksource.c:223
#6 0xffffffff808fb1d7 in timercb (et=0xffffffff8100cf20, arg=<value
optimized out>) at /usr/src/sys/kern/kern_clocksource.c:352
#7 0xfffff800b6429a00 in ?? ()
#8 0xffffffff81051080 in vm_page_array ()
#9 0xffffffff81051098 in vm_page_queue_free_mtx ()
#10 0xfffffe0101818920 in ?? ()
#11 0xffffffff805399c0 in __mtx_lock_sleep (c=<value optimized out>,
tid=Error accessing memory address 0xffffffffffffffac: Bad add\
ress.
) at /usr/src/sys/kern/kern_mutex.c:590
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb) q
[root@avoriaz ~]#
I don't find the requested frame
Henri
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e>