From owner-freebsd-stable@freebsd.org Thu Nov 10 15:20:50 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECEBBC3AA51 for ; Thu, 10 Nov 2016 15:20:50 +0000 (UTC) (envelope-from hlh@restart.be) Received: from tignes.restart.be (tignes.restart.be [5.135.182.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tignes.restart.be", Issuer "CA master" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B9DA229; Thu, 10 Nov 2016 15:20:50 +0000 (UTC) (envelope-from hlh@restart.be) X-Comment: SPF check N/A for local connections - client-ip=2001:41d0:8:bdbe:1:1::; helo=restart.be; envelope-from=hlh@restart.be; receiver=avg@freebsd.org DKIM-Filter: OpenDKIM Filter v2.10.3 tignes.restart.be 3tF6F65B5fzrl5 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=restart.be; s=tignes; t=1478791242; bh=tFKD70WLlQ2GzweiYtKFvhhtGSauVMrm34vbG4Jz0iI=; h=Subject:To:References:From:Date:In-Reply-To; z=Subject:=20Re:=20Freebsd=2011.0=20RELEASE=20-=20ZFS=20deadlock|To :=20Andriy=20Gapon=20,=20freebsd-stable@FreeBSD.o rg|References:=20<0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> =0D=0A=20=0D=0A= 20<43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org>=0D=0A=20=0D=0A=20<9d1f9a76-5a8 d-6eca-9a50-907d55099847@FreeBSD.org>|From:=20Henri=20Hennebert=20 |Date:=20Thu,=2010=20Nov=202016=2016:20:40=20+0100 |In-Reply-To:=20<9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> ; b=iVxZbcszYLx4SHShN6sLhLyU9Vf8Pb200SfYYKz4nq5bwGH0YnSZ17tqUjeYgJCEO 3jvacpMCjW98OgZvZBLqo8E7lz4gLR+L58xGwaCUpTLybbUEyXcteWx++aiK4nbVZQ /2e+lKXkRn52y4Kxdx4N7MeRkumlCSdh8diLv0f77akHqJ5ckf1y5Gr5cN0mnLeatC EBNKrrvbUP/7L9Co8WKg8X+ajoayskBYZuW3n3dpMaBXvv355RSA3rJO8VFiEeKI1X SbmBDSIBKAxyhkI3nIBpyZXcF6IaG4ebeGCqPHKHC6N6JT1EkOfedCRTpxkLcOMoBP mJxGpTAUKWEbQ== Received: from restart.be (avoriaz.restart.be [IPv6:2001:41d0:8:bdbe:1:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.restart.be", Issuer "CA master" (verified OK)) by tignes.restart.be (Postfix) with ESMTPS id 3tF6F65B5fzrl5; Thu, 10 Nov 2016 16:20:42 +0100 (CET) Received: from chamonix.restart.bel (chamonix.restart.bel [IPv6:2001:41d0:8:bdbe:1:9:0:0]) (authenticated bits=0) by restart.be (8.15.2/8.15.2) with ESMTPSA id uAAFKflE036495 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 10 Nov 2016 16:20:41 +0100 (CET) (envelope-from hlh@restart.be) Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock To: Andriy Gapon , freebsd-stable@FreeBSD.org References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org> <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> From: Henri Hennebert Message-ID: <6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be> Date: Thu, 10 Nov 2016 16:20:40 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2016 15:20:51 -0000 On 11/10/2016 15:00, Andriy Gapon wrote: > On 10/11/2016 12:30, Henri Hennebert wrote: >> On 11/10/2016 11:21, Andriy Gapon wrote: >>> On 09/11/2016 15:58, Eric van Gyzen wrote: >>>> On 11/09/2016 07:48, Henri Hennebert wrote: >>>>> I encounter a strange deadlock on >>>>> >>>>> FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #0 r308260: >>>>> Fri Nov 4 02:51:33 CET 2016 >>>>> root@avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ amd64 >>>>> >>>>> This system is exclusively running on zfs. >>>>> >>>>> After 3 or 4 days, `periodic daily` is locked in the directory >>>>> /usr/local/news/bin >>>>> >>>>> [root@avoriaz ~]# ps xa|grep find >>>>> 85656 - D 0:01.13 find / ( ! -fstype local -o -fstype rdonly ) -prune >>>>> -o ( -name [#,]* -o -name .#* -o -name a.out -o -nam >>>>> 462 1 S+ 0:00.00 grep find >>>>> [root@avoriaz ~]# procstat -f 85656 >>>>> PID COMM FD T V FLAGS REF OFFSET PRO NAME >>>>> 85656 find text v r r------- - - - /usr/bin/find >>>>> 85656 find cwd v d r------- - - - /usr/local/news/bin >>>>> 85656 find root v d r------- - - - / >>>>> 85656 find 0 v c r------- 3 0 - /dev/null >>>>> 85656 find 1 p - rw------ 1 0 - - >>>>> 85656 find 2 v r -w------ 7 17 - - >>>>> 85656 find 3 v d r------- 1 0 - /home/root >>>>> 85656 find 4 v d r------- 1 0 - /home/root >>>>> 85656 find 5 v d r----n-- 1 533545184 - /usr/local/news/bin >>>>> [root@avoriaz ~]# >>>>> >>>>> If I try `ls /usr/local/news/bin` it is also locked. >>>>> >>>>> After `shutdown -r now` the system remain locked after the line '0 0 0 0 0 0' >>>>> >>>>> After a reset and reboot I can access /usr/local/news/bin. >>>>> >>>>> I delete this directory and reinstall the package `portupgrade -fu news/inn` >>>>> >>>>> 5 days later `periodic daily`is locked on the same directory :-o >>>>> >>>>> Any idea? >>>> >>>> I can't help with the deadlock, but someone who _can_ help will probably ask for >>>> the output of "procstat -kk PID" with the PID of the "find" process. >>> >>> In fact, it's procstat -kk -a. With just one thread we would see that a thread >>> is blocked on something, but we won't see why that something can not be acquired. >>> >>> >> I attach the result, > > Interesting. I can not spot any suspicious thread that would hold the vnode > lock. Could you please run kgdb (just like that, no arguments), then execute > 'bt' command and then select a frame when _vn_lock is called with 'fr N' > command. Then please 'print *vp' and share the result. > I Think I miss something in your request: [root@avoriaz ~]# kgdb GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. done. --- clip --- Loaded symbols for /boot/kernel/accf_data.ko Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. done. Loaded symbols for /boot/kernel/daemon_saver.ko #0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500, flags=) at /usr/src/sys/kern/sched_ule.c:1973 1973 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500, flags=) at /usr/src/sys/kern/sched_ule.c:1973 #1 0xffffffff80566b15 in tc_fill_vdso_timehands32 (vdso_th32=0x0) at /usr/src/sys/kern/kern_tc.c:2121 #2 0xffffffff80555227 in timekeep_push_vdso () at /usr/src/sys/kern/kern_sharedpage.c:174 #3 0xffffffff80566226 in tc_windup () at /usr/src/sys/kern/kern_tc.c:1426 #4 0xffffffff804eaa41 in hardclock_cnt (cnt=1, usermode=) at /usr/src/sys/kern/kern_clock.c:589 #5 0xffffffff808fac74 in handleevents (now=, fake=0) at /usr/src/sys/kern/kern_clocksource.c:223 #6 0xffffffff808fb1d7 in timercb (et=0xffffffff8100cf20, arg=) at /usr/src/sys/kern/kern_clocksource.c:352 #7 0xfffff800b6429a00 in ?? () #8 0xffffffff81051080 in vm_page_array () #9 0xffffffff81051098 in vm_page_queue_free_mtx () #10 0xfffffe0101818920 in ?? () #11 0xffffffff805399c0 in __mtx_lock_sleep (c=, tid=Error accessing memory address 0xffffffffffffffac: Bad add\ ress. ) at /usr/src/sys/kern/kern_mutex.c:590 Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) q [root@avoriaz ~]# I don't find the requested frame Henri