From owner-freebsd-stable@freebsd.org Sun Nov 13 13:47:08 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB816C3F76D for ; Sun, 13 Nov 2016 13:47:08 +0000 (UTC) (envelope-from hlh@restart.be) Received: from tignes.restart.be (tignes.restart.be [IPv6:2001:41d0:8:bdbe:0:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tignes.restart.be", Issuer "CA master" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8AD41122F; Sun, 13 Nov 2016 13:47:08 +0000 (UTC) (envelope-from hlh@restart.be) X-Comment: SPF check N/A for local connections - client-ip=2001:41d0:8:bdbe:1:1::; helo=restart.be; envelope-from=hlh@restart.be; receiver=avg@freebsd.org DKIM-Filter: OpenDKIM Filter v2.10.3 tignes.restart.be 3tGw1k2Vb7zrFy DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=restart.be; s=tignes; t=1479044826; bh=/Rv38RicVY98+P3PJTt1EVE41CazKZ4MpViWO3Ih4vw=; h=Subject:To:References:Cc:From:Date:In-Reply-To; z=Subject:=20Re:=20Freebsd=2011.0=20RELEASE=20-=20ZFS=20deadlock|To :=20Andriy=20Gapon=20,=20freebsd-stable@FreeBSD.o rg|References:=20<0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> =0D=0A=20=0D=0A= 20<43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org>=0D=0A=20=0D=0A=20<9d1f9a76-5a8 d-6eca-9a50-907d55099847@FreeBSD.org>=0D=0A=20<6bc95dce-31e1-3013- bfe3-7c2dd80f9d1e@restart.be>=0D=0A=20=0D=0A=20<23a66749-f138-1f1a-afae-c775f906ff 37@restart.be>=0D=0A=20<8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeB SD.org>=0D=0A=20<6d991cea-b420-531e-12cc-001e4aeed66b@restart.be>= 0D=0A=20<67f2e8bd-bff0-f808-7557-7dabe5cad78c@FreeBSD.org>=0D=0A=2 0<1cb09c54-5f0e-2259-a41a-fefe76b4fe8b@restart.be>=0D=0A=20=0D=0A=20<9f20020b-e2f1 -862b-c3fc-dc6ff94e301e@restart.be>=0D=0A=20=0D=0A=20<599c5a5b-aa08-2030-34f3-23ff 19d09a9b@restart.be>=0D=0A=20<32686283-948a-6faf-7ded-ed8fcd23affb @FreeBSD.org>=0D=0A=20|Cc:=20Konstantin=20Belousov=20|From:=20Henr i=20Hennebert=20|Date:=20Sun,=2013=20Nov=202016=20 14:47:04=20+0100|In-Reply-To:=20; b=jvKaTxb4z3PxDiZaeR9NW6/2d3WFolAcBb5DqiN8Nsna6nfwbHPun/DgrRLFgWEV/ q7HZk5zypbSnvjXC5Q9893D3EINnwE33c6JQT3MrT3F6xWMkMB31OG1MqgGsewnKBo gSmEjsiDlbP7jdqeT7ffd5O0In0q6jrWJGRZykn8eBLTJADXMJlEbAv0ONTeQ6aapi JJMjJCsxPEd95eeVrrW+teA2fWtHt+F5gnEMbsInCfKZJ1dCyA3biGANOBnrLqONkr qthMDUlkYqfaBKa+RnRtNun5AmF9KY2cKY0fiPX8imBjQ82cTOYztSKHcG8qPUMsv/ OyAa430K/DvhA== Received: from restart.be (avoriaz.restart.be [IPv6:2001:41d0:8:bdbe:1:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.restart.be", Issuer "CA master" (verified OK)) by tignes.restart.be (Postfix) with ESMTPS id 3tGw1k2Vb7zrFy; Sun, 13 Nov 2016 14:47:05 +0100 (CET) Received: from chamonix.restart.bel (chamonix.restart.bel [IPv6:2001:41d0:8:bdbe:1:9:0:0]) (authenticated bits=0) by restart.be (8.15.2/8.15.2) with ESMTPSA id uADDl4gQ071938 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 13 Nov 2016 14:47:05 +0100 (CET) (envelope-from hlh@restart.be) Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock To: Andriy Gapon , freebsd-stable@FreeBSD.org References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org> <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> <6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be> <23a66749-f138-1f1a-afae-c775f906ff37@restart.be> <8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeBSD.org> <6d991cea-b420-531e-12cc-001e4aeed66b@restart.be> <67f2e8bd-bff0-f808-7557-7dabe5cad78c@FreeBSD.org> <1cb09c54-5f0e-2259-a41a-fefe76b4fe8b@restart.be> <9f20020b-e2f1-862b-c3fc-dc6ff94e301e@restart.be> <599c5a5b-aa08-2030-34f3-23ff19d09a9b@restart.be> <32686283-948a-6faf-7ded-ed8fcd23affb@FreeBSD.org> Cc: Konstantin Belousov From: Henri Hennebert Message-ID: Date: Sun, 13 Nov 2016 14:47:04 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2016 13:47:09 -0000 On 11/13/2016 14:28, Henri Hennebert wrote: > This 2 threads are innd processes. In core.txt.4: > > 8 14789 29165 0 24 4 40040 6612 zfs DN - 0:00.00 [innd] > 8 29165 1 0 20 0 42496 6888 select Ds - 0:01.33 [innd] > 8 49778 29165 0 24 4 40040 6900 zfs DN - 0:00.00 [innd] > 8 82034 29165 0 24 4 132 0 zfs DN - 0:00.00 [innd] > > the corresponding info treads are: > > 687 Thread 101243 (PID=49778: innd) sched_switch > (td=0xfffff800b642b500, newtd=0xfffff8000285ea00, flags= out>) at /usr/src/sys/kern/sched_ule.c:1973 > 681 Thread 101147 (PID=14789: innd) sched_switch > (td=0xfffff80065f4e500, newtd=0xfffff8000285f000, flags= out>) at /usr/src/sys/kern/sched_ule.c:1973 > 669 Thread 101250 (PID=82034: innd) sched_switch > (td=0xfffff800b6429000, newtd=0xfffff8000285ea00, flags= out>) at /usr/src/sys/kern/sched_ule.c:1973 > 665 Thread 101262 (PID=29165: innd) sched_switch > (td=0xfffff800b6b54a00, newtd=0xfffff8000285ea00, flags= out>) at /usr/src/sys/kern/sched_ule.c:1973 > In case it may help, I have a look at innd. This processes use 2 execv: one to execute /bin/sh and the other to execute itself: /* ** Re-exec ourselves. */ static const char * CCxexec(char *av[]) { char *innd; char *p; int i; if (CCargv == NULL) return "1 no argv!"; innd = concatpath(innconf->pathbin, "innd"); /* Get the pathname. */ p = av[0]; if (*p == '\0' || strcmp(p, "innd") == 0) CCargv[0] = innd; else return "1 Bad value"; #ifdef DO_PERL PLmode(Mode, OMshutdown, av[0]); #endif #ifdef DO_PYTHON PYmode(Mode, OMshutdown, av[0]); #endif JustCleanup(); syslog(L_NOTICE, "%s execv %s", LogName, CCargv[0]); /* Close all fds to protect possible fd leaking accross successive innds. */ for (i=3; i<30; i++) close(i); execv(CCargv[0], CCargv); syslog(L_FATAL, "%s cant execv %s %m", LogName, CCargv[0]); _exit(1); /* NOTREACHED */ return "1 Exit failed"; } The culprit may be /usr/local/news/bin/innd, remember that find is locked in /usr/local/news/bin Henri