From owner-freebsd-current@freebsd.org Mon May 16 21:22:43 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80DA9B3ED7A for ; Mon, 16 May 2016 21:22:43 +0000 (UTC) (envelope-from gwq_uk@yahoo.com) Received: from nm43-vm4.bullet.mail.bf1.yahoo.com (nm43-vm4.bullet.mail.bf1.yahoo.com [216.109.114.223]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3C9DA136F for ; Mon, 16 May 2016 21:22:42 +0000 (UTC) (envelope-from gwq_uk@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1463433639; bh=bbhfNwWQ7jqkkrrGNk1p6nl0gkdMYAQ2ieXTuk6qEb0=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=JSGohhh7LyqncBGEEULdCItZjuQ92SyOhRktwy+RPfPm2EiKyQNzGbfcnCi6VgWJwKlhVt0cR8f2TMud7MTP9JMe4zi9Rg2SGLU8gd+Xu+wehrnuBv+7bD7EyoNCkClEbKVPL32JTrTIAgCamePtkQPSvnnO/PpvXFC6mHaMzj2PuVupN5PYPxl7dtHPORxdEB9RvTo6mDYsAA6romgD6L7jvasV8WFNMnVkGCrJYu3HTL76g7BN1y7ORtWS4hjhH9uOxgrsADtxf56NdovWC9MuwiFWgYSeVCr6/7hmlPrYBtY97o0ZAp1Aas4lpIch81DbAMttAuHIlQAF1DdRQQ== Received: from [66.196.81.171] by nm43.bullet.mail.bf1.yahoo.com with NNFMP; 16 May 2016 21:20:39 -0000 Received: from [98.139.212.212] by tm17.bullet.mail.bf1.yahoo.com with NNFMP; 16 May 2016 21:20:39 -0000 Received: from [127.0.0.1] by omp1021.mail.bf1.yahoo.com with NNFMP; 16 May 2016 21:20:39 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 138344.54720.bm@omp1021.mail.bf1.yahoo.com X-YMail-OSG: kWBUnrUVM1mSM7WQarKANv0QvuaLEfO6VzHah994aTmW.LR8aDAgG3CHYKoD07A IbV50bVybdLx9_hoozgFIvB9OBsF52rS861VzsNLpDT.Xxkjna0liPSqzD9pKEEebjFQi83IGTVu 4BZzDlWvGk7m4IPg0JfDT4xwPbJiAG1Q7oAphGuJmYqe.1ObVHx.tCpu_QKsqJvzsHcaVPL1FNrI x6zWDSqcS4kTRj8zYV391X87u94jJb830ApnfJdvILTwkhEOg1N.o.kMPKo78QBlt_Mon2Xf8wsg 7PTZQHENViOC2EceGS9_o.YCJ4fM7PzOdPd3aZeTS3T_cpHE_2jAh.HKxPuGVtQcoBZY9zCyyc_Y 1CwUI.i_bCaPObO5siZPs3nNFh4EwhULAi3vCR.th8HjAyvTaaMYO4b93BE1GPqdPh84S2NtoPIG a6vCJ65gyedvsfVrANnpwKpmG5M9ydgJaasWNN2Z5KMSnSQe3SklOx3gM8.qYFctwyx7mEgYG_l7 uOXTJsVzB1Sc0mIVRwhhK7Q-- Received: from jws106101.mail.bf1.yahoo.com by sendmailws159.mail.bf1.yahoo.com; Mon, 16 May 2016 21:20:38 +0000; 1463433638.487 Date: Mon, 16 May 2016 21:20:28 +0000 (UTC) From: Greg Quinlan Reply-To: Greg Quinlan To: Freebsd current Message-ID: <1701232627.3438539.1463433628224.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <474864611.2829850.1463347476264.JavaMail.yahoo@mail.yahoo.com> References: <474864611.2829850.1463347476264.JavaMail.yahoo.ref@mail.yahoo.com> <474864611.2829850.1463347476264.JavaMail.yahoo@mail.yahoo.com> Subject: Re: Kernel panic - help MIME-Version: 1.0 X-Mailman-Approved-At: Mon, 16 May 2016 22:27:38 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 May 2016 21:22:43 -0000 And again, at almost the same time ... kernel panic!!! May 17 03:06:18 dns0 syslogd: kernel boot file is /boot/kernel/kernel May 17 03:06:18 dns0 kernel: panic: softdep_deallocate_dependencies: dangli= ng deps May 17 03:06:18 dns0 kernel: cpuid =3D 1 May 17 03:06:18 dns0 kernel: KDB: stack backtrace: May 17 03:06:18 dns0 kernel: #0 0xffffffff80ae4737 at kdb_backtrace+0x67 May 17 03:06:18 dns0 kernel: #1 0xffffffff80a9f502 at vpanic+0x182 May 17 03:06:18 dns0 kernel: #2 0xffffffff80a9f373 at panic+0x43 May 17 03:06:18 dns0 kernel: #3 0xffffffff80d5fa81 at softdep_deallocate_de= pendencies+0x71 May 17 03:06:18 dns0 kernel: #4 0xffffffff80b47eb9 at brelse+0x1b9 May 17 03:06:18 dns0 kernel: #5 0xffffffff80b456c2 at bufwrite+0x182 May 17 03:06:18 dns0 kernel: #6 0xffffffff80d8245c at ffs_write+0x39c May 17 03:06:18 dns0 kernel: #7 0xffffffff81093459 at VOP_WRITE_APV+0x139 May 17 03:06:18 dns0 kernel: #8 0xffffffff80dce82a at vnode_pager_generic_p= utpages+0x38a May 17 03:06:18 dns0 kernel: #9 0xffffffff810953b6 at VOP_PUTPAGES_APV+0xa6 May 17 03:06:18 dns0 kernel: #10 0xffffffff80dcc4be at vnode_pager_putpages= +0xfe May 17 03:06:18 dns0 kernel: #11 0xffffffff80dc19ca at vm_pageout_flush+0xe= a May 17 03:06:18 dns0 kernel: #12 0xffffffff80db848a at vm_object_page_colle= ct_flush+0x21a May 17 03:06:18 dns0 kernel: #13 0xffffffff80db81c7 at vm_object_page_clean= +0x197 May 17 03:06:18 dns0 kernel: #14 0xffffffff80db7c79 at vm_object_terminate+= 0x99 May 17 03:06:18 dns0 kernel: #15 0xffffffff80dccb96 at vnode_destroy_vobjec= t+0xc6 May 17 03:06:18 dns0 kernel: #16 0xffffffff80d8a336 at ufs_reclaim+0x26 May 17 03:06:18 dns0 kernel: #17 0xffffffff810944b6 at VOP_RECLAIM_APV+0xa6 May 17 03:06:18 dns0 kernel: Uptime: 16h54m12s May 17 03:06:18 dns0 kernel: Dumping 896 out of 8113 MB:..2%..11%..22%..31%= ..42%..52%..61%..72%..81%..92% On Monday, 16 May 2016, 7:24, Greg Quinlan wrote: =20 FreeBSD dns0 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r299817M: Sun May 15 18:= 55:50 AEST 2016=C2=A0 =C2=A0 root@dns0:/usr/obj/usr/src/sys/GENERIC=C2=A0 a= md64 =3D=3D=3D8<=3D=3D=3D May 16 03:03:48 dns0 syslogd: kernel boot file is /boot/kernel/kernel May 16 03:03:48 dns0 kernel: panic: softdep_deallocate_dependencies: dangli= ng deps May 16 03:03:48 dns0 kernel: cpuid =3D 0 May 16 03:03:48 dns0 kernel: KDB: stack backtrace: May 16 03:03:48 dns0 kernel: #0 0xffffffff80ae4737 at kdb_backtrace+0x67 May 16 03:03:48 dns0 kernel: #1 0xffffffff80a9f502 at vpanic+0x182 May 16 03:03:48 dns0 kernel: #2 0xffffffff80a9f373 at panic+0x43 May 16 03:03:48 dns0 kernel: #3 0xffffffff80d5fa81 at softdep_deallocate_de= pendencies+0x71 May 16 03:03:48 dns0 kernel: #4 0xffffffff80b47eb9 at brelse+0x1b9 May 16 03:03:48 dns0 kernel: #5 0xffffffff80b456c2 at bufwrite+0x182 May 16 03:03:48 dns0 kernel: #6 0xffffffff80d8245c at ffs_write+0x39c May 16 03:03:48 dns0 kernel: #7 0xffffffff81093459 at VOP_WRITE_APV+0x139 May 16 03:03:48 dns0 kernel: #8 0xffffffff80dce82a at vnode_pager_generic_p= utpages+0x38a May 16 03:03:48 dns0 kernel: #9 0xffffffff810953b6 at VOP_PUTPAGES_APV+0xa6 May 16 03:03:48 dns0 kernel: #10 0xffffffff80dcc4be at vnode_pager_putpages= +0xfe May 16 03:03:48 dns0 kernel: #11 0xffffffff80dc19ca at vm_pageout_flush+0xe= a May 16 03:03:48 dns0 kernel: #12 0xffffffff80db848a at vm_object_page_colle= ct_flush+0x21a May 16 03:03:48 dns0 kernel: #13 0xffffffff80db81c7 at vm_object_page_clean= +0x197 May 16 03:03:48 dns0 kernel: #14 0xffffffff80db7c79 at vm_object_terminate+= 0x99 May 16 03:03:48 dns0 kernel: #15 0xffffffff80dccb96 at vnode_destroy_vobjec= t+0xc6 May 16 03:03:48 dns0 kernel: #16 0xffffffff80d8a336 at ufs_reclaim+0x26 May 16 03:03:48 dns0 kernel: #17 0xffffffff810944b6 at VOP_RECLAIM_APV+0xa6 May 16 03:03:48 dns0 kernel: Uptime: 7h31m3s May 16 03:03:48 dns0 kernel: Dumping 953 out of 8113 MB:..2%..11%..21%..31%= ..41%..51%..61%..71%..81%..91%=C2=A0 Not sure if the above is useful, seems to happen early morning. Thanks Greg From owner-freebsd-current@freebsd.org Tue May 17 06:24:02 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B89B6B3EA32 for ; Tue, 17 May 2016 06:24:02 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id D623B1F34; Tue, 17 May 2016 06:24:01 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA12078; Tue, 17 May 2016 09:23:54 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1b2YQ5-000Ijh-Vg; Tue, 17 May 2016 09:23:54 +0300 To: FreeBSD Current , Edward Tomasz Napierala From: Andriy Gapon Subject: panic: destroying non-empty racct: 2113536 allocated for resource 4 Message-ID: Date: Tue, 17 May 2016 09:22:57 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2016 06:24:02 -0000 To be fair I got this panic after some exotic sequence of events: running poudriere, sending SIGSTOP to one of build processes, forgetting about it, seeing poudriere timeout that job, sending SIGCONT... This is amd64 head r297350. Some details: (kgdb) bt #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:295 #1 0xffffffff8062d7ef in kern_reboot (howto=) at /usr/src/sys/kern/kern_shutdown.c:363 #2 0xffffffff8062de38 in vpanic (fmt=, ap=0xfffffe0519b73920) at /usr/src/sys/kern/kern_shutdown.c:639 #3 0xffffffff8062db43 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:572 #4 0xffffffff8061ef1c in racct_destroy_locked (racctp=) at /usr/src/sys/kern/kern_racct.c:478 #5 0xffffffff8061ee45 in racct_destroy (racct=0xfffff802f6301518) at /usr/src/sys/kern/kern_racct.c:495 #6 0xffffffff805fdd3c in prison_racct_free_locked (prr=0xfffff802f6301400) at /usr/src/sys/kern/kern_jail.c:4564 #7 0xffffffff805fdc8d in prison_racct_free (prr=0xfffff802f6301400) at /usr/src/sys/kern/kern_jail.c:4583 #8 0xffffffff805fddee in prison_racct_detach (pr=0xfffff802b0730000) at /usr/src/sys/kern/kern_jail.c:4658 #9 0xffffffff805fb2cb in prison_deref (pr=, flags=3) at /usr/src/sys/kern/kern_jail.c:2663 #10 0xffffffff805fca25 in prison_remove_one (pr=) at /usr/src/sys/kern/kern_jail.c:2358 #11 0xffffffff805fc8e4 in sys_jail_remove (td=, uap=) at /usr/src/sys/kern/kern_jail.c:2313 #12 0xffffffff80820ddd in syscallenter (td=0xfffff801146019e0, sa=0xfffffe0519b73b80) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #13 0xffffffff808209af in amd64_syscall (td=0xfffff801146019e0, traced=0) at /usr/src/sys/amd64/amd64/trap.c:943 RACCT_RSS is 4. (kgdb) p *prr $5 = { prr_next = { le_next = 0xfffff80382fe4400, le_prev = 0xfffff8017ac90600 }, prr_name = "basejail-default-job-03", '\000' , prr_refcount = 0, prr_racct = 0xfffff802e3f520b0 } (kgdb) p *prr->prr_racct $6 = { r_resources = {13884177072, 0, 0, 0, 2113536, 0 , 13611325009, 0}, r_rule_links = { lh_first = 0x0 } } Could it be that somehow the CONT'd process failed to deduct its resources from the jail's resources because the jail was already marked for destruction or something like that? -- Andriy Gapon