From owner-freebsd-net@FreeBSD.ORG Fri Jan 27 15:30:11 2012 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 581D61065676 for ; Fri, 27 Jan 2012 15:30:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 36CA08FC12 for ; Fri, 27 Jan 2012 15:30:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q0RFUBEM038236 for ; Fri, 27 Jan 2012 15:30:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q0RFUBJa038232; Fri, 27 Jan 2012 15:30:11 GMT (envelope-from gnats) Date: Fri, 27 Jan 2012 15:30:11 GMT Message-Id: <201201271530.q0RFUBJa038232@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: "Pavel Polyakov" Cc: Subject: Re: kern/159621: [tcp] [panic] panic: soabort: so_count X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Pavel Polyakov List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jan 2012 15:30:11 -0000 The following reply was made to PR kern/159621; it has been noted by GNATS. From: "Pavel Polyakov" To: bug-followup@freebsd.org, bsd@kobyla.org Cc: Subject: Re: kern/159621: [tcp] [panic] panic: soabort: so_count Date: Fri, 27 Jan 2012 16:44:28 +0200 Dump header from device /dev/ada1p20 Architecture: amd64 Architecture Version: 1 Dump Length: 223744B (0 MB) Blocksize: 512 Dumptime: Tue Jan 24 05:32:13 2012 Hostname: shared Magic: FreeBSD Text Dump Version String: FreeBSD 8.2-STABLE #5 r230331: Mon Jan 23 22:05:26 UTC 2012 Panic String: soabort: so_count Dump Parity: 1380996126 Bounds: 1 Dump Status: good ichwd0: timer reloaded panic: soabort: so_count cpuid = 9 KDB: enter: panic 0xffffff0008b293b0: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL by thread 0xffffff0008657000 (pid 20) 0xffffff00bb6a8000: tag ufs, type VDIR usecount 1, writecount 0, refcount 758 mountedhere 0 flags () v_object 0xffffff00af22d948 ref 0 pages 6032 lock type ufs: EXCL by thread 0xffffff01c252a8c0 (pid 73631) with exclusive waiters pending ino 70625292, on dev mirror/shared0home 0xffffff00b47c0ce8: tag ufs, type VREG usecount 4, writecount 4, refcount 1128 mountedhere 0 flags () v_object 0xffffff0014183360 ref 0 pages 8984 lock type ufs: SHARED (count 4) ino 6584279, on dev mirror/shared0home 0xffffff00bc8b8938: tag ufs, type VREG usecount 4, writecount 0, refcount 5 mountedhere 0 flags () v_object 0xffffff017f4e9438 ref 2 pages 9 lock type ufs: SHARED (count 1) ino 103787584, on dev mirror/shared0home db:0:kdb.enter.panic> run lockinfo db:1:lockinfo> show locks exclusive rw tcpinp (tcpinp) r = 0 (0xffffff01c2425670) locked @ /usr/src/sys/netinet/tcp_input.c:946 exclusive rw tcp (tcp) r = 0 (0xffffffff80b35108) locked @ /usr/src/sys/netinet/tcp_usrreq.c:1507 db:1:locks> show alllocks Process 75843 (php) thread 0xffffff01c2b9a8c0 (100468) exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff81ede351c8) locked @ /usr/src/sys/vm/vm_pager.c:311 shared lockmgr ufs (ufs) r = 0 (0xffffff00bc8b89d0) locked @ /usr/src/sys/kern/vfs_subr.c:2170 Process 75837 (php) thread 0xffffff00b723e000 (100232) exclusive sleep mutex pmap (pmap) r = 0 (0xffffff0008f98d70) locked @ /usr/src/sys/amd64/amd64/pmap.c:4363 exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0 (0xffffffff80b41880) locked @ /usr/src/sys/vm/vm_object.c:1101 exclusive sleep mutex vm object (standard object) r = 0 (0xffffff00b41de0d8) locked @ /usr/src/sys/vm/vm_object.c:1048 shared sx user map (user map) r = 0 (0xffffff0008f98cb8) locked @ /usr/src/sys/vm/vm_map.c:2023 Process 75836 (php) thread 0xffffff01c2e228c0 (100490) exclusive sleep mutex vm object (standard object) r = 0 (0xffffff00bb23dbd0) locked @ /usr/src/sys/vm/vm_object.c:1048 shared sx user map (user map) r = 0 (0xffffff0008a44078) locked @ /usr/src/sys/vm/vm_map.c:2023 Process 75148 (php) thread 0xffffff010568c8c0 (100744) shared lockmgr ufs (ufs) r = 0 (0xffffff00b47c0d80) locked @ /usr/src/sys/kern/vfs_vnops.c:549 Process 74926 (exim-4.77-0) thread 0xffffff00b71ea000 (100323) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff01fc50c8f0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 74366 (exim-4.77-0) thread 0xffffff00b713c000 (100677) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0060a908f0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 74205 (php) thread 0xffffff01c2c2f8c0 (100694) shared lockmgr ufs (ufs) r = 0 (0xffffff00b47c0d80) locked @ /usr/src/sys/kern/vfs_vnops.c:549 Process 74118 (exim-4.77-0) thread 0xffffff00b7d748c0 (100685) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff01fc990e40) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 74105 (php) thread 0xffffff01c262d460 (100442) exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff81ee2c5e90) locked @ /usr/src/sys/kern/vfs_bio.c:1891 shared lockmgr ufs (ufs) r = 0 (0xffffff00b47c0d80) locked @ /usr/src/sys/kern/vfs_vnops.c:549 Process 73987 (exim-4.77-0) thread 0xffffff00b715a460 (100223) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00600c68f0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 73728 (php) thread 0xffffff0008f14000 (100195) shared lockmgr ufs (ufs) r = 0 (0xffffff00b47c0d80) locked @ /usr/src/sys/kern/vfs_vnops.c:549 Process 73631 (exim-4.77-0) thread 0xffffff01c252a8c0 (100353) exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff81ee2d62b8) locked @ /usr/src/sys/kern/vfs_bio.c:1891 exclusive sx dirhash (dirhash) r = 0 (0xffffff0092e50000) locked @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:216 exclusive lockmgr ufs (ufs) r = 0 (0xffffff00bb6a8098) locked @ /usr/src/sys/kern/vfs_subr.c:2170 Process 73552 (exim-4.77-0) thread 0xffffff0008bda000 (100206) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff005c0b80f8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 73547 (exim-4.77-0) thread 0xffffff01c252e000 (100349) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff01c27d6648) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 2234 (zabbix_agentd) thread 0xffffff0008c018c0 (100272) exclusive sleep mutex sigio lock (sigio lock) r = 0 (0xffffffff8096bb60) locked @ /usr/src/sys/kern/kern_descrip.c:1095 Process 20 (syncer) thread 0xffffff0008657000 (100115) exclusive lockmgr syncer (syncer) r = 0 (0xffffff0008b29448) locked @ /usr/src/sys/kern/vfs_subr.c:1770 Process 12 (intr) thread 0xffffff0002b068c0 (100077) exclusive sleep mutex em0:tx(0) (em0:tx(0)) r = 0 (0xffffff0002561808) locked @ /usr/src/sys/modules/em/../../dev/e1000/if_em.c:1523 Process 12 (intr) thread 0xffffff0002b07460 (100075) exclusive rw tcpinp (tcpinp) r = 0 (0xffffff01c2425670) locked @ /usr/src/sys/netinet/tcp_input.c:946 exclusive rw tcp (tcp) r = 0 (0xffffffff80b35108) locked @ /usr/src/sys/netinet/tcp_usrreq.c:1507 db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.panic> show pcpu cpuid = 9 dynamic pcpu = 0xffffff807f70d780 curthread = 0xffffff0002b07460: pid 12 "irq256: em0:rx 0" curpcb = 0xffffff8241018d10 fpcurthread = none idlethread = 0xffffff0002369460: tid 100009 "idle: cpu9" curpmap = 0xffffffff8096b670 tssp = 0xffffffff80b7aaa8 commontssp = 0xffffffff80b7aaa8 rsp0 = 0xffffff8241018d10 gs32p = 0xffffffff80b798e0 ldt = 0xffffffff80b79920 tss = 0xffffffff80b79910 spin locks held: db:0:kdb.enter.panic> bt Tracing pid 12 tid 100075 td 0xffffff0002b07460 kdb_enter() at kdb_enter+0x3b panic() at panic+0x17b soabort() at soabort+0x9c syncache_expand() at syncache_expand+0x2ca tcp_input() at tcp_input+0xf05 ip_input() at ip_input+0xb3 netisr_dispatch_src() at netisr_dispatch_src+0x9e ether_demux() at ether_demux+0x176 ether_input() at ether_input+0x198 em_rxeof() at em_rxeof+0x19d em_msix_rx() at em_msix_rx+0x24 intr_event_execute_handlers() at intr_event_execute_handlers+0x67 ithread_loop() at ithread_loop+0xad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8241018d00, rbp = 0 --- 6 0 0 0 SL crypto_r 0xffffffff80e03ce0 [crypto returns] 5 0 0 0 SL crypto_w 0xffffffff80e03ca0 [crypto] 13 0 0 0 SL - 0xffffffff8096e5c4 [yarrow] 4 0 0 0 SL - 0xffffffff8096ab68 [g_down] 3 0 0 0 SL - 0xffffffff8096ab60 [g_up] 2 0 0 0 SL - 0xffffffff8096ab50 [g_event] 12 0 0 0 RL (threaded) intr