From owner-freebsd-bugs Thu Mar 6 13:50:22 2003 Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B095037B401 for ; Thu, 6 Mar 2003 13:50:16 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8D3B43FA3 for ; Thu, 6 Mar 2003 13:50:15 -0800 (PST) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.6/8.12.6) with ESMTP id h26LoFNS047550 for ; Thu, 6 Mar 2003 13:50:15 -0800 (PST) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.6/8.12.6/Submit) id h26LoFlG047549; Thu, 6 Mar 2003 13:50:15 -0800 (PST) Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D13EA37B401 for ; Thu, 6 Mar 2003 13:49:17 -0800 (PST) Received: from edgemaster.zombie.org (edgemaster.creighton.edu [147.134.112.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id D859543F93 for ; Thu, 6 Mar 2003 13:49:14 -0800 (PST) (envelope-from smkelly@zombie.org) Received: by edgemaster.zombie.org (Postfix, from userid 1001) id CF35539839; Thu, 6 Mar 2003 15:49:13 -0600 (CST) Message-Id: <20030306214913.CF35539839@edgemaster.zombie.org> Date: Thu, 6 Mar 2003 15:49:13 -0600 (CST) From: Sean Kelly Reply-To: Sean Kelly To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Subject: kern/48985: Repeated panic which appears to be TCP related Sender: owner-freebsd-bugs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org >Number: 48985 >Category: kern >Synopsis: Repeated panic which appears to be TCP related >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Mar 06 13:50:15 PST 2003 >Closed-Date: >Last-Modified: >Originator: Sean Kelly >Release: FreeBSD 5.0-CURRENT i386 >Organization: >Environment: System: FreeBSD edgemaster.zombie.org 5.0-CURRENT FreeBSD 5.0-CURRENT #0: Tue Mar 4 20:30:35 CST 2003 smkelly@edgemaster.zombie.org:/usr/obj/usr/src/sys/EDGEMASTER i386 >Description: After upgrading from a 5.0-CURRENT kernel from Feb 11 to one from March 4, I began to see random yet repeated panics related to locking in netinet/tcp_input.c Script started on Wed Mar 5 19:51:06 2003 edgemaster# gdb -k /boot/kernel/kernel.debug vmcore.4 GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-undermydesk-freebsd"... panic: bwrite: buffer is not busy??? panic messages: --- panic: lock (sleep mutex) tcp not locked @ /usr/src/sys/netinet/tcp_input.c:2190 Stack backtrace: syncing disks, buffers remaining... panic: bwrite: buffer is not busy??? Uptime: 20m40s Dumping 1279 MB ata1: resetting devices .. done [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] 16 32 48[CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960 976 992 1008 1024 1040 1056 1072 1088 1104 1120 1136 1152 1168 1184 1200 1216 1232 1248 1264 --- #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:239 239 dumping++; (kgdb) bt full #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:239 No locals. #1 0xc01cd66a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:371 No locals. #2 0xc01cd8d3 in panic () at /usr/src/sys/kern/kern_shutdown.c:542 td = (struct thread *) 0xc281aa50 bootopt = 260 newpanic = 0 buf = "bwrite: buffer is not busy???\0ked @ /usr/src/sys/netinet/tcp_input.c:2190", '\0' #3 0xc020e142 in bwrite (bp=0xd30a4778) at /usr/src/sys/kern/vfs_bio.c:795 oldflags = 537002148 newbp = (struct buf *) 0xc767d5b4 #4 0xc020fe5c in vfs_bio_awrite (bp=0xd30a4778) at /usr/src/sys/kern/vfs_bio.c:1692 i = 1 j = 0 lblkno = 0 vp = (struct vnode *) 0xc767d5b4 ncl = 0 nwritten = 16384 size = 16384 maxcl = 8 #5 0xc02c0dca in ffs_fsync (ap=0xdf976a00) at /usr/src/sys/ufs/ffs/ffs_vnops.c:257 vp = (struct vnode *) 0xc767d5b4 ip = (struct inode *) 0xd30a4778 bp = (struct buf *) 0xd30a4778 nbp = (struct buf *) 0x0 error = 0 wait = 0 passes = 4 skipmeta = 0 lbn = 1 #6 0xc02bff1e in ffs_sync (mp=0xc697a000, waitfor=2, cred=0xc2806e80, td=0xc037f6a0) at vnode_if.h:612 nvp = (struct vnode *) 0xc767d490 vp = (struct vnode *) 0xc767d5b4 devvp = (struct vnode *) 0xc767d5b4 ip = (struct inode *) 0x0 ump = (struct ufsmount *) 0xc699b300 fs = (struct fs *) 0xc6970000 error = 0 count = 0 wait = 0 lockreq = 18 allerror = 0 #7 0xc022261b in sync (td=0xc037f6a0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:138 mp = (struct mount *) 0xc697a000 nmp = (struct mount *) 0x0 asyncflag = 0 #8 0xc01cd29c in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:280 bp = (struct buf *) 0x0 iter = -1031689648 nbusy = -1031700352 pbusy = -1031689648 subiter = -1031700352 #9 0xc01cd8d3 in panic () at /usr/src/sys/kern/kern_shutdown.c:542 td = (struct thread *) 0xc281aa50 bootopt = 256 newpanic = 1 buf = "bwrite: buffer is not busy???\0ked @ /usr/src/sys/netinet/tcp_input.c:2190", '\0' #10 0xc01ee013 in witness_unlock (lock=0xc038f9cc, flags=8, file=0xc035a374 "/usr/src/sys/netinet/tcp_input.c", line=2190) at /usr/src/sys/kern/subr_witness.c:951 lock_list = (struct lock_list_entry **) 0xc03e3540 instance = (struct lock_instance *) 0xc03e3554 class = (struct lock_class *) 0xc0384160 s = 1664 i = 0 j = -1070007860 #11 0xc01c4952 in _mtx_unlock_flags (m=0xc03e3554, opts=0, file=0xc038f9cc "`A8À\a\2125À\a\2125À", line=-1069664960) at /usr/src/sys/kern/kern_mutex.c:357 No locals. #12 0xc0255ec9 in tcp_input (m=0xc038f9cc, off0=20) at /usr/src/sys/netinet/tcp_input.c:2324 th = (struct tcphdr *) 0xc34c1824 ip = (struct ip *) 0xc34c1810 ipov = (struct ipovly *) 0xc03e3540 inp = (struct inpcb *) 0xc6cce804 optp = (u_char *) 0xc34c1838 "\001\001\b\n\026û\034\022" optlen = 12 len = -1069664940 tlen = 1409 off = -1069664940 drop_hdrlen = 52 tp = (struct tcpcb *) 0xc75ae42c thflags = 1 so = (struct socket *) 0xc758b300 todrop = -1069664940 acked = -1069664940 ourfinisacked = -1069664940 needoutput = 0 tiwin = 3225302336 to = {to_flags = 1, to_tsval = 385555474, to_tsecr = 123912, to_cc = 0, to_ccecho = 0, to_mss = 0, to_requested_s_scale = 0 '\0', to_pad = 0 '\0'} taop = (struct rmxp_tao *) 0xc03e3554 tao_noncached = {tao_cc = 1461, tao_ccsent = 49209, tao_mssopt = 11371} headlocked = 0 next_hop = (struct sockaddr_in *) 0x0 rstreason = -1069664940 ip6 = (struct ip6_hdr *) 0x0 isipv6 = 0 #13 0xc024e6ea in ip_input (m=0xc283ff00) at /usr/src/sys/netinet/ip_input.c:944 ip = (struct ip *) 0xc34c1810 fp = (struct ipq *) 0xc0353989 ia = (struct in_ifaddr *) 0xc6979400 ifa = (struct ifaddr *) 0x0 i = 0 hlen = 20 checkif = 0 sum = 0 pkt_dst = {s_addr = 1148225171} divert_info = 0 args = {m = 0xc283ff00, oif = 0x0, next_hop = 0x0, rule = 0x0, eh = 0x0, ro = 0xdf976ce0, dst = 0xc03e6174, flags = 227, f_id = { dst_ip = 2475061316, src_ip = 3560425173, dst_port = 49209, src_port = 11371, proto = 6 '\006', flags = 24 '\030'}, divert_rule = 0, retval = 3224731712} #14 0xc02348a2 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:230 ni = (struct netisr *) 0xc038cb90 m = (struct mbuf *) 0xc283ff00 bits = 0 i = 0 #15 0xc01ae6c2 in ithread_loop (arg=0xc2818200) at /usr/src/sys/kern/kern_intr.c:536 ithd = (struct ithd *) 0xc2818200 ih = (struct intrhand *) 0xc280f180 td = (struct thread *) 0xc281aa50 p = (struct proc *) 0xc28199ec #16 0xc01ad902 in fork_exit (callout=0xc01ae570 , arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:871 td = (struct thread *) 0x0 p = (struct proc *) 0xc28199ec (kgdb) edgemaster# Script done on Wed Mar 5 19:51:29 2003 >How-To-Repeat: I'm not sure. I've been bit by it about three times now, though. Each time, I was in the middle of reading e-mail. In the process of the panic, I also had my ~/.gnupg/pubring.pgp file totally wiped and the contents was filled with ^@s. That leads me to believe that the panics were happening when keyservers were being talked to to get keys. >Fix: Hope that somebody with more kernel locking skills will find the problem before I do (if I do). >Release-Note: >Audit-Trail: >Unformatted: To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message