From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 29 21:08:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 525CE10656D5 for ; Wed, 29 Oct 2008 21:08:54 +0000 (UTC) (envelope-from jrytoung@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.29]) by mx1.freebsd.org (Postfix) with ESMTP id BF6ED8FC14 for ; Wed, 29 Oct 2008 21:08:53 +0000 (UTC) (envelope-from jrytoung@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so76655ywe.13 for ; Wed, 29 Oct 2008 14:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=lD6eWArV/32IESEXwTijWJ6DsD4DE12mNfRNWRCu650=; b=jzrsB5+njVFSRXhYYy+cnQoOLEDxJVNUXsjuYjupJbcM0AMcHZdrA+HqDILsP5KgWy 5F9/Bsrr8PBy2rD5U7ZOO2ohzyFxWSjtuDR5VtfjXPuhNU8mEOfYS3me5D2/Wb71Klhs HE8G7Asxc2swK4j9ALnDMXGiepR3UssA6BXbk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=yD9uLvGhh9Og4fYuhGwVO9z6e3RyQQmFpAkkYUn0ulP02w6Kxk9g/rGIeLdknD/KSp MChEo2uqz9pS4JOkrnZdnAWrxzqPbpAEnwCsKf2nprHWBqduAa30qwdF028W/ZHehWdI y6QDSh8wr38Ah8X7OJ59ghJrUdDHV+Ds/RJCc= Received: by 10.90.91.9 with SMTP id o9mr7813257agb.117.1225313159838; Wed, 29 Oct 2008 13:45:59 -0700 (PDT) Received: by 10.90.86.20 with HTTP; Wed, 29 Oct 2008 13:45:59 -0700 (PDT) Message-ID: <86068e730810291345r738242b0lb8130bf6bd011015@mail.gmail.com> Date: Wed, 29 Oct 2008 13:45:59 -0700 From: "Jerry Toung" To: "Robert Watson" , freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: crash at in_pcb.c X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Oct 2008 21:08:54 -0000 Hello List, I can realiably reproduce this crash. We have a deamon that accept several connections per sec. We use iperf and Microsoft Web application stress 1.0 to push traffic to the FreeBSD box. Without further delay, the crash dump is below. I've been troubleshooting, but I am no longer sure if this is a race condition or a stack corruption. The socket pointer between frame 12 and 11 is different. This is on 6.2, but the code for 7.0 is identical, so I think it still applies. Any hint, patching or troubleshooting this is appreciated. Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x2aef0210 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0769098 stack pointer = 0x28:0xef781bc0 frame pointer = 0x28:0xef781bd0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1166 (ndaemon) trap number = 12 panic: page fault cpuid = 0 Uptime: 8h32m25s Dumping 3325 MB (3 chunks) #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) l *0xc0769098 0xc0769098 is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:923). 918 /usr/src/sys/netinet/in_pcb.c: No such file or directory. in /usr/src/sys/netinet/in_pcb.c (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc06c2812 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412 #2 0xc06c2bbd in panic (fmt=0xc0940872 "%s") at /usr/src/sys/kern/kern_shutdown.c:573 #3 0xc08f3e4e in trap_fatal (frame=0xef781b80, eva=720306704) at /usr/src/sys/i386/i386/trap.c:838 #4 0xc08f3b57 in trap_pfault (frame=0xef781b80, usermode=0, eva=720306704) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc08f3745 in trap (frame= {tf_fs = -277348344, tf_es = 40, tf_ds = -913309656, tf_edi = 6, tf_esi = 0, tf_ebp = -277341232, tf_isp = -277341268, tf_ebx = -1062683820, tf_edx = 720306704, tf_ecx = 14063, tf_eax = 720306704, tf_trapno = 12, tf_err = 0, tf_eip = -1065971560, tf_cs = 32, tf_eflags = 66050, tf_esp = 0, tf_ss = -1062683820}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc08dddba in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = 0}, lport_arg=720306704, wild_okay=1) at /usr/src/sys/netinet/in_pcb.c:923 #8 0xc0768452 in in_pcbbind_setup (inp=0xc97150b4, nam=0x36ef, laddrp=0xc97150ec, lportp=0xc97150ce, cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:464 #9 0xc0767f56 in in_pcbbind (inp=0xc97150b4, nam=0x2aef0210, cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:240 #10 0xc077f272 in tcp_connect (tp=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/netinet/tcp_usrreq.c:864 #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/netinet/tcp_usrreq.c:369 #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/kern/uipc_socket.c:558 #13 0xc07046a8 in kern_connect (td=0xc990e180, fd=89, sa=0xc98a1ba0) at /usr/src/sys/kern/uipc_syscalls.c:536 #14 0xc070460f in connect (td=0xc990e180, uap=0xef781d04) at /usr/src/sys/kern/uipc_syscalls.c:505 #15 0xc08f4193 in syscall (frame= {tf_fs = 135725115, tf_es = 59, tf_ds = -1088487365, tf_edi = 135745024, tf_esi = -1089511444, tf_ebp = -1089514536, tf_isp = -277340828, tf_ebx = 671753396, tf_edx = 0, tf_ecx = 135524256, tf_eax = 98, tf_trapno = 0, tf_err = 2, tf_eip = 674451435, tf_cs = 51, tf_eflags = 642, tf_esp = -1089514580, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984 #16 0xc08dde0f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #17 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 7 #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr = 0}, lport_arg=720306704, wild_okay=1) at /usr/src/sys/netinet/in_pcb.c:923 923 in /usr/src/sys/netinet/in_pcb.c (kgdb) i loc phd = (struct inpcbport *) 0x2aef0210 tmphd = (struct inpcbport *) 0x2aef0210 match = (struct inpcb *) 0x0 inp = (struct inpcb *) 0x2aef0210 tmpinp = (struct inpcb *) 0x2aef0210 matchwild = 6 wildcard = -1062683820 lport = 14063 (kgdb) p phd $1 = (struct inpcbport *) 0x2aef0210 (kgdb) p phd->phd_port Cannot access memory at address 0x2aef021c (kgdb) f 12 #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/kern/uipc_socket.c:558 558 /usr/src/sys/kern/uipc_socket.c: No such file or directory. in /usr/src/sys/kern/uipc_socket.c (kgdb) p so $2 = (struct socket *) 0xc97b39bc (kgdb) p nam $3 = (struct sockaddr *) 0xc98a1ba0 (kgdb) p td $4 = (struct thread *) 0xc990e180 (kgdb) l 553 in /usr/src/sys/kern/uipc_socket.c (kgdb) f 11 #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0, td=0xc990e180) at /usr/src/sys/netinet/tcp_usrreq.c:369 369 /usr/src/sys/netinet/tcp_usrreq.c: No such file or directory. in /usr/src/sys/netinet/tcp_usrreq.c (kgdb)