From owner-freebsd-stable@FreeBSD.ORG Thu Dec 14 08:41:10 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E1F0616A4C2 for ; Thu, 14 Dec 2006 08:41:09 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 84FAF43CB8 for ; Thu, 14 Dec 2006 08:39:35 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id EE21E470C6; Thu, 14 Dec 2006 03:41:08 -0500 (EST) Date: Thu, 14 Dec 2006 08:41:08 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alex Povolotsky In-Reply-To: <457FA5D7.9060502@webmail.sub.ru> Message-ID: <20061214083924.C72681@fledge.watson.org> References: <457FA5D7.9060502@webmail.sub.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-stable@freebsd.org Subject: Re: Strange panics X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2006 08:41:10 -0000 On Wed, 13 Dec 2006, Alex Povolotsky wrote: > I'm getting lots of panics on quite lightly loaded box under 6.1-RELEASE. > > The box runned two jails without any flaws for several weeks, after adding > third one it begins to panics at least once a day. > > I have no idea on what could cause this. > > Here is my dmesg.boot (partially, I can send it all) This problem should be fixed in 6.2-RC1 and forward, I believe. If you're unable to upgrade, I may be able to provide patch backports, but would recommend an upgrade generally. This panic occurs because of a race condition in reset handling with respect to socket options, and tends to occur more frequently on boxes with long-lived connections that get reset and applications that frequently check or set socket options. Robert N M Watson Computer Laboratory University of Cambridge > > > > Copyright (c) 1992-2006 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD 6.1-RELEASE #0: Tue Dec 12 08:44:28 MSK 2006 > tarkhil@box2s.sub.ru:/usr/obj/usr/src/sys/SMP > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2793.01-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0xf43 Stepping = 3 > Features=0xbfebfbff ,TM,PBE> > Features2=0x641d> > AMD Features=0x20100000 > Logical CPUs per core: 2 > real memory = 2147352576 (2047 MB) > avail memory = 2096328704 (1999 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 6 > cpu3 (AP): APIC ID: 7 > > and here is kgbd output > > Fatal trap 12: page fault while in kernel mode > cpuid = 2; apic id = 06 > fault virtual address = 0xac > fault code = supervisor write, page not present > instruction pointer = 0x20:0xc06ee9d0 > stack pointer = 0x28:0xe91e2b18 > frame pointer = 0x28:0xe91e2b34 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 16260 (tcpserver) > trap number = 12 > panic: page fault > cpuid = 2 > Uptime: 19h33m1s > > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc06507c9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402 > #2 0xc0650af1 in panic (fmt=0xc08958ae "%s") at > /usr/src/sys/kern/kern_shutdown.c:558 > #3 0xc084a2cc in trap_fatal (frame=0xe91e2ad8, eva=172) at > /usr/src/sys/i386/i386/trap.c:836 > #4 0xc084a00b in trap_pfault (frame=0xe91e2ad8, usermode=0, eva=172) at > /usr/src/sys/i386/i386/trap.c:744 > #5 0xc0849c45 in trap (frame= > {tf_fs = -1065156600, tf_es = -1056636888, tf_ds = 40, tf_edi = 55, > tf_esi = 0, tf_ebp = -383898828, tf_isp = -383898876, tf_ebx = -383898480, > tf_edx = -944133120, tf_ecx = 0, tf_eax = 4, tf_trapno = 12, tf_err = 2, > tf_eip = -1066473008, tf_cs = 32, tf_eflags = 66182, tf_esp = -880992256, > tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:434 > #6 0xc0836c4a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 > #7 0xc06ee9d0 in ip_ctloutput (so=0x4, sopt=0xe91e2c90) at atomic.h:146 > #8 0xc06feb73 in tcp_ctloutput (so=0xc811b6f4, sopt=0xe91e2c90) at > /usr/src/sys/netinet/tcp_usrreq.c:1038 > #9 0xc068bdf8 in sosetopt (so=0xc811b6f4, sopt=0xe91e2c90) at > /usr/src/sys/kern/uipc_socket.c:1560 > #10 0xc0691115 in kern_setsockopt (td=0xc7b9ac00, s=0, level=4, name=4, > val=0xc7b9ac00, valseg=UIO_USERSPACE, valsize=0) > at /usr/src/sys/kern/uipc_syscalls.c:1351 > #11 0xc0691046 in setsockopt (td=0xc7b9ac00, uap=0x4) at > /usr/src/sys/kern/uipc_syscalls.c:1307 > #12 0xc084a613 in syscall (frame= > {tf_fs = -1078001605, tf_es = 59, tf_ds = -1078001605, tf_edi = > -1077942204, tf_esi = 3, tf_ebp = -1077942408, tf_isp = -383898268, tf_ebx = > 0, tf_edx = 2, tf_ecx = 134545464, tf_eax = 105, tf_trapno = 12, tf_err = 2, > tf_eip = 186394799, tf_cs = 51, tf_eflags = 642, tf_esp = -1077942452, tf_ss > = 59}) at /usr/src/sys/i386/i386/trap.c:981 > #13 0xc0836c9f in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:200 > #14 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > > > === cut === > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0xac > fault code = supervisor write, page not present > instruction pointer = 0x20:0xc06ee9d0 > stack pointer = 0x28:0xe8f77b18 > frame pointer = 0x28:0xe8f77b34 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 86881 (tcpserver) > trap number = 12 > panic: page fault > cpuid = 0 > Uptime: 1h55m49s > Dumping 2047 MB (2 chunks) > chunk 0: 1MB (158 pages) ... ok > chunk 1: 2047MB (524000 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 > 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 > 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 > 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 > 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 > 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 > 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 > 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 > > #0 doadump () at pcpu.h:165 > 165 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc06507c9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402 > #2 0xc0650af1 in panic (fmt=0xc08958ae "%s") at > /usr/src/sys/kern/kern_shutdown.c:558 > #3 0xc084a2cc in trap_fatal (frame=0xe8f77ad8, eva=172) at > /usr/src/sys/i386/i386/trap.c:836 > #4 0xc084a00b in trap_pfault (frame=0xe8f77ad8, usermode=0, eva=172) at > /usr/src/sys/i386/i386/trap.c:744 > #5 0xc0849c45 in trap (frame= > {tf_fs = -939130872, tf_es = -1056636888, tf_ds = 40, tf_edi = 55, > tf_esi = 0, tf_ebp = -386434252, tf_isp = -386434300, tf_ebx = -386433904, > tf_edx = -951618816, tf_ecx = 0, tf_eax = 4, tf_trapno = 12, tf_err = 2, > tf_eip = -1066473008, tf_cs = 32, tf_eflags = 66182, tf_esp = -953071872, > tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:434 > #6 0xc0836c4a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 > #7 0xc06ee9d0 in ip_ctloutput (so=0x4, sopt=0xe8f77c90) at atomic.h:146 > #8 0xc06feb73 in tcp_ctloutput (so=0xc8c0142c, sopt=0xe8f77c90) at > /usr/src/sys/netinet/tcp_usrreq.c:1038 > #9 0xc068bdf8 in sosetopt (so=0xc8c0142c, sopt=0xe8f77c90) at > /usr/src/sys/kern/uipc_socket.c:1560 > #10 0xc0691115 in kern_setsockopt (td=0xc7477300, s=0, level=4, name=4, > val=0xc7477300, valseg=UIO_USERSPACE, valsize=0) > at /usr/src/sys/kern/uipc_syscalls.c:1351 > #11 0xc0691046 in setsockopt (td=0xc7477300, uap=0x4) at > /usr/src/sys/kern/uipc_syscalls.c:1307 > #12 0xc084a613 in syscall (frame= > {tf_fs = -1078001605, tf_es = 59, tf_ds = -1078001605, tf_edi = > -1077942088, tf_esi = 3, tf_ebp = -1077942296, tf_isp = -386433692, tf_ebx = > 0, tf_edx = 2, tf_ecx = 134545464, tf_eax = 105, tf_trapno = 12, tf_err = 2, > tf_eip = 186394799, tf_cs = 51, tf_eflags = 646, tf_esp = -1077942340, tf_ss > = 59}) at /usr/src/sys/i386/i386/trap.c:981 > #13 0xc0836c9f in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:200 > #14 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > === cut === > > NIC is Intel (em). > > Can anyone provide any help with this? > > Alex. > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >