Date: Tue, 6 May 2008 14:22:04 +0100 (BST) From: matthew.seaman@thebunker.net To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/123463: repeatable crash related to ipsec-tools Message-ID: <200805061322.m46DM4hT001184@obol.hosted-at.thebunker.net> Resent-Message-ID: <200805061400.m46E06W9088646@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 123463 >Category: kern >Synopsis: repeatable crash related to ipsec-tools >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue May 06 14:00:06 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Matthew Seaman >Release: FreeBSD 7.0-RELEASE-p1 amd64 >Organization: The Bunker >Environment: System: FreeBSD obol.hosted-at.thebunker.net 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #3: Sun May 4 10:46:11 BST 2008 root@obol.hosted-at.thebunker.net:/usr/obj/usr/src/sys/OBOL amd64 >Description: I have a new HP DL140G3 server runing RELENG_7_0 which has been stable up to now. However the combination of configuring it as an IPSec tunnel end-point and then turning on some Nagios monitoring via the tunnel causes the machine to crash within a few minutes. kgdb backtrace attached from the latest crash attached. I'm using racoon from security/ipsec-tools for IKE -- I had tried previously using security/isakmpd but in that case I found the process would run fine for maybe 20 minutes, then get into a loop where it chewed up lots of RAM very fast, until the kernel killed it. >How-To-Repeat: >Fix: --- kgdb.out begins here --- Script started on Tue May 6 14:01:27 2008 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid = 3; apic id = 03 instruction pointer = 0x8:0xffffffff80706048 stack pointer = 0x10:0xffffffffae5cbf30 frame pointer = 0x10:0xffffff0001e1c300 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 779 (snmpd) trap number = 9 panic: general protection fault cpuid = 3 GEOM_MIRROR: Device gm0: rebuilding provider da0 stopped. Uptime: 7m23s Physical memory: 2034 MB Dumping 314 MB: 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11 #0 doadump () at pcpu.h:194 194 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:194 #1 0x0000000000000004 in ?? () #2 0xffffffff8045b9cf in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff8045bdf8 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:563 #4 0xffffffff8071f8ca in trap_fatal (frame=0xffffff00014f8350, eva=18446742974219888848) at /usr/src/sys/amd64/amd64/trap.c:724 #5 0xffffffff80720388 in trap (frame=0xffffffffae5cbe80) at /usr/src/sys/amd64/amd64/trap.c:526 #6 0xffffffff8070738e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #7 0xffffffff80706048 in bus_dmamap_load_mbuf_sg (dmat=0xffffff00012a7a00, map=0x0, m0=Variable "m0" is not available. ) at /usr/src/sys/amd64/amd64/busdma_machdep.c:816 #8 0xffffffff80270a85 in bge_start_locked (ifp=0xffffff0001277000) at /usr/src/sys/dev/bge/if_bge.c:3390 #9 0xffffffff802714a7 in bge_start (ifp=0xffffff0001277000) at /usr/src/sys/dev/bge/if_bge.c:3572 #10 0xffffffff804e977e in ether_output_frame (ifp=0xffffff0001277000, m=0xffffff0001977100) at /usr/src/sys/net/if_ethersubr.c:405 #11 0xffffffff804e9cdf in ether_output (ifp=0xffffff0001277000, m=0xffffff0001977100, dst=Variable "dst" is not available. ) at /usr/src/sys/net/if_ethersubr.c:374 #12 0xffffffff805333e9 in ip_output (m=0xffffff0001977100, opt=Variable "opt" is not available. ) at /usr/src/sys/netinet/ip_output.c:583 #13 0xffffffff805bf747 in ipsec_process_done (m=0xffffff000177bc00, isr=0xffffff000174f800) at /usr/src/sys/netipsec/ipsec_output.c:177 #14 0xffffffff805cd8f8 in esp_output_cb (crp=0xffffff0001e24cb8) at /usr/src/sys/netipsec/xform_esp.c:965 #15 0xffffffff80606109 in crypto_done (crp=0xffffff0001e24cb8) at /usr/src/sys/opencrypto/crypto.c:1148 #16 0xffffffff8060934c in swcr_process (dev=Variable "dev" is not available. ) at /usr/src/sys/opencrypto/cryptosoft.c:975 #17 0xffffffff80606e89 in crypto_invoke (cap=Variable "cap" is not available. ) at cryptodev_if.h:53 #18 0xffffffff80607974 in crypto_dispatch (crp=0xffffff0001e24cb8) at /usr/src/sys/opencrypto/crypto.c:798 #19 0xffffffff805cdf91 in esp_output (m=0xffffff000161c360, isr=0xffffff000174f800, mp=Variable "mp" is not available. ) at /usr/src/sys/netipsec/xform_esp.c:875 #20 0xffffffff805bf95b in ipsec4_process_packet (m=0xffffff000177bc00, isr=0xffffff000174f800, flags=Variable "flags" is not available. ) at /usr/src/sys/netipsec/ipsec_output.c:486 #21 0xffffffff805312e7 in ip_ipsec_output (m=0xffffffffae5cc8b8, inp=0xffffff000168c360, flags=0xffffffffae5cc8ac, error=0xffffffffae5cc8f8, ro=Variable "ro" is not available. ) at /usr/src/sys/netinet/ip_ipsec.c:331 #22 0xffffffff80532814 in ip_output (m=0xffffff000177bc00, opt=Variable "opt" is not available. ) at /usr/src/sys/netinet/ip_output.c:418 #23 0xffffffff80594ab3 in udp_send (so=Variable "so" is not available. ) at /usr/src/sys/netinet/udp_usrreq.c:972 #24 0xffffffff804abb60 in sosend_dgram (so=0xffffff0001aadae0, addr=0xffffff000161c090, uio=Variable "uio" is not available. ) at /usr/src/sys/kern/uipc_socket.c:1053 #25 0xffffffff804af176 in kern_sendit (td=0xffffff00014f8350, s=11, mp=0xffffffffae5ccb10, flags=0, control=0x0, segflg=Variable "segflg" is not available. ) at /usr/src/sys/kern/uipc_syscalls.c:789 #26 0xffffffff804b1c6a in sendit (td=0xffffff00014f8350, s=11, mp=0xffffffffae5ccb10, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:730 #27 0xffffffff804b1d4a in sendto (td=Variable "td" is not available. ) at /usr/src/sys/kern/uipc_syscalls.c:841 #28 0xffffffff8071fedc in syscall (frame=0xffffffffae5ccc70) at /usr/src/sys/amd64/amd64/trap.c:852 #29 0xffffffff8070759b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:290 #30 0x00000008018d607c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) up #1 0x0000000000000004 in ?? () (kgdb) up #2 0xffffffff8045b9cf in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 409 doadump(); (kgdb) up #3 0xffffffff8045bdf8 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:563 563 boot(bootopt); (kgdb) up #4 0xffffffff8071f8ca in trap_fatal (frame=0xffffff00014f8350, eva=18446742974219888848) at /usr/src/sys/amd64/amd64/trap.c:724 724 panic("%s", trap_msg[type]); (kgdb) up #5 0xffffffff80720388 in trap (frame=0xffffffffae5cbe80) at /usr/src/sys/amd64/amd64/trap.c:526 526 trap_fatal(frame, 0); (kgdb) up #6 0xffffffff8070738e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 169 call trap Current language: auto; currently asm (kgdb) up #7 0xffffffff80706048 in bus_dmamap_load_mbuf_sg (dmat=0xffffff00012a7a00, map=0x0, m0=Variable "m0" is not available. ) at /usr/src/sys/amd64/amd64/busdma_machdep.c:816 816 if (m->m_len > 0) { Current language: auto; currently c (kgdb) list 811 int first = 1; 812 bus_addr_t lastaddr = 0; 813 struct mbuf *m; 814 815 for (m = m0; m != NULL && error == 0; m = m->m_next) { 816 if (m->m_len > 0) { 817 error = _bus_dmamap_load_buffer(dmat, map, 818 m->m_data, m->m_len, 819 NULL, flags, &lastaddr, 820 segs, nsegs, first); (kgdb) quit Script done on Tue May 6 14:05:50 2008 --- kgdb.out ends here --- --- dmesg.boot begins here --- Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-RELEASE-p1 #3: Sun May 4 10:46:11 BST 2008 root@obol.hosted-at.thebunker.net:/usr/obj/usr/src/sys/OBOL Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz (1995.01-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Stepping = 11 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> Cores per package: 4 usable memory = 2133483520 (2034 MB) avail memory = 2058792960 (1963 MB) ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) hptrr: HPT RocketRAID controller driver v1.1 (May 4 2008 10:46:04) acpi0: <PTLTD RSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 p4tcc0: <CPU Frequency Thermal Control> on cpu0 cpu1: <ACPI CPU> on acpi0 p4tcc1: <CPU Frequency Thermal Control> on cpu1 cpu2: <ACPI CPU> on acpi0 p4tcc2: <CPU Frequency Thermal Control> on cpu2 cpu3: <ACPI CPU> on acpi0 p4tcc3: <CPU Frequency Thermal Control> on cpu3 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> at device 0.3 on pci1 pci7: <ACPI PCI bus> on pcib4 mpt0: <LSILogic SAS/SATA Adapter> port 0x2000-0x20ff mem 0xdc210000-0xdc213fff,0xdc200000-0xdc20ffff irq 24 at device 1.0 on pci7 mpt0: [ITHREAD] mpt0: MPI Version=1.5.14.0 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 pcib5: <ACPI PCI-PCI bridge> at device 3.0 on pci0 pci8: <ACPI PCI bus> on pcib5 pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0 pci12: <ACPI PCI bus> on pcib6 pcib7: <PCI-PCI bridge> at device 5.0 on pci0 pci13: <PCI bus> on pcib7 pcib8: <ACPI PCI-PCI bridge> at device 6.0 on pci0 pci14: <ACPI PCI bus> on pcib8 pcib9: <PCI-PCI bridge> at device 7.0 on pci0 pci15: <PCI bus> on pcib9 pcib10: <ACPI PCI-PCI bridge> at device 28.0 on pci0 pci22: <ACPI PCI bus> on pcib10 bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xdc300000-0xdc30ffff irq 16 at device 0.0 on pci22 miibus0: <MII bus> on bge0 brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:1e:0b:5a:b2:e4 bge0: [ITHREAD] pcib11: <ACPI PCI-PCI bridge> at device 28.1 on pci0 pci23: <ACPI PCI bus> on pcib11 bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xdc400000-0xdc40ffff irq 17 at device 0.0 on pci23 miibus1: <MII bus> on bge1 brgphy1: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: Ethernet address: 00:1e:0b:5a:b2:e5 bge1: [ITHREAD] uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0 usb0: USB revision 1.0 uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 23 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1 usb1: USB revision 1.0 uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2 usb2: USB revision 1.0 uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2 uhub2: 2 ports with 2 removable, self powered ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xdc000000-0xdc0003ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: <Intel 63XXESB USB 2.0 controller> on ehci0 usb3: USB revision 2.0 uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3 uhub3: 6 ports with 6 removable, self powered pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci24: <ACPI PCI bus> on pcib12 vgapci0: <VGA-compatible display> mem 0xde000000-0xdeffffff,0xdc500000-0xdc503fff,0xdc800000-0xdcffffff irq 17 at device 2.0 on pci24 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] atapci1: <Intel 63XXESB2 SATA300 controller> port 0x1890-0x1897,0x1884-0x1887,0x1888-0x188f,0x1880-0x1883,0x1870-0x187f mem 0xdc000400-0xdc0007ff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] ata2: <ATA channel 0> on atapci1 ata2: [ITHREAD] ata3: <ATA channel 1> on atapci1 ata3: [ITHREAD] pci0: <serial bus, SMBus> at device 31.3 (no driver attached) acpi_button0: <Power Button> on acpi0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] cryptosoft0: <software crypto> on motherboard orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xdc000-0xdffff on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: cannot reserve I/O port range sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ukbd0: <ServerEngines SE USB Device, class 0/0, rev 1.10/0.01, addr 2> on uhub2 kbd2 at ukbd0 ums0: <ServerEngines SE USB Device, class 0/0, rev 1.10/0.01, addr 2> on uhub2 ums0: 8 buttons and Z dir. Timecounters tick every 1.000 msec Fast IPsec: Initialized Security Association Processing. hptrr: no controller detected. acd0: CDRW <DW-224E-V/C.CA> at ata0-master UDMA33 da0 at mpt0 bus 0 target 1 lun 0 da0: <ATA FB160C4081 HPF0> Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing Enabled da0: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) da1 at mpt0 bus 0 target 2 lun 0 da1: <ATA FB160C4081 HPF0> Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing Enabled da1: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! GEOM_MIRROR: Device mirror/gm0 launched (1/2). GEOM_MIRROR: Device gm0: rebuilding provider da0. Trying to mount root from ufs:/dev/mirror/gm0s1a WARNING: / was not properly dismounted --- dmesg.boot ends here --- --- OBOL begins here --- # # Kernel config for FreeBSD 7.0+ server # include GENERIC ident OBOL nooptions SCHED_4BSD options SCHED_ULE device crypto device cryptodev options IPSEC options IPSEC_DEBUG options ALTQ # # That's All Folks! # --- OBOL ends here --- >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200805061322.m46DM4hT001184>