Date: Fri, 17 Dec 2010 23:50:02 +0600 (NOVT) From: Eugene Grosbein <egrosbein@rdtc.ru> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/153255: 8.2-PRERELEASE repeatable kernel panic under heavy network load Message-ID: <201012171750.oBHHo2IE063461@k-45-pc-1.sd.rdtc.ru> Resent-Message-ID: <201012171810.oBHIAB6p028461@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 153255
>Category: kern
>Synopsis: 8.2-PRERELEASE repeatable kernel panic under heavy network load
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Fri Dec 17 18:10:11 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: Eugene Grosbein
>Release: FreeBSD 8.2-PRERELEASE amd64
>Organization:
RDTC JSC
>Environment:
System: FreeBSD k-45-pc-1.sd.rdtc.ru 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Mon Dec 13 14:49:37 NOVT 2010 root@k-45-pc-1.sd.rdtc.ru:/usr/local/obj/home/src/sys/PPPOE amd64
>Description:
I run four PPPoE network access servers based on
FreeBSD 8.2-PRERELEASE/amd64 and mpd-5.5 (/usr/ports/net/mpd5)
that extensively use NETGRAPH. Machines use very different hardware
but same software: they are NanoBSD-based and loaded with same disk image
and nearly same configuration (just distinct IP addresses).
All servers panic under heavy load when they have 900-1800 PPPoE clients
connected and more than 500Mbps going from uplink to clients
and more than 500Mbps going from clients to uplink same time.
All panics are pretty similar: that's GPF when mpd is active process.
I've got full crashdump for total 4GB of router memory.
Here comes gdb backtrace. Then dmesg output, kernel config file,
boot.conf and sysctl.conf.
Script started on Fri Dec 17 23:32:55 2010
kgdb kernel.debug /home/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 9: general protection fault while in kernel mode
cpuid = 2; apic id = 04
instruction pointer = 0x20:0xffffffff803df4e6
stack pointer = 0x28:0xffffff80eb5fc8d0
frame pointer = 0x28:0xffffff80eb5fc900
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1951 (mpd5)
trap number = 9
panic: general protection fault
cpuid = 2
Uptime: 1d1h36m42s
Dumping 4087 MB (3 chunks)
chunk 0: 1MB (155 pages) (CTRL-C to abort) ... ok
chunk 1: 3831MB (980592 pages) 3815 3799 3783 3767 3751 3735 3719 3703 3687 3671 3655 3639 3623 3607 3591 3575 3559 3543 3527 3511 3495 3479 3463 3447 3431 3415 3399 3383 3367 3351 3335 3319 3303 3287 3271 3255 3239 3223 3207 3191 3175 3159 3143 3127 3111 3095 3079 3063 3047 3031 3015 2999 2983 2967 2951 2935 2919 2903 2887 2871 2855 2839 2823 2807 2791 2775 2759 2743 2727 2711 2695 2679 2663 2647 2631 2615 2599 2583 2567 2551 2535 2519 2503 2487 2471 2455 2439 2423 2407 2391 2375 2359 2343 2327 2311 2295 2279 2263 2247 2231 2215 2199 2183 2167 2151 2135 2119 2103 2087 2071 2055 2039 2023 2007 1991 1975 1959 1943 1927 1911 1895 1879 1863 1847 1831 1815 1799 1783 1767 1751 1735 1719 1703 1687 1671 1655 1639 1623 1607 1591 1575 1559 1543 1527 1511 1495 1479 1463 1447 1431 1415 1399 1383 1367 1351 1335 1319 1303 1287 1271 1255 1239 1223 1207 1191 1175 1159 1143 1127 1111 1095 1079 1063 1047 1031 1015 999 983 967 951 935 919 903 887 871 855 839 823 807 791 775 759 743 727 711 !
695 679 663 647 631 615 599 583 567 551 535 519 503 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 ... ok
chunk 2: 256MB (65536 pages) 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1
Reading symbols from /boot/kernel/if_lagg.ko...done.
Loaded symbols for /boot/kernel/if_lagg.ko
Reading symbols from /boot/kernel/ipmi.ko...done.
Loaded symbols for /boot/kernel/ipmi.ko
#0 doadump () at pcpu.h:224
224 __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:224
#1 0xffffffff802feb0e in boot (howto=260) at /home/src/sys/kern/kern_shutdown.c:419
#2 0xffffffff802fef23 in panic (fmt=0x0) at /home/src/sys/kern/kern_shutdown.c:592
#3 0xffffffff8049ff4d in trap_fatal (frame=0xffffff0012f12000, eva=Variable "eva" is not available.
) at /home/src/sys/amd64/amd64/trap.c:783
#4 0xffffffff804a04ba in trap (frame=0xffffff80eb5fc820) at /home/src/sys/amd64/amd64/trap.c:592
#5 0xffffffff80488a24 in calltrap () at /home/src/sys/amd64/amd64/exception.S:224
#6 0xffffffff803df4e6 in in_pcbrele (inp=0xffffff0012c7b930) at /home/src/sys/netinet/in_pcb.c:1540
#7 0xffffffff803e0ba1 in in_pcbfree (inp=0xffffff0012c7b930) at /home/src/sys/netinet/in_pcb.c:1024
#8 0xffffffff8040dca7 in udp_detach (so=Variable "so" is not available.
) at /home/src/sys/netinet/udp_usrreq.c:1561
#9 0xffffffff80361624 in sofree (so=0xffffff00d2fecaa0) at /home/src/sys/kern/uipc_socket.c:616
#10 0xffffffff80362101 in soclose (so=0xffffff00d2fecaa0) at /home/src/sys/kern/uipc_socket.c:708
#11 0xffffffff802c69f3 in _fdrop (fp=0xffffff00d2868870, td=Variable "td" is not available.
) at file.h:293
#12 0xffffffff802c7c45 in closef (fp=0xffffff00d2868870, td=0xffffff00212ca460) at /home/src/sys/kern/kern_descrip.c:2117
#13 0xffffffff802c8443 in kern_close (td=0xffffff00212ca460, fd=Variable "fd" is not available.
) at /home/src/sys/kern/kern_descrip.c:1162
#14 0xffffffff8033cb85 in syscallenter (td=0xffffff00212ca460, sa=0xffffff80eb5fcba0) at /home/src/sys/kern/subr_trap.c:315
#15 0xffffffff804a038b in syscall (frame=0xffffff80eb5fcc40) at /home/src/sys/amd64/amd64/trap.c:888
#16 0xffffffff80488d02 in Xfast_syscall () at /home/src/sys/amd64/amd64/exception.S:377
#17 0x00000008018e377c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) frame 6
#6 0xffffffff803df4e6 in in_pcbrele (inp=0xffffff0012c7b930) at /home/src/sys/netinet/in_pcb.c:1540
1540 free(phd, M_PCB);
(kgdb) l
1535
1536 LIST_REMOVE(inp, inp_hash);
1537 LIST_REMOVE(inp, inp_portlist);
1538 if (LIST_FIRST(&phd->phd_pcblist) == NULL) {
1539 LIST_REMOVE(phd, phd_hash);
1540 free(phd, M_PCB);
1541 }
1542 inp->inp_flags &= ~INP_INHASHLIST;
1543 }
1544 LIST_REMOVE(inp, inp_list);
(kgdb) p M_PCB
$1 = {{ks_next = 0xffffffff80684d40, ks_magic = 877983977, ks_shortdesc = 0xffffffff8054212f "pcb", ks_handle = 0xffffff0001648000}}
(kgdb) p inp->inp_phd
$2 = (struct inpcbport *) 0xffffff0078c304c0
(kgdb) p *inp->inp_phd
$3 = {phd_hash = {le_next = 0x0, le_prev = 0x420f81ef00080800}, phd_pcblist = {lh_first = 0xffffffff8035b850}, phd_port = 36864}
(kgdb) quit
Script done on Fri Dec 17 23:36:05 2010
Now dmesg output:
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-PRERELEASE #1: Mon Dec 13 11:48:13 NOVT 2010
root@k-45-pc-1.sd.rdtc.ru:/usr/local/obj/nanobsd.pppoe/home/src/sys/PPPOE8 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz (2809.81-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106e5 Family = 6 Model = 1e Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x98e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 4294967296 (4096 MB)
avail memory = 4117843968 (3927 MB)
ACPI APIC Table: <070710 APIC1713>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
ioapic0: Changing APIC ID to 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
ichwd module loaded
acpi0: <070710 XSDT1713> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, eff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xD7, should be 0xD6 (20101013/tbutils-354)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 Network Connection version - 2.0.7> port 0xb880-0xb89f mem 0xf7ca0000-0xf7cbffff,0xf7000000-0xf73fffff,0xf7cd8000-0xf7cdbfff irq 16 at device 0.0 on pci1
igb0: Using MSIX interrupts with 5 vectors
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: Ethernet address: 00:1b:21:72:6c:10
igb1: <Intel(R) PRO/1000 Network Connection version - 2.0.7> port 0xbc00-0xbc1f mem 0xf7ce0000-0xf7cfffff,0xf7800000-0xf7bfffff,0xf7cdc000-0xf7cdffff irq 17 at device 0.1 on pci1
igb1: Using MSIX interrupts with 5 vectors
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: Ethernet address: 00:1b:21:72:6c:11
pci0: <base peripheral> at device 8.0 (no driver attached)
pci0: <base peripheral> at device 8.1 (no driver attached)
pci0: <base peripheral> at device 8.2 (no driver attached)
pci0: <base peripheral> at device 8.3 (no driver attached)
pci0: <base peripheral> at device 16.0 (no driver attached)
pci0: <base peripheral> at device 16.1 (no driver attached)
ehci0: <Intel PCH USB 2.0 controller USB-B> mem 0xf6bfe000-0xf6bfe3ff irq 16 at device 26.0 on pci0
ehci0: [ITHREAD]
usbus0: EHCI version 1.0
usbus0: <Intel PCH USB 2.0 controller USB-B> on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci5: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection 7.1.8> port 0xec00-0xec1f mem 0xf7fe0000-0xf7ffffff,0xf7f00000-0xf7f7ffff,0xf7fdc000-0xf7fdffff irq 19 at device 0.0 on pci4
em0: Using MSIX interrupts with 3 vectors
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: Ethernet address: 00:1b:21:68:56:51
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib4
atapci0: <VIA ATA controller> port 0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f irq 16 at device 0.0 on pci3
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci2: <ACPI PCI bus> on pcib5
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xc800-0xc8ff mem 0xf5fff000-0xf5ffffff,0xf5ff8000-0xf5ffbfff irq 17 at device 0.0 on pci2
re0: Using 1 MSI messages
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 48:5b:39:f7:5a:03
re0: [FILTER]
ehci1: <Intel PCH USB 2.0 controller USB-A> mem 0xf6bfd000-0xf6bfd3ff irq 23 at device 29.0 on pci0
ehci1: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <Intel PCH USB 2.0 controller USB-A> on ehci1
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel 5 Series/3400 Series PCH SATA300 controller> port 0xa880-0xa887,0xa800-0xa803,0xa480-0xa487,0xa400-0xa403,0xa080-0xa09f mem 0xf6bfb000-0xf6bfb7ff irq 21 at device 31.2 on pci0
atapci1: [ITHREAD]
atapci1: AHCI called from vendor specific driver
atapci1: AHCI v1.30 controller with 6 3Gbps ports, PM supported
ata4: <ATA channel 0> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci1
ata5: [ITHREAD]
ata6: <ATA channel 2> on atapci1
ata6: [ITHREAD]
ata7: <ATA channel 3> on atapci1
ata7: [ITHREAD]
ata8: <ATA channel 4> on atapci1
ata8: [ITHREAD]
ata9: <ATA channel 5> on atapci1
ata9: [ITHREAD]
ichsmb0: <Intel PCH SMBus controller> port 0x400-0x41f mem 0xf6bfc000-0xf6bfc0ff irq 18 at device 31.3 on pci0
ichsmb0: [ITHREAD]
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart0: console (115200,n,8,1)
ichwd0: <Intel H55 watchdog timer> on isa0
ichwd0: Intel H55 watchdog timer (ICH10 or equivalent)
orm0: <ISA Option ROM> at iomem 0xc8000-0xc8fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
ipfw2 initialized, divert loadable, nat loadable, rule-based forwarding enabled, default to deny, logging disabled
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ad0: 61057MB <KINGSTON SNV425S264GB C091126a> at ata4-master UDMA100 SATA 3Gb/s
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
ugen0.1: <Intel> at usbus0ugen1.1: <Intel> at usbus1
uhub0:
<Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <Intel> at usbus0
uhub2: <Intel Integrated Rate Matching Hub, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <Intel> at usbus1
uhub3: <Intel Integrated Rate Matching Hub, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
Trying to mount root from ufs:/dev/ad0s1a
Setting hostuuid: 142d9a80-791c-11df-86b7-485b39f75a03.
Setting hostid: 0x950b3200.
Entropy harvesting:
interrupts
ethernet
point_to_point
kickstart
.
Starting file system checks:
/dev/ad0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s1a: clean, 702718 free (1166 frags, 87694 blocks, 0.1% fragmentation)
/dev/ad0s3: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s3: clean, 100806 free (30 frags, 12597 blocks, 0.0% fragmentation)
Mounting local file systems:
.
Setting hostname: m-19-pc-2.sd.rdtc.ru
.
Starting Network: lo0 igb0 igb1 em0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet 127.0.0.1 netmask 0xff000000
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:1b:21:72:6c:10
media: Ethernet autoselect
status: no carrier
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:1b:21:72:6c:11
media: Ethernet autoselect
status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:1b:21:68:56:51
inet 62.231.191.246 netmask 0xfffffffc broadcast 62.231.191.247
media: Ethernet autoselect
status: no carrier
add net default: gateway 62.231.191.245
Additional routing options:
IP gateway=YES
.
Starting zebra.
Starting ospfd.
Dec 17 22:02:36 ospfd[602]: ASBR[Status:1]: Update
/etc/rc: WARNING: run_rc_command: cannot run /sbin/devd
Generating host.conf.
Firewall rules loaded.
Creating and/or trimming log files
.
Starting syslogd.
savecore: reboot after panic: general protection fault
savecore: no dump, not enough free space on device (450468 available, need 4185138)
savecore: unsaved dumps found but not saved
No crash dumps in /var/crash.
ELF ldconfig path: /lib /usr/lib /usr/local/lib
32-bit compatibility ldconfig path:
Clearing /tmp (X related).
Starting watchdogd.
Starting local daemons:
.
Updating motd:
.
Starting ntpd.
Starting sshd.
igb0: link state changed to UP
vlan990: link state changed to UP
vlan952: link state changed to UP
vlan922: link state changed to UP
vlan884: link state changed to UP
vlan854: link state changed to UP
vlan816: link state changed to UP
vlan786: link state changed to UP
vlan718: link state changed to UP
vlan680: link state changed to UP
vlan650: link state changed to UP
vlan544: link state changed to UP
vlan476: link state changed to UP
vlan34: link state changed to UP
vlan68: link state changed to UP
vlan170: link state changed to UP
vlan204: link state changed to UP
vlan272: link state changed to UP
vlan340: link state changed to UP
vlan991: link state changed to UP
vlan953: link state changed to UP
vlan885: link state changed to UP
vlan855: link state changed to UP
vlan817: link state changed to UP
vlan787: link state changed to UP
vlan719: link state changed to UP
vlan681: link state changed to UP
vlan651: link state changed to UP
vlan583: link state changed to UP
vlan545: link state changed to UP
vlan515: link state changed to UP
vlan477: link state changed to UP
vlan103: link state changed to UP
vlan239: link state changed to UP
vlan341: link state changed to UP
vlan988: link state changed to UP
vlan954: link state changed to UP
vlan886: link state changed to UP
vlan852: link state changed to UP
vlan818: link state changed to UP
vlan784: link state changed to UP
vlan750: link state changed to UP
vlan682: link state changed to UP
vlan648: link state changed to UP
vlan546: link state changed to UP
vlan478: link state changed to UP
vlan372: link state changed to UP
vlan70: link state changed to UP
vlan236: link state changed to UP
vlan342: link state changed to UP
vlan989: link state changed to UP
vlan955: link state changed to UP
vlan887: link state changed to UP
vlan853: link state changed to UP
vlan819: link state changed to UP
vlan785: link state changed to UP
vlan683: link state changed to UP
vlan649: link state changed to UP
vlan479: link state changed to UP
vlan441: link state changed to UP
vlan411: link state changed to UP
vlan373: link state changed to UP
vlan986: link state changed to UP
vlan956: link state changed to UP
vlan926: link state changed to UP
vlan880: link state changed to UP
vlan850: link state changed to UP
vlan820: link state changed to UP
vlan790: link state changed to UP
vlan714: link state changed to UP
vlan684: link state changed to UP
vlan654: link state changed to UP
vlan608: link state changed to UP
vlan472: link state changed to UP
vlan412: link state changed to UP
vlan38: link state changed to UP
vlan336: link state changed to UP
vlan370: link state changed to UP
vlan987: link state changed to UP
vlan957: link state changed to UP
vlan881: link state changed to UP
vlan851: link state changed to UP
vlan821: link state changed to UP
vlan791: link state changed to UP
vlan685: link state changed to UP
vlan655: link state changed to UP
vlan519: link state changed to UP
vlan473: link state changed to UP
vlan447: link state changed to UP
vlan337: link state changed to UP
vlan371: link state changed to UP
vlan984: link state changed to UP
vlan958: link state changed to UP
vlan924: link state changed to UP
vlan882: link state changed to UP
vlan848: link state changed to UP
vlan788: link state changed to UP
vlan712: link state changed to UP
vlan686: link state changed to UP
vlan652: link state changed to UP
vlan516: link state changed to UP
vlan504: link state changed to UP
vlan474: link state changed to UP
vlan444: link state changed to UP
vlan278: link state changed to UP
vlan338: link state changed to UP
vlan368: link state changed to UP
vlan985: link state changed to UP
vlan959: link state changed to UP
vlan925: link state changed to UP
vlan883: link state changed to UP
vlan849: link state changed to UP
vlan823: link state changed to UP
vlan789: link state changed to UP
vlan687: link state changed to UP
vlan653: link state changed to UP
vlan475: link state changed to UP
vlan37: link state changed to UP
vlan339: link state changed to UP
vlan369: link state changed to UP
vlan982: link state changed to UP
vlan944: link state changed to UP
vlan892: link state changed to UP
vlan862: link state changed to UP
vlan824: link state changed to UP
vlan794: link state changed to UP
vlan672: link state changed to UP
vlan522: link state changed to UP
vlan502: link state changed to UP
vlan468: link state changed to UP
vlan382: link state changed to UP
vlan42: link state changed to UP
vlan76: link state changed to UP
vlan162: link state changed to UP
vlan196: link state changed to UP
vlan983: link state changed to UP
vlan945: link state changed to UP
vlan915: link state changed to UP
vlan893: link state changed to UP
vlan863: link state changed to UP
vlan825: link state changed to UP
vlan795: link state changed to UP
vlan741: link state changed to UP
vlan673: link state changed to UP
vlan643: link state changed to UP
vlan503: link state changed to UP
vlan469: link state changed to UP
vlan383: link state changed to UP
vlan9: link state changed to UP
vlan43: link state changed to UP
vlan197: link state changed to UP
vlan231: link state changed to UP
vlan315: link state changed to UP
vlan349: link state changed to UP
vlan980: link state changed to UP
vlan946: link state changed to UP
vlan894: link state changed to UP
vlan860: link state changed to UP
vlan826: link state changed to UP
vlan792: link state changed to UP
vlan742: link state changed to UP
vlan674: link state changed to UP
vlan520: link state changed to UP
vlan470: link state changed to UP
vlan380: link state changed to UP
vlan40: link state changed to UP
vlan78: link state changed to UP
vlan312: link state changed to UP
vlan350: link state changed to UP
vlan981: link state changed to UP
vlan947: link state changed to UP
vlan895: link state changed to UP
vlan861: link state changed to UP
vlan827: link state changed to UP
vlan793: link state changed to UP
vlan521: link state changed to UP
vlan471: link state changed to UP
vlan403: link state changed to UP
vlan381: link state changed to UP
vlan161: link state changed to UP
vlan351: link state changed to UP
vlan978: link state changed to UP
vlan948: link state changed to UP
vlan888: link state changed to UP
vlan858: link state changed to UP
vlan828: link state changed to UP
vlan798: link state changed to UP
vlan646: link state changed to UP
vlan556: link state changed to UP
vlan526: link state changed to UP
vlan498: link state changed to UP
vlan464: link state changed to UP
vlan438: link state changed to UP
vlan404: link state changed to UP
vlan378: link state changed to UP
vlan72: link state changed to UP
vlan192: link state changed to UP
vlan318: link state changed to UP
vlan979: link state changed to UP
vlan949: link state changed to UP
vlan889: link state changed to UP
vlan859: link state changed to UP
vlan829: link state changed to UP
vlan799: link state changed to UP
vlan677: link state changed to UP
vlan647: link state changed to UP
vlan527: link state changed to UP
vlan465: link state changed to UP
vlan439: link state changed to UP
vlan107: link state changed to UP
vlan193: link state changed to UP
vlan319: link state changed to UP
vlan976: link state changed to UP
vlan950: link state changed to UP
vlan916: link state changed to UP
vlan890: link state changed to UP
vlan856: link state changed to UP
vlan830: link state changed to UP
vlan796: link state changed to UP
vlan678: link state changed to UP
vlan644: link state changed to UP
vlan496: link state changed to UP
vlan376: link state changed to UP
vlan74: link state changed to UP
vlan104: link state changed to UP
vlan194: link state changed to UP
vlan316: link state changed to UP
vlan977: link state changed to UP
vlan951: link state changed to UP
vlan917: link state changed to UP
vlan891: link state changed to UP
vlan857: link state changed to UP
vlan831: link state changed to UP
vlan797: link state changed to UP
vlan679: link state changed to UP
vlan645: link state changed to UP
vlan497: link state changed to UP
vlan437: link state changed to UP
vlan45: link state changed to UP
vlan195: link state changed to UP
vlan287: link state changed to UP
vlan317: link state changed to UP
vlan975: link state changed to UP
vlan869: link state changed to UP
vlan839: link state changed to UP
vlan801: link state changed to UP
vlan765: link state changed to UP
vlan697: link state changed to UP
vlan667: link state changed to UP
vlan599: link state changed to UP
vlan531: link state changed to UP
vlan495: link state changed to UP
vlan85: link state changed to UP
vlan153: link state changed to UP
vlan187: link state changed to UP
vlan255: link state changed to UP
vlan257: link state changed to UP
vlan325: link state changed to UP
vlan359: link state changed to UP
vlan974: link state changed to UP
vlan868: link state changed to UP
vlan838: link state changed to UP
vlan764: link state changed to UP
vlan696: link state changed to UP
vlan666: link state changed to UP
vlan530: link state changed to UP
vlan494: link state changed to UP
vlan84: link state changed to UP
vlan186: link state changed to UP
vlan254: link state changed to UP
vlan290: link state changed to UP
vlan324: link state changed to UP
vlan358: link state changed to UP
vlan973: link state changed to UP
vlan871: link state changed to UP
vlan837: link state changed to UP
vlan803: link state changed to UP
vlan767: link state changed to UP
vlan699: link state changed to UP
vlan529: link state changed to UP
vlan493: link state changed to UP
vlan463: link state changed to UP
vlan19: link state changed to UP
vlan49: link state changed to UP
vlan185: link state changed to UP
vlan289: link state changed to UP
vlan327: link state changed to UP
vlan357: link state changed to UP
vlan972: link state changed to UP
vlan904: link state changed to UP
vlan870: link state changed to UP
vlan836: link state changed to UP
vlan802: link state changed to UP
vlan698: link state changed to UP
vlan664: link state changed to UP
vlan528: link state changed to UP
vlan492: link state changed to UP
vlan462: link state changed to UP
vlan394: link state changed to UP
vlan48: link state changed to UP
vlan184: link state changed to UP
vlan222: link state changed to UP
vlan326: link state changed to UP
vlan356: link state changed to UP
vlan971: link state changed to UP
vlan865: link state changed to UP
vlan835: link state changed to UP
vlan805: link state changed to UP
vlan775: link state changed to UP
vlan761: link state changed to UP
vlan701: link state changed to UP
vlan491: link state changed to UP
vlan457: link state changed to UP
vlan397: link state changed to UP
vlan21: link state changed to UP
vlan191: link state changed to UP
vlan251: link state changed to UP
vlan261: link state changed to UP
vlan321: link state changed to UP
vlan355: link state changed to UP
vlan1000: link state changed to UP
vlan970: link state changed to UP
vlan864: link state changed to UP
vlan834: link state changed to UP
vlan804: link state changed to UP
vlan774: link state changed to UP
vlan760: link state changed to UP
vlan670: link state changed to UP
vlan534: link state changed to UP
vlan490: link state changed to UP
vlan456: link state changed to UP
vlan80: link state changed to UP
vlan114: link state changed to UP
vlan190: link state changed to UP
vlan216: link state changed to UP
vlan260: link state changed to UP
vlan294: link state changed to UP
vlan320: link state changed to UP
vlan354: link state changed to UP
vlan969: link state changed to UP
vlan943: link state changed to UP
vlan867: link state changed to UP
vlan833: link state changed to UP
vlan807: link state changed to UP
vlan773: link state changed to UP
vlan763: link state changed to UP
vlan669: link state changed to UP
vlan533: link state changed to UP
vlan489: link state changed to UP
vlan459: link state changed to UP
vlan399: link state changed to UP
vlan189: link state changed to UP
vlan323: link state changed to UP
vlan353: link state changed to UP
vlan968: link state changed to UP
vlan866: link state changed to UP
vlan832: link state changed to UP
vlan806: link state changed to UP
vlan762: link state changed to UP
vlan668: link state changed to UP
vlan488: link state changed to UP
vlan458: link state changed to UP
vlan398: link state changed to UP
vlan52: link state changed to UP
vlan188: link state changed to UP
vlan322: link state changed to UP
vlan352: link state changed to UP
vlan967: link state changed to UP
vlan899: link state changed to UP
vlan877: link state changed to UP
vlan847: link state changed to UP
vlan809: link state changed to UP
vlan779: link state changed to UP
vlan757: link state changed to UP
vlan689: link state changed to UP
vlan659: link state changed to UP
vlan569: link state changed to UP
vlan539: link state changed to UP
vlan487: link state changed to UP
vlan385: link state changed to UP
vlan265: link state changed to UP
vlan876: link state changed to UP
vlan846: link state changed to UP
vlan808: link state changed to UP
vlan756: link state changed to UP
vlan688: link state changed to UP
vlan658: link state changed to UP
vlan486: link state changed to UP
vlan384: link state changed to UP
vlan58: link state changed to UP
vlan264: link state changed to UP
vlan332: link state changed to UP
vlan965: link state changed to UP
vlan879: link state changed to UP
vlan845: link state changed to UP
vlan811: link state changed to UP
vlan759: link state changed to UP
vlan691: link state changed to UP
vlan657: link state changed to UP
vlan485: link state changed to UP
vlan455: link state changed to UP
vlan417: link state changed to UP
vlan387: link state changed to UP
vlan245: link state changed to UP
vlan267: link state changed to UP
vlan335: link state changed to UP
vlan964: link state changed to UP
vlan930: link state changed to UP
vlan878: link state changed to UP
vlan844: link state changed to UP
vlan810: link state changed to UP
vlan776: link state changed to UP
vlan758: link state changed to UP
vlan724: link state changed to UP
vlan690: link state changed to UP
vlan656: link state changed to UP
vlan484: link state changed to UP
vlan454: link state changed to UP
vlan386: link state changed to UP
vlan94: link state changed to UP
vlan214: link state changed to UP
vlan266: link state changed to UP
vlan334: link state changed to UP
vlan364: link state changed to UP
vlan963: link state changed to UP
vlan873: link state changed to UP
vlan843: link state changed to UP
vlan813: link state changed to UP
vlan783: link state changed to UP
vlan693: link state changed to UP
vlan663: link state changed to UP
vlan543: link state changed to UP
vlan483: link state changed to UP
vlan449: link state changed to UP
vlan389: link state changed to UP
vlan63: link state changed to UP
vlan183: link state changed to UP
vlan269: link state changed to UP
vlan329: link state changed to UP
vlan363: link state changed to UP
vlan962: link state changed to UP
vlan932: link state changed to UP
vlan902: link state changed to UP
vlan842: link state changed to UP
vlan812: link state changed to UP
vlan782: link state changed to UP
vlan752: link state changed to UP
vlan692: link state changed to UP
vlan662: link state changed to UP
vlan542: link state changed to UP
vlan482: link state changed to UP
vlan388: link state changed to UP
vlan208: link state changed to UP
vlan268: link state changed to UP
vlan328: link state changed to UP
vlan362: link state changed to UP
vlan961: link state changed to UP
vlan875: link state changed to UP
vlan841: link state changed to UP
vlan815: link state changed to UP
vlan755: link state changed to UP
vlan695: link state changed to UP
vlan661: link state changed to UP
vlan541: link state changed to UP
vlan481: link state changed to UP
vlan451: link state changed to UP
vlan391: link state changed to UP
vlan211: link state changed to UP
vlan241: link state changed to UP
vlan271: link state changed to UP
vlan331: link state changed to UP
vlan361: link state changed to UP
vlan960: link state changed to UP
vlan874: link state changed to UP
vlan840: link state changed to UP
vlan814: link state changed to UP
vlan694: link state changed to UP
vlan660: link state changed to UP
vlan540: link state changed to UP
vlan480: link state changed to UP
vlan450: link state changed to UP
vlan420: link state changed to UP
vlan390: link state changed to UP
vlan90: link state changed to UP
vlan210: link state changed to UP
vlan240: link state changed to UP
vlan330: link state changed to UP
vlan360: link state changed to UP
em0: link state changed to UP
Starting cron.
Starting bsnmpd.
Starting background file system checks in 60 seconds.
Fri Dec 17 22:02:42 NOVT 2010
The kernel is mostly monolithic; if_lagg.ko and ipmi.ko are loaded
but unused - this particular box has no lagg devices configured and no IPMI hardware.
Here comes kernel config file:
cpu HAMMER
ident PPPOE
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for devices.
# Use the following to compile in values accessible to the kernel
# through getenv() (or kenv(1) in userland). The format of the file
# is 'variable=value', see kenv(1)
#
# env "GENERIC.env"
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
#options INET6 # IPv6 communications protocols
#options SCTP # Stream Control Transmission Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options UFS_GJOURNAL # Enable gjournal-based UFS journaling
options MD_ROOT # MD is a potential root device
#options NFSCLIENT # Network Filesystem Client
#options NFSSERVER # Network Filesystem Server
#options NFSLOCKD # Network Lock Manager
options NFS_ROOT # NFS usable as /, requires NFSCLIENT
#options MSDOSFS # MSDOS Filesystem
#options CD9660 # ISO 9660 Filesystem
#options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options GEOM_JOURNAL
options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty)
options COMPAT_FREEBSD32 # Compatible with i386 binaries
#options COMPAT_FREEBSD4 # Compatible with FreeBSD4
#options COMPAT_FREEBSD5 # Compatible with FreeBSD5
#options COMPAT_FREEBSD6 # Compatible with FreeBSD6
#options COMPAT_FREEBSD7 # Compatible with FreeBSD7
#options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options P1003_1B_SEMAPHORES # POSIX-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed.
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
options AUDIT # Security event auditing
options MAC # TrustedBSD MAC Framework
#options FLOWTABLE # per-cpu routing cache
#options KDTRACE_FRAME # Ensure frames are compiled in
#options KDTRACE_HOOKS # Kernel DTrace hooks
options INCLUDE_CONFIG_FILE # Include this file in kernel
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
# CPU frequency control
device cpufreq
# Bus support.
device acpi
device pci
# Floppy drives
#device fdc
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
#device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
#device atapifd # ATAPI floppy drives
#device atapist # ATAPI tape drives
#options ATA_STATIC_ID # Static device numbering
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
#device ch # SCSI media changers
device da # Direct Access (disks)
#device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
#device ses # SCSI Environmental Services (and SAF-TE)
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
#device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
#device agp # support several AGP chipsets
# Serial (COM) ports
device uart # Generic UART driver
# PCI Ethernet NICs.
device em # Intel PRO/1000 Gigabit Ethernet Family
device igb
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
device re
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device vlan # 802.1Q VLAN support
#device tun # Packet tunnel.
device pty # BSD-style compatibility pseudo ttys
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
#device faith # IPv6-to-IPv4 relaying (translation)
device firmware # firmware assist module
device snp
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
#options USB_DEBUG # enable debug msgs
options USB_VERBOSE
device uhci # UHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
device ukbd # Keyboard
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
device ucom
# USB support for Technologies ARK3116 based serial adapters
device uark
# USB support for Belkin F5U103 and compatible serial adapters
device ubsa
# USB support for serial adapters based on the FT8U100AX and FT8U232AM
device uftdi
# USB support for some Windows CE based serial communication.
device uipaq
# USB support for Prolific PL-2303 serial adapters
device uplcom
# USB support for Silicon Laboratories CP2101/CP2102 based USB serial adapters
device uslcom
#options IPSEC
#device crypto
options NETGRAPH
options NETGRAPH_ETHER
options NETGRAPH_IFACE
options NETGRAPH_MPPC_ENCRYPTION
options NETGRAPH_PPP
options NETGRAPH_PPPOE
options NETGRAPH_SOCKET
options NETGRAPH_TCPMSS
options NETGRAPH_TEE
options NETGRAPH_VJC
options IPFIREWALL
options IPFIREWALL_FORWARD
#options IPDIVERT
options DUMMYNET
options VFS_AIO
device smbus
device smb
device ichsmb
device iicbus
device iicbb
device ic
device iic
device iicsmb
device coretemp
device ichwd
device nvram
#options DEVICE_POLLING
#options KDB
#options KDB_TRACE
#options KDB_UNATTENDED
#options DDB
#options DDB_NUMSYM
Now /boot/loader.conf:
console="vidconsole comconsole"
comconsole_speed=115200
aiboost_load="YES"
beastie_disable="YES"
autoboot_delay="2"
hw.em.rxd=4096
hw.em.txd=4096
hw.igb.rxd=4096
hw.igb.txd=4096
vm.kmem_size=3G
machdep.hyperthreading_allowed=0
if_lagg_load="YES"
ipmi_load="YES"
# netgraph queue sizes tuning, see vmstat -z|egrep 'ITEM|NetGraph'
net.graph.maxdata=65536
net.graph.maxalloc=65536
net.isr.maxthreads=4
# default value
#net.isr.maxqlimit=10240
# for rtsock
net.route.netisr_maxqlen=4096
# for other protocols (IP & PPPoE?)
net.isr.defaultqlimit=4096
Now /etc/sysctl.conf:
dev.em.0.rx_int_delay=200
dev.em.0.tx_int_delay=200
dev.em.0.rx_abs_int_delay=4000
dev.em.0.tx_abs_int_delay=4000
dev.em.0.rx_processing_limit=4096
net.inet.ip.intr_queue_maxlen=4096
net.inet.ip.fastforwarding=1
net.inet.ip.dummynet.pipe_slot_limit=1000
net.inet.ip.dummynet.io_fast=1
kern.ipc.nmbclusters=100000
kern.ipc.nmbjumbop=100000
kern.ipc.maxsockbuf=83886080
net.graph.maxdgram=8388608
net.graph.recvspace=8388608
net.isr.direct=0
net.isr.direct_force=0
net.inet.ip.redirect=0
net.inet.ip.ttl=128
net.inet.ip.random_id=1
net.inet.tcp.blackhole=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.path_mtu_discovery=0
net.inet.icmp.drop_redirect=1
net.inet.tcp.ecn.enable=1
net.inet.ip.ttl=130
debug.minidump=0
>How-To-Repeat:
It takes one-three days for panic to occur
and seem to depend from load: more load, less time to panic.
>Fix:
Unknown.
Eugene Grosbein
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201012171750.oBHHo2IE063461>