Date: Wed, 5 Dec 2007 01:32:36 +0700 (KRAT) From: Eugene Grosbein <eugen@kuzbass.ru> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/118432: [ng_nat] kernel libalias: repeatable panic (double fault) Message-ID: <200712041832.lB4IWaEv069092@grosbein.pp.ru> Resent-Message-ID: <200712041840.lB4Ie2DP066055@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 118432
>Category: kern
>Synopsis: [ng_nat] kernel libalias: repeatable panic (double fault)
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Dec 04 18:40:01 UTC 2007
>Closed-Date:
>Last-Modified:
>Originator: Eugene Grosbein
>Release: FreeBSD 6.3-PRERELEASE i386
>Organization:
Svyaz-Service JSC
>Environment:
System: FreeBSD gw.grosbein.pp.ru 6.3-PRERELEASE FreeBSD 6.3-PRERELEASE #2: Tue Dec 4 14:02:57 UTC 2007
>Description:
My home router panices instantly if I run BitchX IRC client
at the desktop which traffic flows through the panicing router.
And I've got nice crashdump. Note that is does not panics
when there is no BitchX running but lots of other traffic:
SMTP/HTTP/SSH/CVSup etc.
Here is kgdb's output:
Script started on Wed Dec 5 01:13:38 2007
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal double fault:
eip = 0xc0557f11
esp = 0xc4e2e974
ebp = 0xc4e3e9a4
panic: double fault
KDB: stack backtrace:
kdb_backtrace(100,c1091300,0,0,0,...) at 0xc04c5949 = kdb_backtrace+0x29
panic(c064dba7,c064df97,c4e3e9a4,0,0,...) at 0xc04ac2e4 = panic+0xa4
dblfault_handler() at 0xc0605702 = dblfault_handler+0x52
--- trap 0x17, eip = 0xc0557f11, esp = 0xc4e2e974, ebp = 0xc4e3e9a4 ---
AliasHandleIrcOut(c12f8000,c1145800,c15dc800,800) at 0xc0557f11 = AliasHandleIrcOut+0x21
TcpAliasOut(c12f8000,c1145800,800,1) at 0xc0554997 = TcpAliasOut+0x327
LibAliasOutTry(c12f8000,c1145800,800,1,c4e3ea50,...) at 0xc0555115 = LibAliasOutTry+0x155
LibAliasOut(c12f8000,c1145800,800) at 0xc0554fb3 = LibAliasOut+0x13
ng_nat_rcvdata(c1217480,c12ef390,0,c12bc600,c12bc654,...) at 0xc0528e0b = ng_nat_rcvdata+0xeb
ng_apply_item(c12bc600,c12ef390,1,c12ef390,c4e3ebe8,...) at 0xc0526bc5 = ng_apply_item+0x95
ng_snd_item(c12ef390,0) at 0xc0526a64 = ng_snd_item+0x484
ng_ipfw_input(c4e3ebe8,0,c4e3eae0,0,c119c400,...) at 0xc0528a6c = ng_ipfw_input+0x12c
ipfw_check_out(0,c4e3ebe8,c11b6800,2,0) at 0xc053e5f3 = ipfw_check_out+0x2a3
pfil_run_hooks(c068ef80,c4e3ec54,c11b6800,2,0) at 0xc0518b2f = pfil_run_hooks+0xcf
ip_fastforward(c119c400) at 0xc0537c11 = ip_fastforward+0x411
ether_demux(c10fb000,c119c400,c10f70b4,c4e3ecb0,c0453758,...) at 0xc05165bf = ether_demux+0x26f
ether_input(c10fb000,c119c400,c10f7018,0,c0625c86,...) at 0xc0516339 = ether_input+0x219
fxp_intr_body(c10f7000,c10fb000,40,ffffffff) at 0xc0453758 = fxp_intr_body+0x1a8
fxp_intr(c10f7000) at 0xc0453494 = fxp_intr+0x94
ithread_execute_handlers(c1096218,c1083800) at 0xc0498c31 = ithread_execute_handlers+0xe1
ithread_loop(c10f38b0,c4e3ed38,c10f38b0,c0498d10,0,...) at 0xc0498d7e = ithread_loop+0x6e
fork_exit(c0498d10,c10f38b0,c4e3ed38) at 0xc0497e28 = fork_exit+0xa8
fork_trampoline() at 0xc05f67fc = fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xc4e3ed6c, ebp = 0 ---
Uptime: 14m4s
Dumping 47 MB (2 chunks)
chunk 0: 1MB (160 pages) ... ok
chunk 1: 47MB (12032 pages) 32 16
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc04ac076 in boot (howto=260) at /usr/local/smallworld/usr/src/sys/kern/kern_shutdown.c:409
#2 0xc04ac34b in panic (fmt=0xc064dba7 "double fault") at /usr/local/smallworld/usr/src/sys/kern/kern_shutdown.c:565
#3 0xc0605702 in dblfault_handler () at /usr/local/smallworld/usr/src/sys/i386/i386/trap.c:867
#4 0xc0557f11 in AliasHandleIrcOut (la=0xc12f8000, pip=0xc1145800, lnk=0xc15dc800, maxsize=2048) at alias_local.h:353
#5 0xc0554997 in TcpAliasOut (la=0xc12f8000, pip=0xc1145800, maxpacketsize=2048, create=1)
at /usr/local/smallworld/usr/src/sys/netinet/libalias/alias.c:999
#6 0xc0555115 in LibAliasOutTry (la=0xc12f8000, ptr=0xc1145800 "E", maxpacketsize=2048, create=1)
at /usr/local/smallworld/usr/src/sys/netinet/libalias/alias.c:1322
#7 0xc0554fb3 in LibAliasOut (la=0xc12f8000, ptr=0xc1145800 "E", maxpacketsize=2048)
at /usr/local/smallworld/usr/src/sys/netinet/libalias/alias.c:1263
#8 0xc0528e0b in ng_nat_rcvdata (hook=0xc1217480, item=0xc12ef390)
at /usr/local/smallworld/usr/src/sys/netgraph/ng_nat.c:295
#9 0xc0526bc5 in ng_apply_item (node=0xc12bc600, item=0xc12ef390, rw=1)
at /usr/local/smallworld/usr/src/sys/netgraph/ng_base.c:2395
#10 0xc0526a64 in ng_snd_item (item=0xc12ef390, flags=0) at /usr/local/smallworld/usr/src/sys/netgraph/ng_base.c:2323
#11 0xc0528a6c in ng_ipfw_input (m0=0xc4e3ebe8, dir=-1055631340, fwa=0xc4e3eae0, tee=-1053887600)
at /usr/local/smallworld/usr/src/sys/netgraph/ng_ipfw.c:310
#12 0xc053e5f3 in ipfw_check_out (arg=0x0, m0=0xc4e3ebe8, ifp=0xc11b6800, dir=2, inp=0x0)
at /usr/local/smallworld/usr/src/sys/netinet/ip_fw_pfil.c:317
#13 0xc0518b2f in pfil_run_hooks (ph=0xc068ef80, mp=0xc4e3ec54, ifp=0xc11b6800, dir=2, inp=0x0)
at /usr/local/smallworld/usr/src/sys/net/pfil.c:139
#14 0xc0537c11 in ip_fastforward (m=0xc119c400) at /usr/local/smallworld/usr/src/sys/netinet/ip_fastfwd.c:437
#15 0xc05165bf in ether_demux (ifp=0xc10fb000, m=0xc119c400) at /usr/local/smallworld/usr/src/sys/net/if_ethersubr.c:769
#16 0xc0516339 in ether_input (ifp=0xc10fb000, m=0xc119c400) at /usr/local/smallworld/usr/src/sys/net/if_ethersubr.c:623
#17 0xc0453758 in fxp_intr_body (sc=0xc10f7000, ifp=0xc10fb000, statack=180 '´', count=-1)
at /usr/local/smallworld/usr/src/sys/dev/fxp/if_fxp.c:1715
#18 0xc0453494 in fxp_intr (xsc=0xc10f7000) at /usr/local/smallworld/usr/src/sys/dev/fxp/if_fxp.c:1536
#19 0xc0498c31 in ithread_execute_handlers (p=0xc1096218, ie=0xc1083800)
at /usr/local/smallworld/usr/src/sys/kern/kern_intr.c:682
#20 0xc0498d7e in ithread_loop (arg=0xc10f38b0) at /usr/local/smallworld/usr/src/sys/kern/kern_intr.c:766
#21 0xc0497e28 in fork_exit (callout=0xc0498d10 <ithread_loop>, arg=0xc10f38b0, frame=0xc4e3ed38)
at /usr/local/smallworld/usr/src/sys/kern/kern_fork.c:788
#22 0xc05f67fc in fork_trampoline () at /usr/local/smallworld/usr/src/sys/i386/i386/exception.s:208
(kgdb) bt full
#0 doadump () at pcpu.h:165
No locals.
#1 0xc04ac076 in boot (howto=260) at /usr/local/smallworld/usr/src/sys/kern/kern_shutdown.c:409
first_buf_printf = 1
#2 0xc04ac34b in panic (fmt=0xc064dba7 "double fault") at /usr/local/smallworld/usr/src/sys/kern/kern_shutdown.c:565
td = (struct thread *) 0xc1091300
bootopt = 260
newpanic = 1
ap = 0xc06ac1b0 "\227ßdÀ¤éãÄ"
buf = "double fault", '\0' <repeats 243 times>
#3 0xc0605702 in dblfault_handler () at /usr/local/smallworld/usr/src/sys/i386/i386/trap.c:867
No locals.
#4 0xc0557f11 in AliasHandleIrcOut (la=0xc12f8000, pip=0xc1145800, lnk=0xc15dc800, maxsize=2048) at alias_local.h:353
hlen = Cannot access memory at address 0xc4e2e990
(kgdb) frame 4
#4 0xc0557f11 in AliasHandleIrcOut (la=0xc12f8000, pip=0xc1145800, lnk=0xc15dc800, maxsize=2048) at alias_local.h:353
353 alias_local.h: No such file or directory.
in alias_local.h
(kgdb) quit
Script done on Wed Dec 5 01:13:59 2007
>How-To-Repeat:
I do not use any kernel modules. Here is kernel config file:
options INCLUDE_CONFIG_FILE
machine i386
cpu I586_CPU
ident GW
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_DIRHASH # Improve performance on big directories
options MD_ROOT # MD is a potential root device
options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
device pci
device ata
device atadisk # ATA disk drives
options ATA_STATIC_ID # Static device numbering
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device vga # VGA video card driver
device sc
device sio # 8250, 16[45]50 based serial ports
device miibus # MII bus support
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device speaker
device bpf # Berkeley packet filter
options AUTO_EOI_1
options MAXCONS=8
options ALT_BREAK_TO_DEBUGGER
options CONSPEED=115200 # speed for serial console
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_FORWARD
options IPDIVERT
options DUMMYNET
options IPSEC
options IPSEC_ESP
options IPSEC_FILTERGIF
options NFSCLIENT #Network File System client
options LIBALIAS
options NETGRAPH # netgraph(4) system
options NETGRAPH_IPFW
options NETGRAPH_NAT
options NETGRAPH_SOCKET
options INVARIANTS
options INVARIANT_SUPPORT
options KDB
options KDB_TRACE
options KDB_UNATTENDED
options DDB
options DDB_NUMSYM
options GDB
Then, I use fastforwarding, here is my /etc/sysctl.conf:
net.inet.ip.fastforwarding=1
net.inet.ip.fw.one_pass=0
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
net.inet.udp.recvspace=65536
Also, I use ipfw, ng_ipfw and ng_nat here.
# ipfw list
00050 netgraph 1 ip from any to any in recv fxp1
00050 netgraph 2 ip from any to any out xmit fxp1
00060 netgraph 3 ip from any to any in recv gif0
00060 netgraph 4 ip from any to any out xmit gif0
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
65000 allow ip from any to any
65535 deny ip from any to any
ngctl shows:
+ ls
There are 4 total nodes:
Name: ngctl923 Type: socket ID: 00000006 Num hooks: 0
Name: uplink2 Type: nat ID: 00000005 Num hooks: 2
Name: uplink1 Type: nat ID: 00000003 Num hooks: 2
Name: ipfw Type: ipfw ID: 00000001 Num hooks: 4
The nodes are created with a following rcNG startup script for ng_nat:
ftp://www.kuzbass.ru/pub/freebsd/ng_nat.gz
My /etc/rc.conf contains next tunnables for ng_nat script:
uplink=fxp1
ng_nat_enable="YES"
ng_nat_nodes="uplink1 uplink2"
ng_nat_uplink1_interface="$uplink"
ng_nat_uplink1_ipfw_rules="50 50"
ng_nat_uplink1_cookies="1 2"
ng_nat_uplink2_interface="gif0"
ng_nat_uplink2_ipfw_rules="60 60"
ng_nat_uplink2_cookies="3 4"
So you can run ng_nat to repeat my configuration.
Feel free to request additional details.
>Fix:
Unknown
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200712041832.lB4IWaEv069092>