From owner-freebsd-stable@FreeBSD.ORG Sun Nov 23 09:16:32 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BFAA716A4CE; Sun, 23 Nov 2003 09:16:32 -0800 (PST) Received: from shellma.zin.lublin.pl (shellma.zin.lublin.pl [212.182.126.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id 01CCF43FEA; Sun, 23 Nov 2003 09:16:30 -0800 (PST) (envelope-from pawmal-posting@freebsd.lublin.pl) Received: by shellma.zin.lublin.pl (Postfix, from userid 1018) id 2D22F5F103; Sun, 23 Nov 2003 18:22:59 +0100 (CET) Date: Sun, 23 Nov 2003 18:22:59 +0100 From: Pawel Malachowski To: freebsd-stable@freebsd.org Message-ID: <20031123172259.GB74783@shellma.zin.lublin.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.4.1i cc: freebsd-bugs@freebsd.org Subject: Panic on 4.9-PRE (mbuf/m_copydata/ippr_ftp_process related) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: freebsd-bugs@freebsd.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Nov 2003 17:16:32 -0000 Hello, My router caught kernel panic after 11 days of working. System is 4.9-PRERELEASE and runs without problems since 28 Sep 2003. There are two outgoing interfaces and about 10 internal interfaces (mostly vlans); ipfw2 fwd is used to help with routing a bit (2 ISPs, no BGP); dummynet shaping happens at external and some of internal devices: % ipfw pipe show | wc -l 1134 I will update to recent 4.9-STABLE, however I'm posting backtrace here cause it may be hard for me to reproduce this panic (this is the first time I'm seeing it). Any ideas what should I look for? Here comes backtrace: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xc fault code = supervisor read, page not present instruction pointer = 0x8:0xc0255b50 stack pointer = 0x10:0xce4a2cbc frame pointer = 0x10:0xce4a2cc8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 76908 (trafd) interrupt mask = net tty trap number = 12 panic: page fault syncing disks... done Uptime: 11d4h46m8s dumping to dev #ad/0x30001, offset 1573024 dump ata0: resetting devices .. done [...] --- #0 dumpsys () at ../../kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) bt #0 dumpsys () at ../../kern/kern_shutdown.c:487 #1 0xc0238d33 in boot (howto=256) at ../../kern/kern_shutdown.c:316 #2 0xc0239158 in poweroff_wait (junk=0xc043516c, howto=-1069331345) at ../../kern/kern_shutdown.c:595 #3 0xc03b0d8a in trap_fatal (frame=0xce4a2c7c, eva=12) at ../../i386/i386/trap.c:974 #4 0xc03b0a5d in trap_pfault (frame=0xce4a2c7c, usermode=0, eva=12) at ../../i386/i386/trap.c:867 #5 0xc03b061b in trap (frame={tf_fs = 2097168, tf_es = 16, tf_ds = -834011120, tf_edi = 2, tf_esi = 0, tf_ebp = -833999672, tf_isp = -833999704, tf_ebx = 2, tf_edx = -1032830968, tf_ecx = 120, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1071293616, tf_cs = 8, tf_eflags = 66050, tf_esp = 2, tf_ss = -1032830860}) at ../../i386/i386/trap.c:466 #6 0xc0255b50 in m_copydata (m=0xc1178700, off=120, len=2, cp=0xc2704078 "") at ../../kern/uipc_mbuf.c:985 #7 0xc0157b10 in ippr_ftp_process (fin=0xce4a2df0, ip=0xc11b9810, nat=0xc1d1dc00, ftp=0xc2704000, rv=0) at ../../contrib/ipfilter/netinet/ip_ftp_pxy.c:1052 #8 0xc0157cd6 in ippr_ftp_out (fin=0xce4a2df0, ip=0xc11b9810, aps=0xc1aa7a80, nat=0xc1d1dc00) at ../../contrib/ipfilter/netinet/ip_ftp_pxy.c:1165 #9 0xc01591e1 in appr_check (ip=0xc11b9810, fin=0xce4a2df0, nat=0xc1d1dc00) at ../../contrib/ipfilter/netinet/ip_proxy.c:341 #10 0xc0156426 in ip_natout (ip=0xc11b9810, fin=0xce4a2df0) at ../../contrib/ipfilter/netinet/ip_nat.c:2555 #11 0xc014f4ea in fr_check (ip=0xc11b9810, hlen=20, ifp=0xc15c6800, out=1, mp=0xce4a2ea0) at ../../contrib/ipfilter/netinet/fil.c:1154 #12 0xc0295bb9 in ip_output (m0=0xc2541c80, opt=0x0, ro=0xc2541cac, flags=1, imo=0x0, inp=0x0) at ../../netinet/ip_output.c:964 #13 0xc17edb18 in ?? () #14 0xc17eddce in ?? () #15 0xc17ee267 in ?? () #16 0xc023edb1 in softclock () at ../../kern/kern_timeout.c:131 #17 0xc03a3543 in doreti_swi () #18 0x8049c91 in ?? () #19 0x8049edf in ?? () #20 0x804a5e6 in ?? () #21 0x804b530 in ?? () #22 0x28079e89 in ?? () #23 0x280799db in ?? () #24 0x80498b6 in ?? () #25 0x804926d in ?? () (kgdb) up 6 #6 0xc0255b50 in m_copydata (m=0xc1178700, off=120, len=2, cp=0xc2704078 "") at ../../kern/uipc_mbuf.c:985 985 while (len > 0) { (kgdb) list 980 if (off < m->m_len) 981 break; 982 off -= m->m_len; 983 m = m->m_next; 984 } 985 while (len > 0) { 986 KASSERT(m != NULL, ("m_copydata, length > size of mbuf chain")); 987 count = min(m->m_len - off, len); 988 bcopy(mtod(m, caddr_t) + off, cp, count); 989 len -= count; (kgdb) p m $1 = (struct mbuf *) 0x0 (kgdb) up #7 0xc0157b10 in ippr_ftp_process (fin=0xce4a2df0, ip=0xc11b9810, nat=0xc1d1dc00, ftp=0xc2704000, rv=0) at ../../contrib/ipfilter/netinet/ip_ftp_pxy.c:1052 1052 m_copydata(m, off, len, wptr); (kgdb) list 1047 bcopy((char *)m + off, wptr, len); 1048 #else 1049 # if SOLARIS 1050 copyout_mblk(m, off, len, wptr); 1051 # else 1052 m_copydata(m, off, len, wptr); 1053 # endif 1054 #endif 1055 mlen -= len; 1056 off += len; (kgdb) p m $2 = (mb_t *) 0xc1178700 % netstat -m -M vmcore.26 -N /kernel.debug 156/2816/10048 mbufs in use (current/peak/max): 156 mbufs allocated to data 130/1656/2512 mbuf clusters in use (current/peak/max) 4016 Kbytes allocated to network (53% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines ipnat rules are like this: map rl0 172.27.192.0/20 -> x/32 proxy port ftp ftp/tcp map rl0 172.27.192.0/20 -> x/32 portmap tcp/udp auto map rl0 172.27.192.0/20 -> x/32 map rl0 10.0.0.0/8 -> y/32 proxy port ftp ftp/tcp map rl0 10.0.0.0/8 -> y/32 portmap tcp/udp auto map rl0 10.0.0.0/8 -> y/32 map fxp0 172.27.192.0/20 -> z/32 proxy port ftp ftp/tcp map fxp0 172.27.192.0/20 -> z/32 portmap tcp/udp auto map fxp0 172.27.192.0/20 -> z/32 map fxp0 10.0.0.0/8 -> z/32 proxy port ftp ftp/tcp map fxp0 10.0.0.0/8 -> z/32 portmap tcp/udp auto map fxp0 10.0.0.0/8 -> z/32 map rl0 127.0.0.1/32 -> x/32 proxy port ftp ftp/tcp map rl0 127.0.0.1/32 -> x/32 portmap tcp/udp auto map rl0 127.0.0.1/32 -> x/32 map fxp0 127.0.0.1/32 -> z/32 proxy port ftp ftp/tcp map fxp0 127.0.0.1/32 -> z/32 portmap tcp/udp auto map fxp0 127.0.0.1/32 -> z/32 rdr fxp0 from SBD1/32 to z/32 port = XXX -> 10.1.X.X port XXX tcp rdr rl0 from SBD2/32 to x/32 port = XXX -> 10.1.X.X port XXX tcp rdr fxp0 from any to z/32 port = X -> 10.1.X.X port X tcp . . . (similar rdrs) . . rdr xl0 from X/24 to any port = 80 -> 172.27.X.X port 81 tcp TIA, -- Paweł Małachowski