From owner-freebsd-bugs@FreeBSD.ORG Fri Aug 22 17:50:15 2003 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B610C16A4C0 for ; Fri, 22 Aug 2003 17:50:15 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id D80A043FAF for ; Fri, 22 Aug 2003 17:50:14 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with ESMTP id h7N0oEUp024414 for ; Fri, 22 Aug 2003 17:50:14 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.9/8.12.9/Submit) id h7N0oE0n024413; Fri, 22 Aug 2003 17:50:14 -0700 (PDT) Resent-Date: Fri, 22 Aug 2003 17:50:14 -0700 (PDT) Resent-Message-Id: <200308230050.h7N0oE0n024413@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Pawel Malachowski Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F02716A4C0 for ; Fri, 22 Aug 2003 17:41:24 -0700 (PDT) Received: from shellma.zin.lublin.pl (shellma.zin.lublin.pl [212.182.126.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2DB3443FBD for ; Fri, 22 Aug 2003 17:41:22 -0700 (PDT) (envelope-from pm@zin.lublin.pl) Received: by shellma.zin.lublin.pl (Postfix, from userid 1018) id 5FBDE5F103; Sat, 23 Aug 2003 02:50:46 +0200 (CEST) Message-Id: <20030823005046.5FBDE5F103@shellma.zin.lublin.pl> Date: Sat, 23 Aug 2003 02:50:46 +0200 (CEST) From: Pawel Malachowski To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Subject: kern/55886: mbuf exhaustion can cause panic X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Pawel Malachowski List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Aug 2003 00:50:16 -0000 >Number: 55886 >Category: kern >Synopsis: mbuf exhaustion can cause panic >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Aug 22 17:50:13 PDT 2003 >Closed-Date: >Last-Modified: >Originator: Pawe³ Ma³achowski >Release: FreeBSD 4.7-RELEASE-p11 i386 >Organization: ZiN >Environment: System: FreeBSD 4.8, 5.1 >Description: After mbuf exhaustion it is possible, that kernel will panic because of page fault. This was observed by me on two different machines: 1. Celeron/333 with 128MB RAM, running latest GENERIC 4.8-STABLE or 5.1-RELEASE 2. PIII/750 with 768MB RAM, runnig latest customized 4.8-STABLE and by Adam Liberacki from polish FreeBSD users group on his: 3. PIII/600 with 256MB RAM, running 5.1-CURRENT After pingflooding localhost, machine crashes (1. within one minute after mbuf exhaustion, 2. within few minutes, 3. after 4 hours) in exactly the same way (from 4.8-STABLE): Fatal trap 12: page fault while in kernel mode fault virtual address = 0x8 fault code = supervisor read, page not present instruction pointer = 0x8:0xc0299fb7 stack pointer = 0x10:0xe3b8ddfc frame pointer = 0x10:0xe3b8de10 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 24445 (ping) interrupt mask = trap number = 12 panic: page fault syncing disks... done Uptime: 10d0h19m26s dumping to dev #da/0x20009, offset 2621464 [cut] #0 dumpsys () at ../../kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) bt #0 dumpsys () at ../../kern/kern_shutdown.c:487 #1 0xc023911f in boot (howto=256) at ../../kern/kern_shutdown.c:316 #2 0xc0239544 in poweroff_wait (junk=0xc0437c2c, howto=-1069320401) at ../../kern/kern_shutdown.c:595 #3 0xc03b304a in trap_fatal (frame=0xe3b8ddbc, eva=8) at ../../i386/i386/trap.c:974 #4 0xc03b2d1d in trap_pfault (frame=0xe3b8ddbc, usermode=0, eva=8) at ../../i386/i386/trap.c:867 #5 0xc03b28db in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = -474480624, tf_edi = -526853952, tf_esi = 20, tf_ebp = -474423792, tf_isp = -474423832, tf_ebx = -1027443712, tf_edx = 0, tf_ecx = 0, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1071013961, tf_cs = 8, tf_eflags = 66118, tf_esp = -1027443712, tf_ss = -1044738560}) at ../../i386/i386/trap.c:466 #6 0xc0299fb7 in rip_output (m=0x0, so=0xdf9c28c0, dst=16777343) at ../../netinet/raw_ip.c:260 #7 0xc029a503 in rip_send (so=0xdf9c28c0, flags=0, m=0xc2c27400, nam=0xc409f410, control=0x0, p=0xe3ad9d00) at ../../netinet/raw_ip.c:612 #8 0xc025833b in sosend (so=0xdf9c28c0, addr=0xc409f410, uio=0xe3b8decc, top=0xc2c27400, control=0x0, flags=0, p=0xe3ad9d00) at ../../kern/uipc_socket.c:609 #9 0xc025b7bb in sendit (p=0xe3ad9d00, s=3, mp=0xe3b8df0c, flags=0) at ../../kern/uipc_syscalls.c:590 #10 0xc025b8be in sendto (p=0xe3ad9d00, uap=0xe3b8df80) at ../../kern/uipc_syscalls.c:643 #11 0xc03b32f9 in syscall2 (frame={tf_fs = -1078067153, tf_es = -1078067153, tf_ds = -1078067153, tf_edi = 500, tf_esi = 134753312, tf_ebp = -1078003760, tf_isp = -474423340, tf_ebx = 65445, tf_edx = 16409, tf_ecx = 134818756, tf_eax = 133, tf_trapno = 7, tf_err = 2, tf_eip = 134530472, tf_cs = 31, tf_eflags = 663, tf_esp = -1078003836, tf_ss = 47}) at ../../i386/i386/trap.c:1175 #12 0xc03a44c5 in Xint0x80_syscall () #13 0x80492e1 in ?? () #14 0x804813e in ?? () (kgdb) up 6 #6 0xc0299fb7 in rip_output (m=0x0, so=0xdf9c28c0, dst=16777343) at ../../netinet/raw_ip.c:260 260 ip = mtod(m, struct ip *); (kgdb) list 255 if (m->m_pkthdr.len + sizeof(struct ip) > IP_MAXPACKET) { 256 m_freem(m); 257 return(EMSGSIZE); 258 } 259 M_PREPEND(m, sizeof(struct ip), M_WAIT); 260 ip = mtod(m, struct ip *); 261 ip->ip_tos = inp->inp_ip_tos; 262 ip->ip_off = 0; 263 ip->ip_p = inp->inp_ip_p; 264 ip->ip_len = m->m_pkthdr.len; (kgdb) p m $1 = (struct mbuf *) 0x0 (kgdb) up #7 0xc029a503 in rip_send (so=0xdf9c28c0, flags=0, m=0xc2c27400, nam=0xc409f410, control=0x0, p=0xe3ad9d00) at ../../netinet/raw_ip.c:612 612 return rip_output(m, so, dst); (kgdb) p m $2 = (struct mbuf *) 0xc2c27400 (kgdb) p *m $3 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc2c2742c "\b", mh_len = 212, mh_type = 0, mh_flags = 2}, M_dat = {MH = {MH_pkthdr = { rcvif = 0x0, len = 65445, header = 0x2, csum_flags = 0, csum_data = 5, tags = {slh_first = 0x0}}, MH_dat = {MH_ext = { ext_buf = 0x95010008
, ext_free = 0x5f45f7d, ext_size = 1061458863, ext_ref = 0xbe5e8}, MH_databuf = "\b\000\001\225}_ô\005¯\223D?èå\013\000\b\t\n\013\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\e\034\035\036\037 !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_bcdefghijklmnopqrstuvwxyz{|}~\177\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237 ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿"...}}, M_databuf = "\000\000\000\000¥ÿ\000\000\002\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\b\000\001\225}_ô\005¯\223D?èå\013\000\b\t\n\013\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\e\034\035\036\037 !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_bcdefghijklmnopqrstuvwxyz{|}~\177\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237 ¡¢£¤¥¦§"...}} It also seems, that before M_PREPEND at line 259, *m is still valid. On 5.1, it panics earlier, at rip_send(). 2nd machine: % netstat -m -M vmcore.2 26391/26432/26432 mbufs in use (current/peak/max): 25035 mbufs allocated to data 678 mbufs allocated to ancillary data 678 mbufs allocated to socket names and addresses 6576/6608/6608 mbuf clusters in use (current/peak/max) 19824 Kbytes allocated to network (100% of mb_map in use) 5808737 requests for memory denied 155238 requests for memory delayed 143324 calls to protocol drain routines 1st machine (with kern.ipc.nmbclusters="8192" on loader.conf): % netstat -m -M vmcore.78 32767/32768/32768 mbufs in use (current/peak/max): 10968 mbufs allocated to data 10897 mbufs allocated to ancillary data 10902 mbufs allocated to socket names and addresses 4023/4024/8192 mbuf clusters in use (current/peak/max) 16240 Kbytes allocated to network (66% of mb_map in use) 323769 requests for memory denied 178 requests for memory delayed 228 calls to protocol drain routines >How-To-Repeat: Try running this on a machine with 128MB memory and GENERIC autosized mbuf-related values. Run as root and without login.conf limits. It is very easy to reproduce for me, however, as discussed at polish FreeBSD usenet group, some people claim, they are not able to reproduce that. #!/bin/sh while [ 1 ] do ping -q -i 0 -s 65437 localhost >/dev/null 2>&1 & done Wait patiently. :) >Fix: Workaround: increase kern.ipc.nmbclusters="32768" for example, after that I can't reproduce panic on 1. machine. >Release-Note: >Audit-Trail: >Unformatted: