From owner-freebsd-stable@FreeBSD.ORG Mon Aug 15 10:45:16 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C4151065673; Mon, 15 Aug 2011 10:45:16 +0000 (UTC) (envelope-from prvs=1208040d95=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id EB69E8FC1C; Mon, 15 Aug 2011 10:45:15 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 15 Aug 2011 11:33:24 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 15 Aug 2011 11:33:24 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014604141.msg; Mon, 15 Aug 2011 11:33:23 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1208040d95=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> <4E43E272.1060204@FreeBSD.org> <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> <4E440865.1040500@FreeBSD.org> <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> <4E441314.6060606@FreeBSD.org> <2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk> <4E48D967.9060804@FreeBSD.org> Date: Mon, 15 Aug 2011 11:34:02 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Aug 2011 10:45:16 -0000 ----- Original Message ----- From: "Andriy Gapon" >> We have 352 thread entries starting with:- >> #0 sched_switch (td=0xffffffff8083e4e0, newtd=0xffffff0012d838c0, >> flags=Variable "flags" is not available. >> 23 with:- >> cpustop_handler () at atomic.h:285 >> and 16 with:- >> #0 fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:562 > > I would like to get a full output of thread apply all bt. http://blog.multplay.co.uk/dropzone/freebsd/panic-2011-08-14-1524.txt >> The main message being:- >> panic: double fault >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> <118>Aug 14 15:13:33 amsbld15 syslogd: exiting on signal 15 > > So this line, does it indicate a shutdown of a jail or of the whole system? This specific panic was caused by me running "reboot" after all jails (~40) where shutdown, which is slightly different from what my collegue was seeing last friday, where the machines where panicing when the jails themselves where stopped. I may have a crash from one of these if needed. >> Fatal double fault >> rip = 0xffffffff8053b691 > > Can you please provide output of 'list *0xffffffff8053b691' in kgdb? (kgdb) list *0xffffffff8053b691 0xffffffff8053b691 is in vm_fault (/usr/src/sys/vm/vm_fault.c:239). 234 /* 235 * Find the backing store object and offset into it to begin the 236 * search. 237 */ 238 fs.map = map; 239 result = vm_map_lookup(&fs.map, vaddr, fault_type, &fs.entry, 240 &fs.first_object, &fs.first_pindex, &prot, &wired); 241 if (result != KERN_SUCCESS) { 242 if (result != KERN_PROTECTION_FAILURE || 243 (fault_flags & VM_FAULT_WIRE_MASK) != VM_FAULT_USER_WIRE) { > >> rsp = 0xffffff8d8f356fb0 >> rbp = 0xffffff8d8f357210 >> cpuid = 2; apic id = 02 >> panic: double fault >> cpuid = 2 >> KDB: stack backtrace: >> #0 0xffffffff803bb75e at kdb_backtrace+0x5e >> #1 0xffffffff8038956e at panic+0x2ae >> #2 0xffffffff805802b6 at dblfault_handler+0x96 >> #3 0xffffffff8056900d at Xdblfault+0xad > > I think (not 100% sure) that with DDB in kernel we could get a better backtrace > here, possibly with pre-dblfault stack frames, because DDB backend is a bit more > smarter than the trivial stack(9) printer. I've added this into the the kernel on my test machine and will try to get it panic over the next few days. Seems to need a few days on uptime before the panics start happening. In addition to increasing KSTACK_PAGES to 12, if you believe this may be stack exhaustion, do you want me to remove this increase? >> stack: 0xffffff8d8f357000, 4 > > One thing I can say is that this looks like like a double-fault because of stack > exhaustion (the most typical cause): rsp value is below td_kstack. > > Can you please also provide the following information: > p *((struct pcb *)((char *)0xffffff8d8f357000 + KSTACK_PAGES * PAGE_SIZE) - 1) > where KSTACK_PAGES is a value of KSTACK_PAGES option (amd64 default is 4) and > PAGE_SIZE is 4096. (kgdb) p *((struct pcb *)((char *)0xffffff8d8f357000 + 4 * 4096) - 1) $1 = {pcb_r15 = -2138686968, pcb_r14 = -1070655224792, pcb_r13 = 0, pcb_r12 = -1070655225856, pcb_rbp = -491518580864, pcb_rsp = -491518580952, pcb_rbx = -1099195460512, pcb_rip = -2143622375, pcb_fsbase = 34365428376, pcb_gsbase = 0, pcb_kgsbase = 0, pcb_cr0 = 0, pcb_cr2 = 0, pcb_cr3 = 12406784, pcb_cr4 = 0, pcb_dr0 = 0, pcb_dr1 = 0, pcb_dr2 = 0, pcb_dr3 = 0, pcb_dr6 = 0, pcb_dr7 = 0, pcb_flags = 0, pcb_initial_fpucw = 895, pcb_onfault = 0x0, pcb_gs32sd = {sd_lolimit = 0, sd_lobase = 0, sd_type = 0, sd_dpl = 0, sd_p = 0, sd_hilimit = 0, sd_xx = 0, sd_long = 0, sd_def32 = 0, sd_gran = 0, sd_hibase = 0}, pcb_tssp = 0x0, pcb_save = 0xffffff8d8f35ae00, pcb_full_iret = 0 '\0', pcb_gdt = {rd_limit = 0, rd_base = 0}, pcb_idt = {rd_limit = 0, rd_base = 0}, pcb_ldt = {rd_limit = 0, rd_base = 0}, pcb_tr = 0, pcb_user_save = {sv_env = {en_cw = 895, en_sw = 0, en_tw = 0 '\0', en_zero = 0 '\0', en_opcode = 0, en_rip = 0, en_rdp = 0, en_mxcsr = 8096, en_mxcsr_mask = 65535}, sv_fp = {{fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}, {fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}, {fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}, {fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}, {fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}, {fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}, {fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}, {fp_acc = {fp_bytes = "\000\000\000\000\000\000\000\000\000"}, fp_pad = "\000\000\000\000\000"}}, sv_xmm = {{xmm_bytes = "\000\000\000\b\030\212rA\000\000\000\000\000\000\000"}, { xmm_bytes = '\0' } }, sv_pad = '\0' }} Thanks for your help on this, as its way over my head ;-) Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.