From owner-freebsd-stable@FreeBSD.ORG Tue Feb 28 17:30:14 2012 Return-Path: Delivered-To: stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7F451065672 for ; Tue, 28 Feb 2012 17:30:14 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from mail.allbsd.org (gatekeeper-int.allbsd.org [IPv6:2001:2f0:104:e002::2]) by mx1.freebsd.org (Postfix) with ESMTP id 390F98FC1A for ; Tue, 28 Feb 2012 17:30:13 +0000 (UTC) Received: from alph.allbsd.org (p1012-ipbf2105funabasi.chiba.ocn.ne.jp [114.148.160.12]) (authenticated bits=128) by mail.allbsd.org (8.14.4/8.14.4) with ESMTP id q1SHToQZ080776; Wed, 29 Feb 2012 02:30:01 +0900 (JST) (envelope-from hrs@FreeBSD.org) Received: from localhost (localhost [IPv6:::1]) (authenticated bits=0) by alph.allbsd.org (8.14.4/8.14.4) with ESMTP id q1SHTnRq041807; Wed, 29 Feb 2012 02:29:50 +0900 (JST) (envelope-from hrs@FreeBSD.org) Date: Wed, 29 Feb 2012 02:26:51 +0900 (JST) Message-Id: <20120229.022651.1585266709145027511.hrs@allbsd.org> To: kostikbel@gmail.com From: Hiroki Sato In-Reply-To: <20120228130838.GN55074@deviant.kiev.zoral.com.ua> References: <20120224150259.GV55074@deviant.kiev.zoral.com.ua> <20120225.025828.128418237042325597.hrs@allbsd.org> <20120228130838.GN55074@deviant.kiev.zoral.com.ua> X-PGPkey-fingerprint: BDB3 443F A5DD B3D0 A530 FFD7 4F2C D3D8 2793 CF2D X-Mailer: Mew version 6.4 on Emacs 23.3 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Multipart/Signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="--Security_Multipart0(Wed_Feb_29_02_26_51_2012_049)--" Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97 at gatekeeper.allbsd.org X-Virus-Status: Clean X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (mail.allbsd.org [133.31.130.32]); Wed, 29 Feb 2012 02:30:06 +0900 (JST) X-Spam-Status: No, score=-99.8 required=13.0 tests=BAYES_00, CONTENT_TYPE_PRESENT, FAKEDWORD_ONE, FAKEDWORD_VERTICALLINE, RCVD_IN_PBL, RCVD_IN_RP_RNBL, SPF_SOFTFAIL, USER_IN_WHITELIST autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on gatekeeper.allbsd.org Cc: stable@FreeBSD.org Subject: Re: another panic in 8.3-PRERELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2012 17:30:14 -0000 ----Security_Multipart0(Wed_Feb_29_02_26_51_2012_049)-- Content-Type: Multipart/Mixed; boundary="--Next_Part(Wed_Feb_29_02_26_51_2012_369)--" Content-Transfer-Encoding: 7bit ----Next_Part(Wed_Feb_29_02_26_51_2012_369)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Konstantin Belousov wrote in <20120228130838.GN55074@deviant.kiev.zoral.com.ua>: ko> I can see the race in how the wiring of the sysctl buffers is done, but the ko> race can only realize for the multithreaded process. ko> ko> Can you, please, further show me two things: ko> - the p/x *(td->td_pcb) ko> - (this is somewhat laborous) Please find the vm map entry in the process ko> vm_map which covers the range [0x800e96000, 0x800ea6a79) and print it out. ko> You need to walk the td->td_proc->p_vmspace.vm_map.header list using ko> the next link, looking for the entry start/end values. The results and gdb commands I used are attached. In the linked-list there seem two entries that covers the range. -- Hiroki ----Next_Part(Wed_Feb_29_02_26_51_2012_369)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="result.txt" GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 4; apic id = 04 fault virtual address = 0x800e96000 fault code = supervisor write data, protection violation instruction pointer = 0x20:0xffffffff809440cb stack pointer = 0x28:0xffffff86c63890b0 frame pointer = 0x28:0xffffff86c6389100 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 47211 (top) lock order reversal: (Giant after non-sleepable) 1st 0xffffff0244b85568 process lock (process lock) @ /usr/src/sys/kern/kern_proc.c:1211 2nd 0xffffffff80d74c80 Giant (Giant) @ /usr/src/sys/dev/usb/input/ukbd.c:2018 KDB: stack backtrace: Dumping 23903 out of 24550 MB:..1%..11%..21%..31% (CTRL-C to abort) (CTRL-C to abort) ..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_mirror.ko Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done. done. Loaded symbols for /boot/kernel/ipfw.ko #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:263 263 if (textdump_pending) #16 0xffffffff80675e3a in __sysctl (td=0xffffff0396ec5460, uap=0xffffff86c6389bc0) at /usr/src/sys/kern/kern_sysctl.c:1491 1491 error = userland_sysctl(td, name, uap->namelen, -------- p/x *(td->td_pcb): $1 = {pcb_r15 = 0xffffff03969bf470, pcb_r14 = 0x0, pcb_r13 = 0xffffffff80d7f540, pcb_r12 = 0xffffff00057a18c0, pcb_rbp = 0xffffff86c6389700, pcb_rsp = 0xffffff86c63896a8, pcb_rbx = 0xffffff0396ec5460, pcb_rip = 0xffffffff80691367, pcb_fsbase = 0x800542398, pcb_gsbase = 0x0, pcb_kgsbase = 0x0, pcb_cr0 = 0x0, pcb_cr2 = 0x0, pcb_cr3 = 0x6793f000, pcb_cr4 = 0x0, pcb_dr0 = 0x0, pcb_dr1 = 0x0, pcb_dr2 = 0x0, pcb_dr3 = 0x0, pcb_dr6 = 0x0, pcb_dr7 = 0x0, pcb_gdt = {rd_limit = 0x0, rd_base = 0x0}, pcb_idt = { rd_limit = 0x0, rd_base = 0x0}, pcb_ldt = {rd_limit = 0x0, rd_base = 0x0}, pcb_tr = 0x0, pcb_flags = 0x18, pcb_initial_fpucw = 0x37f, pcb_onfault = 0xffffffff809440f0, pcb_gs32sd = {sd_lolimit = 0x0, sd_lobase = 0x0, sd_type = 0x0, sd_dpl = 0x0, sd_p = 0x0, sd_hilimit = 0x0, sd_xx = 0x0, sd_long = 0x0, sd_def32 = 0x0, sd_gran = 0x0, sd_hibase = 0x0}, pcb_tssp = 0x0, pcb_save = 0xffffff86c6389e00, pcb_user_save = {sv_env = {en_cw = 0x37f, en_sw = 0x0, en_tw = 0x0, en_zero = 0x0, en_opcode = 0x0, en_rip = 0x0, en_rdp = 0x0, en_mxcsr = 0x1fa4, en_mxcsr_mask = 0xffff}, sv_fp = {{ fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = { fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, { fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = { fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {fp_acc = {fp_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, fp_pad = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, sv_xmm = {{xmm_bytes = { 0x0 }}, {xmm_bytes = {0x0 }}, { xmm_bytes = {0x0 }}, {xmm_bytes = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xe0, 0x3f, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, {xmm_bytes = {0x0 }} }, sv_pad = { 0x0 }}} -------- #11 0xffffffff8065f6a6 in sysctl_out_proc_copyout (ki=0xffffff86c6389470, req=0xffffff86c63899c0) at /usr/src/sys/kern/kern_proc.c:1085 1085 error = SYSCTL_OUT(req, ki, sizeof(struct kinfo_proc)); -------- range start: $2 = 0x800e96000 range end: $3 = 0x800ea6a79 -------- #16 0xffffffff80675e3a in __sysctl (td=0xffffff0396ec5460, uap=0xffffff86c6389bc0) at /usr/src/sys/kern/kern_sysctl.c:1491 1491 error = userland_sysctl(td, name, uap->namelen, -------- td->td_proc->p_vmspace.vm_map.header: $4 = 0xffffff03d98bedc8 ::start $5 = 0x1000 ::end $6 = 0x800000000000 -------- -------- next: $7 = 0xffffff01f943bb40 ::start $8 = 0x400000 ::end $9 = 0x40c000 -------- -------- next: $10 = 0xffffff01f94cb780 ::start $11 = 0x50c000 ::end $12 = 0x50d000 -------- -------- next: $13 = 0xffffff01f9452690 ::start $14 = 0x50d000 ::end $15 = 0x600000 -------- -------- next: $16 = 0xffffff01f9452ca8 ::start $17 = 0x80050c000 ::end $18 = 0x80053c000 -------- -------- next: $19 = 0xffffff007d349ca8 ::start $20 = 0x80053c000 ::end $21 = 0x800544000 -------- -------- next: $22 = 0xffffff007d3295a0 ::start $23 = 0x80063c000 ::end $24 = 0x800644000 -------- -------- next: $25 = 0xffffff000cf09ac8 ::start $26 = 0x800644000 ::end $27 = 0x800653000 -------- -------- next: $28 = 0xffffff01f9581348 ::start $29 = 0x800653000 ::end $30 = 0x800697000 -------- -------- next: $31 = 0xffffff04d28094b0 ::start $32 = 0x800697000 ::end $33 = 0x800796000 -------- -------- next: $34 = 0xffffff01f9698708 ::start $35 = 0x800796000 ::end $36 = 0x8007a0000 -------- -------- next: $37 = 0xffffff01f94cb708 ::start $38 = 0x8007a0000 ::end $39 = 0x8007be000 -------- -------- next: $40 = 0xffffff012beda348 ::start $41 = 0x8007be000 ::end $42 = 0x8008be000 -------- -------- next: $43 = 0xffffff01f94cc780 ::start $44 = 0x8008be000 ::end $45 = 0x8008c0000 -------- -------- next: $46 = 0xffffff007d330528 ::start $47 = 0x8008c0000 ::end $48 = 0x8008c8000 -------- -------- next: $49 = 0xffffff03f03347f8 ::start $50 = 0x8008c8000 ::end $51 = 0x8009c8000 -------- -------- next: $52 = 0xffffff012beda960 ::start $53 = 0x8009c8000 ::end $54 = 0x8009c9000 -------- -------- next: $55 = 0xffffff01f94b2348 ::start $56 = 0x8009c9000 ::end $57 = 0x800ad2000 -------- -------- next: $58 = 0xffffff052b8144b0 ::start $59 = 0x800ad2000 ::end $60 = 0x800bd1000 -------- -------- next: $61 = 0xffffff007d349d20 ::start $62 = 0x800bd1000 ::end $63 = 0x800bf0000 -------- -------- next: $64 = 0xffffff01f94b2ca8 ::start $65 = 0x800bf0000 ::end $66 = 0x800c0b000 -------- -------- next: $67 = 0xffffff01f943b1e0 ::start $68 = 0x800e00000 ::end $69 = 0x800e96000 ::this entry covers the range $70 = {prev = 0xffffff01f94b2ca8, next = 0xffffff00054f7960, left = 0xffffff01f94b2ca8, right = 0x0, start = 0x800e00000, end = 0x800e96000, avail_ssize = 0x0, adj_free = 0x0, max_free = 0x7fff0c000, object = {vm_object = 0xffffff0342935000, sub_map = 0xffffff0342935000}, offset = 0x210000, eflags = 0x0, protection = 0x3, max_protection = 0x7, inheritance = 0x1, wired_count = 0x0, lastr = 0x2c2, uip = 0x0} -------- -------- next: $71 = 0xffffff00054f7960 ::start $72 = 0x800e96000 ::end $73 = 0x800ea7000 ::this entry covers the range $74 = {prev = 0xffffff01f943b1e0, next = 0xffffff056f97b690, left = 0xffffff01f943b1e0, right = 0xffffff056f97b690, start = 0x800e96000, end = 0x800ea7000, avail_ssize = 0x0, adj_free = 0x0, max_free = 0x7ff7fefe0000, object = {vm_object = 0xffffff0342935000, sub_map = 0xffffff0342935000}, offset = 0x2a6000, eflags = 0x0, protection = 0x3, max_protection = 0x7, inheritance = 0x1, wired_count = 0x1, lastr = 0x2c2, uip = 0x0} -------- -------- next: $75 = 0xffffff056f97b690 ::start $76 = 0x800ea7000 ::end $77 = 0x801000000 -------- -------- next: $78 = 0xffffff01f94cc8e8 ::start $79 = 0x7ffffffe0000 ::end $80 = 0x800000000000 -------- ----Next_Part(Wed_Feb_29_02_26_51_2012_369)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="gdb.cmd" set height 0 f 16 echo --------\n echo p/x *(td->td_pcb):\n p/x *(td->td_pcb) echo --------\n f 11 set $start = req->oldptr set $end = $start + req->oldlen echo --------\n echo range start:\n p/x $start echo range end:\n p/x $end echo --------\n f 16 set $h = &td->td_proc->p_vmspace.vm_map.header set $p = $h set $x = 1 echo --------\n echo td->td_proc->p_vmspace.vm_map.header:\n p/x $h echo ::start\n p/x $h->start echo ::end\n p/x $h->end set $map = 0 if ($p->start >= $start) if ($p->start < $end) set $map = 1 end end if ($p->end >= $start) if ($p->end < $end) set $map = 1 end end if ($map > 0) echo ::this entry covers the range\n p/x *$p set $map = 0 end echo --------\n set $p = $p->next while ($x > 0) echo --------\n echo next:\n p/x $p echo ::start\n p/x $p->start echo ::end\n p/x $p->end set $map = 0 if ($p->start >= $start) if ($p->start < $end) set $map = 1 end end if ($p->end >= $start) if ($p->end < $end) set $map = 1 end end if ($map > 0) echo ::this entry covers the range\n p/x *$p set $map = 0 end set $p = $p->next if ($p == $h) set $x = 0 end echo --------\n end quit ----Next_Part(Wed_Feb_29_02_26_51_2012_369)---- ----Security_Multipart0(Wed_Feb_29_02_26_51_2012_049)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEABECAAYFAk9NDlsACgkQTyzT2CeTzy3TZgCfdpFiMmQ+aaD2XhQMs69Zcd4d 8K0An1HF6L/sW5MbZ/J5o2+929h3WvtB =FQ1R -----END PGP SIGNATURE----- ----Security_Multipart0(Wed_Feb_29_02_26_51_2012_049)----