From owner-freebsd-ppc@FreeBSD.ORG Sun Jun 9 15:47:31 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 313FE53B; Sun, 9 Jun 2013 15:47:31 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from smtpauth4.wiscmail.wisc.edu (wmauth4.doit.wisc.edu [144.92.197.145]) by mx1.freebsd.org (Postfix) with ESMTP id F23191E6A; Sun, 9 Jun 2013 15:47:30 +0000 (UTC) MIME-version: 1.0 Received: from avs-daemon.smtpauth4.wiscmail.wisc.edu by smtpauth4.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MO400A00TOA3S00@smtpauth4.wiscmail.wisc.edu>; Sun, 09 Jun 2013 10:47:24 -0500 (CDT) X-Spam-PmxInfo: Server=avs-4, Version=6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2013.6.9.153918, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from wanderer.tachypleus.net (adsl-76-208-69-84.dsl.mdsnwi.sbcglobal.net [76.208.69.84]) by smtpauth4.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MO400H4OUIX1230@smtpauth4.wiscmail.wisc.edu>; Sun, 09 Jun 2013 10:47:24 -0500 (CDT) Date: Sun, 09 Jun 2013 10:47:21 -0500 From: Nathan Whitehorn Subject: Re: Strange panic on ppc64 In-reply-to: To: Justin Hibbits Message-id: <51B4A389.4020607@freebsd.org> References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130420 Thunderbird/17.0.5 Content-Type: text/plain; CHARSET=US-ASCII Content-Transfer-Encoding: 7BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 15:47:31 -0000 On 06/08/13 17:33, Justin Hibbits wrote: > > > > On Sat, Jun 8, 2013 at 7:54 AM, Nathan Whitehorn > > wrote: > > On 06/08/13 09:21, Justin Hibbits wrote: >> >> >> >> On Wed, Jun 5, 2013 at 9:47 AM, Justin Hibbits >> > wrote: >> >> Will do, when I get it panicking again. >> >> - Justin >> >> On Jun 5, 2013 9:46 AM, "Nathan Whitehorn" >> > wrote: >> >> On 06/04/13 22:35, Justin Hibbits wrote: >> >> After a string of seemingly random hangs, I added >> invariants (but not >> witness) to my custom kernel config, and I get the >> following panic, >> recreated from a fuzzy cell phone picture: >> >> >> [thread pid -1 tid 1006665719 ] >> Stopped at 0: illegal instruction 0 >> db> panic: mutex ohci1 owned at >> /usr/home/chmeee/freebsd/head/sys/dev/usb/usb_transfer.c:2280 >> cpuid = 0 >> Uptime: 9h8m1s >> >> ... >> panic: msleep1 >> cpu = 0 >> KDB: enter: panic >> [ thread pid -1 tid 100665719 ] >> .... >> >> The first question I have is how the hell it got such >> a strange PID/TID, >> memory corruption my guess, something is stomping on >> the pcpu or something, >> and I think these hangs have only happened since I >> added a lot more memory >> (up to 12G from 4G, Andreas Tobler was seeing hangs >> as well), so it might >> be something in the moea64 pmap code, but that's pure >> speculation on my >> part. Then the other panic messages, owned mutex and >> panic in msleep1. I >> enabled more trace code, so hopefully the next time >> it panics I can collect >> better data. >> >> - Justin >> _______________________________________________ >> freebsd-ppc@freebsd.org >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-ppc >> To unsubscribe, send any mail to >> "freebsd-ppc-unsubscribe@freebsd.org >> " >> >> >> Could you post the output from show reg? It looks like it >> tried to jump to a null pointer there. >> -Nathan >> >> >> Well, it's hard to do get that output, because I just hit that >> 'mutex owned' panic, and here's the backtrace: > > > The mutex thing is spurious -- it was already panicing and then > paniced again trying to panic. Can you get the backtrace for the > original panic (it should be different) and the values of the > registers? > -Nathan > > > Here you go: > > [ thread pid -1 tid 1006665719 ] > Stopped at 0: illegal instruction 0 > db:0:kdb.enter.default> show reg > r0 0 > r1 0 > r2 0xab63d0 M_MACTEMP > r3 0xbb12e0 > r4 0x741f18 .ofwcall+0xa8 > r5 0 > r6 0xa4f1a8 > r7 0x1 > r8 0x1 > r9 0xc10500 __pcpu > r10 0x1c35ec0 > r11 0 > r12 0x2000d032 > r13 0x342eb000 > r14 0x10014200 > r15 0xffffffffffffcb58 > r16 0x2 > r17 0x2 > r18 0xffffffffffffcb50 > r19 0 > r20 0xc000000013231478 > r21 0xc00000014c0ce200 > r22 0 > r23 0x64 dbsize+0x10 > r24 0xc00000014c0cdf70 > r25 0xb62cb8 smp_no_rendevous_barrier > r26 0 > r27 0x741f18 .ofwcall+0xa8 > r28 0x741f18 .ofwcall+0xa8 > r29 0x2000d032 > r30 0x9000000000001032 > r31 0xc0cad8 mac_labeled > srr0 0x102ca4 k_trap+0x28 > srr1 0x9000000000001032 > lr 0x102c74 u_trap+0x10 > ctr 0xff846d78 > cr 0x2000f1b0 > xer 0 > dar 0xfffffffffffffd60 > dsisr 0x42000000 > 0: illegal instruction 0 > db:0:kdb.enter.default> bt > Tracing pid -1 tid 1006665719 td 0 > (nothing) Well, that is all kinds of messed up. It appears to have halted while handling a userland trap due to an implicit branch caused by bad translations when it restores the kernel SRs. Could you see what 'show pcpu' does? Does that information look valid at all? I suspect it has become corrupted somehow. -Nathan From owner-freebsd-ppc@FreeBSD.ORG Sun Jun 9 21:21:15 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9CA7636B; Sun, 9 Jun 2013 21:21:15 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-bk0-x236.google.com (mail-bk0-x236.google.com [IPv6:2a00:1450:4008:c01::236]) by mx1.freebsd.org (Postfix) with ESMTP id BE5E812F4; Sun, 9 Jun 2013 21:21:14 +0000 (UTC) Received: by mail-bk0-f54.google.com with SMTP id it16so1015653bkc.27 for ; Sun, 09 Jun 2013 14:21:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=uzal/NX3KyMnJ2+B4R+xLBu93nYj18nSgvW2h+Q+v0w=; b=RHQ3ogGQlAruPq4zPC7BZHnjmmHguJFFSIXv/ELAxP5cl+osnH0eEBI3lmtjRd8w// TXo6F6IVR0xEGSsTGoNXWanEUOm+PyWPvbPqEIDwY3hzvc4mnjZxUcGhl0BKU6lqIoBu u3cehhLnQMd66btR0NXOtysMrpgGy7y8B0dJEFZ5hmPYrJeA6H8VvWQDOT45K7s7Cqjo +UMXnhnMrIqBjZ6Tud2ThfEdXeIBmBuYGqTo2SVDmTJdwmsy0xZ5is7OWvC751l/E3Mq mEJ0sklBjMBBe1jOyWZlw0uon6A08LdrnthwXowI3dp6FzaEUz/1AKEFHPLVJp9u4PLs 12Yg== MIME-Version: 1.0 X-Received: by 10.205.1.1 with SMTP id no1mr1028404bkb.131.1370812873799; Sun, 09 Jun 2013 14:21:13 -0700 (PDT) Sender: chmeeedalf@gmail.com Received: by 10.204.236.132 with HTTP; Sun, 9 Jun 2013 14:21:12 -0700 (PDT) In-Reply-To: <51B4A389.4020607@freebsd.org> References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> Date: Sun, 9 Jun 2013 14:21:12 -0700 X-Google-Sender-Auth: l7Xn40_207D7t7fVkIu362ohYiM Message-ID: Subject: Re: Strange panic on ppc64 From: Justin Hibbits To: Nathan Whitehorn Content-Type: multipart/mixed; boundary=90e6ba25db9365b67f04debf3ebf X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 21:21:15 -0000 --90e6ba25db9365b67f04debf3ebf Content-Type: text/plain; charset=UTF-8 On Sun, Jun 9, 2013 at 8:47 AM, Nathan Whitehorn wrote: > On 06/08/13 17:33, Justin Hibbits wrote: > > > > > On Sat, Jun 8, 2013 at 7:54 AM, Nathan Whitehorn wrote: > >> On 06/08/13 09:21, Justin Hibbits wrote: >> >> >> >> >> On Wed, Jun 5, 2013 at 9:47 AM, Justin Hibbits wrote: >> >>> Will do, when I get it panicking again. >>> >>> - Justin >>> On Jun 5, 2013 9:46 AM, "Nathan Whitehorn" >>> wrote: >>> >>>> On 06/04/13 22:35, Justin Hibbits wrote: >>>> >>>>> After a string of seemingly random hangs, I added invariants (but not >>>>> witness) to my custom kernel config, and I get the following panic, >>>>> recreated from a fuzzy cell phone picture: >>>>> >>>>> >>>>> [thread pid -1 tid 1006665719 ] >>>>> Stopped at 0: illegal instruction 0 >>>>> db> panic: mutex ohci1 owned at >>>>> /usr/home/chmeee/freebsd/head/sys/dev/usb/usb_transfer.c:2280 >>>>> cpuid = 0 >>>>> Uptime: 9h8m1s >>>>> >>>>> ... >>>>> panic: msleep1 >>>>> cpu = 0 >>>>> KDB: enter: panic >>>>> [ thread pid -1 tid 100665719 ] >>>>> .... >>>>> >>>>> The first question I have is how the hell it got such a strange >>>>> PID/TID, >>>>> memory corruption my guess, something is stomping on the pcpu or >>>>> something, >>>>> and I think these hangs have only happened since I added a lot more >>>>> memory >>>>> (up to 12G from 4G, Andreas Tobler was seeing hangs as well), so it >>>>> might >>>>> be something in the moea64 pmap code, but that's pure speculation on my >>>>> part. Then the other panic messages, owned mutex and panic in >>>>> msleep1. I >>>>> enabled more trace code, so hopefully the next time it panics I can >>>>> collect >>>>> better data. >>>>> >>>>> - Justin >>>>> _______________________________________________ >>>>> freebsd-ppc@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-ppc >>>>> To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org" >>>>> >>>> >>>> Could you post the output from show reg? It looks like it tried to jump >>>> to a null pointer there. >>>> -Nathan >>>> >>> >> Well, it's hard to do get that output, because I just hit that 'mutex >> owned' panic, and here's the backtrace: >> >> >> >> The mutex thing is spurious -- it was already panicing and then paniced >> again trying to panic. Can you get the backtrace for the original panic (it >> should be different) and the values of the registers? >> -Nathan >> > > Here you go: > > [ thread pid -1 tid 1006665719 ] > Stopped at 0: illegal instruction 0 > db:0:kdb.enter.default> show reg > r0 0 > r1 0 > r2 0xab63d0 M_MACTEMP > r3 0xbb12e0 > r4 0x741f18 .ofwcall+0xa8 > r5 0 > r6 0xa4f1a8 > r7 0x1 > r8 0x1 > r9 0xc10500 __pcpu > r10 0x1c35ec0 > r11 0 > r12 0x2000d032 > r13 0x342eb000 > r14 0x10014200 > r15 0xffffffffffffcb58 > r16 0x2 > r17 0x2 > r18 0xffffffffffffcb50 > r19 0 > r20 0xc000000013231478 > r21 0xc00000014c0ce200 > r22 0 > r23 0x64 dbsize+0x10 > r24 0xc00000014c0cdf70 > r25 0xb62cb8 smp_no_rendevous_barrier > r26 0 > r27 0x741f18 .ofwcall+0xa8 > r28 0x741f18 .ofwcall+0xa8 > r29 0x2000d032 > r30 0x9000000000001032 > r31 0xc0cad8 mac_labeled > srr0 0x102ca4 k_trap+0x28 > srr1 0x9000000000001032 > lr 0x102c74 u_trap+0x10 > ctr 0xff846d78 > cr 0x2000f1b0 > xer 0 > dar 0xfffffffffffffd60 > dsisr 0x42000000 > 0: illegal instruction 0 > db:0:kdb.enter.default> bt > Tracing pid -1 tid 1006665719 td 0 > (nothing) > > > Well, that is all kinds of messed up. It appears to have halted while > handling a userland trap due to an implicit branch caused by bad > translations when it restores the kernel SRs. Could you see what 'show > pcpu' does? Does that information look valid at all? I suspect it has > become corrupted somehow. > -Nathan > > Here's the full log from dconschat, from bootup to panic. Unfortunately, not everything I wanted to print would print, and I can't type anything once it panics, because it panics when reading the keyboard, so I have to add everything as a ddb enter script. Here's what I've added so far (doesn't do everything as you can see from the transcript): script kdb.enter.default=show reg; bt; show pcpu; ps; run lockinfo; alltrace; show all procs; show files; show malloc; show allchains - Justin --90e6ba25db9365b67f04debf3ebf Content-Type: application/octet-stream; name="zhabar.dcons" Content-Disposition: attachment; filename="zhabar.dcons" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hhqq62m50 U2NyaXB0IHN0YXJ0ZWQgb24gU3VuIEp1biAgOSAwOToxMDoyMCAyMDEzCmNvbW1hbmQ6IGRjb25z Y2hhdCAtdCAwMC0xNC01MS1mZi1mZS0zMy1jYS1iNiAtZSAjIC1yCltkY29ucyBjb25uZWN0ZWRd DQ0KcmVmZXRjaGVkIGRlY29kZQ0KcGNpMTA6IDxPRlcgUENJIGJ1cz4gb24gcGNpYjEwDQpwY2kx MDogZG9tYWluPTEsIHBoeXNpY2FsIGJ1cz0zDQpmb3VuZC0+CXZlbmRvcj0weDExNjYsIGRldj0w eDAyNDAsIHJldmlkPTB4MDANCglkb21haW49MSwgYnVzPTMsIHNsb3Q9MTIsIGZ1bmM9MA0KCWNs YXNzPTAxLTAxLThmLCBoZHJ0eXBlPTB4MDAsIG1mZGV2PTENCgljbWRyZWc9MHgwMDA2LCBzdGF0 cmVnPTB4MDIyMCwgY2FjaGVsbnN6PTAgKGR3b3JkcykNCglsYXR0aW1lcj0weDEwICg0ODAgbnMp LCBtaW5nbnQ9MHgwMCAoMCBucyksIG1heGxhdD0weDAwICgwIG5zKQ0KCW1hcFsxMF06IHR5cGUg SS9PIFBvcnQsIHJhbmdlIDMyLCBiYXNlIDAsIHNpemUgIDMsIHBvcnQgZGlzYWJsZWQNCgltYXBb MTRdOiB0eXBlIEkvTyBQb3J0LCByYW5nZSAzMiwgYmFzZSAwLCBzaXplICAyLCBwb3J0IGRpc2Fi bGVkDQoJbWFwWzE4XTogdHlwZSBJL08gUG9ydCwgcmFuZ2UgMzIsIGJhc2UgMCwgc2l6ZSAgMywg cG9ydCBkaXNhYmxlZA0KCW1hcFsxY106IHR5cGUgSS9PIFBvcnQsIHJhbmdlIDMyLCBiYXNlIDAs IHNpemUgIDIsIHBvcnQgZGlzYWJsZWQNCgltYXBbMjBdOiB0eXBlIEkvTyBQb3J0LCByYW5nZSAz MiwgYmFzZSAwLCBzaXplICA0LCBwb3J0IGRpc2FibGVkDQoJbWFwWzI0XTogdHlwZSBNZW1vcnks IHJhbmdlIDMyLCBiYXNlIDB4ZmE0MDIwMDAsIHNpemUgMTMsIGVuYWJsZWQNCnBjaWIxMDogYWxs b2NhdGVkIG1lbW9yeSByYW5nZSAoMHhmYTQwMjAwMC0weGZhNDAzZmZmKSBmb3IgcmlkIDI0IG9m IHBjaTE6MzoxMjowDQpmb3VuZC0+CXZlbmRvcj0weDEwNmIsIGRldj0weDAwNTAsIHJldmlkPTB4 MDANCglkb21haW49MSwgYnVzPTMsIHNsb3Q9MTMsIGZ1bmM9MA0KCWNsYXNzPWZmLTAwLTAwLCBo ZHJ0eXBlPTB4MDAsIG1mZGV2PTANCgljbWRyZWc9MHgwMDA0LCBzdGF0cmVnPTB4ODIwMCwgY2Fj aGVsbnN6PTE2IChkd29yZHMpDQoJbGF0dGltZXI9MHgyMCAoOTYwIG5zKSwgbWluZ250PTB4MDAg KDAgbnMpLCBtYXhsYXQ9MHgwMCAoMCBucykNCgltYXBbMTBdOiB0eXBlIE1lbW9yeSwgcmFuZ2Ug MzIsIGJhc2UgMHhmYTQwNDAwMCwgc2l6ZSAxNCwgbWVtb3J5IGRpc2FibGVkDQpwY2liMTA6IGFs bG9jYXRlZCBtZW1vcnkgcmFuZ2UgKDB4ZmE0MDQwMDAtMHhmYTQwN2ZmZikgZm9yIHJpZCAxMCBv ZiBwY2kxOjM6MTM6MA0KZm91bmQtPgl2ZW5kb3I9MHgxMDZiLCBkZXY9MHgwMDUyLCByZXZpZD0w eDAwDQoJZG9tYWluPTEsIGJ1cz0zLCBzbG90PTE0LCBmdW5jPTANCgljbGFzcz0wYy0wMC0xMCwg aGRydHlwZT0weDAwLCBtZmRldj0wDQoJY21kcmVnPTB4MDAwMCwgc3RhdHJlZz0weDAyOTAsIGNh Y2hlbG5zej0xNiAoZHdvcmRzKQ0KCWxhdHRpbWVyPTB4ZjggKDc0NDAgbnMpLCBtaW5nbnQ9MHgw YyAoMzAwMCBucyksIG1heGxhdD0weDE4ICg2MDAwIG5zKQ0KCWludHBpbj1hLCBpcnE9MA0KCXBv d2Vyc3BlYyAyICBzdXBwb3J0cyBEMCBEMSBEMiBEMyAgY3VycmVudCBEMA0KCW1hcFsxMF06IHR5 cGUgTWVtb3J5LCByYW5nZSAzMiwgYmFzZSAweGZhNDAwMDAwLCBzaXplIDEyLCBtZW1vcnkgZGlz YWJsZWQNCnBjaWIxMDogYWxsb2NhdGVkIG1lbW9yeSByYW5nZSAoMHhmYTQwMDAwMC0weGZhNDAw ZmZmKSBmb3IgcmlkIDEwIG9mIHBjaTE6MzoxNDowDQphdGFwY2kwOiA8U2VydmVyV29ya3MgSzIg U0FUQTE1MCBjb250cm9sbGVyPiBtZW0gMHhmYTQwMjAwMC0weGZhNDAzZmZmIGlycSAxMCBhdCBk ZXZpY2UgMTIuMCBvbiBwY2kxMA0KcGNpYjE6IGZhaWxlZCB0byByZXNlcnZlIHJlc291cmNlIGZv ciBwY2liMTANCnBjaWIxMDogZmFpbGVkIHRvIGFsbG9jYXRlIGluaXRpYWwgSS9PIHBvcnQgd2lu ZG93ICgwLTB4ZmZmZmZmZmYsMHgxMDAwKQ0KYXRhcGNpMDogMHgxMCBieXRlcyBvZiByaWQgMHgy MCByZXMgNCBmYWlsZWQgKDAsIDB4ZmZmZmZmZmZmZmZmZmZmZikuDQpvZndfcGNpIG1hcGRldjog c3RhcnQgZmE0MDIwMDAsIGxlbiA4MTkyDQphdGEyOiA8QVRBIGNoYW5uZWw+IGF0IGNoYW5uZWwg MCBvbiBhdGFwY2kwDQphdGEzOiA8QVRBIGNoYW5uZWw+IGF0IGNoYW5uZWwgMSBvbiBhdGFwY2kw DQphdGE0OiA8QVRBIGNoYW5uZWw+IGF0IGNoYW5uZWwgMiBvbiBhdGFwY2kwDQphdGE1OiA8QVRB IGNoYW5uZWw+IGF0IGNoYW5uZWwgMyBvbiBhdGFwY2kwDQpvZndfcGNpIG1hcGRldjogc3RhcnQg ZmE0MDQwMDAsIGxlbiAxNjM4NA0KYXRhMDogPFNoYXN0YSBLYXVhaSBBVEEgQ29udHJvbGxlcj4g bWVtIDB4ZmE0MDQwMDAtMHhmYTQwN2ZmZiBpcnEgMzggYXQgZGV2aWNlIDEzLjAgb24gcGNpMTAN CmZ3b2hjaTA6IHZlbmRvcj0xMDZiLCBkZXY9NTINCmZ3b2hjaTA6IHZlbmRvcj0xMDZiLCBkZXY9 NTINCmZ3b2hjaTA6IDwxMzk0IE9wZW4gSG9zdCBDb250cm9sbGVyIEludGVyZmFjZT4gbWVtIDB4 ZmE0MDAwMDAtMHhmYTQwMGZmZiBpcnEgMzkgYXQgZGV2aWNlIDE0LjAgb24gcGNpMTANCm9md19w Y2kgbWFwZGV2OiBzdGFydCBmYTQwMDAwMCwgbGVuIDQwOTYNCmZ3b2hjaTA6IE9IQ0kgdmVyc2lv biAxLjAgKFJPTT0wKQ0KZndvaGNpMDogTm8uIG9mIElzb2Nocm9ub3VzIGNoYW5uZWxzIGlzIDgu DQpmd29oY2kwOiBFVUk2NCAwMDoxNDo1MTpmZjpmZTozMzpjYTpiNg0KZndvaGNpMDogaW52YWxp ZCBzcGVlZCA3IChmaXhlZCB0byAzKS4NCmZ3b2hjaTA6IFBoeSAxMzk0YSBhdmFpbGFibGUgUzgw MCwgMyBwb3J0cy4NCmZ3b2hjaTA6IExpbmsgUzgwMCwgbWF4X3JlYyA0MDk2IGJ5dGVzLg0KZmly ZXdpcmUwOiA8SUVFRTEzOTQoRmlyZVdpcmUpIGJ1cz4gb24gZndvaGNpMA0KZGNvbnNfY3JvbTA6 IDxkY29ucyBjb25maWd1cmF0aW9uIFJPTT4gb24gZmlyZXdpcmUwDQpkY29uc19jcm9tMDogYnVz X2FkZHIgMHgzMGU4YzAwMA0KZndlMDogPEV0aGVybmV0IG92ZXIgRmlyZVdpcmU+IG9uIGZpcmV3 aXJlMA0KaWZfZndlMDogRmFrZSBFdGhlcm5ldCBhZGRyZXNzOiAwMjoxNDo1MTozMzpjYTpiNg0K ZndlMDogYnBmIGF0dGFjaGVkDQpmd2UwOiBFdGhlcm5ldCBhZGRyZXNzOiAwMjoxNDo1MTozMzpj YTpiNg0Kc2JwMDogPFNCUC0yL1NDU0kgb3ZlciBGaXJlV2lyZT4gb24gZmlyZXdpcmUwDQpmd29o Y2kwOiBJbml0aWF0ZSBidXMgcmVzZXQNCmZ3b2hjaTA6IGZ3b2hjaV9pbnRyX2NvcmU6IEJVUyBy ZXNldA0KZndvaGNpMDogZndvaGNpX2ludHJfY29yZTogbm9kZV9pZD0weDAwMDAwMDAxLCBTZWxm SUQgQ291bnQ9MSwgbm9uIENZQ0xFTUFTVEVSIG1vZGUNCnNjMTogbm8gdmlkZW8gYWRhcHRlciBm b3VuZC4NCnNtdTA6IDxBcHBsZSBTeXN0ZW0gTWFuYWdlbWVudCBVbml0PiBvbiBuZXh1czANCnNt dTA6IEZhbjogRFJJVkUgQkFZIEEgSU5UQUtFIHR5cGU6IDANCnNtdTA6IEZhbjogQkFDS1NJREUg dHlwZTogMA0Kc211MDogRmFuOiBDUFUgQSBJTlRBS0UgdHlwZTogMA0Kc211MDogRmFuOiBDUFUg QiBJTlRBS0UgdHlwZTogMA0Kc211MDogRmFuOiBDUFUgQSBFWEhBVVNUIHR5cGU6IDANCnNtdTA6 IEZhbjogQ1BVIEIgRVhIQVVTVCB0eXBlOiAwDQpzbXUwOiBGYW46IEVYUEFOU0lPTiBTTE9UUyBJ TlRBS0UgdHlwZTogMA0Kc211MDogcmVnaXN0ZXJlZCBhcyBhIHRpbWUtb2YtZGF5IGNsb2NrIChy ZXNvbHV0aW9uIDEwMDB1cywgYWRqdXN0bWVudCAwLjAwMDUwMDAwMHMpDQppaWNoYjI6IDxTTVUg STJDIGNvbnRyb2xsZXI+IG9uIHNtdTANCmlpY2J1czI6IDxPRlcgSTJDIGJ1cz4gb24gaWljaGIy DQpzbXVzYXQwOiA8U01VIFNhdGVsbGl0ZSBTZW5zb3JzPiBhdCBhZGRyIDB4YjAgb24gaWljYnVz Mg0KaWljYnVzMjogPHVua25vd24gY2FyZD4gYXQgYWRkciAweGQ0DQppaWNoYjM6IDxTTVUgSTJD IGNvbnRyb2xsZXI+IG9uIHNtdTANCmlpY2J1czM6IDxPRlcgSTJDIGJ1cz4gb24gaWljaGIzDQpz YzE6IG5vIHZpZGVvIGFkYXB0ZXIgZm91bmQuDQpuZXh1czA6IDxhbGlhc2VzPiwgdHlwZSAodW5r bm93bikgKG5vIGRyaXZlciBhdHRhY2hlZCkNCnNjMDogPFN5c3RlbSBjb25zb2xlPiBvbiBuZXh1 czANCnNjMDogVW5rbm93biA8MTYgdmlydHVhbCBjb25zb2xlcywgZmxhZ3M9MHgzMDA+DQpzYzA6 IGZiMCwga2JkMCwgdGVybWluYWwgZW11bGF0b3I6IHNjdGVrZW4gKHRla2VuIHRlcm1pbmFsKQ0K cHJvY2ZzIHJlZ2lzdGVyZWQNClRpbWVjb3VudGVyICJ0aW1lYmFzZSIgZnJlcXVlbmN5IDMzMzMz MzMzIEh6IHF1YWxpdHkgMA0KRXZlbnQgdGltZXIgImRlY3JlbWVudGVyIiBmcmVxdWVuY3kgMzMz MzMzMzMgSHogcXVhbGl0eSAxMDAwDQpUaW1lY291bnRlcnMgdGljayBldmVyeSAxLjAwMCBtc2Vj DQpmaXJld2lyZTA6IDMgbm9kZXMsIG1heGhvcCA8PSAyIGNhYmxlIElSTSBpcm0oMikgDQp2bGFu OiBpbml0aWFsaXplZCwgdXNpbmcgaGFzaCB0YWJsZXMgd2l0aCBjaGFpbmluZw0KdGNwX2luaXQ6 IG5ldC5pbmV0LnRjcC50Y2JoYXNoc2l6ZSBhdXRvIHR1bmVkIHRvIDEzMTA3Mg0KbG8wOiBicGYg YXR0YWNoZWQNCm1heDY2OTAwOiAyIHNlbnNvcnMgZGV0ZWN0ZWQuDQptYXg2NjkwMDogU2Vuc29y cw0KbWF4NjY5MDA6IExvY2F0aW9uIDogQkFDS1NJREUgSUQ6IDYNCm1heDY2OTAwOiBMb2NhdGlv biA6IEtPRElBSyBESU9ERSBJRDogNw0KbWF4NjY5MDE6IDIgc2Vuc29ycyBkZXRlY3RlZC4NCm1h eDY2OTAxOiBTZW5zb3JzDQptYXg2NjkwMTogTG9jYXRpb24gOiBUVU5ORUwgSUQ6IDENCm1heDY2 OTAxOiBMb2NhdGlvbiA6IFRVTk5FTCBIRUFUU0lOSyBJRDogMg0KYmdlMTogbGluayBzdGF0ZSBj aGFuZ2VkIHRvIFVQDQp1c2J1czA6IDEyTWJwcyBGdWxsIFNwZWVkIFVTQiB2MS4wDQp1c2J1czE6 IDEyTWJwcyBGdWxsIFNwZWVkIFVTQiB2MS4wDQp1Z2VuMC4xOiA8TkVDPiBhdCB1c2J1czANCnVo dWIwOiA8TkVDIE9IQ0kgcm9vdCBIVUIsIGNsYXNzIDkvMCwgcmV2IDEuMDAvMS4wMCwgYWRkciAx PiBvbiB1c2J1czANCnVnZW4xLjE6IDxORUM+IGF0IHVzYnVzMQ0KdWh1YjE6IDxORUMgT0hDSSBy b290IEhVQiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAwLCBhZGRyIDE+IG9uIHVzYnVzMQ0KdXNi dXMyOiA0ODBNYnBzIEhpZ2ggU3BlZWQgVVNCIHYyLjANCmF0YTI6IGhhcmQgcmVzZXQgLi4udWdl bjIuMTogPE5FQz4gYXQgdXNidXMyDQp1aHViMjogPE5FQyBFSENJIHJvb3QgSFVCLCBjbGFzcyA5 LzAsIHJldiAyLjAwLzEuMDAsIGFkZHIgMT4gb24gdXNidXMyDQoNCmF0YTI6IFNBVEEgY29ubmVj dCB0aW1lPTEwbXMgc3RhdHVzPTAwMDAwMTEzDQphdGEyOiByZXNldCB0cDEgbWFzaz0wMSBvc3Rh dDA9NTAgb3N0YXQxPTAwDQpmd29oY2kwOiB0eGQgZXJyPSAzIG1pc3MgQWNrIGVycg0KdWh1YjA6 IDMgcG9ydHMgd2l0aCAzIHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkDQp1aHViMTogMiBwb3J0cyB3 aXRoIDIgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQNCmZpcmV3aXJlMDogZndfZXhwbG9yZV9ub2Rl OiBmd2Rldi0+c3BlZWQoUzgwMCkgZGVjcmVtZW50ZWQgZHVlIHRvIG5lZ290aWF0aW9uDQphdGEy OiBzdGF0MD0weDUwIGVycj0weDAxIGxzYj0weDAwIG1zYj0weDAwZmlyZXdpcmUwOiBOZXcgUzQw MCBkZXZpY2UgSUQ6MDAwYTk1ZmZmZTlmN2FlOA0KDQphdGEyOiByZXNldCB0cDIgc3RhdDA9NTAg c3RhdDE9MDAgZGV2aWNlcz0weDENCmF0YTM6IGhhcmQgcmVzZXQgLi4uDQphdGEzOiBTQVRBIGNv bm5lY3QgdGltZW91dCBzdGF0dXM9MDAwMDAwMDQNCmF0YTQ6IGhhcmQgcmVzZXQgLi4uDQpwY20w OiBNaXhlciAidm9sIjoNCmF0YTQ6IFNBVEEgY29ubmVjdCB0aW1lb3V0IHN0YXR1cz0wMDAwMDAw MA0KYXRhNTogaGFyZCByZXNldCAuLi4NCnVodWIyOiA1IHBvcnRzIHdpdGggNSByZW1vdmFibGUs IHNlbGYgcG93ZXJlZA0KYXRhNTogU0FUQSBjb25uZWN0IHRpbWVvdXQgc3RhdHVzPTAwMDAwMDAw DQphdGEwOiByZXNldCB0cDEgbWFzaz0wMyBvc3RhdDA9MDAgb3N0YXQxPTAwDQphdGEwOiBzdGF0 MD0weDAwIGVycj0weDAxIGxzYj0weDE0IG1zYj0weGViDQphdGEwOiBzdGF0MT0weDAwIGVycj0w eDAwIGxzYj0weDAwIG1zYj0weDAwDQphdGEwOiByZXNldCB0cDIgc3RhdDA9MDAgc3RhdDE9MDAg ZGV2aWNlcz0weDEwMDAwDQpwYXNzMCBhdCBhdGEyIGJ1cyAwIHNjYnVzMCB0YXJnZXQgMCBsdW4g MA0KcGFzczA6IDxXREMgV0QyNTAwSlMtNDFNVkIxIDEwLjAyRTAxPiBBVEEtNyBTQVRBIDIueCBk ZXZpY2UNCnBhc3MwOiBTZXJpYWwgTnVtYmVyIFdELVdDQU5ZMTEyMjQ1MA0KcGFzczA6IDE1MC4w MDBNQi9zIHRyYW5zZmVycyAoU0FUQSAxLngsIFVETUE1LCBQSU8gODE5MmJ5dGVzKQ0KcGFzczEg YXQgYXRhMCBidXMgMCBzY2J1czQgdGFyZ2V0IDAgbHVuIDANCnBhc3MxOiA8UElPTkVFUiBEVkQt UlcgIERWUi0xMTBEIEFBMDY+IFJlbW92YWJsZSBDRC1ST00gU0NTSS0wIGRldmljZSANCnBhc3Mx OiBTZXJpYWwgTnVtYmVyIEVJREwwMjY4NTdXTA0KcGFzczE6IDY2LjcwME1CL3MgdHJhbnNmZXJz IChVRE1BNCwgQVRBUEkgMTJieXRlcywgUElPIDY1NTM0Ynl0ZXMpDQphZGEwIGF0IGF0YTIgYnVz IEdFT006IG5ldyBkaXNrIGNkMA0KR0VPTTogbmV3IGRpc2sgYWRhMA0KY2QwIGF0IGF0YTAgYnVz IDAgc2NidXM0IHRhcmdldCAwIGx1biAwDQpjZDA6IDxQSU9ORUVSIERWRC1SVyAgRFZSLTExMEQg QUEwNj4gUmVtb3ZhYmxlIENELVJPTSBTQ1NJLTAgZGV2aWNlIA0KY2QwOiBTZXJpYWwgTnVtYmVy IEVJREwwMjY4NTdXTA0KY2QwOiA2Ni43MDBNQi9zIHRyYW5zZmVycyAoVURNQTQsIEFUQVBJIDEy Ynl0ZXMsIFBJTyA2NTUzNGJ5dGVzKQ0KY2QwOiBBdHRlbXB0IHRvIHF1ZXJ5IGRldmljZSBzaXpl IGZhaWxlZDogTk9UIFJFQURZLCBNZWRpdW0gbm90IHByZXNlbnQNCjAgc2NidXMwIHRhcmdldCAw IGx1biAwDQphZGEwOiA8V0RDIFdEMjUwMEpTLTQxTVZCMSAxMC4wMkUwMT4gQVRBLTcgU0FUQSAy LnggZGV2aWNlDQphZGEwOiBTZXJpYWwgTnVtYmVyIFdELVdDQU5ZMTEyMjQ1MA0KYWRhMDogMTUw LjAwME1CL3MgdHJhbnNmZXJzIChTQVRBIDEueCwgVURNQTUsIFBJTyA4MTkyYnl0ZXMpDQphZGEw OiAyMzg0NzVNQiAoNDg4Mzk3MTY4IDUxMiBieXRlIHNlY3RvcnM6IDF1Z2VuMi4yOiA8dmVuZG9y IDB4MDQyND4gYXQgdXNidXMyDQp1aHViMzogPHZlbmRvciAweDA0MjQgcHJvZHVjdCAweDI1MTQs IGNsYXNzIDkvMCwgcmV2IDIuMDAvMC4wMCwgYWRkciAyPiBvbiB1c2J1czINCjZIIDYzUy91aHVi MzogTVRUIGVuYWJsZWQNClQgMTYzODNDKQ0KYWRhMDogUHJldmlvdXNseSB3YXMga25vd24gYXMg YWQwDQpBZGRpbmcgQ1BVIDAsIHBpcj0wLCBhd2FrZT0xDQpXYWtpbmcgdXAgQ1BVIDEgKGRldj1m Zjg5ZWI3MCkNCkFkZGluZyBDUFUgMSwgcGlyPTEsIGF3YWtlPTENClNNUDogQVAgQ1BVICMxIGxh dW5jaGVkDQp1Z2VuMS4yOiA8TWl0c3VtaSBFbGVjdHJpYz4gYXQgdXNidXMxDQp1aHViNDogPE1p dHN1bWkgRWxlY3RyaWMgSHViIGluIEFwcGxlIEV4dGVuZGVkIFVTQiBLZXlib2FyZCwgY2xhc3Mg OS8wLCByZXYgMS4xMC8xLjIyLCBhZGRyIDI+IG9uIHVzYnVzMQ0KUm9vdCBtb3VudCB3YWl0aW5n IGZvcjogdXNidXMyDQp1aHViNDogMyBwb3J0cyB3aXRoIDIgcmVtb3ZhYmxlLCBidXMgcG93ZXJl ZA0KdWh1YjM6IDQgcG9ydHMgd2l0aCA0IHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkDQpUcnlpbmcg dG8gbW91bnQgcm9vdCBmcm9tIHVmczovZGV2L2FkYTBzMyBbcnddLi4uDQpXQVJOSU5HOiAvIHdh cyBub3QgcHJvcGVybHkgZGlzbW91bnRlZA0KV0FSTklORzogLzogbW91bnQgcGVuZGluZyBlcnJv cjogYmxvY2tzIDAgZmlsZXMgNTYyDQpzdGFydF9pbml0OiB0cnlpbmcgL3NiaW4vaW5pdA0KdWdl bjEuMzogPE1pdHN1bWkgRWxlY3RyaWM+IGF0IHVzYnVzMQ0KdWtiZDA6IDxNaXRzdW1pIEVsZWN0 cmljIEFwcGxlIEV4dGVuZGVkIFVTQiBLZXlib2FyZCwgY2xhc3MgMC8wLCByZXYgMS4xMC8xLjIy LCBhZGRyIDM+IG9uIHVzYnVzMQ0Ka2JkOiBuZXcgYXJyYXkgc2l6ZSA0DQprYmQxIGF0IHVrYmQw DQprYmQxOiB1a2JkMCwgZ2VuZXJpYyAoMCksIGNvbmZpZzoweDAsIGZsYWdzOjB4M2QwMDAwDQp1 aGlkMDogPE1pdHN1bWkgRWxlY3RyaWMgQXBwbGUgRXh0ZW5kZWQgVVNCIEtleWJvYXJkLCBjbGFz cyAwLzAsIHJldiAxLjEwLzEuMjIsIGFkZHIgMz4gb24gdXNidXMxDQp1Z2VuMS40OiA8QnJvYWRj b20gQ29ycD4gYXQgdXNidXMxDQpmd29oY2kwOiBmd29oY2lfaW50cl9jb3JlOiBCVVMgcmVzZXQN CmZ3b2hjaTA6IGZ3b2hjaV9pbnRyX2NvcmU6IG5vZGVfaWQ9MHgwMDAwMDAwMSwgU2VsZklEIENv dW50PTIsIENZQ0xFTUFTVEVSIG1vZGUNCmZpcmV3aXJlMDogMiBub2RlcywgbWF4aG9wIDw9IDEg Y2FibGUgSVJNIGlybSgxKSAgKG1lKSANCmZpcmV3aXJlMDogYnVzIG1hbmFnZXIgMSANCmZpcmV3 aXJlMDogZndfYXR0YWNoX2RldjpSZW1vdmluZyBtaXNzaW5nIGRldmljZSBJRDowMDBhOTVmZmZl OWY3YWU4DQpiZ2UwOiBsaW5rIHN0YXRlIGNoYW5nZWQgdG8gRE9XTg0KYmdlMTogbGluayBzdGF0 ZSBjaGFuZ2VkIHRvIERPV04NCmJnZTE6IGxpbmsgc3RhdGUgY2hhbmdlZCB0byBVUA0KdWJ0MDog PEJyb2FkY29tIENvcnAgQVNVUyBCbHVldG9vdGggRG9uZ2xlLCBjbGFzcyAyMjQvMSwgcmV2IDIu MDAvMS4wMCwgYWRkciA0PiBvbiB1c2J1czENCldBUk5JTkc6IGF0dGVtcHQgdG8gZG9tYWluX2Fk ZChibHVldG9vdGgpIGFmdGVyIGRvbWFpbmZpbmFsaXplKCkNCldBUk5JTkc6IGF0dGVtcHQgdG8g ZG9tYWluX2FkZChuZXRncmFwaCkgYWZ0ZXIgZG9tYWluZmluYWxpemUoKQ0KZndvaGNpMDogZndv aGNpX2ludHJfY29yZTogQlVTIHJlc2V0DQpmd29oY2kwOiBmd29oY2lfaW50cl9jb3JlOiBub2Rl X2lkPTB4MDAwMDAwMDIsIFNlbGZJRCBDb3VudD0zLCBDWUNMRU1BU1RFUiBtb2RlDQpmaXJld2ly ZTA6IDMgbm9kZXMsIG1heGhvcCA8PSAyIGNhYmxlIElSTSBpcm0oMikgIChtZSkgDQpmaXJld2ly ZTA6IGJ1cyBtYW5hZ2VyIDIgDQpmd29oY2kwOiB0eGQgZXJyPSAzIG1pc3MgQWNrIGVycg0KZmly ZXdpcmUwOiBmd19leHBsb3JlX25vZGU6IGZ3ZGV2LT5zcGVlZChTODAwKSBkZWNyZW1lbnRlZCBk dWUgdG8gbmVnb3RpYXRpb24NCmZpcmV3aXJlMDogTmV3IFM0MDAgZGV2aWNlIElEOjAwMGE5NWZm ZmU5ZjdhZTgNClsgdGhyZWFkIHBpZCAtMSB0aWQgMTAwNjY2NTcxOSBdDQpTdG9wcGVkIGF0ICAg ICAgMDogICAgICBpbGxlZ2FsIGluc3RydWN0aW9uIDANCmRiOjA6a2RiLmVudGVyLmRlZmF1bHQ+ IHNob3cgcmVnDQpyMCAgICAgICAgICAgICAgICAgICAwDQpyMSAgICAgICAgICAgICAgICAgICAw DQpyMiAgICAgICAgICAgIDB4YWI2M2QwICBNX01BQ1RFTVANCnIzICAgICAgICAgICAgMHhiYjEy ZTANCnI0ICAgICAgICAgICAgMHg3NDFmMTggIC5vZndjYWxsKzB4YTgNCnI1ICAgICAgICAgICAg ICAgICAgIDANCnI2ICAgICAgICAgICAgMHhhNGYxYTgNCnI3ICAgICAgICAgICAgICAgICAweDEN CnI4ICAgICAgICAgICAgICAgICAweDENCnI5ICAgICAgICAgICAgMHhjMTA1MDAgIF9fcGNwdQ0K cjEwICAgICAgICAgIDB4MWMzNWVjMA0KcjExICAgICAgICAgICAgICAgICAgMA0KcjEyICAgICAg ICAgMHgyMDAwZDAzMg0KcjEzICAgICAgICAgMHgzNDIyZDAwMA0KcjE0ICAgICAgICAgMHgxMDAx NDIwMA0KcjE1ICAgICAgICAgMHhmZmZmZmZmZmZmZmZjYjU4DQpyMTYgICAgICAgICAgICAgICAg MHgyDQpyMTcgICAgICAgICAgICAgICAgMHgyDQpyMTggICAgICAgICAweGZmZmZmZmZmZmZmZmNi NTANCnIxOSAgICAgICAgICAgICAgICAgIDANCnIyMCAgICAgICAgIDB4YzAwMDAwMDAxMzIzMTQ3 OA0KcjIxICAgICAgICAgMHhjMDAwMDAwMTRjMDY5MjAwDQpyMjIgICAgICAgICAgICAgICAgICAw DQpyMjMgICAgICAgICAgICAgICAweDY0ICBkYnNpemUrMHgxMA0KcjI0ICAgICAgICAgMHhjMDAw MDAwMTRjMDY4ZjcwDQpyMjUgICAgICAgICAgIDB4YjYyY2I4ICBzbXBfbm9fcmVuZGV2b3VzX2Jh cnJpZXINCnIyNiAgICAgICAgICAgICAgICAgIDANCnIyNyAgICAgICAgICAgMHg3NDFmMTggIC5v ZndjYWxsKzB4YTgNCnIyOCAgICAgICAgICAgMHg3NDFmMTggIC5vZndjYWxsKzB4YTgNCnIyOSAg ICAgICAgIDB4MjAwMGQwMzINCnIzMCAgICAgICAgIDB4OTAwMDAwMDAwMDAwMTAzMg0KcjMxICAg ICAgICAgICAweGMwY2FkOCAgbWFjX2xhYmVsZWQNCnNycjAgICAgICAgICAgMHgxMDJjYTQgIGtf dHJhcCsweDI4DQpzcnIxICAgICAgICAweDkwMDAwMDAwMDAwMDEwMzINCmxyICAgICAgICAgICAg MHgxMDJjNzQgIHVfdHJhcCsweDEwDQpjdHIgICAgICAgICAweGZmODQ2ZDc4DQpjciAgICAgICAg ICAweDIwMDBmMWIwDQp4ZXIgICAgICAgICAgICAgICAgICAwDQpkYXIgICAgICAgICAweGZmZmZm ZmZmZmZmZmZkNjANCmRzaXNyICAgICAgIDB4NDIwMDAwMDANCjA6ICAgICAgaWxsZWdhbCBpbnN0 cnVjdGlvbiAwDQpkYjowOmtkYi5lbnRlci5kZWZhdWx0PiAgYnQNClRyYWNpbmcgcGlkIC0xIHRp ZCAxMDA2NjY1NzE5IHRkIDANCmRiOjA6a2RiLmVudGVyLmRlZmF1bHQ+ICBzaG93IHBjcHUNCmNw dWlkICAgICAgICA9IDANCmR5bmFtaWMgcGNwdSA9IDB4YmZmZmZmZmZmZjRlYWEwMA0KY3VydGhy ZWFkICAgID0gMHgzNDIyZDAwMDogcGlkIDEwMzMgInBvd2VyZCINCmN1cnBjYiAgICAgICA9IDB4 YzAwMDAwMDE0YzA2OWJkMA0KZnBjdXJ0aHJlYWQgID0gbm9uZQ0KaWRsZXRocmVhZCAgID0gMHgy ZmE1ODQ5MDogdGlkIDEwMDAwMyAiaWRsZTogY3B1MCINClBQQzogaHdyZWYgICA9IDB4ZmY4OWQ2 ODANClBQQzogaXBpbWFzayA9IDANClBQQzogcGlyICAgICA9IDANCmRiOjA6a2RiLmVudGVyLmRl ZmF1bHQ+ICBwcw0KICBwaWQgIHBwaWQgIHBncnAgICB1aWQgICBzdGF0ZSAgIHdtZXNnICAgICAg ICAgd2NoYW4gICAgICAgIGNtZA0KIDI3MjcgIDI3MjYgIDI3MjcgIDEwMDEgIFNzKyAgICAgdHR5 aW4gICAgMHgzMzlkYjhhOCB0Y3NoDQogMjcyNiAgMjcyNCAgMjcyNCAgMTAwMSAgUyAgICAgICBz ZWxlY3QgICAweDNjMWFlZGMwIHNzaGQNCiAyNzI0ICAxMDYwICAyNzI0ICAgICAwICBTcyAgICAg IHNlbGVjdCAgIDB4M2MxYWVjNDAgc3NoZA0KIDExMTkgICAgIDEgICAgIDEgICAgIDAgIFMgICAg ICAgdHR5ZGNkICAgMHgyZmFhMDRlOCBnZXR0eQ0KIDExMTggICAgIDEgIDExMTggICAgIDAgIFNz KyAgICAgdHR5aW4gICAgMHgzMzlkYjBhOCBnZXR0eQ0KIDExMTcgICAgIDEgIDExMTcgICAgIDAg IFNzKyAgICAgdHR5aW4gICAgMHgzMzlkYjRhOCBnZXR0eQ0KIDExMTYgICAgIDEgIDExMTYgICAg IDAgIFNzKyAgICAgdHR5aW4gICAgMHgyZmFhMTRhOCBnZXR0eQ0KIDExMTUgICAgIDEgIDExMTUg ICAgIDAgIFNzKyAgICAgdHR5aW4gICAgMHgyZmFhMThhOCBnZXR0eQ0KIDExMTQgICAgIDEgIDEx MTQgICAgIDAgIFNzKyAgICAgdHR5aW4gICAgMHgyZmFhMWNhOCBnZXR0eQ0KIDExMTMgICAgIDEg IDExMTMgICAgIDAgIFNzKyAgICAgdHR5aW4gICAgMHgzMzlkOTBhOCBnZXR0eQ0KIDExMTIgICAg IDEgIDExMTIgICAgIDAgIFNzKyAgICAgdHR5aW4gICAgMHgzMzlkOTRhOCBnZXR0eQ0KIDExMTEg ICAgIDEgIDExMTEgICAgIDAgIFNzKyAgICAgdHR5aW4gICAgMHgzMzlkOThhOCBnZXR0eQ0KIDEw NzAgICAgIDEgIDEwNzAgICAgIDAgIFNzICAgICAgbmFuc2xwICAgMHhiZTBhYTEgY3Jvbg0KIDEw NjYgICAgIDEgIDEwNjYgICAgMjUgIFNzICAgICAgcGF1c2UgICAgMHgzNDIzZTllOCBzZW5kbWFp bA0KIDEwNjMgICAgIDEgIDEwNjMgICAgIDAgIFNzICAgICAgc2VsZWN0ICAgMHgzNDNlOWRjMCBz ZW5kbWFpbA0KIDEwNjAgICAgIDEgIDEwNjAgICAgIDAgIFNzICAgICAgc2VsZWN0ICAgMHgzNDI5 YjU0MCBzc2hkDQogMTAzMyAgICAgMSAgMTAzMyAgICAgMCAgUnMgICAgICBDUFUgMCAgICAgICAg ICAgICAgICAgICAgICAgcG93ZXJkDQogMTAzMCAgICAgMSAgMTAzMCAgICAgMCAgU3MgICAgICBz ZWxlY3QgICAweDM0M2U4MWMwIG50cGQNCiAgOTU0ICAgICAxICAgOTU0ICAgICAwICBTcyAgICAg IHNlbGVjdCAgIDB4MzQzNzQ1NDAgc3lzbG9nZA0KICA4NDMgICAgIDEgICA4NDMgICAgIDAgIFNz ICAgICAgc2VsZWN0ICAgMHgzYzFhZTljMCBkZXZkDQogIDgwMCAgICAgMCAgICAgMCAgICAgMCAg REwgICAgICAodGhyZWFkZWQpICAgICAgICAgICAgICAgICAgW25nX3F1ZXVlXQ0KMTAwMDc5ICAg ICAgICAgICAgICAgICAgIEQgICAgICAgc2xlZXAgICAgMHhjMDAwMDAwMDE0MmFjMWQwIFtuZ19x dWV1ZTFdDQoxMDAwNjMgICAgICAgICAgICAgICAgICAgRCAgICAgICBzbGVlcCAgICAweGMwMDAw MDAwMTQyYWMxZDAgW25nX3F1ZXVlMF0NCiAgNzk0ICAgICAxICAgNzk0ICAgIDY1ICBTcyAgICAg IHNlbGVjdCAgIDB4MzQzNzQ4NDAgZGhjbGllbnQNCiAgNzU2ICAgICAxICAgNzU2ICAgICAwICBT cyAgICAgIHNlbGVjdCAgIDB4MzQzNzQ3NDAgZGhjbGllbnQNCiAgIDE4ICAgICAwICAgICAwICAg ICAwICBETCAgICAgIHNtdSAgICAgIDB4YzAwMDAwMDE0YzAxNTdiOCBbcG1hY190aGVybWFsXQ0K ICAgMTcgICAgIDAgICAgIDAgICAgIDAgIERMICAgICAgc2RmbHVzaCAgMHhjMGNmMDAgW3NvZnRk ZXBmbHVzaF0NCiAgIDE2ICAgICAwICAgICAwICAgICAwICBETCAgICAgIHN5bmNlciAgIDB4YzAy MjIwIFtzeW5jZXJdDQogICAgOSAgICAgMCAgICAgMCAgICAgMCAgREwgICAgICB2bHJ1d3QgICAw eDM0MWIyMDAwIFt2bmxydV0NCiAgICA4ICAgICAwICAgICAwICAgICAwICBETCAgICAgIHBzbGVl cCAgIDB4YzAxZGE0IFtidWZkYWVtb25dDQogICAgNyAgICAgMCAgICAgMCAgICAgMCAgREwgICAg ICBwZ3plcm8gICAweGMwZjQyNCBbcGFnZXplcm9dDQogICAgNiAgICAgMCAgICAgMCAgICAgMCAg REwgICAgICBwc2xlZXAgICAweGMwZWNlMCBbdm1kYWVtb25dDQogICAgNSAgICAgMCAgICAgMCAg ICAgMCAgREwgICAgICBwc2xlZXAgICAweGMwZWNhOCBbcGFnZWRhZW1vbl0NCiAgICA0ICAgICAw ICAgICAwICAgICAwICBETCAgICAgIGNjYl9zY2FuIDB4YmI5MzQwIFt4cHRfdGhyZF0NCiAgICAz ICAgICAwICAgICAwICAgICAwICBETCAgICAgIHdhaXRpbmdfIDB4YzA3N2MwIFtzY3RwX2l0ZXJh dG9yXQ0KICAgIDIgICAgIDAgICAgIDAgICAgIDAgIFNMICAgICAgLSAgICAgICAgMHhjMDAwMDAw MDEzMmE2MDAwIFtmdzBfcHJvYmVdDQogICAxNSAgICAgMCAgICAgMCAgICAgMCAgREwgICAgICAo dGhyZWFkZWQpICAgICAgICAgICAgICAgICAgW3VzYl0NCjEwMDA0MSAgICAgICAgICAgICAgICAg ICBEICAgICAgIC0gICAgICAgIDB4YzAwMDAwMDAxMzJhM2UxOCBbdXNidXMyXQ0KMTAwMDQwICAg ICAgICAgICAgICAgICAgIEQgICAgICAgLSAgICAgICAgMHhjMDAwMDAwMDEzMmEzZGMwIFt1c2J1 czJdDQoxMDAwMzkgICAgICAgICAgICAgICAgICAgRCAgICAgICAtICAgICAgICAweGMwMDAwMDAw MTMyYTNkNjggW3VzYnVzMl0NCjEwMDAzOCAgICAgICAgICAgICAgICAgICBEICAgICAgIC0gICAg ICAgIDB4YzAwMDAwMDAxMzJhM2QxMCBbdXNidXMyXQ0KMTAwMDM3ICAgICAgICAgICAgICAgICAg IEQgICAgICAgLSAgICAgICAgMHhjMDAwMDAwMDEzMjliNDYwIFt1c2J1czFdDQoxMDAwMzYgICAg ICAgICAgICAgICAgICAgRCAgICAgICAtICAgICAgICAweGMwMDAwMDAwMTMyOWI0MDggW3VzYnVz MV0NCjEwMDAzNSAgICAgICAgICAgICAgICAgICBEICAgICAgIC0gICAgICAgIDB4YzAwMDAwMDAx MzI5YjNiMCBbdXNidXMxXQ0KMTAwMDM0ICAgICAgICAgICAgICAgICAgIEQgICAgICAgLSAgICAg ICAgMHhjMDAwMDAwMDEzMjliMzU4IFt1c2J1czFdDQoxMDAwMzMgICAgICAgICAgICAgICAgICAg RCAgICAgICAtICAgICAgICAweGMwMDAwMDAwMTMyOTg0NjAgW3VzYnVzMF0NCjEwMDAzMiAgICAg ICAgICAgICAgICAgICBEICAgICAgIC0gICAgICAgIDB4YzAwMDAwMDAxMzI5ODQwOCBbdXNidXMw XQ0KMTAwMDMxICAgICAgICAgICAgICAgICAgIEQgICAgICAgLSAgICAgICAgMHhjMDAwMDAwMDEz Mjk4M2IwIFt1c2J1czBdDQoxMDAwMzAgICAgICAgICAgICAgICAgICAgRCAgICAgICAtICAgICAg ICAweGMwMDAwMDAwMTMyOTgzNTggW3VzYnVzMF0NCiAgIDE0ICAgICAwICAgICAwICAgICAwICBE TCAgICAgIC0gICAgICAgIDB4YmMxMzkwIFt5YXJyb3ddDQogICAxMyAgICAgMCAgICAgMCAgICAg MCAgREwgICAgICAodGhyZWFkZWQpICAgICAgICAgICAgICAgICAgW2dlb21dDQoxMDAwMTEgICAg ICAgICAgICAgICAgICAgRCAgICAgICAtICAgICAgICAweGJkYjAzOCBbZ19kb3duXQ0KMTAwMDEw ICAgICAgICAgICAgICAgICAgIEQgICAgICAgLSAgICAgICAgMHhiZGIwMzAgW2dfdXBdDQoxMDAw MDkgICAgICAgICAgICAgICAgICAgRCAgICAgICAtICAgICAgICAweGJkYjAyMCBbZ19ldmVudF0N CiAgIDEyICAgICAwICAgICAwICAgICAwICBXTCAgICAgICh0aHJlYWRlZCkgICAgICAgICAgICAg ICAgICBbaW50cl0NCjEwMDA0NyAgICAgICAgICAgICAgICAgICBJICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBbaXJxNTM6IHNtdWRvb3JiZWxsMF0NCjEwMDA0NCAgICAgICAgICAg ICAgICAgICBJICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbaXJxMzk6IGZ3b2hj aTBdDQoxMDAwNDMgICAgICAgICAgICAgICAgICAgSSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgW2lycTM4OiBhdGEwXQ0KMTAwMDQyICAgICAgICAgICAgICAgICAgIEkgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtpcnExMDogYXRhcGNpMF0NCjEwMDAyOSAgICAg ICAgICAgICAgICAgICBJICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbaXJxNzA6 IG9oY2kwIG9oY2kxK10NCjEwMDAyOCAgICAgICAgICAgICAgICAgICBJICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBbaXJxMTE6IHBjbTBdDQoxMDAwMjcgICAgICAgICAgICAgICAg ICAgSSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW2lycTI3OiBpaWNoYjFdDQox MDAwMjYgICAgICAgICAgICAgICAgICAgSSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgW3N3aTA6IHVhcnQgdWFydF0NCjEwMDAyNSAgICAgICAgICAgICAgICAgICBJICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBbaXJxNzU6IGFvYWdwaW80XQ0KMTAwMDI0ICAgICAg ICAgICAgICAgICAgIEkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtpcnE3OTog YW9hZ3BpbzFdDQoxMDAwMjMgICAgICAgICAgICAgICAgICAgSSAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgW2lycTk6IGJnZTFdDQoxMDAwMjIgICAgICAgICAgICAgICAgICAgSSAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW2lycTg6IGJnZTBdDQoxMDAwMjEgICAg ICAgICAgICAgICAgICAgSSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW2lycTA6 IGlpY2hiMF0NCjEwMDAyMCAgICAgICAgICAgICAgICAgICBJICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBbc3dpMjogY2FtYmlvXQ0KMTAwMDE4ICAgICAgICAgICAgICAgICAgIEkg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtzd2k2OiB0YXNrIHF1ZXVlXQ0KMTAw MDE3ICAgICAgICAgICAgICAgICAgIEkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IFtzd2k2OiBHaWFudCB0YXNrcV0NCjEwMDAxNCAgICAgICAgICAgICAgICAgICBJICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBbc3dpNTogZmFzdCB0YXNrcV0NCjEwMDAwOCAgICAg ICAgICAgICAgICAgICBJICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbc3dpMTog bmV0aXNyIDBdDQoxMDAwMDcgICAgICAgICAgICAgICAgICAgSSAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgW3N3aTM6IHZtXQ0KMTAwMDA2ICAgICAgICAgICAgICAgICAgIEkgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtzd2k0OiBjbG9ja10NCjEwMDAwNSAgICAg ICAgICAgICAgICAgICBJICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbc3dpNDog Y2xvY2tdDQogICAxMSAgICAgMCAgICAgMCAgICAgMCAgUkwgICAgICAodGhyZWFkZWQpICAgICAg ICAgICAgICAgICAgW2lkbGVdDQoxMDAwMDQgICAgICAgICAgICAgICAgICAgUnVuICAgICBDUFUg MSAgICAgICAgICAgICAgICAgICAgICAgW2lkbGU6IGNwdTFdDQoxMDAwMDMgICAgICAgICAgICAg ICAgICAgQ2FuUnVuICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW2lkbGU6IGNwdTBdDQog ICAgMSAgICAgMCAgICAgMSAgICAgMCAgU0xzICAgICB3YWl0ICAgICAweDJmYTU2OTQwIFtpbml0 XQ0KICAgMTAgICAgIDAgICAgIDAgICAgIDAgIERMICAgICAgYXVkaXRfd28gMHhjMGMxMDggW2F1 ZGl0XQ0KICAgIDAgICAgIDAgICAgIDAgICAgIDAgIERMcyAgICAgKHRocmVhZGVkKSAgICAgICAg ICAgICAgICAgIFtrZXJuZWxdDQoxMDAwNDUgICAgICAgICAgICAgICAgICAgRCAgICAgICAtICAg ICAgICAweDMzOTQwYTAwIFtmdzBfdGFza3FdDQoxMDAwMTkgICAgICAgICAgICAgICAgICAgRCAg ICAgICAtICAgICAgICAweDJmYTczYjAwIFtrcXVldWUgdGFza3FdDQoxMDAwMTYgICAgICAgICAg ICAgICAgICAgRCAgICAgICAtICAgICAgICAweDJmYjVlMjAwIFt0aHJlYWQgdGFza3FdDQoxMDAw MTUgICAgICAgICAgICAgICAgICAgRCAgICAgICAtICAgICAgICAweDJmYTRiMjAwIFtmZnNfdHJp bSB0YXNrcV0NCjEwMDAxMiAgICAgICAgICAgICAgICAgICBEICAgICAgIC0gICAgICAgIDB4MmZh NGEyMDAgW2Zpcm13YXJlIHRhc2txXQ0KMTAwMDAwICAgICAgICAgICAgICAgICAgIEQgICAgICAg c2NoZWQgICAgMHhiZGI2MjAgW3N3YXBwZXJdDQpkYjowOmtkYi5lbnRlci5kZWZhdWx0PiAgcnVu IGxvY2tpbmZvDQpkYjoxOmxvY2tpbmZvPiBzaG93IGxvY2tzDQpObyBzdWNoIGNvbW1hbmQNCmRi OjE6bG9ja3M+ICBzaG93IGFsbGxvY2tzDQpObyBzdWNoIGNvbW1hbmQNCmRiOjE6YWxsbG9ja3M+ ICBzaG93IGxvY2tlZHZub2RzDQpMb2NrZWQgdm5vZGVzDQoNCltkY29uc2NoYXQgZXhpdGluZy4u Ll0NCgpTY3JpcHQgZG9uZSBvbiBTdW4gSnVuICA5IDEzOjE3OjEzIDIwMTMK --90e6ba25db9365b67f04debf3ebf-- From owner-freebsd-ppc@FreeBSD.ORG Sun Jun 9 22:40:13 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 43CFFC55; Sun, 9 Jun 2013 22:40:13 +0000 (UTC) (envelope-from superbisquit@gmail.com) Received: from mail-oa0-x22f.google.com (mail-oa0-x22f.google.com [IPv6:2607:f8b0:4003:c02::22f]) by mx1.freebsd.org (Postfix) with ESMTP id E9FC51956; Sun, 9 Jun 2013 22:40:12 +0000 (UTC) Received: by mail-oa0-f47.google.com with SMTP id m1so3083026oag.6 for ; Sun, 09 Jun 2013 15:40:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Y51sPaZkR36gCvfwD39dWMRx2gWFkcCatnDCR95cqOc=; b=Mt6s1VIVf/OPiyihTjgvEAyejfOVkNsKOOgfkbiRxZ1WEAJ6OKw0NPS7rTCR2wi26V k6r2rIVjSxTq0JfqkSkMj0gRMPKwcrQSZ/9IFjQk0a0F/hAv2Z8vBY0WIoF8hY06jVuU k4f8fWl9VsSg13TX2CkJKMAWwPEY2yWD2MhXjWPVWcV9r50zljQBYYCwUiYTt7ZPHNI9 14PFpORbWFAaW26SPJu3pFI2YDLXJiZV5Jmt5yIgfp+Pv6PXdEyWFLiQU9ZevBgorHHS 5kx/cPAEgiJpdi3pJQ1GDgwxbvpSaJqugCdJCWqOnWcMO3LgpiNFhd7hXniv5oBZZTdu +kBA== MIME-Version: 1.0 X-Received: by 10.60.62.103 with SMTP id x7mr5925184oer.6.1370817611915; Sun, 09 Jun 2013 15:40:11 -0700 (PDT) Received: by 10.182.53.231 with HTTP; Sun, 9 Jun 2013 15:40:11 -0700 (PDT) In-Reply-To: References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> Date: Sun, 9 Jun 2013 18:40:11 -0400 Message-ID: Subject: Re: Strange panic on ppc64 From: Super Bisquit To: Justin Hibbits , Nathan Whitehorn , FreeBSD PowerPC ML Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 22:40:13 -0000 See if you have enough time to ssh into the box. It may not be so stupid of an idea. You could monitor from another machine. I'm not sure if the gnome desktop recording works with ssh but that may help. Again, this is just an idea. On Sun, Jun 9, 2013 at 5:21 PM, Justin Hibbits wrote: > On Sun, Jun 9, 2013 at 8:47 AM, Nathan Whitehorn >wrote: > > > On 06/08/13 17:33, Justin Hibbits wrote: > > > > > > > > > > On Sat, Jun 8, 2013 at 7:54 AM, Nathan Whitehorn >wrote: > > > >> On 06/08/13 09:21, Justin Hibbits wrote: > >> > >> > >> > >> > >> On Wed, Jun 5, 2013 at 9:47 AM, Justin Hibbits >wrote: > >> > >>> Will do, when I get it panicking again. > >>> > >>> - Justin > >>> On Jun 5, 2013 9:46 AM, "Nathan Whitehorn" > >>> wrote: > >>> > >>>> On 06/04/13 22:35, Justin Hibbits wrote: > >>>> > >>>>> After a string of seemingly random hangs, I added invariants (but not > >>>>> witness) to my custom kernel config, and I get the following panic, > >>>>> recreated from a fuzzy cell phone picture: > >>>>> > >>>>> > >>>>> [thread pid -1 tid 1006665719 ] > >>>>> Stopped at 0: illegal instruction 0 > >>>>> db> panic: mutex ohci1 owned at > >>>>> /usr/home/chmeee/freebsd/head/sys/dev/usb/usb_transfer.c:2280 > >>>>> cpuid = 0 > >>>>> Uptime: 9h8m1s > >>>>> > >>>>> ... > >>>>> panic: msleep1 > >>>>> cpu = 0 > >>>>> KDB: enter: panic > >>>>> [ thread pid -1 tid 100665719 ] > >>>>> .... > >>>>> > >>>>> The first question I have is how the hell it got such a strange > >>>>> PID/TID, > >>>>> memory corruption my guess, something is stomping on the pcpu or > >>>>> something, > >>>>> and I think these hangs have only happened since I added a lot more > >>>>> memory > >>>>> (up to 12G from 4G, Andreas Tobler was seeing hangs as well), so it > >>>>> might > >>>>> be something in the moea64 pmap code, but that's pure speculation on > my > >>>>> part. Then the other panic messages, owned mutex and panic in > >>>>> msleep1. I > >>>>> enabled more trace code, so hopefully the next time it panics I can > >>>>> collect > >>>>> better data. > >>>>> > >>>>> - Justin > >>>>> _______________________________________________ > >>>>> freebsd-ppc@freebsd.org mailing list > >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-ppc > >>>>> To unsubscribe, send any mail to " > freebsd-ppc-unsubscribe@freebsd.org" > >>>>> > >>>> > >>>> Could you post the output from show reg? It looks like it tried to > jump > >>>> to a null pointer there. > >>>> -Nathan > >>>> > >>> > >> Well, it's hard to do get that output, because I just hit that 'mutex > >> owned' panic, and here's the backtrace: > >> > >> > >> > >> The mutex thing is spurious -- it was already panicing and then paniced > >> again trying to panic. Can you get the backtrace for the original panic > (it > >> should be different) and the values of the registers? > >> -Nathan > >> > > > > Here you go: > > > > [ thread pid -1 tid 1006665719 ] > > Stopped at 0: illegal instruction 0 > > db:0:kdb.enter.default> show reg > > r0 0 > > r1 0 > > r2 0xab63d0 M_MACTEMP > > r3 0xbb12e0 > > r4 0x741f18 .ofwcall+0xa8 > > r5 0 > > r6 0xa4f1a8 > > r7 0x1 > > r8 0x1 > > r9 0xc10500 __pcpu > > r10 0x1c35ec0 > > r11 0 > > r12 0x2000d032 > > r13 0x342eb000 > > r14 0x10014200 > > r15 0xffffffffffffcb58 > > r16 0x2 > > r17 0x2 > > r18 0xffffffffffffcb50 > > r19 0 > > r20 0xc000000013231478 > > r21 0xc00000014c0ce200 > > r22 0 > > r23 0x64 dbsize+0x10 > > r24 0xc00000014c0cdf70 > > r25 0xb62cb8 smp_no_rendevous_barrier > > r26 0 > > r27 0x741f18 .ofwcall+0xa8 > > r28 0x741f18 .ofwcall+0xa8 > > r29 0x2000d032 > > r30 0x9000000000001032 > > r31 0xc0cad8 mac_labeled > > srr0 0x102ca4 k_trap+0x28 > > srr1 0x9000000000001032 > > lr 0x102c74 u_trap+0x10 > > ctr 0xff846d78 > > cr 0x2000f1b0 > > xer 0 > > dar 0xfffffffffffffd60 > > dsisr 0x42000000 > > 0: illegal instruction 0 > > db:0:kdb.enter.default> bt > > Tracing pid -1 tid 1006665719 td 0 > > (nothing) > > > > > > Well, that is all kinds of messed up. It appears to have halted while > > handling a userland trap due to an implicit branch caused by bad > > translations when it restores the kernel SRs. Could you see what 'show > > pcpu' does? Does that information look valid at all? I suspect it has > > become corrupted somehow. > > -Nathan > > > > > Here's the full log from dconschat, from bootup to panic. Unfortunately, > not everything I wanted to print would print, and I can't type anything > once it panics, because it panics when reading the keyboard, so I have to > add everything as a ddb enter script. Here's what I've added so far > (doesn't do everything as you can see from the transcript): > > script kdb.enter.default=show reg; bt; show pcpu; ps; run lockinfo; > alltrace; show all procs; show files; show malloc; show allchains > > - Justin > > _______________________________________________ > freebsd-ppc@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ppc > To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org" > From owner-freebsd-ppc@FreeBSD.ORG Sun Jun 9 22:52:24 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9FAF6234; Sun, 9 Jun 2013 22:52:24 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-bk0-x22f.google.com (mail-bk0-x22f.google.com [IPv6:2a00:1450:4008:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id E7D3E1AC9; Sun, 9 Jun 2013 22:52:23 +0000 (UTC) Received: by mail-bk0-f47.google.com with SMTP id jg1so2787739bkc.20 for ; Sun, 09 Jun 2013 15:52:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=m44sUnQjzs3HAjejg3GQ/yAhRYIY3rcTYRr5JoVAbr8=; b=NvHvfRKjqmp6j2IEBzEP5szZXNwPjFM5bh96LPT1AI8kKc8BPsvMs+aCcfQauynaX5 sWAa7xcJIvgXMYPDX2ZaHZFZDUWfl9mAAvxBU1QY2ECFpQ3+Fav+rJgU3/fnhStQIG3h cD+8djJZqnH8RNYche8RoNwRAzpozGtDmo8pyELymik7fu/heH4JasvIhIz1rjpamk1Z wd9sEjz2m+GsIL4caj9htzthVkd6qT5HJOo1Z7UFtSaXGd7QQ/kyinD4PkRepSfPGzJH TxoAylsC4C6DGyofe9C2wmmEHFEGtkeAngGlYPQEiLNvSBuSfqCbRSOzF7/XKAp+6XJe rAeA== MIME-Version: 1.0 X-Received: by 10.205.36.138 with SMTP id ta10mr1123211bkb.4.1370818342993; Sun, 09 Jun 2013 15:52:22 -0700 (PDT) Sender: chmeeedalf@gmail.com Received: by 10.204.236.132 with HTTP; Sun, 9 Jun 2013 15:52:22 -0700 (PDT) In-Reply-To: References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> Date: Sun, 9 Jun 2013 15:52:22 -0700 X-Google-Sender-Auth: dg9P9WX0ixKQCpeaY24mh0IoRjQ Message-ID: Subject: Re: Strange panic on ppc64 From: Justin Hibbits To: Super Bisquit Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 22:52:24 -0000 On Sun, Jun 9, 2013 at 3:40 PM, Super Bisquit wrote: > See if you have enough time to ssh into the box. It may not be so stupid > of an idea. You could monitor from another machine. I'm not sure if the > gnome desktop recording works with ssh but that may help. Again, this is > just an idea. > > > On Sun, Jun 9, 2013 at 5:21 PM, Justin Hibbits wrote: > >> On Sun, Jun 9, 2013 at 8:47 AM, Nathan Whitehorn > >wrote: >> >> > On 06/08/13 17:33, Justin Hibbits wrote: >> > >> > >> > >> > >> > On Sat, Jun 8, 2013 at 7:54 AM, Nathan Whitehorn < >> nwhitehorn@freebsd.org>wrote: >> > >> >> On 06/08/13 09:21, Justin Hibbits wrote: >> >> >> >> >> >> >> >> >> >> On Wed, Jun 5, 2013 at 9:47 AM, Justin Hibbits > >wrote: >> >> >> >>> Will do, when I get it panicking again. >> >>> >> >>> - Justin >> >>> On Jun 5, 2013 9:46 AM, "Nathan Whitehorn" >> >>> wrote: >> >>> >> >>>> On 06/04/13 22:35, Justin Hibbits wrote: >> >>>> >> >>>>> After a string of seemingly random hangs, I added invariants (but >> not >> >>>>> witness) to my custom kernel config, and I get the following panic, >> >>>>> recreated from a fuzzy cell phone picture: >> >>>>> >> >>>>> >> >>>>> [thread pid -1 tid 1006665719 ] >> >>>>> Stopped at 0: illegal instruction 0 >> >>>>> db> panic: mutex ohci1 owned at >> >>>>> /usr/home/chmeee/freebsd/head/sys/dev/usb/usb_transfer.c:2280 >> >>>>> cpuid = 0 >> >>>>> Uptime: 9h8m1s >> >>>>> >> >>>>> ... >> >>>>> panic: msleep1 >> >>>>> cpu = 0 >> >>>>> KDB: enter: panic >> >>>>> [ thread pid -1 tid 100665719 ] >> >>>>> .... >> >>>>> >> >>>>> The first question I have is how the hell it got such a strange >> >>>>> PID/TID, >> >>>>> memory corruption my guess, something is stomping on the pcpu or >> >>>>> something, >> >>>>> and I think these hangs have only happened since I added a lot more >> >>>>> memory >> >>>>> (up to 12G from 4G, Andreas Tobler was seeing hangs as well), so it >> >>>>> might >> >>>>> be something in the moea64 pmap code, but that's pure speculation >> on my >> >>>>> part. Then the other panic messages, owned mutex and panic in >> >>>>> msleep1. I >> >>>>> enabled more trace code, so hopefully the next time it panics I can >> >>>>> collect >> >>>>> better data. >> >>>>> >> >>>>> - Justin >> >>>>> _______________________________________________ >> >>>>> freebsd-ppc@freebsd.org mailing list >> >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-ppc >> >>>>> To unsubscribe, send any mail to " >> freebsd-ppc-unsubscribe@freebsd.org" >> >>>>> >> >>>> >> >>>> Could you post the output from show reg? It looks like it tried to >> jump >> >>>> to a null pointer there. >> >>>> -Nathan >> >>>> >> >>> >> >> Well, it's hard to do get that output, because I just hit that 'mutex >> >> owned' panic, and here's the backtrace: >> >> >> >> >> >> >> >> The mutex thing is spurious -- it was already panicing and then >> paniced >> >> again trying to panic. Can you get the backtrace for the original >> panic (it >> >> should be different) and the values of the registers? >> >> -Nathan >> >> >> > >> > Here you go: >> > >> > [ thread pid -1 tid 1006665719 ] >> > Stopped at 0: illegal instruction 0 >> > db:0:kdb.enter.default> show reg >> > r0 0 >> > r1 0 >> > r2 0xab63d0 M_MACTEMP >> > r3 0xbb12e0 >> > r4 0x741f18 .ofwcall+0xa8 >> > r5 0 >> > r6 0xa4f1a8 >> > r7 0x1 >> > r8 0x1 >> > r9 0xc10500 __pcpu >> > r10 0x1c35ec0 >> > r11 0 >> > r12 0x2000d032 >> > r13 0x342eb000 >> > r14 0x10014200 >> > r15 0xffffffffffffcb58 >> > r16 0x2 >> > r17 0x2 >> > r18 0xffffffffffffcb50 >> > r19 0 >> > r20 0xc000000013231478 >> > r21 0xc00000014c0ce200 >> > r22 0 >> > r23 0x64 dbsize+0x10 >> > r24 0xc00000014c0cdf70 >> > r25 0xb62cb8 smp_no_rendevous_barrier >> > r26 0 >> > r27 0x741f18 .ofwcall+0xa8 >> > r28 0x741f18 .ofwcall+0xa8 >> > r29 0x2000d032 >> > r30 0x9000000000001032 >> > r31 0xc0cad8 mac_labeled >> > srr0 0x102ca4 k_trap+0x28 >> > srr1 0x9000000000001032 >> > lr 0x102c74 u_trap+0x10 >> > ctr 0xff846d78 >> > cr 0x2000f1b0 >> > xer 0 >> > dar 0xfffffffffffffd60 >> > dsisr 0x42000000 >> > 0: illegal instruction 0 >> > db:0:kdb.enter.default> bt >> > Tracing pid -1 tid 1006665719 td 0 >> > (nothing) >> > >> > >> > Well, that is all kinds of messed up. It appears to have halted while >> > handling a userland trap due to an implicit branch caused by bad >> > translations when it restores the kernel SRs. Could you see what 'show >> > pcpu' does? Does that information look valid at all? I suspect it has >> > become corrupted somehow. >> > -Nathan >> > >> > >> Here's the full log from dconschat, from bootup to panic. Unfortunately, >> not everything I wanted to print would print, and I can't type anything >> once it panics, because it panics when reading the keyboard, so I have to >> add everything as a ddb enter script. Here's what I've added so far >> (doesn't do everything as you can see from the transcript): >> >> script kdb.enter.default=show reg; bt; show pcpu; ps; run lockinfo; >> alltrace; show all procs; show files; show malloc; show allchains >> >> - Justin >> >> _______________________________________________ >> freebsd-ppc@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-ppc >> To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org" >> > > I do ssh into the box. I can run a buildworld just fine, but as soon as it finishes (and sometimes before it finishes) it panics. If I don't put much pressure on the VM (sit it idle, read man pages, etc), it can last for days. Heavy building, like what I've been doing using poudriere (a very nice piece of software I might add), it crashes. It could simply be bad RAM, since it never crashed until I added my new RAM. However, that's nearly impossible to accurately test, so I'd like to exhaust every other problem before tackling that idea. - Justin From owner-freebsd-ppc@FreeBSD.ORG Mon Jun 10 02:12:30 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 061755B2; Mon, 10 Jun 2013 02:12:30 +0000 (UTC) (envelope-from superbisquit@gmail.com) Received: from mail-ob0-x231.google.com (mail-ob0-x231.google.com [IPv6:2607:f8b0:4003:c01::231]) by mx1.freebsd.org (Postfix) with ESMTP id 9E41215F5; Mon, 10 Jun 2013 02:12:29 +0000 (UTC) Received: by mail-ob0-f177.google.com with SMTP id ta17so9110292obb.8 for ; Sun, 09 Jun 2013 19:12:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=QwN2CQ/T3fqM5mftxO1/63sYY1EveMRn0coccoUcccU=; b=dUSe6b85Sg9qorbrh9/zRyjETQLQBk3I4flJjIUT+H7+o0/2svUR3R3m4WZ502RDFk a0m8qLVFYvo/XgySbcXZuEAUUZ7DURFOa06duZ3dFdYhbatBIXWLwL7lmeVonORP+W8Y ZjVT9xGIdhfFlaGf2HI2QEsnuhdYvkh2jNPnOBwMXGdXbDWPSPwotm1e6r9uvP/ehz/c ZmBT+O09Ac5ohgWfGTt01ivTddGgZKmyAd2YhxzwZvql8KcbzlILbMlHJZhKCRUJYl9t ZBvK60s/B4L+yFNRPfb3kQZoZ78DeEl51p4xb18geDwyAZFxEWxeDuM0I92SK+kNSWsR /FkQ== MIME-Version: 1.0 X-Received: by 10.182.214.39 with SMTP id nx7mr6493038obc.20.1370830349277; Sun, 09 Jun 2013 19:12:29 -0700 (PDT) Received: by 10.182.53.231 with HTTP; Sun, 9 Jun 2013 19:12:29 -0700 (PDT) In-Reply-To: <20130610003923.GA97977@FreeBSD.org> References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> <20130610003923.GA97977@FreeBSD.org> Date: Sun, 9 Jun 2013 22:12:29 -0400 Message-ID: Subject: Re: Strange panic on ppc64 From: Super Bisquit To: FreeBSD PowerPC ML Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Justin Hibbits X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 02:12:30 -0000 Okay. Now I understand. You mean redundancy. Justin, try finding what the build limit is before it crashes \ Start small then build up. Hmmm. It's possible that Linux LiveCD may have the tools for testing RAM. Fedora is an option On Sun, Jun 9, 2013 at 8:39 PM, Alexey Dokuchaev wrote: > Please stop overquoting, thank you all. > > ./danfe > From owner-freebsd-ppc@FreeBSD.ORG Mon Jun 10 11:02:43 2013 Return-Path: Delivered-To: powerpc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6938DA81; Mon, 10 Jun 2013 11:02:43 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id 358691BC0; Mon, 10 Jun 2013 11:02:42 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id r5AB2giM009307; Mon, 10 Jun 2013 07:02:42 -0400 (EDT) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id r5AB2gLt009303; Mon, 10 Jun 2013 11:02:42 GMT (envelope-from tinderbox@freebsd.org) Date: Mon, 10 Jun 2013 11:02:42 GMT Message-Id: <201306101102.r5AB2gLt009303@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on powerpc/powerpc Precedence: bulk X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 11:02:43 -0000 TB --- 2013-06-10 10:23:38 - tinderbox 2.10 running on freebsd-current.sentex.ca TB --- 2013-06-10 10:23:38 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-10 10:23:38 - starting HEAD tinderbox run for powerpc/powerpc TB --- 2013-06-10 10:23:38 - cleaning the object tree TB --- 2013-06-10 10:23:38 - /usr/local/bin/svn stat /src TB --- 2013-06-10 10:23:42 - At svn revision 251600 TB --- 2013-06-10 10:23:43 - building world TB --- 2013-06-10 10:23:43 - CROSS_BUILD_TESTING=YES TB --- 2013-06-10 10:23:43 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-10 10:23:43 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-10 10:23:43 - SRCCONF=/dev/null TB --- 2013-06-10 10:23:43 - TARGET=powerpc TB --- 2013-06-10 10:23:43 - TARGET_ARCH=powerpc TB --- 2013-06-10 10:23:43 - TZ=UTC TB --- 2013-06-10 10:23:43 - __MAKE_CONF=/dev/null TB --- 2013-06-10 10:23:43 - cd /src TB --- 2013-06-10 10:23:43 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Mon Jun 10 10:23:49 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries [...] *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop in /src. TB --- 2013-06-10 11:02:42 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-10 11:02:42 - ERROR: failed to build world TB --- 2013-06-10 11:02:42 - 1799.02 user 335.40 system 2343.96 real http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-powerpc-powerpc.full From owner-freebsd-ppc@FreeBSD.ORG Mon Jun 10 11:06:54 2013 Return-Path: Delivered-To: freebsd-ppc@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0CC276D for ; Mon, 10 Jun 2013 11:06:54 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id F1D0D1C93 for ; Mon, 10 Jun 2013 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5AB6rLb097066 for ; Mon, 10 Jun 2013 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5AB6r7W097064 for freebsd-ppc@FreeBSD.org; Mon, 10 Jun 2013 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 10 Jun 2013 11:06:53 GMT Message-Id: <201306101106.r5AB6r7W097064@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ppc@FreeBSD.org Subject: Current problem reports assigned to freebsd-ppc@FreeBSD.org X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 11:06:54 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o power/178038 ppc clang++ fails to produce a binary in powerpc64 o power/175151 ppc Loader fails to load modules on PPC64 o power/170340 ppc Enabling geom-mirror on 9.1-powerpc64 prevents the sys o power/164225 ppc Boot fails on IBM 7028-6E1 (heap memory claim failed) o power/161947 ppc multimedia/libdvdnav builds broken binaries on big end o power/161045 ppc X doesn't detect ADB mouse up event until another even o power/149009 ppc sysinstall(8) on powerpc fails to install manpages, so o power/140241 ppc [kernel] [patch] Linker set problems on PowerPC EABI o power/135576 ppc gdb cannot debug threaded programs on ppc o power/133503 ppc [sound] Sound stutter after switching ttys o power/131548 ppc ofw_syscons no longer supports 32-bit framebuffer 11 problems total. From owner-freebsd-ppc@FreeBSD.ORG Mon Jun 10 13:20:20 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 01CBD9F7; Mon, 10 Jun 2013 13:20:20 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from smtpauth2.wiscmail.wisc.edu (wmauth2.doit.wisc.edu [144.92.197.222]) by mx1.freebsd.org (Postfix) with ESMTP id C372617AC; Mon, 10 Jun 2013 13:20:19 +0000 (UTC) MIME-version: 1.0 Received: from avs-daemon.smtpauth2.wiscmail.wisc.edu by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MO600300HXFOA00@smtpauth2.wiscmail.wisc.edu>; Mon, 10 Jun 2013 08:20:13 -0500 (CDT) X-Spam-PmxInfo: Server=avs-2, Version=6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2013.6.10.130936, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from wanderer.tachypleus.net (unknown [70.42.94.222]) by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MO600KVLIDOYC00@smtpauth2.wiscmail.wisc.edu>; Mon, 10 Jun 2013 08:20:13 -0500 (CDT) Date: Mon, 10 Jun 2013 08:20:12 -0500 From: Nathan Whitehorn Subject: Re: Strange panic on ppc64 In-reply-to: To: Justin Hibbits Message-id: <51B5D28C.505@freebsd.org> References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130420 Thunderbird/17.0.5 Content-Type: text/plain; CHARSET=US-ASCII Content-Transfer-Encoding: 7BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 13:20:20 -0000 On 06/09/13 16:21, Justin Hibbits wrote: > On Sun, Jun 9, 2013 at 8:47 AM, Nathan Whitehorn > > wrote: > > On 06/08/13 17:33, Justin Hibbits wrote: >> >> >> >> On Sat, Jun 8, 2013 at 7:54 AM, Nathan Whitehorn >> > wrote: >> >> On 06/08/13 09:21, Justin Hibbits wrote: >>> >>> >>> >>> On Wed, Jun 5, 2013 at 9:47 AM, Justin Hibbits >>> > wrote: >>> >>> Will do, when I get it panicking again. >>> >>> - Justin >>> >>> On Jun 5, 2013 9:46 AM, "Nathan Whitehorn" >>> > >>> wrote: >>> >>> On 06/04/13 22:35, Justin Hibbits wrote: >>> >>> After a string of seemingly random hangs, I >>> added invariants (but not >>> witness) to my custom kernel config, and I get >>> the following panic, >>> recreated from a fuzzy cell phone picture: >>> >>> >>> [thread pid -1 tid 1006665719 ] >>> Stopped at 0: illegal instruction 0 >>> db> panic: mutex ohci1 owned at >>> /usr/home/chmeee/freebsd/head/sys/dev/usb/usb_transfer.c:2280 >>> cpuid = 0 >>> Uptime: 9h8m1s >>> >>> ... >>> panic: msleep1 >>> cpu = 0 >>> KDB: enter: panic >>> [ thread pid -1 tid 100665719 ] >>> .... >>> >>> The first question I have is how the hell it got >>> such a strange PID/TID, >>> memory corruption my guess, something is >>> stomping on the pcpu or something, >>> and I think these hangs have only happened since >>> I added a lot more memory >>> (up to 12G from 4G, Andreas Tobler was seeing >>> hangs as well), so it might >>> be something in the moea64 pmap code, but that's >>> pure speculation on my >>> part. Then the other panic messages, owned >>> mutex and panic in msleep1. I >>> enabled more trace code, so hopefully the next >>> time it panics I can collect >>> better data. >>> >>> - Justin >>> _______________________________________________ >>> freebsd-ppc@freebsd.org >>> mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-ppc >>> To unsubscribe, send any mail to >>> "freebsd-ppc-unsubscribe@freebsd.org >>> " >>> >>> >>> Could you post the output from show reg? It looks >>> like it tried to jump to a null pointer there. >>> -Nathan >>> >>> >>> Well, it's hard to do get that output, because I just hit >>> that 'mutex owned' panic, and here's the backtrace: >> >> >> The mutex thing is spurious -- it was already panicing and >> then paniced again trying to panic. Can you get the backtrace >> for the original panic (it should be different) and the >> values of the registers? >> -Nathan >> >> >> Here you go: >> >> [ thread pid -1 tid 1006665719 ] >> Stopped at 0: illegal instruction 0 >> db:0:kdb.enter.default> show reg >> r0 0 >> r1 0 >> r2 0xab63d0 M_MACTEMP >> r3 0xbb12e0 >> r4 0x741f18 .ofwcall+0xa8 >> r5 0 >> r6 0xa4f1a8 >> r7 0x1 >> r8 0x1 >> r9 0xc10500 __pcpu >> r10 0x1c35ec0 >> r11 0 >> r12 0x2000d032 >> r13 0x342eb000 >> r14 0x10014200 >> r15 0xffffffffffffcb58 >> r16 0x2 >> r17 0x2 >> r18 0xffffffffffffcb50 >> r19 0 >> r20 0xc000000013231478 >> r21 0xc00000014c0ce200 >> r22 0 >> r23 0x64 dbsize+0x10 >> r24 0xc00000014c0cdf70 >> r25 0xb62cb8 smp_no_rendevous_barrier >> r26 0 >> r27 0x741f18 .ofwcall+0xa8 >> r28 0x741f18 .ofwcall+0xa8 >> r29 0x2000d032 >> r30 0x9000000000001032 >> r31 0xc0cad8 mac_labeled >> srr0 0x102ca4 k_trap+0x28 >> srr1 0x9000000000001032 >> lr 0x102c74 u_trap+0x10 >> ctr 0xff846d78 >> cr 0x2000f1b0 >> xer 0 >> dar 0xfffffffffffffd60 >> dsisr 0x42000000 >> 0: illegal instruction 0 >> db:0:kdb.enter.default> bt >> Tracing pid -1 tid 1006665719 td 0 >> (nothing) > > Well, that is all kinds of messed up. It appears to have halted > while handling a userland trap due to an implicit branch caused by > bad translations when it restores the kernel SRs. Could you see > what 'show pcpu' does? Does that information look valid at all? I > suspect it has become corrupted somehow. > -Nathan > > > Here's the full log from dconschat, from bootup to panic. > Unfortunately, not everything I wanted to print would print, and I > can't type anything once it panics, because it panics when reading the > keyboard, so I have to add everything as a ddb enter script. Here's > what I've added so far (doesn't do everything as you can see from the > transcript): > > script kdb.enter.default=show reg; bt; show pcpu; ps; run > lockinfo; alltrace; show all procs; show files; show malloc; show > allchains > > - Justin This is now getting interesting. Reading the tea leaves, what has happened is that the kernel has called into Open Firmware. Open Firmware has then crashed early on, before setting up its own trap handlers, which has then flung you back into FreeBSD's handlers with a totally bogus environment, causing a second panic, which then causes a *third* panic when trying to acquire a lock. It would be interesting to know what the OF environment looked like and what commands it was trying to execute (in r3), but that may be tricky to get... -Nathan From owner-freebsd-ppc@FreeBSD.ORG Mon Jun 10 13:31:41 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id AE151F5B; Mon, 10 Jun 2013 13:31:41 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from smtpauth2.wiscmail.wisc.edu (wmauth2.doit.wisc.edu [144.92.197.222]) by mx1.freebsd.org (Postfix) with ESMTP id 871B91857; Mon, 10 Jun 2013 13:31:41 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from avs-daemon.smtpauth2.wiscmail.wisc.edu by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MO600700IHLIK00@smtpauth2.wiscmail.wisc.edu>; Mon, 10 Jun 2013 08:31:41 -0500 (CDT) X-Spam-PmxInfo: Server=avs-2, Version=6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2013.6.10.131821, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from wanderer.tachypleus.net (unknown [70.42.94.222]) by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MO600KBDIWPYC10@smtpauth2.wiscmail.wisc.edu>; Mon, 10 Jun 2013 08:31:38 -0500 (CDT) Message-id: <51B5D539.8050102@freebsd.org> Date: Mon, 10 Jun 2013 08:31:37 -0500 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130420 Thunderbird/17.0.5 To: Justin Hibbits Subject: Re: Strange panic on ppc64 References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> <51B5D28C.505@freebsd.org> In-reply-to: <51B5D28C.505@freebsd.org> Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 13:31:41 -0000 On 06/10/13 08:20, Nathan Whitehorn wrote: > On 06/09/13 16:21, Justin Hibbits wrote: >> On Sun, Jun 9, 2013 at 8:47 AM, Nathan Whitehorn >> > wrote: >> >> On 06/08/13 17:33, Justin Hibbits wrote: >>> >>> >>> On Sat, Jun 8, 2013 at 7:54 AM, Nathan Whitehorn >>> > wrote: >>> >>> On 06/08/13 09:21, Justin Hibbits wrote: >>>> >>>> >>>> On Wed, Jun 5, 2013 at 9:47 AM, Justin Hibbits >>>> > wrote: >>>> >>>> Will do, when I get it panicking again. >>>> >>>> - Justin >>>> >>>> On Jun 5, 2013 9:46 AM, "Nathan Whitehorn" >>>> > >>>> wrote: >>>> >>>> On 06/04/13 22:35, Justin Hibbits wrote: >>>> >>>> After a string of seemingly random hangs, I >>>> added invariants (but not >>>> witness) to my custom kernel config, and I get >>>> the following panic, >>>> recreated from a fuzzy cell phone picture: >>>> >>>> >>>> [thread pid -1 tid 1006665719 ] >>>> Stopped at 0: illegal instruction 0 >>>> db> panic: mutex ohci1 owned at >>>> /usr/home/chmeee/freebsd/head/sys/dev/usb/usb_transfer.c:2280 >>>> cpuid = 0 >>>> Uptime: 9h8m1s >>>> >>>> ... >>>> panic: msleep1 >>>> cpu = 0 >>>> KDB: enter: panic >>>> [ thread pid -1 tid 100665719 ] >>>> .... >>>> >>>> The first question I have is how the hell it got >>>> such a strange PID/TID, >>>> memory corruption my guess, something is >>>> stomping on the pcpu or something, >>>> and I think these hangs have only happened since >>>> I added a lot more memory >>>> (up to 12G from 4G, Andreas Tobler was seeing >>>> hangs as well), so it might >>>> be something in the moea64 pmap code, but that's >>>> pure speculation on my >>>> part. Then the other panic messages, owned >>>> mutex and panic in msleep1. I >>>> enabled more trace code, so hopefully the next >>>> time it panics I can collect >>>> better data. >>>> >>>> - Justin >>>> _______________________________________________ >>>> freebsd-ppc@freebsd.org >>>> mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-ppc >>>> To unsubscribe, send any mail to >>>> "freebsd-ppc-unsubscribe@freebsd.org >>>> " >>>> >>>> >>>> Could you post the output from show reg? It looks >>>> like it tried to jump to a null pointer there. >>>> -Nathan >>>> >>>> >>>> Well, it's hard to do get that output, because I just hit >>>> that 'mutex owned' panic, and here's the backtrace: >>> >>> The mutex thing is spurious -- it was already panicing and >>> then paniced again trying to panic. Can you get the backtrace >>> for the original panic (it should be different) and the >>> values of the registers? >>> -Nathan >>> >>> >>> Here you go: >>> >>> [ thread pid -1 tid 1006665719 ] >>> Stopped at 0: illegal instruction 0 >>> db:0:kdb.enter.default> show reg >>> r0 0 >>> r1 0 >>> r2 0xab63d0 M_MACTEMP >>> r3 0xbb12e0 >>> r4 0x741f18 .ofwcall+0xa8 >>> r5 0 >>> r6 0xa4f1a8 >>> r7 0x1 >>> r8 0x1 >>> r9 0xc10500 __pcpu >>> r10 0x1c35ec0 >>> r11 0 >>> r12 0x2000d032 >>> r13 0x342eb000 >>> r14 0x10014200 >>> r15 0xffffffffffffcb58 >>> r16 0x2 >>> r17 0x2 >>> r18 0xffffffffffffcb50 >>> r19 0 >>> r20 0xc000000013231478 >>> r21 0xc00000014c0ce200 >>> r22 0 >>> r23 0x64 dbsize+0x10 >>> r24 0xc00000014c0cdf70 >>> r25 0xb62cb8 smp_no_rendevous_barrier >>> r26 0 >>> r27 0x741f18 .ofwcall+0xa8 >>> r28 0x741f18 .ofwcall+0xa8 >>> r29 0x2000d032 >>> r30 0x9000000000001032 >>> r31 0xc0cad8 mac_labeled >>> srr0 0x102ca4 k_trap+0x28 >>> srr1 0x9000000000001032 >>> lr 0x102c74 u_trap+0x10 >>> ctr 0xff846d78 >>> cr 0x2000f1b0 >>> xer 0 >>> dar 0xfffffffffffffd60 >>> dsisr 0x42000000 >>> 0: illegal instruction 0 >>> db:0:kdb.enter.default> bt >>> Tracing pid -1 tid 1006665719 td 0 >>> (nothing) >> Well, that is all kinds of messed up. It appears to have halted >> while handling a userland trap due to an implicit branch caused by >> bad translations when it restores the kernel SRs. Could you see >> what 'show pcpu' does? Does that information look valid at all? I >> suspect it has become corrupted somehow. >> -Nathan >> >> >> Here's the full log from dconschat, from bootup to panic. >> Unfortunately, not everything I wanted to print would print, and I >> can't type anything once it panics, because it panics when reading the >> keyboard, so I have to add everything as a ddb enter script. Here's >> what I've added so far (doesn't do everything as you can see from the >> transcript): >> >> script kdb.enter.default=show reg; bt; show pcpu; ps; run >> lockinfo; alltrace; show all procs; show files; show malloc; show >> allchains >> >> - Justin > This is now getting interesting. Reading the tea leaves, what has > happened is that the kernel has called into Open Firmware. Open Firmware > has then crashed early on, before setting up its own trap handlers, > which has then flung you back into FreeBSD's handlers with a totally > bogus environment, causing a second panic, which then causes a *third* > panic when trying to acquire a lock. It would be interesting to know > what the OF environment looked like and what commands it was trying to > execute (in r3), but that may be tricky to get... > -Nathan > _______________________________________________ One other point: you can trace this pretty easily by just putting something like: if (pmap_bootstrapped) printf("Open Firmware call %p\n", args); in the top of openfirmware(). If I understood the debugger output correctly, something should be making a firmware call immediately before the crash. As a random guess about what is happening, it is possible OF is trying to allocate memory for itself. We just ignore the possibility that it might want to do that at present, but that is not necessarily a good assumption. -Nathan From owner-freebsd-ppc@FreeBSD.ORG Mon Jun 10 14:17:12 2013 Return-Path: Delivered-To: powerpc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 589635E0; Mon, 10 Jun 2013 14:17:12 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id 22CD51ABC; Mon, 10 Jun 2013 14:17:11 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id r5AEHB6w053722; Mon, 10 Jun 2013 10:17:11 -0400 (EDT) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id r5AEHBb5053721; Mon, 10 Jun 2013 14:17:11 GMT (envelope-from tinderbox@freebsd.org) Date: Mon, 10 Jun 2013 14:17:11 GMT Message-Id: <201306101417.r5AEHBb5053721@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on powerpc/powerpc Precedence: bulk X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 14:17:12 -0000 TB --- 2013-06-10 13:37:16 - tinderbox 2.10 running on freebsd-current.sentex.ca TB --- 2013-06-10 13:37:16 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-10 13:37:16 - starting HEAD tinderbox run for powerpc/powerpc TB --- 2013-06-10 13:37:16 - cleaning the object tree TB --- 2013-06-10 13:38:14 - /usr/local/bin/svn stat /src TB --- 2013-06-10 13:38:19 - At svn revision 251600 TB --- 2013-06-10 13:38:20 - building world TB --- 2013-06-10 13:38:20 - CROSS_BUILD_TESTING=YES TB --- 2013-06-10 13:38:20 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-10 13:38:20 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-10 13:38:20 - SRCCONF=/dev/null TB --- 2013-06-10 13:38:20 - TARGET=powerpc TB --- 2013-06-10 13:38:20 - TARGET_ARCH=powerpc TB --- 2013-06-10 13:38:20 - TZ=UTC TB --- 2013-06-10 13:38:20 - __MAKE_CONF=/dev/null TB --- 2013-06-10 13:38:20 - cd /src TB --- 2013-06-10 13:38:20 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Mon Jun 10 13:38:28 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries [...] *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop in /src. TB --- 2013-06-10 14:17:11 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-10 14:17:11 - ERROR: failed to build world TB --- 2013-06-10 14:17:11 - 1797.88 user 340.08 system 2394.42 real http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-powerpc-powerpc.full From owner-freebsd-ppc@FreeBSD.ORG Mon Jun 10 17:36:52 2013 Return-Path: Delivered-To: powerpc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DD1C1F71; Mon, 10 Jun 2013 17:36:52 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id A92C617C2; Mon, 10 Jun 2013 17:36:52 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id r5AHapcm098026; Mon, 10 Jun 2013 13:36:51 -0400 (EDT) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id r5AHapve097998; Mon, 10 Jun 2013 17:36:51 GMT (envelope-from tinderbox@freebsd.org) Date: Mon, 10 Jun 2013 17:36:51 GMT Message-Id: <201306101736.r5AHapve097998@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on powerpc/powerpc Precedence: bulk X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 17:36:53 -0000 TB --- 2013-06-10 16:56:31 - tinderbox 2.10 running on freebsd-current.sentex.ca TB --- 2013-06-10 16:56:31 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-10 16:56:31 - starting HEAD tinderbox run for powerpc/powerpc TB --- 2013-06-10 16:56:31 - cleaning the object tree TB --- 2013-06-10 16:57:40 - /usr/local/bin/svn stat /src TB --- 2013-06-10 16:57:45 - At svn revision 251603 TB --- 2013-06-10 16:57:46 - building world TB --- 2013-06-10 16:57:46 - CROSS_BUILD_TESTING=YES TB --- 2013-06-10 16:57:46 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-10 16:57:46 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-10 16:57:46 - SRCCONF=/dev/null TB --- 2013-06-10 16:57:46 - TARGET=powerpc TB --- 2013-06-10 16:57:46 - TARGET_ARCH=powerpc TB --- 2013-06-10 16:57:46 - TZ=UTC TB --- 2013-06-10 16:57:46 - __MAKE_CONF=/dev/null TB --- 2013-06-10 16:57:46 - cd /src TB --- 2013-06-10 16:57:46 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Mon Jun 10 16:57:53 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries [...] *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop in /src. TB --- 2013-06-10 17:36:51 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-10 17:36:51 - ERROR: failed to build world TB --- 2013-06-10 17:36:51 - 1800.62 user 337.37 system 2420.61 real http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-powerpc-powerpc.full From owner-freebsd-ppc@FreeBSD.ORG Tue Jun 11 03:33:11 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9944D418; Tue, 11 Jun 2013 03:33:11 +0000 (UTC) (envelope-from superbisquit@gmail.com) Received: from mail-ob0-x230.google.com (mail-ob0-x230.google.com [IPv6:2607:f8b0:4003:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id 4ACA11891; Tue, 11 Jun 2013 03:33:11 +0000 (UTC) Received: by mail-ob0-f176.google.com with SMTP id v19so11107657obq.21 for ; Mon, 10 Jun 2013 20:33:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=wY878D0FRfqzjnaYoSFnCSPgUiOseFDOjw7zPWs+Jq0=; b=sVqSGuw6j3k8qbEirBV72WVLh/pux307RzwTdS3IWShDy9uC+RJU59onAwKlKcvRK5 TrbpdyHpzFpxuEuhc5/8CQ7U+FctqkK+Mdk7gO44odLeoLRKeK4C3/OemYr7jbBrI1cv 84l8X7LIFOtovSrtyBMxYNvjJze37TWVOuqsygJphEvYrsJc7m8xJ5hcJRHq50j40evq qdTnKEdWorZmELssSGGT0IHFCUZVFu1rljT/szkgs77I2b5P6IS4dnFXAzxaXhJIhfW4 Re4ljYw3Qi7NXOa4TtplurJXB6TY2jxdEowHirrQLmFM6JDL6zuem5sMbhhyDoGxH6H9 LixA== MIME-Version: 1.0 X-Received: by 10.182.65.100 with SMTP id w4mr10378973obs.70.1370921590898; Mon, 10 Jun 2013 20:33:10 -0700 (PDT) Received: by 10.182.53.231 with HTTP; Mon, 10 Jun 2013 20:33:10 -0700 (PDT) In-Reply-To: <51B5D539.8050102@freebsd.org> References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> <51B5D28C.505@freebsd.org> <51B5D539.8050102@freebsd.org> Date: Mon, 10 Jun 2013 23:33:10 -0400 Message-ID: Subject: Re: Strange panic on ppc64 From: Super Bisquit To: Nathan Whitehorn Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Justin Hibbits , FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jun 2013 03:33:11 -0000 1. Is the Open Firmware located on the CPU itself or is it part of the logic board? 2. Would he be able to flash a newer version from a remote machine? The answers to these two questions may help myself and others in the near future. Thanks, Desmond On Mon, Jun 10, 2013 at 9:31 AM, Nathan Whitehorn wrote: > On 06/10/13 08:20, Nathan Whitehorn wrote: > > On 06/09/13 16:21, Justin Hibbits wrote: > >> On Sun, Jun 9, 2013 at 8:47 AM, Nathan Whitehorn > >> > wrote: > >> > >> On 06/08/13 17:33, Justin Hibbits wrote: > >>> > >>> > >>> On Sat, Jun 8, 2013 at 7:54 AM, Nathan Whitehorn > >>> > wrote: > >>> > >>> On 06/08/13 09:21, Justin Hibbits wrote: > >>>> > >>>> > >>>> On Wed, Jun 5, 2013 at 9:47 AM, Justin Hibbits > >>>> > wrote: > >>>> > >>>> Will do, when I get it panicking again. > >>>> > >>>> - Justin > >>>> > >>>> On Jun 5, 2013 9:46 AM, "Nathan Whitehorn" > >>>> > > >>>> wrote: > >>>> > >>>> On 06/04/13 22:35, Justin Hibbits wrote: > >>>> > >>>> After a string of seemingly random hangs, I > >>>> added invariants (but not > >>>> witness) to my custom kernel config, and I get > >>>> the following panic, > >>>> recreated from a fuzzy cell phone picture: > >>>> > >>>> > >>>> [thread pid -1 tid 1006665719 ] > >>>> Stopped at 0: illegal instruction 0 > >>>> db> panic: mutex ohci1 owned at > >>>> > /usr/home/chmeee/freebsd/head/sys/dev/usb/usb_transfer.c:2280 > >>>> cpuid = 0 > >>>> Uptime: 9h8m1s > >>>> > >>>> ... > >>>> panic: msleep1 > >>>> cpu = 0 > >>>> KDB: enter: panic > >>>> [ thread pid -1 tid 100665719 ] > >>>> .... > >>>> > >>>> The first question I have is how the hell it got > >>>> such a strange PID/TID, > >>>> memory corruption my guess, something is > >>>> stomping on the pcpu or something, > >>>> and I think these hangs have only happened since > >>>> I added a lot more memory > >>>> (up to 12G from 4G, Andreas Tobler was seeing > >>>> hangs as well), so it might > >>>> be something in the moea64 pmap code, but that's > >>>> pure speculation on my > >>>> part. Then the other panic messages, owned > >>>> mutex and panic in msleep1. I > >>>> enabled more trace code, so hopefully the next > >>>> time it panics I can collect > >>>> better data. > >>>> > >>>> - Justin > >>>> _______________________________________________ > >>>> freebsd-ppc@freebsd.org > >>>> mailing list > >>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-ppc > >>>> To unsubscribe, send any mail to > >>>> "freebsd-ppc-unsubscribe@freebsd.org > >>>> " > >>>> > >>>> > >>>> Could you post the output from show reg? It looks > >>>> like it tried to jump to a null pointer there. > >>>> -Nathan > >>>> > >>>> > >>>> Well, it's hard to do get that output, because I just hit > >>>> that 'mutex owned' panic, and here's the backtrace: > >>> > >>> The mutex thing is spurious -- it was already panicing and > >>> then paniced again trying to panic. Can you get the backtrace > >>> for the original panic (it should be different) and the > >>> values of the registers? > >>> -Nathan > >>> > >>> > >>> Here you go: > >>> > >>> [ thread pid -1 tid 1006665719 ] > >>> Stopped at 0: illegal instruction 0 > >>> db:0:kdb.enter.default> show reg > >>> r0 0 > >>> r1 0 > >>> r2 0xab63d0 M_MACTEMP > >>> r3 0xbb12e0 > >>> r4 0x741f18 .ofwcall+0xa8 > >>> r5 0 > >>> r6 0xa4f1a8 > >>> r7 0x1 > >>> r8 0x1 > >>> r9 0xc10500 __pcpu > >>> r10 0x1c35ec0 > >>> r11 0 > >>> r12 0x2000d032 > >>> r13 0x342eb000 > >>> r14 0x10014200 > >>> r15 0xffffffffffffcb58 > >>> r16 0x2 > >>> r17 0x2 > >>> r18 0xffffffffffffcb50 > >>> r19 0 > >>> r20 0xc000000013231478 > >>> r21 0xc00000014c0ce200 > >>> r22 0 > >>> r23 0x64 dbsize+0x10 > >>> r24 0xc00000014c0cdf70 > >>> r25 0xb62cb8 smp_no_rendevous_barrier > >>> r26 0 > >>> r27 0x741f18 .ofwcall+0xa8 > >>> r28 0x741f18 .ofwcall+0xa8 > >>> r29 0x2000d032 > >>> r30 0x9000000000001032 > >>> r31 0xc0cad8 mac_labeled > >>> srr0 0x102ca4 k_trap+0x28 > >>> srr1 0x9000000000001032 > >>> lr 0x102c74 u_trap+0x10 > >>> ctr 0xff846d78 > >>> cr 0x2000f1b0 > >>> xer 0 > >>> dar 0xfffffffffffffd60 > >>> dsisr 0x42000000 > >>> 0: illegal instruction 0 > >>> db:0:kdb.enter.default> bt > >>> Tracing pid -1 tid 1006665719 td 0 > >>> (nothing) > >> Well, that is all kinds of messed up. It appears to have halted > >> while handling a userland trap due to an implicit branch caused by > >> bad translations when it restores the kernel SRs. Could you see > >> what 'show pcpu' does? Does that information look valid at all? I > >> suspect it has become corrupted somehow. > >> -Nathan > >> > >> > >> Here's the full log from dconschat, from bootup to panic. > >> Unfortunately, not everything I wanted to print would print, and I > >> can't type anything once it panics, because it panics when reading the > >> keyboard, so I have to add everything as a ddb enter script. Here's > >> what I've added so far (doesn't do everything as you can see from the > >> transcript): > >> > >> script kdb.enter.default=show reg; bt; show pcpu; ps; run > >> lockinfo; alltrace; show all procs; show files; show malloc; show > >> allchains > >> > >> - Justin > > This is now getting interesting. Reading the tea leaves, what has > > happened is that the kernel has called into Open Firmware. Open Firmware > > has then crashed early on, before setting up its own trap handlers, > > which has then flung you back into FreeBSD's handlers with a totally > > bogus environment, causing a second panic, which then causes a *third* > > panic when trying to acquire a lock. It would be interesting to know > > what the OF environment looked like and what commands it was trying to > > execute (in r3), but that may be tricky to get... > > -Nathan > > _______________________________________________ > > One other point: you can trace this pretty easily by just putting > something like: > > if (pmap_bootstrapped) printf("Open Firmware call %p\n", args); > > in the top of openfirmware(). If I understood the debugger output > correctly, something should be making a firmware call immediately before > the crash. > > As a random guess about what is happening, it is possible OF is trying > to allocate memory for itself. We just ignore the possibility that it > might want to do that at present, but that is not necessarily a good > assumption. > -Nathan > _______________________________________________ > freebsd-ppc@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ppc > To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org" > From owner-freebsd-ppc@FreeBSD.ORG Tue Jun 11 04:20:38 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 77DE6944; Tue, 11 Jun 2013 04:20:38 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-bk0-x235.google.com (mail-bk0-x235.google.com [IPv6:2a00:1450:4008:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id D3CF1199C; Tue, 11 Jun 2013 04:20:37 +0000 (UTC) Received: by mail-bk0-f53.google.com with SMTP id e11so3084017bkh.40 for ; Mon, 10 Jun 2013 21:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=4bSlMJywfx8M1mNzH2YydM/Ow6ZMEfUr1emSAVWpPqY=; b=cLBncvhuLMlW9psvWW2T7UKawsJvbjFtxSkLmgj/ekh25RlnWELqiNtQwsQF4vWyLF 0eS/KzmW2jDYFAQWA+83d4x/emwWnVC9zv7Zv6W7a/Qf2WrtUH5lA8iWrGCHpqevNbzt acYvzUNZkXRmz2IUxcuWMrgV5JyqD9K5FAkz4ruDBW+q9BXS+txl4lIt9Ov9sSn0ZeFK Y6MuNMK2V4je4+S+mgOhiybwP9C8hQQUO7RrrY1FjsrMRh85riG/L5QyF/H2wljyfEsM 9x/hdF760i/0HhdEqPK3m2FYa8kvzOYlYO5gTXH8MjlIeYkeT1q5eC72CzuqX5BV3TgK MTBw== MIME-Version: 1.0 X-Received: by 10.204.169.137 with SMTP id z9mr2004779bky.33.1370924436842; Mon, 10 Jun 2013 21:20:36 -0700 (PDT) Sender: chmeeedalf@gmail.com Received: by 10.204.236.132 with HTTP; Mon, 10 Jun 2013 21:20:36 -0700 (PDT) In-Reply-To: <51B5D539.8050102@freebsd.org> References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> <51B5D28C.505@freebsd.org> <51B5D539.8050102@freebsd.org> Date: Mon, 10 Jun 2013 21:20:36 -0700 X-Google-Sender-Auth: LxkD-hI-O6Piq-voJqCP2nU6vf8 Message-ID: Subject: Re: Strange panic on ppc64 From: Justin Hibbits To: Nathan Whitehorn Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jun 2013 04:20:38 -0000 On Mon, Jun 10, 2013 at 6:31 AM, Nathan Whitehorn wrote: > On 06/10/13 08:20, Nathan Whitehorn wrote: > > This is now getting interesting. Reading the tea leaves, what has > > happened is that the kernel has called into Open Firmware. Open Firmware > > has then crashed early on, before setting up its own trap handlers, > > which has then flung you back into FreeBSD's handlers with a totally > > bogus environment, causing a second panic, which then causes a *third* > > panic when trying to acquire a lock. It would be interesting to know > > what the OF environment looked like and what commands it was trying to > > execute (in r3), but that may be tricky to get... > > -Nathan > > _______________________________________________ > > One other point: you can trace this pretty easily by just putting > something like: > > if (pmap_bootstrapped) printf("Open Firmware call %p\n", args); > > in the top of openfirmware(). If I understood the debugger output > correctly, something should be making a firmware call immediately before > the crash. > > As a random guess about what is happening, it is possible OF is trying > to allocate memory for itself. We just ignore the possibility that it > might want to do that at present, but that is not necessarily a good > assumption. > -Nathan > I added that, both on entry and exit. I also have it printing out the name of the ofw call, since the first item is always a pointer to the name. I'll be able to report more tomorrow. - Justin From owner-freebsd-ppc@FreeBSD.ORG Wed Jun 12 21:24:24 2013 Return-Path: Delivered-To: powerpc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AD93BF28; Wed, 12 Jun 2013 21:24:24 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id 60E81133B; Wed, 12 Jun 2013 21:24:23 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id r5CLONOE095034; Wed, 12 Jun 2013 17:24:23 -0400 (EDT) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id r5CLON9q095031; Wed, 12 Jun 2013 21:24:23 GMT (envelope-from tinderbox@freebsd.org) Date: Wed, 12 Jun 2013 21:24:23 GMT Message-Id: <201306122124.r5CLON9q095031@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on powerpc/powerpc Precedence: bulk X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2013 21:24:24 -0000 TB --- 2013-06-12 19:08:31 - tinderbox 2.10 running on freebsd-current.sentex.ca TB --- 2013-06-12 19:08:31 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-12 19:08:31 - starting HEAD tinderbox run for powerpc/powerpc TB --- 2013-06-12 19:08:31 - cleaning the object tree TB --- 2013-06-12 19:08:31 - /usr/local/bin/svn stat /src TB --- 2013-06-12 19:08:35 - At svn revision 251658 TB --- 2013-06-12 19:08:36 - building world TB --- 2013-06-12 19:08:36 - CROSS_BUILD_TESTING=YES TB --- 2013-06-12 19:08:36 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-12 19:08:36 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-12 19:08:36 - SRCCONF=/dev/null TB --- 2013-06-12 19:08:36 - TARGET=powerpc TB --- 2013-06-12 19:08:36 - TARGET_ARCH=powerpc TB --- 2013-06-12 19:08:36 - TZ=UTC TB --- 2013-06-12 19:08:36 - __MAKE_CONF=/dev/null TB --- 2013-06-12 19:08:36 - cd /src TB --- 2013-06-12 19:08:36 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Wed Jun 12 19:08:43 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything [...] *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop in /src. TB --- 2013-06-12 21:24:23 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-12 21:24:23 - ERROR: failed to build world TB --- 2013-06-12 21:24:23 - 7038.75 user 905.14 system 8152.06 real http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-powerpc-powerpc.full From owner-freebsd-ppc@FreeBSD.ORG Thu Jun 13 03:30:58 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CFE99CF; Thu, 13 Jun 2013 03:30:58 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-ie0-x230.google.com (mail-ie0-x230.google.com [IPv6:2607:f8b0:4001:c03::230]) by mx1.freebsd.org (Postfix) with ESMTP id 967501566; Thu, 13 Jun 2013 03:30:58 +0000 (UTC) Received: by mail-ie0-f176.google.com with SMTP id ar20so22748626iec.21 for ; Wed, 12 Jun 2013 20:30:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=DN7S2LZXQh2ose8r5/PNzmmuroaoxwrLfKvNcndA6mk=; b=CZS8/9WmgBwY7v9+64CUQfm51nLuBFVw0TqnTB2nPTR86XLwjFgYG7l9FdmKG7jQyW PqZpsSlqn8YAXxlHunUzs6FCE1/6Y5stFXMZ7qWzlRBk9uhfhEdm1IRFEtZCisDjzoNs lOwBlz/iMJKIc0P01B85q+zboJ3nyApNVVOik9s9F/HI39+e3z7Z/mtcWNhdt+1xYyXA MUflXQwtQjhlQd+dCVNDT5g0bmn2ckz9fFGKQiuDVnsF8JDlZQWL8XyK9mettMb1eZp1 OMGuRH3mgNB9SIAIG+IQAtorzaXgjRr0pzsi0h8JwrtwuteTgr4HWlsQajufWeIitweo QKGA== MIME-Version: 1.0 X-Received: by 10.50.50.195 with SMTP id e3mr4801117igo.109.1371094258357; Wed, 12 Jun 2013 20:30:58 -0700 (PDT) Sender: chmeeedalf@gmail.com Received: by 10.64.33.11 with HTTP; Wed, 12 Jun 2013 20:30:58 -0700 (PDT) In-Reply-To: References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> <51B5D28C.505@freebsd.org> <51B5D539.8050102@freebsd.org> Date: Wed, 12 Jun 2013 20:30:58 -0700 X-Google-Sender-Auth: EgWH-x35g2GUsW9gk6tFanRCNac Message-ID: Subject: Re: Strange panic on ppc64 From: Justin Hibbits To: Nathan Whitehorn Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2013 03:30:58 -0000 On Mon, Jun 10, 2013 at 9:20 PM, Justin Hibbits wrote: > On Mon, Jun 10, 2013 at 6:31 AM, Nathan Whitehorn wrote: > >> On 06/10/13 08:20, Nathan Whitehorn wrote: >> > This is now getting interesting. Reading the tea leaves, what has >> > happened is that the kernel has called into Open Firmware. Open Firmware >> > has then crashed early on, before setting up its own trap handlers, >> > which has then flung you back into FreeBSD's handlers with a totally >> > bogus environment, causing a second panic, which then causes a *third* >> > panic when trying to acquire a lock. It would be interesting to know >> > what the OF environment looked like and what commands it was trying to >> > execute (in r3), but that may be tricky to get... >> > -Nathan >> > _______________________________________________ >> >> One other point: you can trace this pretty easily by just putting >> something like: >> >> if (pmap_bootstrapped) printf("Open Firmware call %p\n", args); >> >> in the top of openfirmware(). If I understood the debugger output >> correctly, something should be making a firmware call immediately before >> the crash. >> >> As a random guess about what is happening, it is possible OF is trying >> to allocate memory for itself. We just ignore the possibility that it >> might want to do that at present, but that is not necessarily a good >> assumption. >> -Nathan >> > > I added that, both on entry and exit. I also have it printing out the > name of the ofw call, since the first item is always a pointer to the > name. I'll be able to report more tomorrow. > > - Justin > Since putting those printf()s in, my machine's been up for close to 48 hours without a hitch, and I've done a buildworld, plus a whole package repo rebuild for my G4 (900 packages), 4 concurrent jobs (load average regularly over 6). I did see a ton of OF getprop calls for the first 12 hours of being up, but none since. I'll try some multiple concurrent buildworlds after poudriere finishes. This is very odd. - Justin From owner-freebsd-ppc@FreeBSD.ORG Thu Jun 13 17:44:36 2013 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EC4D4491; Thu, 13 Jun 2013 17:44:36 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-bk0-x233.google.com (mail-bk0-x233.google.com [IPv6:2a00:1450:4008:c01::233]) by mx1.freebsd.org (Postfix) with ESMTP id 5192D1A0F; Thu, 13 Jun 2013 17:44:36 +0000 (UTC) Received: by mail-bk0-f51.google.com with SMTP id ji1so4606780bkc.10 for ; Thu, 13 Jun 2013 10:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=8QrAkN1uj5VUzhbvY5ChFWgyNJV6R0faylBRF+ybOwA=; b=Sz30CY2/OTsSAB586MStGTCTp+04mdOVFUMq4qg/nIdE2E6gU5Oavs/4RKAIJWUNd8 4wlT6LCD4/pbHWgqGWhjUQvQtWHG99H7hLhwhXrS4Qu61yKlVW0uoH82zQ9GwSjYrGuS CpZvQ6qMpYouQsf5tnexFE5n9osWVvuqQhkfiUKWidQFJ2eex/KuOSyub6DM1cLTbHJV 8AUDndpVirExHX2/F10FbE5yXtMqnYAjIXEcKBRpdG5XV2Zs0Oz83OTlTku7ZQObeQqc 6oYxHzBIVsuGl9WJqIlzpspaHzrvkAl43aEYvlwHjaACuZtdwRhbd8MLEfC+0/BrBWWk JFNg== MIME-Version: 1.0 X-Received: by 10.204.233.137 with SMTP id jy9mr325409bkb.29.1371145475297; Thu, 13 Jun 2013 10:44:35 -0700 (PDT) Sender: chmeeedalf@gmail.com Received: by 10.204.236.132 with HTTP; Thu, 13 Jun 2013 10:44:35 -0700 (PDT) In-Reply-To: References: <51AF6661.3060007@freebsd.org> <51B345BE.5030905@freebsd.org> <51B4A389.4020607@freebsd.org> <51B5D28C.505@freebsd.org> <51B5D539.8050102@freebsd.org> Date: Thu, 13 Jun 2013 10:44:35 -0700 X-Google-Sender-Auth: zPe7Mh4NP4zJyB082cxeIlF4orM Message-ID: Subject: Re: Strange panic on ppc64 From: Justin Hibbits To: Adam Martin Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2013 17:44:37 -0000 On Thu, Jun 13, 2013 at 10:14 AM, Adam Martin wrote: > Could the way it enters OFW have a race, maybe? Prints would put enough > of a delay to ameliorate it. > > You are on SMP, and maybe the ofw entry code was only thought of as single > core? > > Or maybe I'm off the mark as usual. :-P > > -- > A > On Jun 12, 2013 11:31 PM, "Justin Hibbits" wrote: > >> On Mon, Jun 10, 2013 at 9:20 PM, Justin Hibbits > >wrote: >> >> > On Mon, Jun 10, 2013 at 6:31 AM, Nathan Whitehorn < >> nwhitehorn@freebsd.org>wrote: >> > >> >> On 06/10/13 08:20, Nathan Whitehorn wrote: >> >> > This is now getting interesting. Reading the tea leaves, what has >> >> > happened is that the kernel has called into Open Firmware. Open >> Firmware >> >> > has then crashed early on, before setting up its own trap handlers, >> >> > which has then flung you back into FreeBSD's handlers with a totally >> >> > bogus environment, causing a second panic, which then causes a >> *third* >> >> > panic when trying to acquire a lock. It would be interesting to know >> >> > what the OF environment looked like and what commands it was trying >> to >> >> > execute (in r3), but that may be tricky to get... >> >> > -Nathan >> >> > _______________________________________________ >> >> >> >> One other point: you can trace this pretty easily by just putting >> >> something like: >> >> >> >> if (pmap_bootstrapped) printf("Open Firmware call %p\n", args); >> >> >> >> in the top of openfirmware(). If I understood the debugger output >> >> correctly, something should be making a firmware call immediately >> before >> >> the crash. >> >> >> >> As a random guess about what is happening, it is possible OF is trying >> >> to allocate memory for itself. We just ignore the possibility that it >> >> might want to do that at present, but that is not necessarily a good >> >> assumption. >> >> -Nathan >> >> >> > >> > I added that, both on entry and exit. I also have it printing out the >> > name of the ofw call, since the first item is always a pointer to the >> > name. I'll be able to report more tomorrow. >> > >> > - Justin >> > >> >> Since putting those printf()s in, my machine's been up for close to 48 >> hours without a hitch, and I've done a buildworld, plus a whole package >> repo rebuild for my G4 (900 packages), 4 concurrent jobs (load average >> regularly over 6). I did see a ton of OF getprop calls for the first 12 >> hours of being up, but none since. >> >> I'll try some multiple concurrent buildworlds after poudriere finishes. >> This is very odd. >> >> - Justin >> > I'm thinking that may be the case. When I get back from my trip I'll replace the printf()s with sync, and try exercising it again. Oh, and I was wrong about no OFW 'getprop' calls since 12 hours in. During the concurrent buildworlds, I saw a bunch, only while building ports with poudriere did I see none, which is really bizarre. I may also add a sysctl to enable printing stack traces on entering OF, since the only call made since going multiuser is getprop, it'd be nice to know what keeps getting properties. - Justin From owner-freebsd-ppc@FreeBSD.ORG Fri Jun 14 05:20:00 2013 Return-Path: Delivered-To: powerpc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 906CD6CE; Fri, 14 Jun 2013 05:20:00 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-legacy2.sentex.ca (freebsd-legacy2.sentex.ca [IPv6:2607:f3e0:0:3::6502:9c]) by mx1.freebsd.org (Postfix) with ESMTP id 45FC31145; Fri, 14 Jun 2013 05:20:00 +0000 (UTC) Received: from freebsd-legacy2.sentex.ca (localhost [127.0.0.1]) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5) with ESMTP id r5E5JxsP084901; Fri, 14 Jun 2013 05:19:59 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5/Submit) id r5E5JxDm084896; Fri, 14 Jun 2013 05:19:59 GMT (envelope-from tinderbox@freebsd.org) Date: Fri, 14 Jun 2013 05:19:59 GMT Message-Id: <201306140519.r5E5JxDm084896@freebsd-legacy2.sentex.ca> X-Authentication-Warning: freebsd-legacy2.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [releng_8 tinderbox] failure on powerpc/powerpc Precedence: bulk X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jun 2013 05:20:00 -0000 TB --- 2013-06-14 05:14:50 - tinderbox 2.10 running on freebsd-legacy2.sentex.ca TB --- 2013-06-14 05:14:50 - FreeBSD freebsd-legacy2.sentex.ca 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-14 05:14:50 - starting RELENG_8 tinderbox run for powerpc/powerpc TB --- 2013-06-14 05:14:50 - cleaning the object tree TB --- 2013-06-14 05:14:50 - /usr/local/bin/svn stat /src TB --- 2013-06-14 05:14:53 - At svn revision 251732 TB --- 2013-06-14 05:14:54 - building world TB --- 2013-06-14 05:14:54 - CROSS_BUILD_TESTING=YES TB --- 2013-06-14 05:14:54 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-14 05:14:54 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-14 05:14:54 - SRCCONF=/dev/null TB --- 2013-06-14 05:14:54 - TARGET=powerpc TB --- 2013-06-14 05:14:54 - TARGET_ARCH=powerpc TB --- 2013-06-14 05:14:54 - TZ=UTC TB --- 2013-06-14 05:14:54 - __MAKE_CONF=/dev/null TB --- 2013-06-14 05:14:54 - cd /src TB --- 2013-06-14 05:14:54 - /usr/bin/make -B buildworld >>> World build started on Fri Jun 14 05:14:54 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools [...] /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/config/rs6000/sysv4.h:285:1: warning: "PTRDIFF_TYPE" redefined In file included from /obj/powerpc/src/tmp/src/gnu/usr.bin/cc/cc_int/../cc_tools/tm.h:11, from /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:26: /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/config/freebsd.h:76:1: warning: this is the location of the previous definition /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c: In function 'interpret_float': /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: 'CPP_N_DEFAULT' undeclared (first use in this function) /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: (Each undeclared identifier is reported only once /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: for each function it appears in.) *** [c-lex.o] Error code 1 Stop in /src/gnu/usr.bin/cc/cc_int. *** [all] Error code 1 Stop in /src/gnu/usr.bin/cc. *** [cross-tools] Error code 1 Stop in /src. *** [_cross-tools] Error code 1 Stop in /src. *** [buildworld] Error code 1 Stop in /src. TB --- 2013-06-14 05:19:59 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-14 05:19:59 - ERROR: failed to build world TB --- 2013-06-14 05:19:59 - 270.39 user 36.28 system 308.93 real http://tinderbox.freebsd.org/tinderbox-freebsd8-build-RELENG_8-powerpc-powerpc.full From owner-freebsd-ppc@FreeBSD.ORG Fri Jun 14 07:50:11 2013 Return-Path: Delivered-To: powerpc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4E235F64; Fri, 14 Jun 2013 07:50:11 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-legacy2.sentex.ca (freebsd-legacy2.sentex.ca [IPv6:2607:f3e0:0:3::6502:9c]) by mx1.freebsd.org (Postfix) with ESMTP id 0569B1A0B; Fri, 14 Jun 2013 07:50:10 +0000 (UTC) Received: from freebsd-legacy2.sentex.ca (localhost [127.0.0.1]) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5) with ESMTP id r5E7oACp009132; Fri, 14 Jun 2013 07:50:10 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5/Submit) id r5E7oABK009128; Fri, 14 Jun 2013 07:50:10 GMT (envelope-from tinderbox@freebsd.org) Date: Fri, 14 Jun 2013 07:50:10 GMT Message-Id: <201306140750.r5E7oABK009128@freebsd-legacy2.sentex.ca> X-Authentication-Warning: freebsd-legacy2.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [releng_8 tinderbox] failure on powerpc/powerpc Precedence: bulk X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jun 2013 07:50:11 -0000 TB --- 2013-06-14 07:44:57 - tinderbox 2.10 running on freebsd-legacy2.sentex.ca TB --- 2013-06-14 07:44:57 - FreeBSD freebsd-legacy2.sentex.ca 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-14 07:44:57 - starting RELENG_8 tinderbox run for powerpc/powerpc TB --- 2013-06-14 07:44:57 - cleaning the object tree TB --- 2013-06-14 07:44:58 - /usr/local/bin/svn stat /src TB --- 2013-06-14 07:45:00 - At svn revision 251736 TB --- 2013-06-14 07:45:01 - building world TB --- 2013-06-14 07:45:01 - CROSS_BUILD_TESTING=YES TB --- 2013-06-14 07:45:01 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-14 07:45:01 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-14 07:45:01 - SRCCONF=/dev/null TB --- 2013-06-14 07:45:01 - TARGET=powerpc TB --- 2013-06-14 07:45:01 - TARGET_ARCH=powerpc TB --- 2013-06-14 07:45:01 - TZ=UTC TB --- 2013-06-14 07:45:01 - __MAKE_CONF=/dev/null TB --- 2013-06-14 07:45:01 - cd /src TB --- 2013-06-14 07:45:01 - /usr/bin/make -B buildworld >>> World build started on Fri Jun 14 07:45:02 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools [...] /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/config/rs6000/sysv4.h:285:1: warning: "PTRDIFF_TYPE" redefined In file included from /obj/powerpc/src/tmp/src/gnu/usr.bin/cc/cc_int/../cc_tools/tm.h:11, from /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:26: /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/config/freebsd.h:76:1: warning: this is the location of the previous definition /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c: In function 'interpret_float': /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: 'CPP_N_DEFAULT' undeclared (first use in this function) /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: (Each undeclared identifier is reported only once /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: for each function it appears in.) *** [c-lex.o] Error code 1 Stop in /src/gnu/usr.bin/cc/cc_int. *** [all] Error code 1 Stop in /src/gnu/usr.bin/cc. *** [cross-tools] Error code 1 Stop in /src. *** [_cross-tools] Error code 1 Stop in /src. *** [buildworld] Error code 1 Stop in /src. TB --- 2013-06-14 07:50:10 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-14 07:50:10 - ERROR: failed to build world TB --- 2013-06-14 07:50:10 - 270.98 user 38.86 system 312.99 real http://tinderbox.freebsd.org/tinderbox-freebsd8-build-RELENG_8-powerpc-powerpc.full From owner-freebsd-ppc@FreeBSD.ORG Fri Jun 14 11:23:43 2013 Return-Path: Delivered-To: powerpc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2A6B6B36; Fri, 14 Jun 2013 11:23:43 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-legacy2.sentex.ca (freebsd-legacy2.sentex.ca [IPv6:2607:f3e0:0:3::6502:9c]) by mx1.freebsd.org (Postfix) with ESMTP id CE55915FE; Fri, 14 Jun 2013 11:23:42 +0000 (UTC) Received: from freebsd-legacy2.sentex.ca (localhost [127.0.0.1]) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5) with ESMTP id r5EAK6CM033196; Fri, 14 Jun 2013 10:20:06 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5/Submit) id r5EAK6Le033188; Fri, 14 Jun 2013 10:20:06 GMT (envelope-from tinderbox@freebsd.org) Date: Fri, 14 Jun 2013 10:20:06 GMT Message-Id: <201306141020.r5EAK6Le033188@freebsd-legacy2.sentex.ca> X-Authentication-Warning: freebsd-legacy2.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [releng_8 tinderbox] failure on powerpc/powerpc Precedence: bulk X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jun 2013 11:23:43 -0000 TB --- 2013-06-14 10:14:55 - tinderbox 2.10 running on freebsd-legacy2.sentex.ca TB --- 2013-06-14 10:14:55 - FreeBSD freebsd-legacy2.sentex.ca 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-14 10:14:55 - starting RELENG_8 tinderbox run for powerpc/powerpc TB --- 2013-06-14 10:14:55 - cleaning the object tree TB --- 2013-06-14 10:14:56 - /usr/local/bin/svn stat /src TB --- 2013-06-14 10:14:59 - At svn revision 251744 TB --- 2013-06-14 10:15:00 - building world TB --- 2013-06-14 10:15:00 - CROSS_BUILD_TESTING=YES TB --- 2013-06-14 10:15:00 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-14 10:15:00 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-14 10:15:00 - SRCCONF=/dev/null TB --- 2013-06-14 10:15:00 - TARGET=powerpc TB --- 2013-06-14 10:15:00 - TARGET_ARCH=powerpc TB --- 2013-06-14 10:15:00 - TZ=UTC TB --- 2013-06-14 10:15:00 - __MAKE_CONF=/dev/null TB --- 2013-06-14 10:15:00 - cd /src TB --- 2013-06-14 10:15:00 - /usr/bin/make -B buildworld >>> World build started on Fri Jun 14 10:15:00 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools [...] /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/config/rs6000/sysv4.h:285:1: warning: "PTRDIFF_TYPE" redefined In file included from /obj/powerpc/src/tmp/src/gnu/usr.bin/cc/cc_int/../cc_tools/tm.h:11, from /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:26: /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/config/freebsd.h:76:1: warning: this is the location of the previous definition /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c: In function 'interpret_float': /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: 'CPP_N_DEFAULT' undeclared (first use in this function) /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: (Each undeclared identifier is reported only once /src/gnu/usr.bin/cc/cc_int/../../../../contrib/gcc/c-lex.c:644: error: for each function it appears in.) *** [c-lex.o] Error code 1 Stop in /src/gnu/usr.bin/cc/cc_int. *** [all] Error code 1 Stop in /src/gnu/usr.bin/cc. *** [cross-tools] Error code 1 Stop in /src. *** [_cross-tools] Error code 1 Stop in /src. *** [buildworld] Error code 1 Stop in /src. TB --- 2013-06-14 10:20:06 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-14 10:20:06 - ERROR: failed to build world TB --- 2013-06-14 10:20:06 - 270.45 user 37.16 system 310.46 real http://tinderbox.freebsd.org/tinderbox-freebsd8-build-RELENG_8-powerpc-powerpc.full