From owner-freebsd-ppc@freebsd.org Mon May 22 12:08:07 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19184D77925 for ; Mon, 22 May 2017 12:08:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E130F15AC for ; Mon, 22 May 2017 12:08:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by freefall.freebsd.org (Postfix) id 0A2E71576; Mon, 22 May 2017 12:08:06 +0000 (UTC) Delivered-To: freebsd-powerpc@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id E48C81575 for ; Mon, 22 May 2017 12:08:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3603415A4 for ; Mon, 22 May 2017 12:08:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MC84Ri074425 for ; Mon, 22 May 2017 12:08:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-powerpc@FreeBSD.org Subject: [Bug 219455] [patch] fix build of devel/talloc on powerpc64 Date: Mon, 22 May 2017 12:08:04 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: timur@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status keywords bug_severity priority component assigned_to reporter cc flagtypes.name attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 12:08:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219455 Bug ID: 219455 Summary: [patch] fix build of devel/talloc on powerpc64 Product: Ports & Packages Version: Latest Hardware: powerpc OS: Any Status: New Keywords: patch Severity: Affects Some People Priority: --- Component: Individual Port(s) Assignee: timur@FreeBSD.org Reporter: linimon@FreeBSD.org CC: freebsd-powerpc@FreeBSD.org Keywords: patch Assignee: timur@FreeBSD.org CC: freebsd-powerpc@FreeBSD.org Flags: maintainer-feedback?(timur@FreeBSD.org) Created attachment 182801 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D182801&action= =3Dedit patch to devel/talloc/Makefile This patch fixes the build of devel/talloc on powerpc64. It may also fix t= he build on other archs that do not have llvm in base (not yet tested). Build-tested on powerpc64; tested for no-effect on amd64 via 'make patch'. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-ppc@freebsd.org Mon May 22 12:09:31 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C3A1D77A2C for ; Mon, 22 May 2017 12:09:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C9FA51702 for ; Mon, 22 May 2017 12:09:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by freefall.freebsd.org (Postfix) id E1E6C1637; Mon, 22 May 2017 12:09:29 +0000 (UTC) Delivered-To: freebsd-powerpc@localmail.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by freefall.freebsd.org (Postfix) with ESMTPS id CC7271635 for ; Mon, 22 May 2017 12:09:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 23BC416EF for ; Mon, 22 May 2017 12:09:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MC9SeP029581 for ; Mon, 22 May 2017 12:09:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-powerpc@FreeBSD.org Subject: [Bug 219455] [patch] fix build of devel/talloc on powerpc64 Date: Mon, 22 May 2017 12:09:28 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: timur@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 12:09:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219455 --- Comment #1 from Mark Linimon --- Created attachment 182802 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D182802&action= =3Dedit new patchfile to add into devel/talloc/files/ --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-ppc@freebsd.org Tue May 23 06:33:51 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A856D7A1E0 for ; Tue, 23 May 2017 06:33:51 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-13.reflexion.net [208.70.210.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1FD91BB5 for ; Tue, 23 May 2017 06:33:50 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 7457 invoked from network); 23 May 2017 06:33:48 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 23 May 2017 06:33:48 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Tue, 23 May 2017 02:33:48 -0400 (EDT) Received: (qmail 19644 invoked from network); 23 May 2017 06:33:48 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 23 May 2017 06:33:48 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id B6125EC8B7B; Mon, 22 May 2017 23:33:47 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: TARGET_ARCH=powerpc: how to cause a dump from kernel thread without disturbing that thread's stack? Message-Id: Date: Mon, 22 May 2017 23:33:47 -0700 To: FreeBSD PowerPC ML , freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 May 2017 06:33:51 -0000 I've been trying to track down an occasional kernel panic on an old PowerMac G5 so-called "Quad Core" used with 32-bit PowerPC FreeBSD. The panics report absurdly large exception figures and such --and it turns out the content matches some very low memory (not necessarily properly aligned for its starting address). The following is an as-if investigative example. The eventual details may differ. But, presuming that theses tests in these places were appropriate, I'd like to be able to cause a dump without making calls that update the contents of the stack for the kernel thread: I want see the original stack contents in the dump for its history. content. The calls to panic in the example code below would disturb the history available on the kernel-thread's stack, something I'd like to avoid. Is there a known FreeBSD technique for preserving such memory contents when getting a dump at a specific place? void powerpc_interrupt(struct trapframe *framep) { struct thread *td; struct trapframe *oldframe; register_t ee; td =3D curthread; if ((void*)framep <=3D (void*)0x1000) panic("0:bad framep: %p\n", = (void*)framep); if (0x2f00 <=3D framep->exc) panic("0:bad framep->exc: %x (%p)\n", = framep->exc, (void*)framep); CTR2(KTR_INTR, "%s: EXC=3D%x", __func__, framep->exc); switch (framep->exc) { . . . default: /* Re-enable interrupts if applicable. */ ee =3D framep->srr1 & PSL_EE; if (ee !=3D 0) mtmsr(mfmsr() | ee); if ((void*)framep <=3D (void*)0x1000) panic("1:bad framep: %p\n", = (void*)framep); if (0x2f00 <=3D framep->exc) panic("1:bad framep->exc: %x (%p)\n", = framep->exc, (void*)framep); trap(framep); } } =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-ppc@freebsd.org Tue May 23 12:22:51 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5403D78FF8 for ; Tue, 23 May 2017 12:22:51 +0000 (UTC) (envelope-from david.smith@cioutlookreports.com) Received: from mailer57.gate93.rs.smtp.com (mailer57.gate93.rs.smtp.com [74.91.93.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7411F1E85 for ; Tue, 23 May 2017 12:22:50 +0000 (UTC) (envelope-from david.smith@cioutlookreports.com) X-MSFBL: +yfBEJMODJcegnafIr9l31D60qLthk2ioT1NROZQD5I=|eyJyIjoiZnJlZWJzZC1 wcGNAZnJlZWJzZC5vcmciLCJnIjoiQ29tbW9kaXR5SW5zaWRlX2RlZGljYXRlZF9 wb29sIiwiYiI6Ijc0XzkxXzkzXzU3In0= Received: from [192.168.80.22] ([192.168.80.22:51396] helo=rs-ord-mta02-in2.smtp.com) by rs-ord-mta02-out1.smtp.com (envelope-from ) (ecelerity 4.2.1.55028 r(Core:4.2.1.12)) with ESMTP id EA/45-06480-2E424295; Tue, 23 May 2017 12:02:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; d=smtpserver.email; s=smtpcustomer; c=relaxed/simple; q=dns/txt; i=@smtpserver.email; t=1495540962; h=From:Subject:To:Date:MIME-Version:Content-Type; bh=/ijO+p78J/WH8tV+kIo6xVPlvJ9lfTBMnQd07L+u8Kc=; b=XChc26Q+w/TA652HIN9F30cpKPrYFXeiXRmTosdHoizS2J8Habd9zdfs6BwehYLO rso9IJKUohzsEhcDpO5hUJ2bnlfdOdwxFJqwbdCdu37uBpMJKfmwZzwmGsKArauw G8iCbWen6JkO3jKXeJu7K8tEvOUbnZn2svM0YI/3bSFLTvAO0Khx5xjl8rvndd0T lplgMgj/flrY/Rg/tUkfkKlOTJL2YPrFwoN6qONkdo0gfRGZkyQl+b1tYTS+GcHu gSd7AD0boteWm+s6/5Yj4ZLv2m7cbSVNOFu4koQbQoLUBllIxCjuhSH7BCqGIQ29 JKhNhI11qr6QTbxqCQvMNQ==; Received: from [86.4.116.225] ([86.4.116.225:35844] helo=cpc90280-cove16-2-0-cust224.3-1.cable.virginm.net) by rs-ord-mta02-in2.smtp.com (envelope-from ) (ecelerity 4.1.0.46749 r(Core:4.1.0.4)) with ESMTPA id D2/30-13219-1E424295; Tue, 23 May 2017 12:02:42 +0000 MIME-Version: 1.0 From: "David Smith" Reply-To: david.smith@cioutlookreports.com To: freebsd-ppc@freebsd.org Subject: Global Vehicle-to-Everything (V2X) Communications Market out to 2027 X-Mailer: Smart_Send_2_0_138 Date: Tue, 23 May 2017 13:02:32 +0100 Message-ID: <254836362464838762793@Saket> X-Report-Abuse: SMTP.com is an email service provider. Our abuse team cares about your feedback. Please contact abuse@smtp.com for further investigation. X-SMTPCOM-Tracking-Number: 3a59c48f-eb30-48ce-b8e8-7be80649e28c X-SMTPCOM-Sender-ID: 5012537 Feedback-ID: 5012537:SMTPCOM Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 May 2017 12:22:51 -0000 Global Vehicle-to-Everything (V2X) Communications Market out to 2027 (Repor= t) =20 Report Information: Release Date: March 2017 Number of Pages/Slides: 103 Tables and Figures: 67 Report Overview: =20 The global vehicle-to-everything (V2X) communications market is expected to= see significant growth over the next ten years. Some of the major drivers = behind the growth will be stringent safety regulations, increased demand fo= r connectivity and development of autonomous vehicles. Meanwhile, growth wi= ll shift from North America and Western Europe to Asia, especially China. The V2X market has evolved through a raft of regulations, and government an= d private investments. It will see further legislations and renewed market = growth on the back of rapid adoption of V2V and V2I communication technolog= ies. Commodity Inside understands that demand for automotive connectivity is at = its embryonic stage and has an enormous growth potential. V2X communication= technologies will enhance existing safety features and help developing new= applications. With automakers planning to introduce fully autonomous vehic= les by 2021, V2X will be one of the major beneficiaries and will see a surg= e in sales. The deployment of 5G cellular technology will also put pressure on the exis= ting dedicated short-range communications (DSRC) technology. However, both = technologies would have limitations, so both will continue to be used over = the next ten years. The 5G technology will open new opportunities for big d= ata and big data analytics, allowing the fastest transmission of the vast a= mount of live data. The expected high growth and rising adoption have also opened new opportuni= ties for high-tech companies. Vehicles are increasingly becoming vulnerable= to hacking as they are interacting with others vehicles and devices. Conse= quently, this has opened new venues for cyber security companies which are = penetrating quite rapidly in the automotive market. Why this report is unique and a must read for the V2X and automotive indust= ry as a whole=3F Global Vehicle-to-Everything (V2X) Communications Market out to 2027 is a v= aluable resource necessary for examining the global V2X communications mark= et. We have employed a very sophisticated and robust approach to assess the= automotive market by taking into account various demand and supply dynamic= s as well as technological developments. The report has also encompassed di= fferent segmentations of the market and strategic directions for OEMs and T= ier-1 suppliers going forward. The report covers the following key aspects: How will the V2X market perform over the next ten-years=3F What will be the major factors driving the V2X communications market and ho= w they will impact the industry=3F How will the increasing use of communications technology benefit the indust= ry=3F Detailed analysis of the current status of V2X communications supply chain=3F What will be the major drivers behind each type of communication, connectiv= ity, applications, hardware & software and vehicle type and what will be th= eir implications=3F Analysis and ten year projections for V2V, V2I, V2P, V2G and V2N markets. The current and future demand dynamics of OEMs and aftermarket in the V2V c= ommunications segment. Projections of the roadside equipment installation in the V2I market. Detailed discussion on market strategies and competitive landscapes. Analysis of the present and future performance of matured and developing ma= rkets. Why should you read this report=3F Global Vehicle-to-Everything (V2X) Communications Market out to 2027 provid= es you with the following in-depth analysis: The V2X communications market is analysed by five communications segments, = by vehicle type, by connectivity type, by applications and by hardware & so= ftware. Discussion on major drivers and restraints of each communications segment. The full coverage of V2X communications market in terms of sales value. Coverage of the whole world by five regions and ten selected major countrie= s. Detailed analysis of the current and future supply chain. Market projections for roadside units installation and OEM-aftermarket dyna= mics in the V2V segment. 67 tables, figures and charts. All supportive data provided in Excel. Who should buy this report=3F Conventional car manufacturers Electric vehicle manufacturers Hybrid and plug-in hybrid vehicles manufacturers Component manufacturers Suppliers of materials to automotive Automotive technology vendors Software and cyber security companies Mobile network operators Smartphone manufacturers Mobility service providers Utility companies Financial institutions Industry consultants, researchers and analysts Government bodies Why our analyses are robust and authoritative=3F We constantly consult industry experts and incorporate their views in our a= nalysis.=20 We employ both quantitative and qualitative methods to derive robust analys= is.=20 All our forecast data were also supported by our proprietary econometric an= d excel based models.=20 We are completely independent and represent our own views. =20 Table of Contents Chapter 1- Executive Summary 1.1 Developments in the V2X communications market Chapter 2 =96 Introduction and Methodology 2.1 Introduction 2.2 Methodology Chapter 3 =96 V2X Market Segmentations 3.1 V2X market by communication 3.1.1 Vehicle-to-Vehicle (V2V) 3.1.2 Vehicle-to-Infrastructure (V2I) 3.1.3 Vehicle-to-Pedestrian (V2P) 3.1.4 Vehicle-to-Grid (V2G) 3.1.5 Vehicle-to-Network (V2N) 3.2 V2X market by vehicle type 3.3 V2X market by connectivity type 3.4 V2X market by offering 3.5 V2X market by application type Chapter 4 =96 V2X Market by Geography 4.1 Brazil 4.2 Canada 4.3 China 4.4 France 4.5 Germany 4.6 India 4.7 Japan 4.8 South Korea 4.9 UK 4.10 US Chapter 5 =96 Competitive Landscape 5.1 Competitive landscape by major partnerships and M&As 5.3 Company profiles by OEMS and Teir-1 suppliers List of Tables: Table 1.1 Global V2X market by region 2014-2027 ($ billions) Table 1.2 Global V2X market by communication segments 2014-2027 ($ millions) Table 3.1 Global V2V market 2014-2027 ($ millions) Table 3.2 Global OEM and Aftermarket V2V market 2014-2027 ($ millions) Table 3.3 Global V2I market 2014-2027 ($ millions, %) Table 3.4 Global roadside units installations 2014-2027 (=91000 units) Table 3.5 Global V2P market 2014-2027 ($ millions, %) Table 3.6 Global V2P enabled smartphone shipments 2014-2027 (million units) Table 3.7 Global V2G market 2014-2027 ($ millions, %) Table 3.8 Global V2N market 2014-2027 ($ million, %) Table 3.9 Global V2X market by offerings 2014-2027 ($ millions) Table 4.1 Brazilian V2X market 2014-2027 ($ millions, %) Table 4.2 Canadian V2X market 2014-2027 ($ millions, %) Table 4.3 Chinese V2X market 2014-2027 ($ millions, %) Table 4.4 French V2X market 2014-2027 ($ millions, %) Table 4.5 German V2X market 2014-2027 ($ millions, %) Table 4.6 Indian V2X market 2014-2027 ($ millions, %) Table 4.7 Japanese V2X market 2014-2027 ($ millions, %) Table 4.8 South Korean V2X market 2014-2027 ($ millions, %) Table 4.9 UK V2X market 2014-2027 ($ millions, %) Table 4.10 US V2X market 2014-2027 ($ millions, %) =20 List of Figures: Figure 1.1 Global V2X market by region 2014-2027 ($ billions) Figure 1.2 Global V2X market by segment 2014-2027 ($ millions) Figure 1.3 Global V2X market by application 2014-2027 ($ billions) Figure 1.4 Global V2X market by type of vehicle 2014-2027 ($ billions) Figure 1.5 Global V2X market by connectivity 2014-2027 ($ billions) Figure 1.6 Global V2X market by offering 2014-2027 ($ billions) Figure 3.1 Global V2V market 2014-2027 ($ millions) Figure 3.2 Global OEM and aftermarket V2X market 2014-2027 ($ millions) Figure 3.3 Global V2I market 2014-2027 ($ millions) Figure 3.4 Global roadside units installations 2014-2027 (=91000 units) Figure 3.5 Global V2P market ($ millions) Figure 3.6 Global V2P enabled device shipments 2014-2017 (million units) Figure 3.7 Global V2G market 2014-2027 ($ millions) Figure 3.8 Global V2N market 2014-2027 ($ billions) Figure 3.9 Global vehicle sales 2014-2027 (million units) Figure 3.10 Global light vehicles sales 2014-2027 (million units) Figure 3.11 Global medium and high vehicle sales 2014-2027 (million units) Figure 3.12 Global V2X light vehicles market 2014-2027 ($ billions) Figure 3.13 Global V2X medium and light vehicle market 2014-2027 ($ billion= s) Figure 3.14 Global V2X market 2014-2027 ($ billions) Figure 3.15 Global DSRC market 2014-2027 ($ billions) Figure 3.16 Global Cellular market 2014-2027 ($ billions) Figure 3.17 Global V2X market by offerings 2014-2027 ($ billions) Figure 3.18 Global V2X market by application ($ billions) Figure 3.19 Global V2X market share by application (% share) Figure 4.1 Brazilian V2X market 2014-2027 ($ millions) Figure 4.2 Canadian V2X market ($ millions) Figure 4.3 Chinese V2X market 2014-2027 ($ millions) Figure 4.4 French V2X market 2014-2027 ($ millions) Figure 4.5 German V2X market 2014-2027 ($ millions) Figure 4.6 Indian V2X market 2014-2027 ($ millions) Figure 4.7 Japanese V2X market 2014-2027 ($ millions) Figure 4.8 South Korean V2X market 2014-2027 ($ millions) Figure 4.9 UK V2X market 2014-2027 ($ million, %) Figure 4.10 US V2X market 2014-2027 ($ millions) =20 List of charts: Chart 1.1 Opportunities, challenges, readiness, regulations impact, time to= scale production and aftermarket potential in the V2X market by supply cha= in Chart 2.1 V2X market segments by communication type Chart 2.2 V2X communications process Chart 3.1 Comparison of DSRC and cellular applications Chart 3.2 Vehicle sensors by ADAS applications Chart 3.3 V2X application grouped by agency data, safety, mobility and road= weather Chart 4.1 Existing and future V2X applications in Japan Chart 5.1 List of vehicles with V2V and V2I capabilities by major OEMs Chart 5.2 List of major Tier-1 suppliers and their V2X portfolio Chart 5.3 Major partnerships in the V2X market 2015-2017 Chart 5.4 Mergers and acquisitions in the V2X market 2015-2017 =20 Report Pricing Single User License: =A31695 Departmental License (up to 5 Users): =A32995 Global License: =A34995 =20 Ordering process Please contact David Smith on david.smith@cioutlookreports.com And provide the following information: Report Title - Report License - (Single User/Departmental/Global) Name - Email - Job Title - Company - Invoice Address VAT number (EU Only) Please contact me if you have any questions, or wish to purchase a copy I look forward to hearing from you. Kind Regards David Smith Business Intelligence Executive To Unsubscribe send an email with Unsubscribe in the subject line to info@c= s-reports.com Remove From owner-freebsd-ppc@freebsd.org Wed May 24 11:44:13 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4C02D7B51F; Wed, 24 May 2017 11:44:13 +0000 (UTC) (envelope-from rdivacky@vlakno.cz) Received: from vlakno.cz (mail.vlakno.cz [91.217.96.224]) by mx1.freebsd.org (Postfix) with ESMTP id B2202105A; Wed, 24 May 2017 11:44:13 +0000 (UTC) (envelope-from rdivacky@vlakno.cz) Received: by vlakno.cz (Postfix, from userid 1002) id D842E1C7FF01; Wed, 24 May 2017 13:34:41 +0200 (CEST) Date: Wed, 24 May 2017 13:34:41 +0200 From: Roman Divacky To: freebsd-toolchain@freebsd.org Cc: freebsd-ppc@freebsd.org, markmi@dsl-only.net Subject: PowerPC64 C++ exceptions with clang/llvm Message-ID: <20170524113441.GA58092@vlakno.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.8.2 (2017-04-18) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 May 2017 11:44:14 -0000 Hi, Currently C++ exceptions don't work on PowerPC64 when compiled with clang. I looked at why and discovered the following. With the default gcc unwinder and libstdc++ this simple test program: #include int main() { try { std::cout << "Before\n"; throw std::exception(); std::cout << "After\n"; } catch (...) { std::cout << "Exception\n"; } } segfaults right after printing "Before\n". This is because the .eh_frame is corrupted and the personality relocation is wrong: < 1> version 1 cie section offset 56 0x00000038 augmentation zPLR code_alignment_factor 4 data_alignment_factor -8 return_address_register 65 eh aug data len 0xb bytes 0x94 00 00 00 01 00 01 02 ed 14 1b bytes of initial instructions 3 cie length 28 initial instructions 0 DW_CFA_def_cfa r1 0 the personality relocation is "00 00 00 01 00 01 02 ed" encoded as pc relative. The program segfaults because this address is wrong. The 33rd bit is incorrect, the real address should be "00 00 00 00 00 01 02 ed". The same problem applies for LDSA encoding. Both personality and LDSA relocations are produced by CFI instructions in the assembler source code. I verified this theory by hacking the gcc unwinder and libsupc++ &= the address with 0xffffffff; and that made the application work. (the full patch can be found at http://www.vlakno.cz/~rdivacky/ppc64.exceptions.hack.patch). I checked why this relocation is wrong and it turns out it's our old in-tree ld that (wrongly) produces this. Our old gcc does not use CFI to produce these and hardcodes the CIE manually and produces correct address. And for some reason ld is fine with that. When I compile and assemble the above source code using clang++ and only use newer ld linker from ports the resulting binary is correct and works as expected. So this is a bug in our in-tree ld linker. I don't know why ld has this bug but given Mark Millard's report of ld being broken for other stuff (kernel iirc) I am not going to bother trying to fix it. I suspect it might be related to comdats but I didn't really investigate. I didn't try the situation with the LLVM unwinder and libcxxrt, but I hope it might be the same. Thank you, Roman From owner-freebsd-ppc@freebsd.org Thu May 25 04:03:55 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5231D8179C for ; Thu, 25 May 2017 04:03:55 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-15.reflexion.net [208.70.210.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8605C19D7 for ; Thu, 25 May 2017 04:03:55 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 10434 invoked from network); 25 May 2017 04:05:11 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 25 May 2017 04:05:11 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Thu, 25 May 2017 00:03:53 -0400 (EDT) Received: (qmail 8168 invoked from network); 25 May 2017 04:03:53 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 25 May 2017 04:03:53 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id CFE10EC78CC; Wed, 24 May 2017 21:03:52 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: PowerPC64 C++ exceptions with clang/llvm From: Mark Millard In-Reply-To: <20170524113441.GA58092@vlakno.cz> Date: Wed, 24 May 2017 21:03:52 -0700 Cc: FreeBSD Toolchain , freebsd-ppc@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <74D091B2-71CD-42DE-8C1E-8E0BDED1EAFD@dsl-only.net> References: <20170524113441.GA58092@vlakno.cz> To: Roman Divacky X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 04:03:55 -0000 [So far I've been unable to replicate dwarfump reporting anything near "0x94 00 00 00 01 00 01 02 ed 14 1b". The compiled code still gets SIGSEGV: it is only the low level detail that I'm noting as different. Roman has more information from me than I show here.] On 2017-May-24, at 4:34 AM, Roman Divacky wrote: > Hi, >=20 > Currently C++ exceptions don't work on PowerPC64 when compiled with = clang. I > looked at why and discovered the following. >=20 > With the default gcc unwinder and libstdc++ this simple test program: >=20 > #include >=20 > int main() > { > try { > std::cout << "Before\n"; > throw std::exception(); > std::cout << "After\n"; > } catch (...) { > std::cout << "Exception\n"; > } > } >=20 > segfaults right after printing "Before\n". This is because the = .eh_frame is corrupted and > the personality relocation is wrong: >=20 > < 1> version 1 > cie section offset 56 0x00000038 > augmentation zPLR > code_alignment_factor 4 > data_alignment_factor -8 > return_address_register 65 > eh aug data len 0xb bytes 0x94 00 00 00 01 00 01 02 ed 14 1b=20 So far I'm unable to replicate producing anything like the "large one" as I'll call it. I get things like: # clang++ -std=3Dc++11 exception_test_dots.cpp # dwarfdump -v -v -F a.out | more .eh_frame . . . cie: < 0> version 1 . . . eh aug data len 0xb bytes 0x94 00 00 00 00 00 01 03 45 14 1b=20 . . . (No cia<1> at all.) and like: # clang++ -B /usr/local/bin/ -std=3Dc++11 exception_test_dots.cpp # dwarfdump -v -v -F a.out | more .eh_frame . . . cie: < 0> version 1 . . . < 1> version 1 . . . eh aug data len 0xb bytes 0x94 00 00 00 00 00 01 f1 1d 14 1b=20 . . . My context is head -r317820 on a system built with the system clang without building gcc 4.2.1 materials. # uname -paKU FreeBSD FBSDG5L 12.0-CURRENT FreeBSD 12.0-CURRENT r317820M powerpc = powerpc64 1200030 1200030 # ld --version GNU ld 2.17.50 [FreeBSD] 2007-07-03 Copyright 2007 Free Software Foundation, Inc. This program is free software; you may redistribute it under the terms = of the GNU General Public License. This program has absolutely no = warranty. # /usr/local/bin/ld --version GNU ld (GNU Binutils) 2.27 Copyright (C) 2016 Free Software Foundation, Inc. This program is free software; you may redistribute it under the terms = of the GNU General Public License version 3 or (at your option) a later = version. This program has absolutely no warranty. (I have avoided 2.28 so far because of the "return code" consequences on the likes of aarch64.) I've tried variations like omitting -std=3Dc++11. So far I've not found a way to get near: "0x94 00 00 00 01 00 01 02 ed 14 1b" but all the experiments get SIGSEGV when I run the produced program. (I do not have a gcc 4.2.1 build around and so do not have libstdc++ on powerpc64.) > bytes of initial instructions 3 > cie length 28 > initial instructions > 0 DW_CFA_def_cfa r1 0 >=20 > the personality relocation is "00 00 00 01 00 01 02 ed" encoded as pc = relative. The program segfaults > because this address is wrong. The 33rd bit is incorrect, the real = address should be "00 00 00 00 00 01 02 ed". > The same problem applies for LDSA encoding. Both personality and LDSA = relocations are produced by CFI instructions > in the assembler source code. >=20 > I verified this theory by hacking the gcc unwinder and libsupc++ &=3D = the address with 0xffffffff; and that made > the application work. (the full patch can be found at = http://www.vlakno.cz/~rdivacky/ppc64.exceptions.hack.patch). >=20 > I checked why this relocation is wrong and it turns out it's our old = in-tree ld that (wrongly) produces this. > Our old gcc does not use CFI to produce these and hardcodes the CIE = manually and produces correct address. And for > some reason ld is fine with that. >=20 > When I compile and assemble the above source code using clang++ and = only use newer ld linker from ports the resulting > binary is correct and works as expected. >=20 > So this is a bug in our in-tree ld linker. I don't know why ld has = this bug but given Mark Millard's report of ld > being broken for other stuff (kernel iirc) I am not going to bother = trying to fix it. I suspect it might be related > to comdats but I didn't really investigate. >=20 > I didn't try the situation with the LLVM unwinder and libcxxrt, but I = hope it might be the same. Side note: Why I'm at head -r317820 . . . For TARGET_ARCH=3Dpowerpc I've got a periodic panic after booting for which changing the memory layout even a little bit seems to stop the panic. (Something less obvious may still be wrong.) The panic can take minutes but usually is more like hours. Even attempting to add small amounts of debug code tends to make the problem invisible. Thus a binary search for versions is not in the cards for now. So I've been focusing on trying for evidence about what leads to the panic and I've not been updating any of the FreeBSD environments that I have around. =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-ppc@freebsd.org Thu May 25 19:03:05 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 963A1D82A6E for ; Thu, 25 May 2017 19:03:05 +0000 (UTC) (envelope-from andy.silva@snsmarketreports.com) Received: from mailer238.gate85.rs.smtp.com (mailer238.gate85.rs.smtp.com [74.91.85.238]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5609E1E15 for ; Thu, 25 May 2017 19:03:04 +0000 (UTC) (envelope-from andy.silva@snsmarketreports.com) X-MSFBL: zcVdynPD6eflAD+hA/ajBPvoJuP4nVGUsXMxgo2G6Nc=|eyJiIjoiNzRfOTFfODV fMjM4IiwiZyI6IlNuc3RlbGVjb21fZGVkaWNhdGVkX3Bvb2wiLCJyIjoiZnJlZWJ zZC1wcGNAZnJlZWJzZC5vcmcifQ== Received: from [192.168.80.22] ([192.168.80.22:38520] helo=rs-ord-mta02-in2.smtp.com) by rs-ord-mta01-out2.smtp.com (envelope-from ) (ecelerity 4.2.1.55028 r(Core:4.2.1.12)) with ESMTP id A1/85-04808-0B527295; Thu, 25 May 2017 18:42:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; d=snsresearchreports.com; s=snskey; c=relaxed/simple; q=dns/txt; i=@snsresearchreports.com; t=1495737776; h=From:Subject:To:Date:MIME-Version:Content-Type; bh=SceKgTtKknQJ39NPSNeRv4VL7yQkHVcxRFaWdMJJCOA=; b=G5P8hwjZODPQpbyvQvsDXTlTNAdn84N6q27hbOhRG+3WAlqolKEmgdBN2vIL2Xjd NuByK2qvbgsWkNIQas80CyZ7HBUZA9Sb84lJxhDNgw4hvC4o2pusirCULJ+blR6P yKWPDgMeuib9Pu2g0RFwsFuItPbO5M+gir6meobl1PJ0LefNBsnm6Zu3ysqbKpsp CvSD5Kr3mvyVlh8HR2n6Gsny5p1pk36Sq4JsoitZraSN9hfj/m0TTWxGM1adECtb Zk4FQ7sj7jykdL73u8S134i7ZxKYOu6bAaTg+K2lx5Om6wB85kavuI37YP7uyf0R bOVyjY4uEEczJbmggfbBYw==; Received: from [205.250.228.1] ([205.250.228.1:38703] helo=d205-250-228-1.bchsia.telus.net) by rs-ord-mta02-in2.smtp.com (envelope-from ) (ecelerity 4.1.0.46749 r(Core:4.1.0.4)) with ESMTPA id 1F/D4-05884-0B527295; Thu, 25 May 2017 18:42:56 +0000 MIME-Version: 1.0 From: "Andy Silva" Reply-To: andy.silva@snsmarketreports.com To: freebsd-ppc@freebsd.org Subject: 5G Wireless Ecosystem: 2017 - 2030 - Technologies, Applications, Verticals, Strategies & Forecasts (Report) X-Mailer: Smart_Send_2_0_138 Date: Thu, 25 May 2017 11:42:47 -0700 Message-ID: <50404014583601119130808@Ankur> X-Report-Abuse: SMTP.com is an email service provider. Our abuse team cares about your feedback. Please contact abuse@smtp.com for further investigation. X-SMTPCOM-Tracking-Number: 2fc0418b-2c58-42f8-a321-cb163722f7f5 X-SMTPCOM-Sender-ID: 6008902 Feedback-ID: 6008902:SMTPCOM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 19:03:05 -0000 5G Wireless Ecosystem: 2017 =96 2030 =96 Technologies, Applications, Vertic= als, Strategies & Forecasts (Report) Hello Let me offer you the latest SNS Research report to you and your team, "5G W= ireless Ecosystem: 2017 =96 2030 =96 Technologies, Applications, Verticals,= Strategies & Forecasts" Below is the report highlight and if you like I ca= n send you sample pages for your details inside. Despite the lack of sufficient LTE coverage in parts of the world, mobile o= perators and vendors have already embarked on R&D initiatives to develop 5G= , the next evolution in mobile networks. 5G is expected to provide a single= network environment to deliver not only existing mobile broadband and IoT = services, but also new innovations such as self-driving cars, cloud robotic= s, 3D holographic telepresence and remote surgery with haptic feedback. Although 2020 has conventionally been regarded as the headline date for 5G = commercialization, the very first standardized deployments of the technolog= y are expected to be commercialized as early as 2019 with the 3GPP's initia= l 5G specifications set to be implementation-ready by March 2018. Between 2= 019 and 2025, we expect the 5G network infrastructure market to aggressivel= y grow a CAGR of nearly 70%, eventually accounting for $28 Billion in annua= l spending by the end of 2025. These infrastructure investments will be com= plemented by annual shipments of up to 520 Million 5G-capable devices. The report comes with an associated Excel datasheet suite covering quantita= tive data from all numeric forecasts presented in the report, as well as a = 5G deployment tracking database covering over 60 global 5G trials, demos an= d commercial deployment commitments (as of Q1=922017). Report Information: Release Date: March 2017 Number of Pages: 363 Number of Tables and Figures: 117 Key Questions Answered: How big is the opportunity for 5G network infrastructure, user equipment an= d operator services=3F What trends, challenges and barriers will influence the development and ado= ption of 5G=3F How will 5G drive the adoption of AR (Augmented Reality)/VR (Virtual Realit= y) applications such as 3D holographic telepresence and 360 degree streamin= g of live events=3F How have advanced antenna and chip technologies made it possible to utilize= millimeter wave spectrum for mobile communications in 5G networks=3F How can non-orthogonal multiple access schemes such as RSMA (Resource Sprea= d Multiple Access) enable 5G networks to support higher connection densitie= s for Millions of IoT devices=3F What will be the number of 5G subscriptions in 2019 and at what rate will i= t grow=3F Which regions and countries will be the first to adopt 5G=3F Which frequency bands are most likely to be utilized by 5G networks=3F Who are the key 5G vendors and what are their strategies=3F Will 5G networks rely on a disaggregated RAN architecture=3F How will 5G impact the fiber industry=3F Will satellite communications and aerial networking platforms play a wider = role in 5G networks=3F Key Findings: The report has the following key findings: The Unites States and South Korea are spearheading early investments in pre= -standards 5G trial networks, as mobile operators rush to be the first to o= ffer 5G services. SNS Research estimates that by the end of 2017, pre-stand= ards 5G network investments are expected to account for over $250 Million. Following completion of the 3GPP's first phase of 5G specifications in Marc= h 2018, SNS Research expects that early adopters across the globe will simu= ltaneously begin commercializing 5G services in 2019. Between 2019 and 2025, we expect the 5G network infrastructure market to ag= gressively grow a CAGR of nearly 70%, eventually accounting for $28 Billion= in annual spending by the end of 2025. Although early 5G R&D investments have primarily targeted the radio access = segment, network-slicing has recently emerged as necessary "end-to-end" cap= ability to guarantee performance for different 5G applications which may ha= ve contrasting requirements. In order to support diverse usage scenarios, 5G networks are expected to ut= ilize a variety of frequency bands ranging from established sub-6 GHz cellu= lar bands to millimeter wave spectrum. The report covers the following topics: 5G NR (New Radio) and NextGen (Next Generation) system architecture Market drivers and barriers to the adoption of 5G networks 5G requirements, usage scenarios, vertical markets and applications Key enabling technologies including air interface design, higher frequency = radio access, advanced antenna systems, flexible duplex schemes, D2D (Devic= e-to-Device) connectivity, dynamic spectrum access, self-backhauling and network slicing Complementary concepts including NFV, SDN, hyperscale data centers, Cloud R= AN, satellite communications and aerial networking platforms Case studies and review of mobile operator 5G commitments 5G standardization, development and research initiatives Analysis of spectrum availability and allocation strategies for 5G networks Competitive assessment of vendor strategies Review of investments on R&D and pre-standards 5G networks Standardized 5G infrastructure, user equipment and operator service forecas= ts till 2030 Forecast Segmentation: Market forecasts are provided for each of the following submarkets and thei= r subcategories: 5G R&D Investments New Air Interface & Millimeter Wave Radio Access MIMO, Beamforming & Advanced Antenna Technologies Spectrum Sharing, Aggregation & Interference Management Virtualization & Cloud RAN Network Slicing & Other Technologies Pre-Standards 5G Network Investments Pre-Standards Base Stations Pre-Standards User Equipment Transport Networking & Other Investments Standardized 5G Infrastructure Investments 5G NR (New Radio) Distributed Macrocell Base Stations Small Cells RRHs (Remote Radio Heads) C-RAN BBUs (Baseband Units) NextGen (Next Generation) Core Network Fronthaul & Backhaul Networking Standardized 5G User Equipment Investments Handsets Tablets Embedded IoT Modules USB Dongles Routers 5G Operator Services Subscriptions Service Revenue Regional Segmentation Asia Pacific Eastern Europe Latin & Central America Middle East & Africa North America Western Europe Report Pricing: =20 Single User License: USD 2,500 Company Wide License: USD 3,500 =20 Ordering Process: =20 Please provide the following information: Report Title - Report License - (Single User/Company Wide) Name - Email - Job Title - Company - Invoice Address - Please contact me if you have any questions, like to see table of contents,= sample pages or wish to purchase a copy. I look forward to hearing from you. =20 Kind Regards =20 Andy Silva Marketing Executive Signals and Systems Telecom andy.silva@snsresearchreports.com =20 =20 =20 To unsubscribe send an email with unsubscribe in the subject line to: remov= e@snsreports.com =20 From owner-freebsd-ppc@freebsd.org Thu May 25 22:06:41 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D181ED82EE6 for ; Thu, 25 May 2017 22:06:41 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-62.reflexion.net [208.70.210.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9648F1BB4 for ; Thu, 25 May 2017 22:06:41 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 11687 invoked from network); 25 May 2017 21:59:19 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 25 May 2017 21:59:19 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Thu, 25 May 2017 17:59:19 -0400 (EDT) Received: (qmail 10415 invoked from network); 25 May 2017 21:59:19 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 25 May 2017 21:59:19 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 11BB2EC805D; Thu, 25 May 2017 14:59:19 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: llvm FreeBSD powerpc ABI target bug fix: Re: [Bug 26519] Clang 4.0.0's "Target: powerpc-unknown-freebsd11.0" code generation is violating the SVR4 ABI (SEGV can result) From: Mark Millard In-Reply-To: <408D3509-3D62-4413-986B-6C1171FB6138@dsl-only.net> Date: Thu, 25 May 2017 14:59:18 -0700 Cc: Roman Divacky , FreeBSD Toolchain , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <0103401A-CEEA-4992-A45E-E60EA151119B@dsl-only.net> <893ECA11-7C80-4D24-A496-92ADC7978A07@FreeBSD.org> <408D3509-3D62-4413-986B-6C1171FB6138@dsl-only.net> To: Dimitry Andric X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 22:06:41 -0000 [Top post of question with older material listed after for reference.] Is llvm bugzilla's latest 26519 fix going to make it into release/11.1.0 ? This fixes the last known stack handling ABI violation for targeting powerpc FreeBSD (32-bit). (For both powerpc64 and powerpc handling thrown C++ exceptions still needs work but the above does make the rest of buildworld operational for TARGET_ARCH=3Dpowerpc --as far as my basic testing goes anyway.) =3D=3D=3D Mark Millard markmi at dsl-only.net On 2017-May-17, at 3:22 PM, Mark Millard wrote: I got a notice this morning of the latest fix for covering the TARGET_ARCH=3Dpowerpc stack-handling issues for llvm bugzilla 26519: > Comment # 33 on bug 26519 from Krzysztof Parzyszek > The fix has been committed in master in r303257. >=20 > I opened PR33070 to request merging it into 4.0.1. >=20 > You are receiving this mail because: > =E2=80=A2 You reported the bug. I've been using a version of the patch for some time and for buildworld it appears that with it powerpc and powerpc64 have a similar status: the one known area not working is handling of thrown C++ exceptions --for example the required dwarf information is incomplete so programs crash. (I have one powerpc64 patch in use that is not applied upstream or in FreeBSD that is essential for the powerpc64 status. See the later side notes for the tiny patch.) For buildkernel there is a difference for TARGET_ARCH=3Dpowerpc vs. TARGET_ARCH=3Dpowerpc64 : A) powerpc46 works for building and running the kernel and world on the old G5 PowerMacs. B) powerpc FreeBSD on the same machines fails at the /sbin/init attempt and then gets an alignment exception. (I've not tried a G4 or G3 yet.) As of yet I've no clue why (B) is an issue. Side notes: Note 0: The patch I was given a fair time ago that is required for TARGET_ARCH=3Dpowerpc64 is: # svnlite diff /usr/src/contrib/llvm/tools/ Index: /usr/src/contrib/llvm/tools/lld/ELF/Target.cpp =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/contrib/llvm/tools/lld/ELF/Target.cpp (revision = 317820) +++ /usr/src/contrib/llvm/tools/lld/ELF/Target.cpp (working copy) @@ -1070,7 +1070,8 @@ } PPC64TargetInfo::PPC64TargetInfo() { - PltRel =3D GotRel =3D R_PPC64_GLOB_DAT; + GotRel =3D R_PPC64_GLOB_DAT; + PltRel =3D R_PPC64_JMP_SLOT; RelativeRel =3D R_PPC64_RELATIVE; GotEntrySize =3D 8; (Thanks to Roman Divacky.) Note 1: While a pure gcc 4.2.1 buildworld buildkernel with a debug kernel is working for booting and using, a production style kernel gets occasional panics on the old G5 PowerMacs. (The powerpc64 builds work fine on the same machines.) (I've not tried any G4's or a G3 yet.) It also appears that small changes in memory layout details (from trying to get better evidence) change the behavior/failure-mode/ details. I do not expect to find out what is going on any time soon. The same problems existed when buildworld was via clang 4. A kernel from a clang buildkernel does not get far enough for me to see what it would do for the issue. As this issue is more fundamental to general operation it has been getting much of my FreeBSD time. =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-ppc@freebsd.org Thu May 25 23:31:40 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B50EBD81824; Thu, 25 May 2017 23:31:40 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tensor.andric.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76E0A131D; Thu, 25 May 2017 23:31:40 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from [IPv6:2001:470:7a58::4034:2885:b014:e7b0] (unknown [IPv6:2001:470:7a58:0:4034:2885:b014:e7b0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id C739016B07; Fri, 26 May 2017 01:31:30 +0200 (CEST) From: Dimitry Andric Message-Id: <45696549-B3B7-4080-AB2B-7A1DC0966C81@FreeBSD.org> Content-Type: multipart/signed; boundary="Apple-Mail=_28F03236-9184-43E4-A21E-EA1248916F82"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: llvm FreeBSD powerpc ABI target bug fix: Re: [Bug 26519] Clang 4.0.0's "Target: powerpc-unknown-freebsd11.0" code generation is violating the SVR4 ABI (SEGV can result) Date: Fri, 26 May 2017 01:31:18 +0200 In-Reply-To: Cc: FreeBSD Toolchain , FreeBSD PowerPC ML To: Mark Millard References: <0103401A-CEEA-4992-A45E-E60EA151119B@dsl-only.net> <893ECA11-7C80-4D24-A496-92ADC7978A07@FreeBSD.org> <408D3509-3D62-4413-986B-6C1171FB6138@dsl-only.net> X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 23:31:40 -0000 --Apple-Mail=_28F03236-9184-43E4-A21E-EA1248916F82 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On 25 May 2017, at 23:59, Mark Millard wrote: > > Is llvm bugzilla's latest 26519 fix going to make it > into release/11.1.0 ? This fixes the last known stack > handling ABI violation for targeting powerpc FreeBSD > (32-bit). I just committed it in r318906. It should make 11.1-RELEASE. Would be nice if we can finally close PR 206990. :) -Dimitry --Apple-Mail=_28F03236-9184-43E4-A21E-EA1248916F82 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.30 iEYEARECAAYFAlknaVIACgkQsF6jCi4glqPCQwCdH7xbkCUG6e7E4I4eFINg3Qfj 5EwAoKmPAfmwW+maIJkWidS6EDfd6x1f =bvq7 -----END PGP SIGNATURE----- --Apple-Mail=_28F03236-9184-43E4-A21E-EA1248916F82-- From owner-freebsd-ppc@freebsd.org Fri May 26 00:29:14 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B83DD7AF9F for ; Fri, 26 May 2017 00:29:14 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-63.reflexion.net [208.70.210.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AFC18118D for ; Fri, 26 May 2017 00:29:13 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 28661 invoked from network); 26 May 2017 00:29:12 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 26 May 2017 00:29:12 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Thu, 25 May 2017 20:29:12 -0400 (EDT) Received: (qmail 6258 invoked from network); 26 May 2017 00:29:11 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 26 May 2017 00:29:11 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 1A3FAEC7D4D; Thu, 25 May 2017 17:29:11 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: llvm FreeBSD powerpc ABI target bug fix: Re: [Bug 26519] Clang 4.0.0's "Target: powerpc-unknown-freebsd11.0" code generation is violating the SVR4 ABI (SEGV can result) From: Mark Millard In-Reply-To: <45696549-B3B7-4080-AB2B-7A1DC0966C81@FreeBSD.org> Date: Thu, 25 May 2017 17:29:10 -0700 Cc: FreeBSD Toolchain , FreeBSD PowerPC ML Content-Transfer-Encoding: 7bit Message-Id: References: <0103401A-CEEA-4992-A45E-E60EA151119B@dsl-only.net> <893ECA11-7C80-4D24-A496-92ADC7978A07@FreeBSD.org> <408D3509-3D62-4413-986B-6C1171FB6138@dsl-only.net> <45696549-B3B7-4080-AB2B-7A1DC0966C81@FreeBSD.org> To: Dimitry Andric X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 00:29:14 -0000 On 2017-May-25, at 4:31 PM, Dimitry Andric wrote: > On 25 May 2017, at 23:59, Mark Millard wrote: >> >> Is llvm bugzilla's latest 26519 fix going to make it >> into release/11.1.0 ? This fixes the last known stack >> handling ABI violation for targeting powerpc FreeBSD >> (32-bit). > > I just committed it in r318906. It should make 11.1-RELEASE. Would be > nice if we can finally close PR 206990. :) To my knowledge there are no more stack handling problems and use of the update should provide evidence that 206990 can be closed. But it may be a while before I update to direct use of something after head -r317820. . . I've frozen at head -317820 while trying to get evidence for a periodic powerpc kernel panic in that version (kernel built with gcc 4.2.1, worlds via both compilers). Small adjustments that change the kernel memory layout a little tend to change the behavior, usually meaning no obvious symptoms. So updating to some other version likely just hides the problem. Unfortunately it frequently takes hours for the panic to occur and so a long time to conclude the panic is not likely to happen in any test that I have running. It may be that I just will not identify anything that points at anything fairly specific to fix but I've not given up yet. What I know of for powerpc's clang status with the stack handling fixes in place is: A) Handling thrown C++ exceptions crashes for code generated by clang. (powerpc64 also has this property.) B) A clang-based kernel fails to start /sbin/init and then gets a data alignment panic. Past investigations relative to (A) suggest multiple problems are around that contribute, it is not just one simple problem. I've not looked at (B) yet. === Mark Millard markmi at dsl-only.net From owner-freebsd-ppc@freebsd.org Fri May 26 11:00:36 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFAAAD7B669 for ; Fri, 26 May 2017 11:00:36 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-63.reflexion.net [208.70.210.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 75DB01061 for ; Fri, 26 May 2017 11:00:35 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 11194 invoked from network); 26 May 2017 11:00:34 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 26 May 2017 11:00:34 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Fri, 26 May 2017 07:00:34 -0400 (EDT) Received: (qmail 1700 invoked from network); 26 May 2017 11:00:34 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 26 May 2017 11:00:34 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 58486EC805D; Fri, 26 May 2017 04:00:33 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: PowerPC64 C++ exceptions with clang/llvm From: Mark Millard In-Reply-To: <74D091B2-71CD-42DE-8C1E-8E0BDED1EAFD@dsl-only.net> Date: Fri, 26 May 2017 04:00:32 -0700 Cc: Roman Divacky Content-Transfer-Encoding: quoted-printable Message-Id: References: <20170524113441.GA58092@vlakno.cz> <74D091B2-71CD-42DE-8C1E-8E0BDED1EAFD@dsl-only.net> To: FreeBSD Toolchain , FreeBSD PowerPC ML X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:00:36 -0000 [Turns out Roman's working case was based on gcc 4.2.1 based system libraries, not ones built by clang.] On 2017-May-24, at 9:03 PM, Mark Millard wrote: > [So far I've been unable to replicate dwarfump reporting > anything near "0x94 00 00 00 01 00 01 02 ed 14 1b". The > compiled code still gets SIGSEGV: it is only the low > level detail that I'm noting as different. Roman has > more information from me than I show here.] >=20 > On 2017-May-24, at 4:34 AM, Roman Divacky = wrote: >=20 >> Hi, >>=20 >> Currently C++ exceptions don't work on PowerPC64 when compiled with = clang. I >> looked at why and discovered the following. >>=20 >> With the default gcc unwinder and libstdc++ this simple test program: >>=20 >> #include >>=20 >> int main() >> { >> try { >> std::cout << "Before\n"; >> throw std::exception(); >> std::cout << "After\n"; >> } catch (...) { >> std::cout << "Exception\n"; >> } >> } >>=20 >> segfaults right after printing "Before\n". This is because the = .eh_frame is corrupted and >> the personality relocation is wrong: >>=20 >> < 1> version 1 >> cie section offset 56 0x00000038 >> augmentation zPLR >> code_alignment_factor 4 >> data_alignment_factor -8 >> return_address_register 65 >> eh aug data len 0xb bytes 0x94 00 00 00 01 00 01 02 ed 14 1b=20 >=20 > So far I'm unable to replicate producing anything like > the "large one" as I'll call it. >=20 > I get things like: >=20 > # clang++ -std=3Dc++11 exception_test_dots.cpp > # dwarfdump -v -v -F a.out | more >=20 > .eh_frame > . . . > cie: > < 0> version 1 > . . . > eh aug data len 0xb bytes 0x94 00 00 00 00 00 01 03 45 14 1b=20 > . . . >=20 > (No cia<1> at all.) >=20 > and like: >=20 > # clang++ -B /usr/local/bin/ -std=3Dc++11 exception_test_dots.cpp > # dwarfdump -v -v -F a.out | more >=20 > .eh_frame > . . . > cie: > < 0> version 1 > . . . > < 1> version 1 > . . . > eh aug data len 0xb bytes 0x94 00 00 00 00 00 01 f1 1d 14 1b=20 > . . . >=20 > My context is head -r317820 on a system built with > the system clang without building gcc 4.2.1 materials. >=20 > # uname -paKU > FreeBSD FBSDG5L 12.0-CURRENT FreeBSD 12.0-CURRENT r317820M powerpc = powerpc64 1200030 1200030 >=20 > # ld --version > GNU ld 2.17.50 [FreeBSD] 2007-07-03 > Copyright 2007 Free Software Foundation, Inc. > This program is free software; you may redistribute it under the terms = of > the GNU General Public License. This program has absolutely no = warranty. >=20 > # /usr/local/bin/ld --version > GNU ld (GNU Binutils) 2.27 > Copyright (C) 2016 Free Software Foundation, Inc. > This program is free software; you may redistribute it under the terms = of > the GNU General Public License version 3 or (at your option) a later = version. > This program has absolutely no warranty. >=20 > (I have avoided 2.28 so far because of the > "return code" consequences on the likes of > aarch64.) >=20 > I've tried variations like omitting -std=3Dc++11. > So far I've not found a way to get near: >=20 > "0x94 00 00 00 01 00 01 02 ed 14 1b" >=20 > but all the experiments get SIGSEGV when > I run the produced program. >=20 > (I do not have a gcc 4.2.1 build around and so > do not have libstdc++ on powerpc64.) It is still true that I've not replicated the "large one" problems (as I called it). But my environment is not a mix of gcc 4.2.1 and clang based: just clang based, including libc++ and the like. So things are very different from Roman's context. >> bytes of initial instructions 3 >> cie length 28 >> initial instructions >> 0 DW_CFA_def_cfa r1 0 >>=20 >> the personality relocation is "00 00 00 01 00 01 02 ed" encoded as pc = relative. The program segfaults >> because this address is wrong. The 33rd bit is incorrect, the real = address should be "00 00 00 00 00 01 02 ed". >> The same problem applies for LDSA encoding. Both personality and LDSA = relocations are produced by CFI instructions >> in the assembler source code. >>=20 >> I verified this theory by hacking the gcc unwinder and libsupc++ &=3D = the address with 0xffffffff; and that made >> the application work. (the full patch can be found at = http://www.vlakno.cz/~rdivacky/ppc64.exceptions.hack.patch). Roman did not test a libstdc++ built with clang here. He was using a library to support handling thrown C++ exceptions that was built by gcc 4.2.1 . Instead the only clang code tested was in his a.out (or whatever he named the program generated). My past testing indicates that whatever library is supposed to support thrown C++ exceptions is broken when built by clang. (I've a couple of llvm submittals about messed up dwarf information and the like.) In essence the consequences of the builtin's that are involved in creating the supporting code in the library is wrong/incomplete in clang's code and/or dwarf generation as I understand. For example: __builtin_unwind_init() __builtin_eh_return_data_regno (0) __builtin_eh_return __builtin_dwarf_cfa() were involved back when I did the clang 3.8 investigation (that mostly seems to still apply in the C++ exception handling area). I'm not surprised that clang generated code can use the library code that gcc 4.2.1 generated, at least for the small examples involved (and probably far more). >> I checked why this relocation is wrong and it turns out it's our old = in-tree ld that (wrongly) produces this. >> Our old gcc does not use CFI to produce these and hardcodes the CIE = manually and produces correct address. And for >> some reason ld is fine with that. >>=20 >> When I compile and assemble the above source code using clang++ and = only use newer ld linker from ports the resulting >> binary is correct and works as expected. >>=20 >> So this is a bug in our in-tree ld linker. I don't know why ld has = this bug but given Mark Millard's report of ld >> being broken for other stuff (kernel iirc) I am not going to bother = trying to fix it. I suspect it might be related >> to comdats but I didn't really investigate. As my libc++ and the like was built with clang for my tests, the system ld does not work for buildworld for such a context, nor does lld. So I used devel/powerpc64-binutils for buildworld and buildkernel. Given that clang built it all instead of gcc 4.2.1, what the system ld does for the little test program in my context would be expected to be likely to not match Roman's results. But my context gives more information about the status of things for attempting to avoid gcc 4.2.1, since that is what I was doing (gcc 4.2.1 and related material not built or present) . >> I didn't try the situation with the LLVM unwinder and libcxxrt, but I = hope it might be the same. Last I tried the llvm unwinder could not be be built by clang. I submitted llvm bugzilla material for it at the time. lld did not work last I tried either. (In fact Roman supplied one patch that let me get far enough to report some on what else is a problem for lld.) =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-ppc@freebsd.org Sat May 27 02:14:38 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F40EAD83949 for ; Sat, 27 May 2017 02:14:37 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-62.reflexion.net [208.70.210.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD1B81CFB for ; Sat, 27 May 2017 02:14:37 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 15020 invoked from network); 27 May 2017 02:14:30 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 27 May 2017 02:14:30 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Fri, 26 May 2017 22:14:30 -0400 (EDT) Received: (qmail 30784 invoked from network); 27 May 2017 02:14:30 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 02:14:30 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 5C8C9EC7B46; Fri, 26 May 2017 19:14:29 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) Message-Id: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> Date: Fri, 26 May 2017 19:14:28 -0700 To: Justin Hibbits , Nathan Whitehorn , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 02:14:38 -0000 I lucked out and got a vmcore.9 for a random panic that I could manage to backtrace for one of my test builds of -r317820. It appears that not all that much happened before it got the panic so much context was better preserved this time. (I do not explore from ddb as I've had that panic and mess up the dump just made by replacing it. So this is a manual backtrace from the debug.minidump=3D0 style vmcore.9 file. objdump was used on the /boot/kernel/kernel to find code.) Being able to see the problem is very sensitive to kernel memory layout. This is why I'm sticking with -r317820 built production style: the kind of context the problem was first observed in. Attempting a debug kernel build simply did not repeat the problem for days (vs. the usual hours for builds like this). So below is the backtrace: (I do not show what trap_fatal calls: starting with trap_fatal and going toward larger memory addresses. . .) [vmcore.9's offset in file when no 0x prefix] 013e83b0 df 5e 55 50 00 8f 34 5c fa 50 05 af fa 50 05 af = |.^UP..4\.P...P..| 0x008f3454 mr r3,r26 0x008f3458 bl 008f2030 0x008f345c b 008f34c8 013e83c0 fa 50 05 af fa 50 05 af fa 50 05 af fa 50 05 af = |.P...P...P...P..| * 013e83e0 df 5e 54 00 fa 50 05 af fa 50 05 af fa 50 05 af = |.^T..P...P...P..| 013e83f0 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 00 = |.P...P.......^T.| 013e8400 df 5e 54 20 00 53 3a d0 fa 50 05 af fa 50 05 af |.^T = .S:..P...P..| 013e8410 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 20 = |.P...P.......^T | 013e8420 df 5e 54 d0 00 53 3a d0 00 00 00 00 00 00 00 00 = |.^T..S:.........| 013e8430 00 00 00 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |................| 013e8440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| * 013e8460 00 00 00 54 7f ff ff ff 00 00 00 00 ff ff ff aa = |...T............| 013e8470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| * 013e8490 00 d4 c4 5c 00 d4 c4 5c 00 17 99 dd 00 17 9a 5c = |...\...\.......\| 013e84a0 00 00 17 9a 00 d4 c4 6c df 5e 54 ec 00 00 00 54 = |.......l.^T....T| 013e84b0 df 5e 54 d0 7f ff ff ff 05 aa 36 c0 05 aa 39 e8 = |.^T.......6...9.| 013e84c0 00 00 00 00 00 00 00 00 00 d1 ca ac df 5e 54 d0 = |.............^T.| 013e84d0 df 5e 55 80 00 53 3a d0 05 91 d0 00 05 91 d3 28 = |.^U..S:........(| 013e84e0 df 5e 55 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |.^U.............| 013e84f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| 013e8500 00 00 00 00 00 00 00 00 00 d4 bd ec 00 cb 98 98 = |................| 013e8510 00 d4 c4 5c 00 d4 c4 5c 00 17 99 f8 00 17 99 f8 = |...\...\........| 013e8520 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 17 99 = |.....i.y........| 013e8530 f8 1c 2f cc df 5e 55 88 01 47 d6 c0 01 43 a2 00 = |../..^U..G...C..| 013e8540 41 eb 30 00 0a 00 00 00 00 d2 6e 4c df 5e 55 50 = |A.0.......nL.^UP| 013e8550 df 5e 55 80 00 8e 7d 40 df 5e 55 9c 00 00 00 28 = |.^U...}@.^U....(| 0x008e7d28 mfmsr r0 0x008e7d2c or r0,r0,r9 0x008e7d30 mtmsr r0 0x008e7d34 isync 0x008e7d38 mr r3,r25 0x008e7d3c bl 008f222c 0x008e7d40 lwz r11,0(r1) 013e8560 00 00 00 00 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.........@...C..| 013e8570 41 eb 30 00 0a 00 00 00 00 00 00 00 00 08 10 32 = |A.0............2| r0 r1 013e8580 df 5e 56 40 00 10 08 f8 00 00 00 04 df 5e 56 40 = |.^V@.........^V@| The struct trapframe starts is 0x 013e8588 in the vmcore.9 file. The 0x00108f8 is as shown below: 0x001008ec isync 0x001008f0 addi r3,r1,8 0x001008f4 bl 008e7ba4 0x001008f8 mfmsr r3 0x001008fc andi. r3,r3,32767 013e8590 01 47 d6 c0 00 00 00 28 01 47 c6 c0 00 00 00 04 = |.G.....(.G......| r2 r3 r4 r5 013e85a0 00 00 00 04 00 00 00 0f 00 00 00 00 00 d4 c0 3c = |...............<| r6 r7 r8 r9 013e85b0 01 47 d6 c0 df 5e 56 80 14 3d 60 da 00 00 00 00 = |.G...^V..=3D`.....| r10 r11 r12 r13 013e85c0 00 d4 bd ec 00 cb 98 98 00 d4 c4 5c 00 d4 c4 5c = |...........\...\| r14 r15 r16 r17 013e85d0 00 17 99 f8 00 17 99 f8 00 00 17 99 fa 69 b8 79 = |.............i.y| r18 r19 r20 r21 013e85e0 00 17 9a 0a 00 00 17 99 f8 1c 2f cc 00 00 17 9a = |........../.....| r22 r32 r24 r25 013e85f0 06 40 c2 a8 01 43 a2 00 00 eb a7 80 01 47 d6 c0 = |.@...C.......G..| r26 r27 r28 r29 013e8600 00 d1 ca ac df 5e 56 40 00 53 5a d0 20 00 90 44 = |.....^V@.SZ. ..D| r30 r31 lr cr xer ctr srr0 srr1 013e8610 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| Note: objdump shows no code at 0x0090a030. 0x0090a030 is inside what objdump -x reports as the section .hash (Idx 2). exc dar dsisr (booke dbcer0) 013e8620 00 00 07 00 41 eb 30 00 0a 00 00 00 01 47 c6 c0 = |....A.0......G..| The 0x00007000 above is the framep->exc (exception code) (program in this case). 013e8630 00 eb a7 80 01 47 60 10 00 d1 ca ac df 5e 56 40 = |.....G`......^V@| At this point the above does not match the below part of the stack trace. The lr part of struct trapframe was: 0x00535ad0 so showing around that: 00535ab8 stwu r1,-32(r1) 00535abc mflr r0 00535ac0 stw r29,20(r1) 00535ac4 stw r30,24(r1) 00535ac8 stw r31,28(r1) 00535acc stw r0,36(r1) 00535ad0 mr r31,r1 00535ad4 mr r29,r3 Back to the stack backtrace. . . 013e8640 df 5e 56 80 00 53 59 dc 00 17 99 f8 00 17 99 f8 = |.^V..SY.........| 0x005359c8 bl 008ea420 0x005359cc mr r3,r28 0x005359d0 mr r4,r27 0x005359d4 mr r5,r25 0x005359d8 bl 005356ec 0x005359dc mfsprg r9,0 I show around 0x005356ec from the sched_add+0x19c bl that is above because the routine is not referenced in the stack tracce but the above indicates that it should have been called: 0x005356ec stwu r1,-32(r1) 0x005356f0 mflr r0 0x005356f4 stw r28,16(r1) 0x005356f8 stw r29,20(r1) 0x005356fc stw r30,24(r1) 0x00535700 stw r31,28(r1) 0x00535704 stw r0,36(r1) . . . Back to the stack backtrace again. . . 013e8650 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 00 04 = |.....i.y........| 013e8660 f8 1c 2f cc 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |../......@...C..| 013e8670 01 47 c6 c0 01 47 60 10 00 d1 b4 30 df 5e 56 80 = |.G...G`....0.^V.| 013e8680 df 5e 56 b0 00 4a 87 8c 00 d2 5b 10 00 00 00 04 = |.^V..J....[.....| 0x004a8780 mr r3,r28 0x004a8784 li r4,4 0x004a8788 bl 0053583c = 0x004a878c lwz r9,0(r28) 013e8690 df 5e 56 b0 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.^V......@...C..| 013e86a0 00 00 00 00 01 46 81 c0 00 d1 b4 30 df 5e 56 b0 = |.....F.....0.^V.| 013e86b0 df 5e 56 f0 00 4a 97 0c 00 00 00 00 00 00 00 04 = |.^V..J..........| 0x004a9700 bl 005000ec 0x004a9704 mr r3,r27 0x004a9708 bl 004a86bc = 0x004a970c lwz r11,0(r1) 013e86c0 df 5e 56 e0 01 47 d6 c0 01 47 d6 c0 01 45 4d 40 = |.^V..G...G...EM@| 013e86d0 df 5e 56 f0 00 8e a4 44 06 40 c2 a8 00 00 17 9a = |.^V....D.@......| 013e86e0 78 00 00 00 00 e9 56 00 00 d1 c8 20 df 5e 56 f0 |x.....V.... = .^V.| 013e86f0 df 5e 57 50 00 51 79 6c df 5e 58 78 01 47 d6 c0 = |.^WP.Qyl.^Xx.G..| 0x00517960 lwz r3,264(r29) 0x00517964 li r4,0 0x00517968 bl 004a965c 0x0051796c lwz r11,0(r1) 013e8700 01 47 d7 b8 00 00 00 00 00 d1 ab 24 00 00 00 04 = |.G.........$....| 013e8710 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| 013e8720 00 d0 53 00 00 eb a7 80 00 00 00 01 00 00 00 00 = |..S.............| 013e8730 df 5e 59 8c 00 00 00 00 df 5e 58 78 00 00 17 99 = |.^Y......^Xx....| 013e8740 f8 1c 2f cc d0 01 dd 00 00 d2 5b 10 df 5e 57 50 = |../.......[..^WP| 013e8750 df 5e 57 a0 00 8a b2 70 df 5e 57 60 df 5e 57 60 = |.^W....p.^W`.^W`| 0x008ab264 mr r3,r27 0x008ab268 mr r4,r28 0x008ab26c bl 00517540 0x008ab270 li r3,0 013e8760 df 5e 57 a0 df 5e 58 78 05 86 37 00 00 00 00 04 = |.^W..^Xx..7.....| 013e8770 00 00 00 00 05 9b f2 00 00 c9 66 bc 01 47 d6 c0 = |..........f..G..| 013e8780 df 5e 59 8c 00 f6 1d 10 00 00 17 99 f8 1c 2f cc = |.^Y.........../.| 013e8790 d0 01 dd 00 d0 01 dd 30 00 d2 5b 10 df 5e 57 a0 = |.......0..[..^W.| 013e87a0 df 5e 58 20 00 8a d1 10 00 d2 6e 5c df 5e 57 b0 |.^X = ......n\.^W.| 0x008ad100 mr r3,r26 0x008ad104 mr r4,r27 0x008ad108 li r5,0 0x008ad10c bl 008aafc0 0x008ad110 lwz r11,0(r1) 013e87b0 df 5e 57 e0 00 4a 96 00 00 00 17 99 00 00 00 00 = |.^W..J..........| 013e87c0 f8 1c 2f cc 0a 3f a0 12 df 5e 58 78 05 86 37 00 = |../..?...^Xx..7.| 013e87d0 01 48 b1 00 05 86 37 80 00 d4 bd ec 00 cb 98 98 = |.H....7.........| 013e87e0 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| 013e87f0 df 5e 59 e0 00 eb a7 80 00 c9 66 bc 01 47 d6 c0 = |.^Y.......f..G..| 013e8800 df 5e 59 8c df 5e 58 78 01 47 d6 c0 00 00 00 00 = |.^Y..^Xx.G......| 013e8810 00 f6 1d 10 00 00 00 01 00 d2 6b c8 df 5e 58 20 = |..........k..^X | 013e8820 df 5e 58 40 00 8e 1e 48 00 00 00 00 00 eb a7 80 = |.^X@...H........| So 0x13e8820 from vmcore.9 has start of struct trapeframe. The 0x8e1e48 is from: 0x008e1e34 lwz r0,56(r28) 0x008e1e38 mtctr r0 0x008e1e3c mr r3,r28 0x008e1e40 lwz r4,64(r28) 0x008e1e44 bctrl 0x008e1e48 addi r29,r29,-1 013e8830 01 47 d7 94 00 00 00 01 00 d2 6e 4c df 5e 58 40 = |.G........nL.^X@| 013e8840 df 5e 58 70 00 8e 7c 9c 00 d1 ca ac df 5e 58 50 = |.^Xp..|......^XP| 013e8850 00 cd f0 74 00 00 00 01 00 00 00 01 00 eb a7 80 = |...t............| 013e8860 41 eb 30 00 0a 00 00 00 00 00 00 00 00 00 90 32 = |A.0............2| 013e8870 df 5e 59 30 00 10 08 f8 00 04 90 32 df 5e 59 30 = |.^Y0.......2.^Y0| 013e8880 01 47 d6 c0 00 00 00 00 0d 0b bf e3 00 00 00 00 = |.G..............| 013e8890 0d 0b bf e3 00 19 eb 7c 00 00 00 00 00 00 00 44 = |.......|.......D| 013e88a0 01 fc a0 55 00 00 90 32 d0 01 dd 00 00 00 00 00 = |...U...2........| 013e88b0 00 d4 bd ec 00 cb 98 98 00 c9 66 bc 00 c4 5d 48 = |..........f...]H| 013e88c0 00 c9 66 bc 00 d4 c5 3c df 5e 59 e0 00 eb a7 80 = |..f....<.^Y.....| 013e88d0 00 c9 66 bc 01 47 d6 c0 df 5e 59 8c 00 00 00 01 = |..f..G...^Y.....| 013e88e0 00 00 00 01 00 eb a7 80 00 00 00 00 00 8e 3b f8 = |..............;.| srr0 013e88f0 00 d2 6b f0 df 5e 59 30 00 8e 3c 14 40 00 00 42 = |..k..^Y0..<.@..B| The 0x00833c14 from the trap frame is the srr0 (conceptual lr): 0x008e3c04 stw r31,28(r1) 0x008e3c08 stw r0,36(r1) 0x008e3c0c mr r31,r1 0x008e3c10 bcl- 20,4*cr7+so,008e3c14 = 0x008e3c14 mflr r30 0x008e3c18 lwz r0,-32(r30) 013e8900 20 00 00 00 00 8e 3b f8 00 8e 3c 80 00 00 90 32 | = .....;...<....2| exc 013e8910 00 00 09 00 41 eb 30 00 0a 00 00 00 00 00 00 00 = |....A.0.........| The 0x00009000 above is the framep->exc (exception code) (decrementer in this case). 013e8920 eb 10 25 b5 55 7e c4 22 00 00 00 00 00 00 00 04 = |..%.U~."........| 013e8930 df 5e 59 50 00 00 00 01 00 00 00 01 00 eb a7 80 = |.^YP............| (trap [above] means no lr filled in here) 013e8940 00 00 00 00 00 d4 ca 34 00 d2 6b f0 df 5e 59 50 = |.......4..k..^YP| 013e8950 df 5e 59 70 00 8e 31 9c 00 00 00 02 00 eb a7 80 = |.^Yp..1.........| 0x008e318c bl 008ad8b8 0x008e3190 lwz r29,0(r29) 0x008e3194 mtctr r29 0x008e3198 bctrl 0x008e319c bl 008ad7b8 013e8960 00 f2 d5 fc 00 00 00 01 00 d1 ca ac df 5e 59 70 = |.............^Yp| 013e8970 df 5e 5a 50 00 53 6e 7c fa 50 05 af fa 50 05 af = |.^ZP.Sn|.P...P..| 0x00536e78 bl 008e3144 0x00536e7c stw r28,136(r27) 013e8980 fa 50 05 af fa 50 05 af fa 50 05 af ff ff ff fe = |.P...P...P......| 013e8990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| 013e89a0 ff ff ff ff ff ff ff ff ff ff ff ff fa 50 05 af = |.............P..| 013e89b0 fa 50 05 af 00 00 00 02 ff ff ff ff 00 00 01 f0 = |.P..............| 013e89c0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| 013e89d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| 013e89e0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| 013e89f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| 013e8a00 df 5e 5a 20 fa 50 05 af 00 00 00 00 00 00 00 00 |.^Z = .P..........| 013e8a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| * 013e8a30 00 00 00 00 00 53 69 a8 df 5e 5a 98 00 00 00 00 = |.....Si..^Z.....| 013e8a40 01 47 96 e0 01 47 d6 c0 00 d1 b3 70 df 5e 5a 50 = |.G...G.....p.^ZP| 013e8a50 df 5e 5a 80 00 4a 3c b4 df 5e 5a 60 df 5e 5a 60 = |.^Z..J<..^Z`.^Z`| 0x004a3ca0 bl 008ea420 0x004a3ca4 mr r3,r27 0x004a3ca8 mr r4,r26 0x004a3cac mtctr r25 0x004a3cb0 bctrl 0x004a3cb4 lwz r0,108(r28) 013e8a60 df 5e 5a 80 00 00 00 00 00 00 00 00 00 00 00 00 = |.^Z.............| 013e8a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| 013e8a80 00 00 00 00 00 8f 18 d0 00 53 69 a8 00 00 00 00 = |.........Si.....| 0x008f18c0 lwz r3,8(r1) 0x008f18c4 lwz r4,12(r1) 0x008f18c8 lwz r5,16(r1) 0x008f18cc bl 004a3bbc 0x008f18d0 addi r1,r1,16 0x008f18d4 b 001008f8 So that is an example context for the failure. It has taken weeks to get this. It may be some time before I get another for comparison/contrast. And sticking with the same build vs. trying some more to find a better way to get more evidence is not obvious to me at this point. =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-ppc@freebsd.org Sat May 27 05:29:18 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9961CD84F60 for ; Sat, 27 May 2017 05:29:18 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-63.reflexion.net [208.70.210.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5E8D21D1C for ; Sat, 27 May 2017 05:29:17 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 1287 invoked from network); 27 May 2017 05:29:16 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 27 May 2017 05:29:16 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Sat, 27 May 2017 01:29:16 -0400 (EDT) Received: (qmail 19300 invoked from network); 27 May 2017 05:29:16 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 05:29:16 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 9310FEC7B46; Fri, 26 May 2017 22:29:15 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) [better bt; more] Date: Fri, 26 May 2017 22:29:14 -0700 References: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> To: Justin Hibbits , Nathan Whitehorn , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org In-Reply-To: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> Message-Id: <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 05:29:18 -0000 [Additional information that does not need to interlace with the prior material, so see after. A somewhat better backtrace reported by ddb. And so on.] On 2017-May-26, at 7:14 PM, Mark Millard wrote: > I lucked out and got a vmcore.9 for a random > panic that I could manage to backtrace for > one of my test builds of -r317820. It appears > that not all that much happened before it got > the panic so much context was better preserved > this time. >=20 > (I do not explore from ddb as I've had that > panic and mess up the dump just made by > replacing it. So this is a manual backtrace > from the debug.minidump=3D0 style vmcore.9 > file. objdump was used on the > /boot/kernel/kernel to find code.) >=20 > Being able to see the problem is very > sensitive to kernel memory layout. This > is why I'm sticking with -r317820 built > production style: the kind of context > the problem was first observed in. > Attempting a debug kernel build simply > did not repeat the problem for days > (vs. the usual hours for builds like > this). >=20 >=20 > So below is the backtrace: >=20 > (I do not show what trap_fatal calls: > starting with trap_fatal and going toward > larger memory addresses. . .) >=20 > [vmcore.9's > offset > in file > when no > 0x prefix] >=20 > 013e83b0 df 5e 55 50 00 8f 34 5c fa 50 05 af fa 50 05 af = |.^UP..4\.P...P..| > 0x008f3454 mr r3,r26 > 0x008f3458 bl 008f2030 > 0x008f345c b 008f34c8 >=20 > 013e83c0 fa 50 05 af fa 50 05 af fa 50 05 af fa 50 05 af = |.P...P...P...P..| > * > 013e83e0 df 5e 54 00 fa 50 05 af fa 50 05 af fa 50 05 af = |.^T..P...P...P..| > 013e83f0 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 00 = |.P...P.......^T.| > 013e8400 df 5e 54 20 00 53 3a d0 fa 50 05 af fa 50 05 af |.^T = .S:..P...P..| > 013e8410 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 20 = |.P...P.......^T | > 013e8420 df 5e 54 d0 00 53 3a d0 00 00 00 00 00 00 00 00 = |.^T..S:.........| > 013e8430 00 00 00 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |................| > 013e8440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 013e8460 00 00 00 54 7f ff ff ff 00 00 00 00 ff ff ff aa = |...T............| > 013e8470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 013e8490 00 d4 c4 5c 00 d4 c4 5c 00 17 99 dd 00 17 9a 5c = |...\...\.......\| > 013e84a0 00 00 17 9a 00 d4 c4 6c df 5e 54 ec 00 00 00 54 = |.......l.^T....T| > 013e84b0 df 5e 54 d0 7f ff ff ff 05 aa 36 c0 05 aa 39 e8 = |.^T.......6...9.| > 013e84c0 00 00 00 00 00 00 00 00 00 d1 ca ac df 5e 54 d0 = |.............^T.| > 013e84d0 df 5e 55 80 00 53 3a d0 05 91 d0 00 05 91 d3 28 = |.^U..S:........(| > 013e84e0 df 5e 55 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |.^U.............| > 013e84f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > 013e8500 00 00 00 00 00 00 00 00 00 d4 bd ec 00 cb 98 98 = |................| > 013e8510 00 d4 c4 5c 00 d4 c4 5c 00 17 99 f8 00 17 99 f8 = |...\...\........| > 013e8520 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 17 99 = |.....i.y........| > 013e8530 f8 1c 2f cc df 5e 55 88 01 47 d6 c0 01 43 a2 00 = |../..^U..G...C..| > 013e8540 41 eb 30 00 0a 00 00 00 00 d2 6e 4c df 5e 55 50 = |A.0.......nL.^UP| >=20 > 013e8550 df 5e 55 80 00 8e 7d 40 df 5e 55 9c 00 00 00 28 = |.^U...}@.^U....(| > 0x008e7d28 mfmsr r0 > 0x008e7d2c or r0,r0,r9 > 0x008e7d30 mtmsr r0 > 0x008e7d34 isync > 0x008e7d38 mr r3,r25 > 0x008e7d3c bl 008f222c > 0x008e7d40 lwz r11,0(r1) >=20 > 013e8560 00 00 00 00 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.........@...C..| > 013e8570 41 eb 30 00 0a 00 00 00 00 00 00 00 00 08 10 32 = |A.0............2| >=20 > r0 r1 > 013e8580 df 5e 56 40 00 10 08 f8 00 00 00 04 df 5e 56 40 = |.^V@.........^V@| > The struct trapframe starts is 0x 013e8588 in the > vmcore.9 file. The 0x00108f8 is as shown below: >=20 > 0x001008ec isync > 0x001008f0 addi r3,r1,8 > 0x001008f4 bl 008e7ba4 > 0x001008f8 mfmsr r3 > 0x001008fc andi. r3,r3,32767 >=20 > 013e8590 01 47 d6 c0 00 00 00 28 01 47 c6 c0 00 00 00 04 = |.G.....(.G......| > r2 r3 r4 r5 >=20 > 013e85a0 00 00 00 04 00 00 00 0f 00 00 00 00 00 d4 c0 3c = |...............<| > r6 r7 r8 r9 >=20 > 013e85b0 01 47 d6 c0 df 5e 56 80 14 3d 60 da 00 00 00 00 = |.G...^V..=3D`.....| > r10 r11 r12 r13 >=20 > 013e85c0 00 d4 bd ec 00 cb 98 98 00 d4 c4 5c 00 d4 c4 5c = |...........\...\| > r14 r15 r16 r17 >=20 > 013e85d0 00 17 99 f8 00 17 99 f8 00 00 17 99 fa 69 b8 79 = |.............i.y| > r18 r19 r20 r21 >=20 > 013e85e0 00 17 9a 0a 00 00 17 99 f8 1c 2f cc 00 00 17 9a = |........../.....| > r22 r32 r24 r25 >=20 > 013e85f0 06 40 c2 a8 01 43 a2 00 00 eb a7 80 01 47 d6 c0 = |.@...C.......G..| > r26 r27 r28 r29 >=20 > 013e8600 00 d1 ca ac df 5e 56 40 00 53 5a d0 20 00 90 44 = |.....^V@.SZ. ..D| > r30 r31 lr cr >=20 > xer ctr srr0 srr1 > 013e8610 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| > Note: objdump shows no code at 0x0090a030. 0x0090a030 is > inside what objdump -x reports as the section .hash (Idx > 2). >=20 > exc dar dsisr (booke dbcer0) > 013e8620 00 00 07 00 41 eb 30 00 0a 00 00 00 01 47 c6 c0 = |....A.0......G..| > The 0x00007000 above is the framep->exc (exception code) > (program in this case). >=20 > 013e8630 00 eb a7 80 01 47 60 10 00 d1 ca ac df 5e 56 40 = |.....G`......^V@| >=20 > At this point the above does not match the below > part of the stack trace. >=20 > The lr part of struct trapframe was: 0x00535ad0 so > showing around that: >=20 > 00535ab8 stwu r1,-32(r1) > 00535abc mflr r0 > 00535ac0 stw r29,20(r1) > 00535ac4 stw r30,24(r1) > 00535ac8 stw r31,28(r1) > 00535acc stw r0,36(r1) > 00535ad0 mr r31,r1 > 00535ad4 mr r29,r3 >=20 > Back to the stack backtrace. . . >=20 > 013e8640 df 5e 56 80 00 53 59 dc 00 17 99 f8 00 17 99 f8 = |.^V..SY.........| > 0x005359c8 bl 008ea420 > 0x005359cc mr r3,r28 > 0x005359d0 mr r4,r27 > 0x005359d4 mr r5,r25 > 0x005359d8 bl 005356ec > 0x005359dc mfsprg r9,0 >=20 > I show around 0x005356ec > from the sched_add+0x19c bl that is above > because the routine is not referenced > in the stack tracce but the above indicates > that it should have been called: > 0x005356ec stwu r1,-32(r1) > 0x005356f0 mflr r0 > 0x005356f4 stw r28,16(r1) > 0x005356f8 stw r29,20(r1) > 0x005356fc stw r30,24(r1) > 0x00535700 stw r31,28(r1) > 0x00535704 stw r0,36(r1) > . . . >=20 > Back to the stack backtrace again. . . >=20 > 013e8650 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 00 04 = |.....i.y........| > 013e8660 f8 1c 2f cc 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |../......@...C..| > 013e8670 01 47 c6 c0 01 47 60 10 00 d1 b4 30 df 5e 56 80 = |.G...G`....0.^V.| >=20 > 013e8680 df 5e 56 b0 00 4a 87 8c 00 d2 5b 10 00 00 00 04 = |.^V..J....[.....| > 0x004a8780 mr r3,r28 > 0x004a8784 li r4,4 > 0x004a8788 bl 0053583c = > 0x004a878c lwz r9,0(r28) >=20 > 013e8690 df 5e 56 b0 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.^V......@...C..| > 013e86a0 00 00 00 00 01 46 81 c0 00 d1 b4 30 df 5e 56 b0 = |.....F.....0.^V.| >=20 > 013e86b0 df 5e 56 f0 00 4a 97 0c 00 00 00 00 00 00 00 04 = |.^V..J..........| > 0x004a9700 bl 005000ec > 0x004a9704 mr r3,r27 > 0x004a9708 bl 004a86bc = > 0x004a970c lwz r11,0(r1) >=20 > 013e86c0 df 5e 56 e0 01 47 d6 c0 01 47 d6 c0 01 45 4d 40 = |.^V..G...G...EM@| > 013e86d0 df 5e 56 f0 00 8e a4 44 06 40 c2 a8 00 00 17 9a = |.^V....D.@......| > 013e86e0 78 00 00 00 00 e9 56 00 00 d1 c8 20 df 5e 56 f0 = |x.....V.... .^V.| >=20 > 013e86f0 df 5e 57 50 00 51 79 6c df 5e 58 78 01 47 d6 c0 = |.^WP.Qyl.^Xx.G..| > 0x00517960 lwz r3,264(r29) > 0x00517964 li r4,0 > 0x00517968 bl 004a965c > 0x0051796c lwz r11,0(r1) >=20 > 013e8700 01 47 d7 b8 00 00 00 00 00 d1 ab 24 00 00 00 04 = |.G.........$....| > 013e8710 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| > 013e8720 00 d0 53 00 00 eb a7 80 00 00 00 01 00 00 00 00 = |..S.............| > 013e8730 df 5e 59 8c 00 00 00 00 df 5e 58 78 00 00 17 99 = |.^Y......^Xx....| > 013e8740 f8 1c 2f cc d0 01 dd 00 00 d2 5b 10 df 5e 57 50 = |../.......[..^WP| >=20 > 013e8750 df 5e 57 a0 00 8a b2 70 df 5e 57 60 df 5e 57 60 = |.^W....p.^W`.^W`| > 0x008ab264 mr r3,r27 > 0x008ab268 mr r4,r28 > 0x008ab26c bl 00517540 > 0x008ab270 li r3,0 >=20 > 013e8760 df 5e 57 a0 df 5e 58 78 05 86 37 00 00 00 00 04 = |.^W..^Xx..7.....| > 013e8770 00 00 00 00 05 9b f2 00 00 c9 66 bc 01 47 d6 c0 = |..........f..G..| > 013e8780 df 5e 59 8c 00 f6 1d 10 00 00 17 99 f8 1c 2f cc = |.^Y.........../.| > 013e8790 d0 01 dd 00 d0 01 dd 30 00 d2 5b 10 df 5e 57 a0 = |.......0..[..^W.| >=20 > 013e87a0 df 5e 58 20 00 8a d1 10 00 d2 6e 5c df 5e 57 b0 |.^X = ......n\.^W.| > 0x008ad100 mr r3,r26 > 0x008ad104 mr r4,r27 > 0x008ad108 li r5,0 > 0x008ad10c bl 008aafc0 > 0x008ad110 lwz r11,0(r1) >=20 > 013e87b0 df 5e 57 e0 00 4a 96 00 00 00 17 99 00 00 00 00 = |.^W..J..........| > 013e87c0 f8 1c 2f cc 0a 3f a0 12 df 5e 58 78 05 86 37 00 = |../..?...^Xx..7.| > 013e87d0 01 48 b1 00 05 86 37 80 00 d4 bd ec 00 cb 98 98 = |.H....7.........| > 013e87e0 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| > 013e87f0 df 5e 59 e0 00 eb a7 80 00 c9 66 bc 01 47 d6 c0 = |.^Y.......f..G..| > 013e8800 df 5e 59 8c df 5e 58 78 01 47 d6 c0 00 00 00 00 = |.^Y..^Xx.G......| > 013e8810 00 f6 1d 10 00 00 00 01 00 d2 6b c8 df 5e 58 20 = |..........k..^X | >=20 > 013e8820 df 5e 58 40 00 8e 1e 48 00 00 00 00 00 eb a7 80 = |.^X@...H........| > So 0x13e8820 from vmcore.9 has start of struct trapeframe. > The 0x8e1e48 is from: > 0x008e1e34 lwz r0,56(r28) > 0x008e1e38 mtctr r0 > 0x008e1e3c mr r3,r28 > 0x008e1e40 lwz r4,64(r28) > 0x008e1e44 bctrl > 0x008e1e48 addi r29,r29,-1 >=20 > 013e8830 01 47 d7 94 00 00 00 01 00 d2 6e 4c df 5e 58 40 = |.G........nL.^X@| > 013e8840 df 5e 58 70 00 8e 7c 9c 00 d1 ca ac df 5e 58 50 = |.^Xp..|......^XP| > 013e8850 00 cd f0 74 00 00 00 01 00 00 00 01 00 eb a7 80 = |...t............| > 013e8860 41 eb 30 00 0a 00 00 00 00 00 00 00 00 00 90 32 = |A.0............2| > 013e8870 df 5e 59 30 00 10 08 f8 00 04 90 32 df 5e 59 30 = |.^Y0.......2.^Y0| > 013e8880 01 47 d6 c0 00 00 00 00 0d 0b bf e3 00 00 00 00 = |.G..............| > 013e8890 0d 0b bf e3 00 19 eb 7c 00 00 00 00 00 00 00 44 = |.......|.......D| > 013e88a0 01 fc a0 55 00 00 90 32 d0 01 dd 00 00 00 00 00 = |...U...2........| > 013e88b0 00 d4 bd ec 00 cb 98 98 00 c9 66 bc 00 c4 5d 48 = |..........f...]H| > 013e88c0 00 c9 66 bc 00 d4 c5 3c df 5e 59 e0 00 eb a7 80 = |..f....<.^Y.....| > 013e88d0 00 c9 66 bc 01 47 d6 c0 df 5e 59 8c 00 00 00 01 = |..f..G...^Y.....| > 013e88e0 00 00 00 01 00 eb a7 80 00 00 00 00 00 8e 3b f8 = |..............;.| >=20 > srr0 > 013e88f0 00 d2 6b f0 df 5e 59 30 00 8e 3c 14 40 00 00 42 = |..k..^Y0..<.@..B| > The 0x00833c14 from the trap frame is the srr0 (conceptual lr): > 0x008e3c04 stw r31,28(r1) > 0x008e3c08 stw r0,36(r1) > 0x008e3c0c mr r31,r1 > 0x008e3c10 bcl- 20,4*cr7+so,008e3c14 = > 0x008e3c14 mflr r30 > 0x008e3c18 lwz r0,-32(r30) >=20 > 013e8900 20 00 00 00 00 8e 3b f8 00 8e 3c 80 00 00 90 32 | = .....;...<....2| >=20 > exc > 013e8910 00 00 09 00 41 eb 30 00 0a 00 00 00 00 00 00 00 = |....A.0.........| > The 0x00009000 above is the framep->exc (exception code) > (decrementer in this case). >=20 > 013e8920 eb 10 25 b5 55 7e c4 22 00 00 00 00 00 00 00 04 = |..%.U~."........| >=20 > 013e8930 df 5e 59 50 00 00 00 01 00 00 00 01 00 eb a7 80 = |.^YP............| > (trap [above] means no lr filled in here) >=20 > 013e8940 00 00 00 00 00 d4 ca 34 00 d2 6b f0 df 5e 59 50 = |.......4..k..^YP| >=20 > 013e8950 df 5e 59 70 00 8e 31 9c 00 00 00 02 00 eb a7 80 = |.^Yp..1.........| > 0x008e318c bl 008ad8b8 > 0x008e3190 lwz r29,0(r29) > 0x008e3194 mtctr r29 > 0x008e3198 bctrl > 0x008e319c bl 008ad7b8 >=20 > 013e8960 00 f2 d5 fc 00 00 00 01 00 d1 ca ac df 5e 59 70 = |.............^Yp| >=20 > 013e8970 df 5e 5a 50 00 53 6e 7c fa 50 05 af fa 50 05 af = |.^ZP.Sn|.P...P..| > 0x00536e78 bl 008e3144 > 0x00536e7c stw r28,136(r27) >=20 >=20 > 013e8980 fa 50 05 af fa 50 05 af fa 50 05 af ff ff ff fe = |.P...P...P......| > 013e8990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89a0 ff ff ff ff ff ff ff ff ff ff ff ff fa 50 05 af = |.............P..| > 013e89b0 fa 50 05 af 00 00 00 02 ff ff ff ff 00 00 01 f0 = |.P..............| > 013e89c0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89e0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e8a00 df 5e 5a 20 fa 50 05 af 00 00 00 00 00 00 00 00 |.^Z = .P..........| > 013e8a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 013e8a30 00 00 00 00 00 53 69 a8 df 5e 5a 98 00 00 00 00 = |.....Si..^Z.....| > 013e8a40 01 47 96 e0 01 47 d6 c0 00 d1 b3 70 df 5e 5a 50 = |.G...G.....p.^ZP| >=20 > 013e8a50 df 5e 5a 80 00 4a 3c b4 df 5e 5a 60 df 5e 5a 60 = |.^Z..J<..^Z`.^Z`| > 0x004a3ca0 bl 008ea420 > 0x004a3ca4 mr r3,r27 > 0x004a3ca8 mr r4,r26 > 0x004a3cac mtctr r25 > 0x004a3cb0 bctrl > 0x004a3cb4 lwz r0,108(r28) >=20 > 013e8a60 df 5e 5a 80 00 00 00 00 00 00 00 00 00 00 00 00 = |.^Z.............| > 013e8a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >=20 > 013e8a80 00 00 00 00 00 8f 18 d0 00 53 69 a8 00 00 00 00 = |.........Si.....| > 0x008f18c0 lwz r3,8(r1) > 0x008f18c4 lwz r4,12(r1) > 0x008f18c8 lwz r5,16(r1) > 0x008f18cc bl 004a3bbc > 0x008f18d0 addi r1,r1,16 > 0x008f18d4 b 001008f8 >=20 >=20 > So that is an example context for the failure. >=20 > It has taken weeks to get this. It may be some > time before I get another for comparison/contrast. >=20 > And sticking with the same build vs. trying some more > to find a better way to get more evidence is not > obvious to me at this point. I should have mentioned that this is TARGET_ARCH=3Dpowerpc but used on a so-called PowerMac G5 "Quad Core". I've had 2 more examples, with the same 0x0090a030 srr0 and such (vmcore.0 and vmcore.1) (I have added one more little block of code for detecting an earlier problem symptom not being currently seen so the build is slight different from the earlier report.) Looking around in vmcore.0 I find 3 examples of "00 90 a0 30" in areas overlapping with objdump -x's coverage. . . 00c77fd0 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| [ ] (from sorted objdump -x output:) 00c775a8 l O .data 00000dc0 seqprog 00c78368 l O .data 0000000c seeprom_long_ewen [ ] 00f65820 41 eb 30 00 0a 00 00 00 00 90 a0 30 00 08 10 32 = |A.0........0...2| . . . 00f65870 00 90 a0 30 00 08 10 32 00 00 00 00 df 5d 30 00 = |...0...2.....]0.| [ ] (from sorted objdump -x output:) 00f64780 g O .bss 00020000 __pcpu 00f84780 l O .bss 00000004 ap_letgo For vmcore.1 I went ahead and tried exploring some with ddb --and had no new panics. . . (Hand transcriptions of pictures) fatal kernel trap: exception =3D 0x700 (program) srr0 =3D 0x90a030 srr1 =3D 0x81032 lr =3D 0x535ad0 curthread =3D 0x147d6c0 pic =3D 11, comm =3D idle: cpu0 [ thread pid 11 tid 100003 ] Stopped at _etext+0xb8fc: illegal instruction 0 db> bt 0xdf5e55d0: at sched_wakeup+0xa4 0xdf5e55f0: at setrunnable+0x9c 0xdf5e5610: at sleepq_resume_thread+0x17c 0xdf5e5640: at sleepq_timeout+0xc8 0xdf5e5680: at softclock_call_cc+0x1f0 0xdf5e56f0: at callout_process+0x27c 0xdf5e57a0: at timercb+0x4c4 0xdf5e5820: at decr_intr+0xf0 0xdf5e5840: at powerpc_interrupt_0xf4 0xdf5e5870: at kernel DECR trap by cpu_idle_60x+0x88 (so: srr0) srr1=3D0x9032 r1 =3D0xdf5e5930 cr =3D0x40000042 xer =3D0x20000000 ctr =3D0x8e3bf8 saved LR(0xfffffffd) is invalid. db> show reg (but reformatted r0 =3D0x4 r1 =3D0xdf5e5590 r2 =3D0x147d6c0 r3 =3D0x54 testppc64size+0x34 r4 =3D0x591d000 r5 =3D0 r6 =3D0 r7 =3D0xf r8 =3D0 r9 =3D0xd4c03c cold r10 =3D0x147d6c0 r11 =3D0xdf5e55d0 r12 =3D0 r13 =3D0 r14 =3D0xd4bdec sdt_probe_func r15 =3D0xcb9898 std_lockstat___spin__release r16 =3D0xd4c45c callwheelmask r17 =3D0xd4c45c callwheelmask r18 =3D0x55925 r19 =3D0x559a4 r20 =3D0x559 dsmisssize+0x469 r21 =3D0x591d000 r22 =3D0x566430 sleeppq_timeout r23 =3D0x114 dsmisssize+0x24 r24 =3D0 r25 =3D0 r26 =3D0x1 r27 =3D0 r28 =3D0xeba780 tdq_cpu r29 =3D0x147d6c0 r30 =3D0xd1caac r31 =3D0xdf5e5590 srr0 =3D0x90a030 srr1 =3D0x81032 lr =3D0x535ad0 shed_affinity+0x18 ctr =3D0 cr =3D0x20009034 xer =3D0 dar =3D0x419df5d4 dsisr=3D0x24000000 _etext+0xb8fc: illegal instruction 0 Just for completeness: acttrace also showed: Tracing command less pid 1144 tid 100150 td 0x5bc8a20 (CPU 3) 0xd25a59f0: at powrpc_dispatch_intr+0xc8 0xd25a5a20: at openpic_dispatch+0x90 0xd25a5a50: at powerpc_interrupt+0xc0 0xd25a5a80: at user EXI trap by 0x181c68c (so: ssr0) r1 =3D0xffffdb30 cr =3D0x44020624 xer=3D0 ctr=3D0x41989570 Tracing command idle pid 11 tid 100004 td 0x147d360 (CPU 1) saved LR(0x4c) in invalid Tracing command idle pid 11 tid 100005 td 0x147d360 (CPU 2) saved LR(0x4c) in invalid =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-ppc@freebsd.org Sat May 27 07:11:44 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D66CD84973 for ; Sat, 27 May 2017 07:11:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8C5FD154E for ; Sat, 27 May 2017 07:11:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v4R7BhWp098842 for ; Sat, 27 May 2017 07:11:44 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-ppc@FreeBSD.org Subject: [Bug 205458] 11.0-CURRENT/10-STABLE powerpc64: a PowerMac G5 specific sys/powerpc/ofw/ofw_machdep.c change for reliable PowerMac G5 booting (with lots of RAM) Date: Sat, 27 May 2017 07:11:43 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markmi@dsl-only.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jhibbits@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 07:11:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205458 --- Comment #20 from Mark Millard --- (In reply to Justin Hibbits from comment #19) It turns out that if TARGET_ARCH=3Dpowerpc is to work on powerpc64 PowerMac's reliably then it needs to detect the context and also avoid the sprg0 update. It turns out that the problems and evidence that I've been reporting are from the code restoring the openfirmware sprg0 lile the TARGET_ARCH=3Dpowerpc64 code used to do. I'll submit a separate, new report. But I figured the relationship would be good to note here. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-ppc@freebsd.org Sat May 27 07:42:23 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 593D7D8434C for ; Sat, 27 May 2017 07:42:23 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-63.reflexion.net [208.70.210.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2111914E0 for ; Sat, 27 May 2017 07:42:22 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 29459 invoked from network); 27 May 2017 07:42:21 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 27 May 2017 07:42:21 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Sat, 27 May 2017 03:42:21 -0400 (EDT) Received: (qmail 3295 invoked from network); 27 May 2017 07:42:21 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 07:42:21 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 5D05FEC7B46; Sat, 27 May 2017 00:42:20 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) [better bt; more] From: Mark Millard In-Reply-To: <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> Date: Sat, 27 May 2017 00:42:19 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <62BF8E69-E7E6-4C4F-AB33-38B03E903CDA@dsl-only.net> References: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> To: Justin Hibbits , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 07:42:23 -0000 [Top post of the answer to what is wrong. I have submitted 219589 for this.] TARGET_ARCH=3Dpowerpc64 got a fix to bugzilla 205458, avoiding inappropriate restoration of openfirmware's sprg0 value. It turns out that TARGET_ARCH=3Dpowerpc needs to detect when it is running on powerpc64 and do the same thing if it is to avoid trashing memory and running unreliably on PowerMac G5's. The issue is that for powerpc64 it is inappropriate to restore the sprg0 value to its openfirmware value. This is because the FreeBSD real-mode handling is involved instead of the openfirmware's original virtual mode, making openfirware's value simply inappropriate. Quoting Nathan W. from Comment #4 of 205458: > Where this explodes is if OF uses an unmapped SLB entry. > The SLB fault handler runs in real mode and refers to the > PCPU pointer in SPRG0, which blows up the kernel. Having > a value of SPRG0 that works for the kernel is less fatal > than preserving OF's value in this case. I know that part of the code does detect the powerpc64 context vs. not and does things differently to emulate being powerpc like on powerpc64 (such as limiting RAM use as a consequence). The powerpc64 vs. not status needs to be recorded and used to control a sprg0 restoration choice: avoid restoring openfirmware's value on powerpc64; otherwise restore it. =3D=3D=3D Mark Millard markmi at dsl-only.net On 2017-May-26, at 10:29 PM, Mark Millard wrote: [Additional information that does not need to interlace with the prior material, so see after. A somewhat better backtrace reported by ddb. And so on.] On 2017-May-26, at 7:14 PM, Mark Millard wrote: > I lucked out and got a vmcore.9 for a random > panic that I could manage to backtrace for > one of my test builds of -r317820. It appears > that not all that much happened before it got > the panic so much context was better preserved > this time. >=20 > (I do not explore from ddb as I've had that > panic and mess up the dump just made by > replacing it. So this is a manual backtrace > from the debug.minidump=3D0 style vmcore.9 > file. objdump was used on the > /boot/kernel/kernel to find code.) >=20 > Being able to see the problem is very > sensitive to kernel memory layout. This > is why I'm sticking with -r317820 built > production style: the kind of context > the problem was first observed in. > Attempting a debug kernel build simply > did not repeat the problem for days > (vs. the usual hours for builds like > this). >=20 >=20 > So below is the backtrace: >=20 > (I do not show what trap_fatal calls: > starting with trap_fatal and going toward > larger memory addresses. . .) >=20 > [vmcore.9's > offset > in file > when no > 0x prefix] >=20 > 013e83b0 df 5e 55 50 00 8f 34 5c fa 50 05 af fa 50 05 af = |.^UP..4\.P...P..| > 0x008f3454 mr r3,r26 > 0x008f3458 bl 008f2030 > 0x008f345c b 008f34c8 >=20 > 013e83c0 fa 50 05 af fa 50 05 af fa 50 05 af fa 50 05 af = |.P...P...P...P..| > * > 013e83e0 df 5e 54 00 fa 50 05 af fa 50 05 af fa 50 05 af = |.^T..P...P...P..| > 013e83f0 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 00 = |.P...P.......^T.| > 013e8400 df 5e 54 20 00 53 3a d0 fa 50 05 af fa 50 05 af |.^T = .S:..P...P..| > 013e8410 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 20 = |.P...P.......^T | > 013e8420 df 5e 54 d0 00 53 3a d0 00 00 00 00 00 00 00 00 = |.^T..S:.........| > 013e8430 00 00 00 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |................| > 013e8440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 013e8460 00 00 00 54 7f ff ff ff 00 00 00 00 ff ff ff aa = |...T............| > 013e8470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 013e8490 00 d4 c4 5c 00 d4 c4 5c 00 17 99 dd 00 17 9a 5c = |...\...\.......\| > 013e84a0 00 00 17 9a 00 d4 c4 6c df 5e 54 ec 00 00 00 54 = |.......l.^T....T| > 013e84b0 df 5e 54 d0 7f ff ff ff 05 aa 36 c0 05 aa 39 e8 = |.^T.......6...9.| > 013e84c0 00 00 00 00 00 00 00 00 00 d1 ca ac df 5e 54 d0 = |.............^T.| > 013e84d0 df 5e 55 80 00 53 3a d0 05 91 d0 00 05 91 d3 28 = |.^U..S:........(| > 013e84e0 df 5e 55 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |.^U.............| > 013e84f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > 013e8500 00 00 00 00 00 00 00 00 00 d4 bd ec 00 cb 98 98 = |................| > 013e8510 00 d4 c4 5c 00 d4 c4 5c 00 17 99 f8 00 17 99 f8 = |...\...\........| > 013e8520 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 17 99 = |.....i.y........| > 013e8530 f8 1c 2f cc df 5e 55 88 01 47 d6 c0 01 43 a2 00 = |../..^U..G...C..| > 013e8540 41 eb 30 00 0a 00 00 00 00 d2 6e 4c df 5e 55 50 = |A.0.......nL.^UP| >=20 > 013e8550 df 5e 55 80 00 8e 7d 40 df 5e 55 9c 00 00 00 28 = |.^U...}@.^U....(| > 0x008e7d28 mfmsr r0 > 0x008e7d2c or r0,r0,r9 > 0x008e7d30 mtmsr r0 > 0x008e7d34 isync > 0x008e7d38 mr r3,r25 > 0x008e7d3c bl 008f222c > 0x008e7d40 lwz r11,0(r1) >=20 > 013e8560 00 00 00 00 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.........@...C..| > 013e8570 41 eb 30 00 0a 00 00 00 00 00 00 00 00 08 10 32 = |A.0............2| >=20 > r0 r1 > 013e8580 df 5e 56 40 00 10 08 f8 00 00 00 04 df 5e 56 40 = |.^V@.........^V@| > The struct trapframe starts is 0x 013e8588 in the > vmcore.9 file. The 0x00108f8 is as shown below: >=20 > 0x001008ec isync > 0x001008f0 addi r3,r1,8 > 0x001008f4 bl 008e7ba4 > 0x001008f8 mfmsr r3 > 0x001008fc andi. r3,r3,32767 >=20 > 013e8590 01 47 d6 c0 00 00 00 28 01 47 c6 c0 00 00 00 04 = |.G.....(.G......| > r2 r3 r4 r5 >=20 > 013e85a0 00 00 00 04 00 00 00 0f 00 00 00 00 00 d4 c0 3c = |...............<| > r6 r7 r8 r9 >=20 > 013e85b0 01 47 d6 c0 df 5e 56 80 14 3d 60 da 00 00 00 00 = |.G...^V..=3D`.....| > r10 r11 r12 r13 >=20 > 013e85c0 00 d4 bd ec 00 cb 98 98 00 d4 c4 5c 00 d4 c4 5c = |...........\...\| > r14 r15 r16 r17 >=20 > 013e85d0 00 17 99 f8 00 17 99 f8 00 00 17 99 fa 69 b8 79 = |.............i.y| > r18 r19 r20 r21 >=20 > 013e85e0 00 17 9a 0a 00 00 17 99 f8 1c 2f cc 00 00 17 9a = |........../.....| > r22 r32 r24 r25 >=20 > 013e85f0 06 40 c2 a8 01 43 a2 00 00 eb a7 80 01 47 d6 c0 = |.@...C.......G..| > r26 r27 r28 r29 >=20 > 013e8600 00 d1 ca ac df 5e 56 40 00 53 5a d0 20 00 90 44 = |.....^V@.SZ. ..D| > r30 r31 lr cr >=20 > xer ctr srr0 srr1 > 013e8610 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| > Note: objdump shows no code at 0x0090a030. 0x0090a030 is > inside what objdump -x reports as the section .hash (Idx > 2). >=20 > exc dar dsisr (booke dbcer0) > 013e8620 00 00 07 00 41 eb 30 00 0a 00 00 00 01 47 c6 c0 = |....A.0......G..| > The 0x00007000 above is the framep->exc (exception code) > (program in this case). >=20 > 013e8630 00 eb a7 80 01 47 60 10 00 d1 ca ac df 5e 56 40 = |.....G`......^V@| >=20 > At this point the above does not match the below > part of the stack trace. >=20 > The lr part of struct trapframe was: 0x00535ad0 so > showing around that: >=20 > 00535ab8 stwu r1,-32(r1) > 00535abc mflr r0 > 00535ac0 stw r29,20(r1) > 00535ac4 stw r30,24(r1) > 00535ac8 stw r31,28(r1) > 00535acc stw r0,36(r1) > 00535ad0 mr r31,r1 > 00535ad4 mr r29,r3 >=20 > Back to the stack backtrace. . . >=20 > 013e8640 df 5e 56 80 00 53 59 dc 00 17 99 f8 00 17 99 f8 = |.^V..SY.........| > 0x005359c8 bl 008ea420 > 0x005359cc mr r3,r28 > 0x005359d0 mr r4,r27 > 0x005359d4 mr r5,r25 > 0x005359d8 bl 005356ec > 0x005359dc mfsprg r9,0 >=20 > I show around 0x005356ec > from the sched_add+0x19c bl that is above > because the routine is not referenced > in the stack tracce but the above indicates > that it should have been called: > 0x005356ec stwu r1,-32(r1) > 0x005356f0 mflr r0 > 0x005356f4 stw r28,16(r1) > 0x005356f8 stw r29,20(r1) > 0x005356fc stw r30,24(r1) > 0x00535700 stw r31,28(r1) > 0x00535704 stw r0,36(r1) > . . . >=20 > Back to the stack backtrace again. . . >=20 > 013e8650 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 00 04 = |.....i.y........| > 013e8660 f8 1c 2f cc 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |../......@...C..| > 013e8670 01 47 c6 c0 01 47 60 10 00 d1 b4 30 df 5e 56 80 = |.G...G`....0.^V.| >=20 > 013e8680 df 5e 56 b0 00 4a 87 8c 00 d2 5b 10 00 00 00 04 = |.^V..J....[.....| > 0x004a8780 mr r3,r28 > 0x004a8784 li r4,4 > 0x004a8788 bl 0053583c = > 0x004a878c lwz r9,0(r28) >=20 > 013e8690 df 5e 56 b0 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.^V......@...C..| > 013e86a0 00 00 00 00 01 46 81 c0 00 d1 b4 30 df 5e 56 b0 = |.....F.....0.^V.| >=20 > 013e86b0 df 5e 56 f0 00 4a 97 0c 00 00 00 00 00 00 00 04 = |.^V..J..........| > 0x004a9700 bl 005000ec > 0x004a9704 mr r3,r27 > 0x004a9708 bl 004a86bc = > 0x004a970c lwz r11,0(r1) >=20 > 013e86c0 df 5e 56 e0 01 47 d6 c0 01 47 d6 c0 01 45 4d 40 = |.^V..G...G...EM@| > 013e86d0 df 5e 56 f0 00 8e a4 44 06 40 c2 a8 00 00 17 9a = |.^V....D.@......| > 013e86e0 78 00 00 00 00 e9 56 00 00 d1 c8 20 df 5e 56 f0 = |x.....V.... .^V.| >=20 > 013e86f0 df 5e 57 50 00 51 79 6c df 5e 58 78 01 47 d6 c0 = |.^WP.Qyl.^Xx.G..| > 0x00517960 lwz r3,264(r29) > 0x00517964 li r4,0 > 0x00517968 bl 004a965c > 0x0051796c lwz r11,0(r1) >=20 > 013e8700 01 47 d7 b8 00 00 00 00 00 d1 ab 24 00 00 00 04 = |.G.........$....| > 013e8710 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| > 013e8720 00 d0 53 00 00 eb a7 80 00 00 00 01 00 00 00 00 = |..S.............| > 013e8730 df 5e 59 8c 00 00 00 00 df 5e 58 78 00 00 17 99 = |.^Y......^Xx....| > 013e8740 f8 1c 2f cc d0 01 dd 00 00 d2 5b 10 df 5e 57 50 = |../.......[..^WP| >=20 > 013e8750 df 5e 57 a0 00 8a b2 70 df 5e 57 60 df 5e 57 60 = |.^W....p.^W`.^W`| > 0x008ab264 mr r3,r27 > 0x008ab268 mr r4,r28 > 0x008ab26c bl 00517540 > 0x008ab270 li r3,0 >=20 > 013e8760 df 5e 57 a0 df 5e 58 78 05 86 37 00 00 00 00 04 = |.^W..^Xx..7.....| > 013e8770 00 00 00 00 05 9b f2 00 00 c9 66 bc 01 47 d6 c0 = |..........f..G..| > 013e8780 df 5e 59 8c 00 f6 1d 10 00 00 17 99 f8 1c 2f cc = |.^Y.........../.| > 013e8790 d0 01 dd 00 d0 01 dd 30 00 d2 5b 10 df 5e 57 a0 = |.......0..[..^W.| >=20 > 013e87a0 df 5e 58 20 00 8a d1 10 00 d2 6e 5c df 5e 57 b0 |.^X = ......n\.^W.| > 0x008ad100 mr r3,r26 > 0x008ad104 mr r4,r27 > 0x008ad108 li r5,0 > 0x008ad10c bl 008aafc0 > 0x008ad110 lwz r11,0(r1) >=20 > 013e87b0 df 5e 57 e0 00 4a 96 00 00 00 17 99 00 00 00 00 = |.^W..J..........| > 013e87c0 f8 1c 2f cc 0a 3f a0 12 df 5e 58 78 05 86 37 00 = |../..?...^Xx..7.| > 013e87d0 01 48 b1 00 05 86 37 80 00 d4 bd ec 00 cb 98 98 = |.H....7.........| > 013e87e0 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| > 013e87f0 df 5e 59 e0 00 eb a7 80 00 c9 66 bc 01 47 d6 c0 = |.^Y.......f..G..| > 013e8800 df 5e 59 8c df 5e 58 78 01 47 d6 c0 00 00 00 00 = |.^Y..^Xx.G......| > 013e8810 00 f6 1d 10 00 00 00 01 00 d2 6b c8 df 5e 58 20 = |..........k..^X | >=20 > 013e8820 df 5e 58 40 00 8e 1e 48 00 00 00 00 00 eb a7 80 = |.^X@...H........| > So 0x13e8820 from vmcore.9 has start of struct trapeframe. > The 0x8e1e48 is from: > 0x008e1e34 lwz r0,56(r28) > 0x008e1e38 mtctr r0 > 0x008e1e3c mr r3,r28 > 0x008e1e40 lwz r4,64(r28) > 0x008e1e44 bctrl > 0x008e1e48 addi r29,r29,-1 >=20 > 013e8830 01 47 d7 94 00 00 00 01 00 d2 6e 4c df 5e 58 40 = |.G........nL.^X@| > 013e8840 df 5e 58 70 00 8e 7c 9c 00 d1 ca ac df 5e 58 50 = |.^Xp..|......^XP| > 013e8850 00 cd f0 74 00 00 00 01 00 00 00 01 00 eb a7 80 = |...t............| > 013e8860 41 eb 30 00 0a 00 00 00 00 00 00 00 00 00 90 32 = |A.0............2| > 013e8870 df 5e 59 30 00 10 08 f8 00 04 90 32 df 5e 59 30 = |.^Y0.......2.^Y0| > 013e8880 01 47 d6 c0 00 00 00 00 0d 0b bf e3 00 00 00 00 = |.G..............| > 013e8890 0d 0b bf e3 00 19 eb 7c 00 00 00 00 00 00 00 44 = |.......|.......D| > 013e88a0 01 fc a0 55 00 00 90 32 d0 01 dd 00 00 00 00 00 = |...U...2........| > 013e88b0 00 d4 bd ec 00 cb 98 98 00 c9 66 bc 00 c4 5d 48 = |..........f...]H| > 013e88c0 00 c9 66 bc 00 d4 c5 3c df 5e 59 e0 00 eb a7 80 = |..f....<.^Y.....| > 013e88d0 00 c9 66 bc 01 47 d6 c0 df 5e 59 8c 00 00 00 01 = |..f..G...^Y.....| > 013e88e0 00 00 00 01 00 eb a7 80 00 00 00 00 00 8e 3b f8 = |..............;.| >=20 > srr0 > 013e88f0 00 d2 6b f0 df 5e 59 30 00 8e 3c 14 40 00 00 42 = |..k..^Y0..<.@..B| > The 0x00833c14 from the trap frame is the srr0 (conceptual lr): > 0x008e3c04 stw r31,28(r1) > 0x008e3c08 stw r0,36(r1) > 0x008e3c0c mr r31,r1 > 0x008e3c10 bcl- 20,4*cr7+so,008e3c14 = > 0x008e3c14 mflr r30 > 0x008e3c18 lwz r0,-32(r30) >=20 > 013e8900 20 00 00 00 00 8e 3b f8 00 8e 3c 80 00 00 90 32 | = .....;...<....2| >=20 > exc > 013e8910 00 00 09 00 41 eb 30 00 0a 00 00 00 00 00 00 00 = |....A.0.........| > The 0x00009000 above is the framep->exc (exception code) > (decrementer in this case). >=20 > 013e8920 eb 10 25 b5 55 7e c4 22 00 00 00 00 00 00 00 04 = |..%.U~."........| >=20 > 013e8930 df 5e 59 50 00 00 00 01 00 00 00 01 00 eb a7 80 = |.^YP............| > (trap [above] means no lr filled in here) >=20 > 013e8940 00 00 00 00 00 d4 ca 34 00 d2 6b f0 df 5e 59 50 = |.......4..k..^YP| >=20 > 013e8950 df 5e 59 70 00 8e 31 9c 00 00 00 02 00 eb a7 80 = |.^Yp..1.........| > 0x008e318c bl 008ad8b8 > 0x008e3190 lwz r29,0(r29) > 0x008e3194 mtctr r29 > 0x008e3198 bctrl > 0x008e319c bl 008ad7b8 >=20 > 013e8960 00 f2 d5 fc 00 00 00 01 00 d1 ca ac df 5e 59 70 = |.............^Yp| >=20 > 013e8970 df 5e 5a 50 00 53 6e 7c fa 50 05 af fa 50 05 af = |.^ZP.Sn|.P...P..| > 0x00536e78 bl 008e3144 > 0x00536e7c stw r28,136(r27) >=20 >=20 > 013e8980 fa 50 05 af fa 50 05 af fa 50 05 af ff ff ff fe = |.P...P...P......| > 013e8990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89a0 ff ff ff ff ff ff ff ff ff ff ff ff fa 50 05 af = |.............P..| > 013e89b0 fa 50 05 af 00 00 00 02 ff ff ff ff 00 00 01 f0 = |.P..............| > 013e89c0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89e0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e89f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| > 013e8a00 df 5e 5a 20 fa 50 05 af 00 00 00 00 00 00 00 00 |.^Z = .P..........| > 013e8a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 013e8a30 00 00 00 00 00 53 69 a8 df 5e 5a 98 00 00 00 00 = |.....Si..^Z.....| > 013e8a40 01 47 96 e0 01 47 d6 c0 00 d1 b3 70 df 5e 5a 50 = |.G...G.....p.^ZP| >=20 > 013e8a50 df 5e 5a 80 00 4a 3c b4 df 5e 5a 60 df 5e 5a 60 = |.^Z..J<..^Z`.^Z`| > 0x004a3ca0 bl 008ea420 > 0x004a3ca4 mr r3,r27 > 0x004a3ca8 mr r4,r26 > 0x004a3cac mtctr r25 > 0x004a3cb0 bctrl > 0x004a3cb4 lwz r0,108(r28) >=20 > 013e8a60 df 5e 5a 80 00 00 00 00 00 00 00 00 00 00 00 00 = |.^Z.............| > 013e8a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >=20 > 013e8a80 00 00 00 00 00 8f 18 d0 00 53 69 a8 00 00 00 00 = |.........Si.....| > 0x008f18c0 lwz r3,8(r1) > 0x008f18c4 lwz r4,12(r1) > 0x008f18c8 lwz r5,16(r1) > 0x008f18cc bl 004a3bbc > 0x008f18d0 addi r1,r1,16 > 0x008f18d4 b 001008f8 >=20 >=20 > So that is an example context for the failure. >=20 > It has taken weeks to get this. It may be some > time before I get another for comparison/contrast. >=20 > And sticking with the same build vs. trying some more > to find a better way to get more evidence is not > obvious to me at this point. I should have mentioned that this is TARGET_ARCH=3Dpowerpc but used on a so-called PowerMac G5 "Quad Core". I've had 2 more examples, with the same 0x0090a030 srr0 and such (vmcore.0 and vmcore.1) (I have added one more little block of code for detecting an earlier problem symptom not being currently seen so the build is slight different from the earlier report.) Looking around in vmcore.0 I find 3 examples of "00 90 a0 30" in areas overlapping with objdump -x's coverage. . . 00c77fd0 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| [ ] (from sorted objdump -x output:) 00c775a8 l O .data 00000dc0 seqprog 00c78368 l O .data 0000000c seeprom_long_ewen [ ] 00f65820 41 eb 30 00 0a 00 00 00 00 90 a0 30 00 08 10 32 = |A.0........0...2| . . . 00f65870 00 90 a0 30 00 08 10 32 00 00 00 00 df 5d 30 00 = |...0...2.....]0.| [ ] (from sorted objdump -x output:) 00f64780 g O .bss 00020000 __pcpu 00f84780 l O .bss 00000004 ap_letgo For vmcore.1 I went ahead and tried exploring some with ddb --and had no new panics. . . (Hand transcriptions of pictures) fatal kernel trap: exception =3D 0x700 (program) srr0 =3D 0x90a030 srr1 =3D 0x81032 lr =3D 0x535ad0 curthread =3D 0x147d6c0 pic =3D 11, comm =3D idle: cpu0 [ thread pid 11 tid 100003 ] Stopped at _etext+0xb8fc: illegal instruction 0 db> bt 0xdf5e55d0: at sched_wakeup+0xa4 0xdf5e55f0: at setrunnable+0x9c 0xdf5e5610: at sleepq_resume_thread+0x17c 0xdf5e5640: at sleepq_timeout+0xc8 0xdf5e5680: at softclock_call_cc+0x1f0 0xdf5e56f0: at callout_process+0x27c 0xdf5e57a0: at timercb+0x4c4 0xdf5e5820: at decr_intr+0xf0 0xdf5e5840: at powerpc_interrupt_0xf4 0xdf5e5870: at kernel DECR trap by cpu_idle_60x+0x88 (so: srr0) srr1=3D0x9032 r1 =3D0xdf5e5930 cr =3D0x40000042 xer =3D0x20000000 ctr =3D0x8e3bf8 saved LR(0xfffffffd) is invalid. db> show reg (but reformatted r0 =3D0x4 r1 =3D0xdf5e5590 r2 =3D0x147d6c0 r3 =3D0x54 testppc64size+0x34 r4 =3D0x591d000 r5 =3D0 r6 =3D0 r7 =3D0xf r8 =3D0 r9 =3D0xd4c03c cold r10 =3D0x147d6c0 r11 =3D0xdf5e55d0 r12 =3D0 r13 =3D0 r14 =3D0xd4bdec sdt_probe_func r15 =3D0xcb9898 std_lockstat___spin__release r16 =3D0xd4c45c callwheelmask r17 =3D0xd4c45c callwheelmask r18 =3D0x55925 r19 =3D0x559a4 r20 =3D0x559 dsmisssize+0x469 r21 =3D0x591d000 r22 =3D0x566430 sleeppq_timeout r23 =3D0x114 dsmisssize+0x24 r24 =3D0 r25 =3D0 r26 =3D0x1 r27 =3D0 r28 =3D0xeba780 tdq_cpu r29 =3D0x147d6c0 r30 =3D0xd1caac r31 =3D0xdf5e5590 srr0 =3D0x90a030 srr1 =3D0x81032 lr =3D0x535ad0 shed_affinity+0x18 ctr =3D0 cr =3D0x20009034 xer =3D0 dar =3D0x419df5d4 dsisr=3D0x24000000 _etext+0xb8fc: illegal instruction 0 Just for completeness: acttrace also showed: Tracing command less pid 1144 tid 100150 td 0x5bc8a20 (CPU 3) 0xd25a59f0: at powrpc_dispatch_intr+0xc8 0xd25a5a20: at openpic_dispatch+0x90 0xd25a5a50: at powerpc_interrupt+0xc0 0xd25a5a80: at user EXI trap by 0x181c68c (so: ssr0) r1 =3D0xffffdb30 cr =3D0x44020624 xer=3D0 ctr=3D0x41989570 Tracing command idle pid 11 tid 100004 td 0x147d360 (CPU 1) saved LR(0x4c) in invalid Tracing command idle pid 11 tid 100005 td 0x147d360 (CPU 2) saved LR(0x4c) in invalid =3D=3D=3D Mark Millard markmi at dsl-only.net _______________________________________________ freebsd-ppc@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ppc To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org" From owner-freebsd-ppc@freebsd.org Sat May 27 08:17:26 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3585FD84E51 for ; Sat, 27 May 2017 08:17:26 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-63.reflexion.net [208.70.210.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EE9AE14EC for ; Sat, 27 May 2017 08:17:25 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 28158 invoked from network); 27 May 2017 08:18:42 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 27 May 2017 08:18:42 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Sat, 27 May 2017 04:17:24 -0400 (EDT) Received: (qmail 32529 invoked from network); 27 May 2017 08:17:24 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 08:17:24 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 4B1DBEC7B46; Sat, 27 May 2017 01:17:23 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) [better bt; more] From: Mark Millard In-Reply-To: <62BF8E69-E7E6-4C4F-AB33-38B03E903CDA@dsl-only.net> Date: Sat, 27 May 2017 01:17:22 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> <62BF8E69-E7E6-4C4F-AB33-38B03E903CDA@dsl-only.net> To: Justin Hibbits , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 08:17:26 -0000 [I suggest a patch this time.] On 2017-May-27, at 12:42 AM, Mark Millard wrote: > [Top post of the answer to what is wrong. I > have submitted 219589 for this.] >=20 > TARGET_ARCH=3Dpowerpc64 got a fix to > bugzilla 205458, avoiding inappropriate > restoration of openfirmware's sprg0 > value. >=20 > It turns out that TARGET_ARCH=3Dpowerpc > needs to detect when it is running on > powerpc64 and do the same thing if it > is to avoid trashing memory and running > unreliably on PowerMac G5's. >=20 > The issue is that for powerpc64 it is > inappropriate to restore the sprg0 > value to its openfirmware value. This > is because the FreeBSD real-mode > handling is involved instead of the > openfirmware's original virtual mode, > making openfirware's value simply > inappropriate. >=20 > Quoting Nathan W. from Comment > #4 of 205458: >=20 >> Where this explodes is if OF uses an unmapped SLB entry. >> The SLB fault handler runs in real mode and refers to the >> PCPU pointer in SPRG0, which blows up the kernel. Having >> a value of SPRG0 that works for the kernel is less fatal >> than preserving OF's value in this case. >=20 > I know that part of the code does detect > the powerpc64 context vs. not and does > things differently to emulate being powerpc > like on powerpc64 (such as limiting > RAM use as a consequence). >=20 > The powerpc64 vs. not status needs to be > recorded and used to control a sprg0 > restoration choice: avoid restoring > openfirmware's value on powerpc64; > otherwise restore it. I expect that the following patch would fix the problem: # svnlite diff /usr/src/sys/powerpc/ofw/ofw_machdep.c Index: /usr/src/sys/powerpc/ofw/ofw_machdep.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/ofw/ofw_machdep.c (revision 317820) +++ /usr/src/sys/powerpc/ofw/ofw_machdep.c (working copy) @@ -147,7 +147,8 @@ * PCPU data cannot be used until this routine is called ! */ #ifndef __powerpc64__ - __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); + if (cpu_features & PPC_FEATURE_64 !=3D PPC_FEATURE_64) + __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); #endif } #endif This is based on cpu_features already having had PPC_FEATURE_64 masked in before this if things are running on a PowerMac G5 or other powerpc64. =3D=3D=3D Mark Millard markmi at dsl-only.net [The original (better) panic evidence. . .] On 2017-May-26, at 10:29 PM, Mark Millard wrote: > [Additional information that does not need to > interlace with the prior material, so see > after. A somewhat better backtrace reported > by ddb. And so on.] >=20 >=20 > On 2017-May-26, at 7:14 PM, Mark Millard = wrote: >=20 >> I lucked out and got a vmcore.9 for a random >> panic that I could manage to backtrace for >> one of my test builds of -r317820. It appears >> that not all that much happened before it got >> the panic so much context was better preserved >> this time. >>=20 >> (I do not explore from ddb as I've had that >> panic and mess up the dump just made by >> replacing it. So this is a manual backtrace >> from the debug.minidump=3D0 style vmcore.9 >> file. objdump was used on the >> /boot/kernel/kernel to find code.) >>=20 >> Being able to see the problem is very >> sensitive to kernel memory layout. This >> is why I'm sticking with -r317820 built >> production style: the kind of context >> the problem was first observed in. >> Attempting a debug kernel build simply >> did not repeat the problem for days >> (vs. the usual hours for builds like >> this). >>=20 >>=20 >> So below is the backtrace: >>=20 >> (I do not show what trap_fatal calls: >> starting with trap_fatal and going toward >> larger memory addresses. . .) >>=20 >> [vmcore.9's >> offset >> in file >> when no >> 0x prefix] >>=20 >> 013e83b0 df 5e 55 50 00 8f 34 5c fa 50 05 af fa 50 05 af = |.^UP..4\.P...P..| >> 0x008f3454 mr r3,r26 >> 0x008f3458 bl 008f2030 >> 0x008f345c b 008f34c8 >>=20 >> 013e83c0 fa 50 05 af fa 50 05 af fa 50 05 af fa 50 05 af = |.P...P...P...P..| >> * >> 013e83e0 df 5e 54 00 fa 50 05 af fa 50 05 af fa 50 05 af = |.^T..P...P...P..| >> 013e83f0 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 00 = |.P...P.......^T.| >> 013e8400 df 5e 54 20 00 53 3a d0 fa 50 05 af fa 50 05 af |.^T = .S:..P...P..| >> 013e8410 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 20 = |.P...P.......^T | >> 013e8420 df 5e 54 d0 00 53 3a d0 00 00 00 00 00 00 00 00 = |.^T..S:.........| >> 013e8430 00 00 00 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |................| >> 013e8440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8460 00 00 00 54 7f ff ff ff 00 00 00 00 ff ff ff aa = |...T............| >> 013e8470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8490 00 d4 c4 5c 00 d4 c4 5c 00 17 99 dd 00 17 9a 5c = |...\...\.......\| >> 013e84a0 00 00 17 9a 00 d4 c4 6c df 5e 54 ec 00 00 00 54 = |.......l.^T....T| >> 013e84b0 df 5e 54 d0 7f ff ff ff 05 aa 36 c0 05 aa 39 e8 = |.^T.......6...9.| >> 013e84c0 00 00 00 00 00 00 00 00 00 d1 ca ac df 5e 54 d0 = |.............^T.| >> 013e84d0 df 5e 55 80 00 53 3a d0 05 91 d0 00 05 91 d3 28 = |.^U..S:........(| >> 013e84e0 df 5e 55 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |.^U.............| >> 013e84f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> 013e8500 00 00 00 00 00 00 00 00 00 d4 bd ec 00 cb 98 98 = |................| >> 013e8510 00 d4 c4 5c 00 d4 c4 5c 00 17 99 f8 00 17 99 f8 = |...\...\........| >> 013e8520 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 17 99 = |.....i.y........| >> 013e8530 f8 1c 2f cc df 5e 55 88 01 47 d6 c0 01 43 a2 00 = |../..^U..G...C..| >> 013e8540 41 eb 30 00 0a 00 00 00 00 d2 6e 4c df 5e 55 50 = |A.0.......nL.^UP| >>=20 >> 013e8550 df 5e 55 80 00 8e 7d 40 df 5e 55 9c 00 00 00 28 = |.^U...}@.^U....(| >> 0x008e7d28 mfmsr r0 >> 0x008e7d2c or r0,r0,r9 >> 0x008e7d30 mtmsr r0 >> 0x008e7d34 isync >> 0x008e7d38 mr r3,r25 >> 0x008e7d3c bl 008f222c >> 0x008e7d40 lwz r11,0(r1) >>=20 >> 013e8560 00 00 00 00 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.........@...C..| >> 013e8570 41 eb 30 00 0a 00 00 00 00 00 00 00 00 08 10 32 = |A.0............2| >>=20 >> r0 r1 >> 013e8580 df 5e 56 40 00 10 08 f8 00 00 00 04 df 5e 56 40 = |.^V@.........^V@| >> The struct trapframe starts is 0x 013e8588 in the >> vmcore.9 file. The 0x00108f8 is as shown below: >>=20 >> 0x001008ec isync >> 0x001008f0 addi r3,r1,8 >> 0x001008f4 bl 008e7ba4 >> 0x001008f8 mfmsr r3 >> 0x001008fc andi. r3,r3,32767 >>=20 >> 013e8590 01 47 d6 c0 00 00 00 28 01 47 c6 c0 00 00 00 04 = |.G.....(.G......| >> r2 r3 r4 r5 >>=20 >> 013e85a0 00 00 00 04 00 00 00 0f 00 00 00 00 00 d4 c0 3c = |...............<| >> r6 r7 r8 r9 >>=20 >> 013e85b0 01 47 d6 c0 df 5e 56 80 14 3d 60 da 00 00 00 00 = |.G...^V..=3D`.....| >> r10 r11 r12 r13 >>=20 >> 013e85c0 00 d4 bd ec 00 cb 98 98 00 d4 c4 5c 00 d4 c4 5c = |...........\...\| >> r14 r15 r16 r17 >>=20 >> 013e85d0 00 17 99 f8 00 17 99 f8 00 00 17 99 fa 69 b8 79 = |.............i.y| >> r18 r19 r20 r21 >>=20 >> 013e85e0 00 17 9a 0a 00 00 17 99 f8 1c 2f cc 00 00 17 9a = |........../.....| >> r22 r32 r24 r25 >>=20 >> 013e85f0 06 40 c2 a8 01 43 a2 00 00 eb a7 80 01 47 d6 c0 = |.@...C.......G..| >> r26 r27 r28 r29 >>=20 >> 013e8600 00 d1 ca ac df 5e 56 40 00 53 5a d0 20 00 90 44 = |.....^V@.SZ. ..D| >> r30 r31 lr cr >>=20 >> xer ctr srr0 srr1 >> 013e8610 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| >> Note: objdump shows no code at 0x0090a030. 0x0090a030 is >> inside what objdump -x reports as the section .hash (Idx >> 2). >>=20 >> exc dar dsisr (booke dbcer0) >> 013e8620 00 00 07 00 41 eb 30 00 0a 00 00 00 01 47 c6 c0 = |....A.0......G..| >> The 0x00007000 above is the framep->exc (exception code) >> (program in this case). >>=20 >> 013e8630 00 eb a7 80 01 47 60 10 00 d1 ca ac df 5e 56 40 = |.....G`......^V@| >>=20 >> At this point the above does not match the below >> part of the stack trace. >>=20 >> The lr part of struct trapframe was: 0x00535ad0 so >> showing around that: >>=20 >> 00535ab8 stwu r1,-32(r1) >> 00535abc mflr r0 >> 00535ac0 stw r29,20(r1) >> 00535ac4 stw r30,24(r1) >> 00535ac8 stw r31,28(r1) >> 00535acc stw r0,36(r1) >> 00535ad0 mr r31,r1 >> 00535ad4 mr r29,r3 >>=20 >> Back to the stack backtrace. . . >>=20 >> 013e8640 df 5e 56 80 00 53 59 dc 00 17 99 f8 00 17 99 f8 = |.^V..SY.........| >> 0x005359c8 bl 008ea420 >> 0x005359cc mr r3,r28 >> 0x005359d0 mr r4,r27 >> 0x005359d4 mr r5,r25 >> 0x005359d8 bl 005356ec >> 0x005359dc mfsprg r9,0 >>=20 >> I show around 0x005356ec >> from the sched_add+0x19c bl that is above >> because the routine is not referenced >> in the stack tracce but the above indicates >> that it should have been called: >> 0x005356ec stwu r1,-32(r1) >> 0x005356f0 mflr r0 >> 0x005356f4 stw r28,16(r1) >> 0x005356f8 stw r29,20(r1) >> 0x005356fc stw r30,24(r1) >> 0x00535700 stw r31,28(r1) >> 0x00535704 stw r0,36(r1) >> . . . >>=20 >> Back to the stack backtrace again. . . >>=20 >> 013e8650 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 00 04 = |.....i.y........| >> 013e8660 f8 1c 2f cc 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |../......@...C..| >> 013e8670 01 47 c6 c0 01 47 60 10 00 d1 b4 30 df 5e 56 80 = |.G...G`....0.^V.| >>=20 >> 013e8680 df 5e 56 b0 00 4a 87 8c 00 d2 5b 10 00 00 00 04 = |.^V..J....[.....| >> 0x004a8780 mr r3,r28 >> 0x004a8784 li r4,4 >> 0x004a8788 bl 0053583c = >> 0x004a878c lwz r9,0(r28) >>=20 >> 013e8690 df 5e 56 b0 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.^V......@...C..| >> 013e86a0 00 00 00 00 01 46 81 c0 00 d1 b4 30 df 5e 56 b0 = |.....F.....0.^V.| >>=20 >> 013e86b0 df 5e 56 f0 00 4a 97 0c 00 00 00 00 00 00 00 04 = |.^V..J..........| >> 0x004a9700 bl 005000ec >> 0x004a9704 mr r3,r27 >> 0x004a9708 bl 004a86bc = >> 0x004a970c lwz r11,0(r1) >>=20 >> 013e86c0 df 5e 56 e0 01 47 d6 c0 01 47 d6 c0 01 45 4d 40 = |.^V..G...G...EM@| >> 013e86d0 df 5e 56 f0 00 8e a4 44 06 40 c2 a8 00 00 17 9a = |.^V....D.@......| >> 013e86e0 78 00 00 00 00 e9 56 00 00 d1 c8 20 df 5e 56 f0 = |x.....V.... .^V.| >>=20 >> 013e86f0 df 5e 57 50 00 51 79 6c df 5e 58 78 01 47 d6 c0 = |.^WP.Qyl.^Xx.G..| >> 0x00517960 lwz r3,264(r29) >> 0x00517964 li r4,0 >> 0x00517968 bl 004a965c >> 0x0051796c lwz r11,0(r1) >>=20 >> 013e8700 01 47 d7 b8 00 00 00 00 00 d1 ab 24 00 00 00 04 = |.G.........$....| >> 013e8710 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e8720 00 d0 53 00 00 eb a7 80 00 00 00 01 00 00 00 00 = |..S.............| >> 013e8730 df 5e 59 8c 00 00 00 00 df 5e 58 78 00 00 17 99 = |.^Y......^Xx....| >> 013e8740 f8 1c 2f cc d0 01 dd 00 00 d2 5b 10 df 5e 57 50 = |../.......[..^WP| >>=20 >> 013e8750 df 5e 57 a0 00 8a b2 70 df 5e 57 60 df 5e 57 60 = |.^W....p.^W`.^W`| >> 0x008ab264 mr r3,r27 >> 0x008ab268 mr r4,r28 >> 0x008ab26c bl 00517540 >> 0x008ab270 li r3,0 >>=20 >> 013e8760 df 5e 57 a0 df 5e 58 78 05 86 37 00 00 00 00 04 = |.^W..^Xx..7.....| >> 013e8770 00 00 00 00 05 9b f2 00 00 c9 66 bc 01 47 d6 c0 = |..........f..G..| >> 013e8780 df 5e 59 8c 00 f6 1d 10 00 00 17 99 f8 1c 2f cc = |.^Y.........../.| >> 013e8790 d0 01 dd 00 d0 01 dd 30 00 d2 5b 10 df 5e 57 a0 = |.......0..[..^W.| >>=20 >> 013e87a0 df 5e 58 20 00 8a d1 10 00 d2 6e 5c df 5e 57 b0 |.^X = ......n\.^W.| >> 0x008ad100 mr r3,r26 >> 0x008ad104 mr r4,r27 >> 0x008ad108 li r5,0 >> 0x008ad10c bl 008aafc0 >> 0x008ad110 lwz r11,0(r1) >>=20 >> 013e87b0 df 5e 57 e0 00 4a 96 00 00 00 17 99 00 00 00 00 = |.^W..J..........| >> 013e87c0 f8 1c 2f cc 0a 3f a0 12 df 5e 58 78 05 86 37 00 = |../..?...^Xx..7.| >> 013e87d0 01 48 b1 00 05 86 37 80 00 d4 bd ec 00 cb 98 98 = |.H....7.........| >> 013e87e0 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e87f0 df 5e 59 e0 00 eb a7 80 00 c9 66 bc 01 47 d6 c0 = |.^Y.......f..G..| >> 013e8800 df 5e 59 8c df 5e 58 78 01 47 d6 c0 00 00 00 00 = |.^Y..^Xx.G......| >> 013e8810 00 f6 1d 10 00 00 00 01 00 d2 6b c8 df 5e 58 20 = |..........k..^X | >>=20 >> 013e8820 df 5e 58 40 00 8e 1e 48 00 00 00 00 00 eb a7 80 = |.^X@...H........| >> So 0x13e8820 from vmcore.9 has start of struct trapeframe. >> The 0x8e1e48 is from: >> 0x008e1e34 lwz r0,56(r28) >> 0x008e1e38 mtctr r0 >> 0x008e1e3c mr r3,r28 >> 0x008e1e40 lwz r4,64(r28) >> 0x008e1e44 bctrl >> 0x008e1e48 addi r29,r29,-1 >>=20 >> 013e8830 01 47 d7 94 00 00 00 01 00 d2 6e 4c df 5e 58 40 = |.G........nL.^X@| >> 013e8840 df 5e 58 70 00 8e 7c 9c 00 d1 ca ac df 5e 58 50 = |.^Xp..|......^XP| >> 013e8850 00 cd f0 74 00 00 00 01 00 00 00 01 00 eb a7 80 = |...t............| >> 013e8860 41 eb 30 00 0a 00 00 00 00 00 00 00 00 00 90 32 = |A.0............2| >> 013e8870 df 5e 59 30 00 10 08 f8 00 04 90 32 df 5e 59 30 = |.^Y0.......2.^Y0| >> 013e8880 01 47 d6 c0 00 00 00 00 0d 0b bf e3 00 00 00 00 = |.G..............| >> 013e8890 0d 0b bf e3 00 19 eb 7c 00 00 00 00 00 00 00 44 = |.......|.......D| >> 013e88a0 01 fc a0 55 00 00 90 32 d0 01 dd 00 00 00 00 00 = |...U...2........| >> 013e88b0 00 d4 bd ec 00 cb 98 98 00 c9 66 bc 00 c4 5d 48 = |..........f...]H| >> 013e88c0 00 c9 66 bc 00 d4 c5 3c df 5e 59 e0 00 eb a7 80 = |..f....<.^Y.....| >> 013e88d0 00 c9 66 bc 01 47 d6 c0 df 5e 59 8c 00 00 00 01 = |..f..G...^Y.....| >> 013e88e0 00 00 00 01 00 eb a7 80 00 00 00 00 00 8e 3b f8 = |..............;.| >>=20 >> srr0 >> 013e88f0 00 d2 6b f0 df 5e 59 30 00 8e 3c 14 40 00 00 42 = |..k..^Y0..<.@..B| >> The 0x00833c14 from the trap frame is the srr0 (conceptual lr): >> 0x008e3c04 stw r31,28(r1) >> 0x008e3c08 stw r0,36(r1) >> 0x008e3c0c mr r31,r1 >> 0x008e3c10 bcl- 20,4*cr7+so,008e3c14 = >> 0x008e3c14 mflr r30 >> 0x008e3c18 lwz r0,-32(r30) >>=20 >> 013e8900 20 00 00 00 00 8e 3b f8 00 8e 3c 80 00 00 90 32 | = .....;...<....2| >>=20 >> exc >> 013e8910 00 00 09 00 41 eb 30 00 0a 00 00 00 00 00 00 00 = |....A.0.........| >> The 0x00009000 above is the framep->exc (exception code) >> (decrementer in this case). >>=20 >> 013e8920 eb 10 25 b5 55 7e c4 22 00 00 00 00 00 00 00 04 = |..%.U~."........| >>=20 >> 013e8930 df 5e 59 50 00 00 00 01 00 00 00 01 00 eb a7 80 = |.^YP............| >> (trap [above] means no lr filled in here) >>=20 >> 013e8940 00 00 00 00 00 d4 ca 34 00 d2 6b f0 df 5e 59 50 = |.......4..k..^YP| >>=20 >> 013e8950 df 5e 59 70 00 8e 31 9c 00 00 00 02 00 eb a7 80 = |.^Yp..1.........| >> 0x008e318c bl 008ad8b8 >> 0x008e3190 lwz r29,0(r29) >> 0x008e3194 mtctr r29 >> 0x008e3198 bctrl >> 0x008e319c bl 008ad7b8 >>=20 >> 013e8960 00 f2 d5 fc 00 00 00 01 00 d1 ca ac df 5e 59 70 = |.............^Yp| >>=20 >> 013e8970 df 5e 5a 50 00 53 6e 7c fa 50 05 af fa 50 05 af = |.^ZP.Sn|.P...P..| >> 0x00536e78 bl 008e3144 >> 0x00536e7c stw r28,136(r27) >>=20 >>=20 >> 013e8980 fa 50 05 af fa 50 05 af fa 50 05 af ff ff ff fe = |.P...P...P......| >> 013e8990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89a0 ff ff ff ff ff ff ff ff ff ff ff ff fa 50 05 af = |.............P..| >> 013e89b0 fa 50 05 af 00 00 00 02 ff ff ff ff 00 00 01 f0 = |.P..............| >> 013e89c0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89e0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e8a00 df 5e 5a 20 fa 50 05 af 00 00 00 00 00 00 00 00 |.^Z = .P..........| >> 013e8a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8a30 00 00 00 00 00 53 69 a8 df 5e 5a 98 00 00 00 00 = |.....Si..^Z.....| >> 013e8a40 01 47 96 e0 01 47 d6 c0 00 d1 b3 70 df 5e 5a 50 = |.G...G.....p.^ZP| >>=20 >> 013e8a50 df 5e 5a 80 00 4a 3c b4 df 5e 5a 60 df 5e 5a 60 = |.^Z..J<..^Z`.^Z`| >> 0x004a3ca0 bl 008ea420 >> 0x004a3ca4 mr r3,r27 >> 0x004a3ca8 mr r4,r26 >> 0x004a3cac mtctr r25 >> 0x004a3cb0 bctrl >> 0x004a3cb4 lwz r0,108(r28) >>=20 >> 013e8a60 df 5e 5a 80 00 00 00 00 00 00 00 00 00 00 00 00 = |.^Z.............| >> 013e8a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >>=20 >> 013e8a80 00 00 00 00 00 8f 18 d0 00 53 69 a8 00 00 00 00 = |.........Si.....| >> 0x008f18c0 lwz r3,8(r1) >> 0x008f18c4 lwz r4,12(r1) >> 0x008f18c8 lwz r5,16(r1) >> 0x008f18cc bl 004a3bbc >> 0x008f18d0 addi r1,r1,16 >> 0x008f18d4 b 001008f8 >>=20 >>=20 >> So that is an example context for the failure. >>=20 >> It has taken weeks to get this. It may be some >> time before I get another for comparison/contrast. >>=20 >> And sticking with the same build vs. trying some more >> to find a better way to get more evidence is not >> obvious to me at this point. >=20 > I should have mentioned that this is > TARGET_ARCH=3Dpowerpc but used on a so-called PowerMac > G5 "Quad Core". >=20 > I've had 2 more examples, with the same 0x0090a030 > srr0 and such (vmcore.0 and vmcore.1) (I have added > one more little block of code for detecting an earlier > problem symptom not being currently seen so the build > is slight different from the earlier report.) >=20 > Looking around in vmcore.0 I find 3 examples of > "00 90 a0 30" in areas overlapping with objdump -x's > coverage. . . >=20 > 00c77fd0 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| > [ ] >=20 > (from sorted objdump -x output:) > 00c775a8 l O .data 00000dc0 seqprog > 00c78368 l O .data 0000000c seeprom_long_ewen >=20 > [ ] > 00f65820 41 eb 30 00 0a 00 00 00 00 90 a0 30 00 08 10 32 = |A.0........0...2| > . . . > 00f65870 00 90 a0 30 00 08 10 32 00 00 00 00 df 5d 30 00 = |...0...2.....]0.| > [ ] >=20 > (from sorted objdump -x output:) > 00f64780 g O .bss 00020000 __pcpu > 00f84780 l O .bss 00000004 ap_letgo >=20 >=20 > For vmcore.1 I went ahead and tried exploring > some with ddb --and had no new panics. . . > (Hand transcriptions of pictures) >=20 > fatal kernel trap: > exception =3D 0x700 (program) > srr0 =3D 0x90a030 > srr1 =3D 0x81032 > lr =3D 0x535ad0 > curthread =3D 0x147d6c0 > pic =3D 11, comm =3D idle: cpu0 >=20 > [ thread pid 11 tid 100003 ] > Stopped at _etext+0xb8fc: illegal instruction 0 > db> bt > 0xdf5e55d0: at sched_wakeup+0xa4 > 0xdf5e55f0: at setrunnable+0x9c > 0xdf5e5610: at sleepq_resume_thread+0x17c > 0xdf5e5640: at sleepq_timeout+0xc8 > 0xdf5e5680: at softclock_call_cc+0x1f0 > 0xdf5e56f0: at callout_process+0x27c > 0xdf5e57a0: at timercb+0x4c4 > 0xdf5e5820: at decr_intr+0xf0 > 0xdf5e5840: at powerpc_interrupt_0xf4 > 0xdf5e5870: at kernel DECR trap > by cpu_idle_60x+0x88 (so: srr0) > srr1=3D0x9032 > r1 =3D0xdf5e5930 > cr =3D0x40000042 > xer =3D0x20000000 > ctr =3D0x8e3bf8 > saved LR(0xfffffffd) is invalid. >=20 > db> show reg (but reformatted > r0 =3D0x4 > r1 =3D0xdf5e5590 > r2 =3D0x147d6c0 > r3 =3D0x54 testppc64size+0x34 > r4 =3D0x591d000 > r5 =3D0 > r6 =3D0 > r7 =3D0xf > r8 =3D0 > r9 =3D0xd4c03c cold > r10 =3D0x147d6c0 > r11 =3D0xdf5e55d0 > r12 =3D0 > r13 =3D0 > r14 =3D0xd4bdec sdt_probe_func > r15 =3D0xcb9898 std_lockstat___spin__release > r16 =3D0xd4c45c callwheelmask > r17 =3D0xd4c45c callwheelmask > r18 =3D0x55925 > r19 =3D0x559a4 > r20 =3D0x559 dsmisssize+0x469 > r21 =3D0x591d000 > r22 =3D0x566430 sleeppq_timeout > r23 =3D0x114 dsmisssize+0x24 > r24 =3D0 > r25 =3D0 > r26 =3D0x1 > r27 =3D0 > r28 =3D0xeba780 tdq_cpu > r29 =3D0x147d6c0 > r30 =3D0xd1caac > r31 =3D0xdf5e5590 > srr0 =3D0x90a030 > srr1 =3D0x81032 > lr =3D0x535ad0 shed_affinity+0x18 > ctr =3D0 > cr =3D0x20009034 > xer =3D0 > dar =3D0x419df5d4 > dsisr=3D0x24000000 > _etext+0xb8fc: illegal instruction 0 >=20 >=20 > Just for completeness: acttrace also showed: >=20 > Tracing command less pid 1144 tid 100150 td 0x5bc8a20 (CPU 3) > 0xd25a59f0: at powrpc_dispatch_intr+0xc8 > 0xd25a5a20: at openpic_dispatch+0x90 > 0xd25a5a50: at powerpc_interrupt+0xc0 > 0xd25a5a80: at user EXI trap > by 0x181c68c (so: ssr0) > r1 =3D0xffffdb30 > cr =3D0x44020624 > xer=3D0 > ctr=3D0x41989570 >=20 > Tracing command idle pid 11 tid 100004 td 0x147d360 (CPU 1) > saved LR(0x4c) in invalid >=20 > Tracing command idle pid 11 tid 100005 td 0x147d360 (CPU 2) > saved LR(0x4c) in invalid From owner-freebsd-ppc@freebsd.org Sat May 27 08:32:26 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B88AD81838 for ; Sat, 27 May 2017 08:32:26 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-63.reflexion.net [208.70.210.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F58E1489 for ; Sat, 27 May 2017 08:32:25 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 28594 invoked from network); 27 May 2017 08:32:24 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 27 May 2017 08:32:24 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Sat, 27 May 2017 04:32:24 -0400 (EDT) Received: (qmail 2212 invoked from network); 27 May 2017 08:32:23 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 08:32:23 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 00EDEEC7B46; Sat, 27 May 2017 01:32:22 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) [better bt; more] Date: Sat, 27 May 2017 01:32:22 -0700 References: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> <62BF8E69-E7E6-4C4F-AB33-38B03E903CDA@dsl-only.net> To: Justin Hibbits , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org In-Reply-To: Message-Id: <67C79408-0DDC-4A22-BFEC-6EC5DD80F076@dsl-only.net> X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 08:32:26 -0000 [It helps to type all my ()'s. Trying again, using idiom from elsewhere in the file. . .] On 2017-May-27, at 1:17 AM, Mark Millard wrote: > [I suggest a patch this time.] >=20 > On 2017-May-27, at 12:42 AM, Mark Millard wrote: >=20 >> [Top post of the answer to what is wrong. I >> have submitted 219589 for this.] >>=20 >> TARGET_ARCH=3Dpowerpc64 got a fix to >> bugzilla 205458, avoiding inappropriate >> restoration of openfirmware's sprg0 >> value. >>=20 >> It turns out that TARGET_ARCH=3Dpowerpc >> needs to detect when it is running on >> powerpc64 and do the same thing if it >> is to avoid trashing memory and running >> unreliably on PowerMac G5's. >>=20 >> The issue is that for powerpc64 it is >> inappropriate to restore the sprg0 >> value to its openfirmware value. This >> is because the FreeBSD real-mode >> handling is involved instead of the >> openfirmware's original virtual mode, >> making openfirware's value simply >> inappropriate. >>=20 >> Quoting Nathan W. from Comment >> #4 of 205458: >>=20 >>> Where this explodes is if OF uses an unmapped SLB entry. >>> The SLB fault handler runs in real mode and refers to the >>> PCPU pointer in SPRG0, which blows up the kernel. Having >>> a value of SPRG0 that works for the kernel is less fatal >>> than preserving OF's value in this case. >>=20 >> I know that part of the code does detect >> the powerpc64 context vs. not and does >> things differently to emulate being powerpc >> like on powerpc64 (such as limiting >> RAM use as a consequence). >>=20 >> The powerpc64 vs. not status needs to be >> recorded and used to control a sprg0 >> restoration choice: avoid restoring >> openfirmware's value on powerpc64; >> otherwise restore it. >=20 > I expect that the following patch would > fix the problem: >=20 > # svnlite diff /usr/src/sys/powerpc/ofw/ofw_machdep.c > Index: /usr/src/sys/powerpc/ofw/ofw_machdep.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/powerpc/ofw/ofw_machdep.c (revision 317820) > +++ /usr/src/sys/powerpc/ofw/ofw_machdep.c (working copy) > @@ -147,7 +147,8 @@ > * PCPU data cannot be used until this routine is called ! > */ > #ifndef __powerpc64__ > - __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); > + if (cpu_features & PPC_FEATURE_64 !=3D PPC_FEATURE_64) > + __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); > #endif > } > #endif >=20 > This is based on cpu_features already having had > PPC_FEATURE_64 masked in before this if things > are running on a PowerMac G5 or other powerpc64. Fixing the ()'s mistake above: # svnlite diff /usr/src/sys/powerpc/ofw/ofw_machdep.c Index: /usr/src/sys/powerpc/ofw/ofw_machdep.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/ofw/ofw_machdep.c (revision 317820) +++ /usr/src/sys/powerpc/ofw/ofw_machdep.c (working copy) @@ -147,7 +147,8 @@ * PCPU data cannot be used until this routine is called ! */ #ifndef __powerpc64__ - __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); + if (!(cpu_features & PPC_FEATURE_64)) + __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); #endif } #endif =3D=3D=3D Mark Millard markmi at dsl-only.net [The original (better) panic evidence. . .] On 2017-May-26, at 10:29 PM, Mark Millard wrote: > [Additional information that does not need to > interlace with the prior material, so see > after. A somewhat better backtrace reported > by ddb. And so on.] >=20 >=20 > On 2017-May-26, at 7:14 PM, Mark Millard = wrote: >=20 >> I lucked out and got a vmcore.9 for a random >> panic that I could manage to backtrace for >> one of my test builds of -r317820. It appears >> that not all that much happened before it got >> the panic so much context was better preserved >> this time. >>=20 >> (I do not explore from ddb as I've had that >> panic and mess up the dump just made by >> replacing it. So this is a manual backtrace >> from the debug.minidump=3D0 style vmcore.9 >> file. objdump was used on the >> /boot/kernel/kernel to find code.) >>=20 >> Being able to see the problem is very >> sensitive to kernel memory layout. This >> is why I'm sticking with -r317820 built >> production style: the kind of context >> the problem was first observed in. >> Attempting a debug kernel build simply >> did not repeat the problem for days >> (vs. the usual hours for builds like >> this). >>=20 >>=20 >> So below is the backtrace: >>=20 >> (I do not show what trap_fatal calls: >> starting with trap_fatal and going toward >> larger memory addresses. . .) >>=20 >> [vmcore.9's >> offset >> in file >> when no >> 0x prefix] >>=20 >> 013e83b0 df 5e 55 50 00 8f 34 5c fa 50 05 af fa 50 05 af = |.^UP..4\.P...P..| >> 0x008f3454 mr r3,r26 >> 0x008f3458 bl 008f2030 >> 0x008f345c b 008f34c8 >>=20 >> 013e83c0 fa 50 05 af fa 50 05 af fa 50 05 af fa 50 05 af = |.P...P...P...P..| >> * >> 013e83e0 df 5e 54 00 fa 50 05 af fa 50 05 af fa 50 05 af = |.^T..P...P...P..| >> 013e83f0 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 00 = |.P...P.......^T.| >> 013e8400 df 5e 54 20 00 53 3a d0 fa 50 05 af fa 50 05 af |.^T = .S:..P...P..| >> 013e8410 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 20 = |.P...P.......^T | >> 013e8420 df 5e 54 d0 00 53 3a d0 00 00 00 00 00 00 00 00 = |.^T..S:.........| >> 013e8430 00 00 00 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |................| >> 013e8440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8460 00 00 00 54 7f ff ff ff 00 00 00 00 ff ff ff aa = |...T............| >> 013e8470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8490 00 d4 c4 5c 00 d4 c4 5c 00 17 99 dd 00 17 9a 5c = |...\...\.......\| >> 013e84a0 00 00 17 9a 00 d4 c4 6c df 5e 54 ec 00 00 00 54 = |.......l.^T....T| >> 013e84b0 df 5e 54 d0 7f ff ff ff 05 aa 36 c0 05 aa 39 e8 = |.^T.......6...9.| >> 013e84c0 00 00 00 00 00 00 00 00 00 d1 ca ac df 5e 54 d0 = |.............^T.| >> 013e84d0 df 5e 55 80 00 53 3a d0 05 91 d0 00 05 91 d3 28 = |.^U..S:........(| >> 013e84e0 df 5e 55 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |.^U.............| >> 013e84f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> 013e8500 00 00 00 00 00 00 00 00 00 d4 bd ec 00 cb 98 98 = |................| >> 013e8510 00 d4 c4 5c 00 d4 c4 5c 00 17 99 f8 00 17 99 f8 = |...\...\........| >> 013e8520 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 17 99 = |.....i.y........| >> 013e8530 f8 1c 2f cc df 5e 55 88 01 47 d6 c0 01 43 a2 00 = |../..^U..G...C..| >> 013e8540 41 eb 30 00 0a 00 00 00 00 d2 6e 4c df 5e 55 50 = |A.0.......nL.^UP| >>=20 >> 013e8550 df 5e 55 80 00 8e 7d 40 df 5e 55 9c 00 00 00 28 = |.^U...}@.^U....(| >> 0x008e7d28 mfmsr r0 >> 0x008e7d2c or r0,r0,r9 >> 0x008e7d30 mtmsr r0 >> 0x008e7d34 isync >> 0x008e7d38 mr r3,r25 >> 0x008e7d3c bl 008f222c >> 0x008e7d40 lwz r11,0(r1) >>=20 >> 013e8560 00 00 00 00 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.........@...C..| >> 013e8570 41 eb 30 00 0a 00 00 00 00 00 00 00 00 08 10 32 = |A.0............2| >>=20 >> r0 r1 >> 013e8580 df 5e 56 40 00 10 08 f8 00 00 00 04 df 5e 56 40 = |.^V@.........^V@| >> The struct trapframe starts is 0x 013e8588 in the >> vmcore.9 file. The 0x00108f8 is as shown below: >>=20 >> 0x001008ec isync >> 0x001008f0 addi r3,r1,8 >> 0x001008f4 bl 008e7ba4 >> 0x001008f8 mfmsr r3 >> 0x001008fc andi. r3,r3,32767 >>=20 >> 013e8590 01 47 d6 c0 00 00 00 28 01 47 c6 c0 00 00 00 04 = |.G.....(.G......| >> r2 r3 r4 r5 >>=20 >> 013e85a0 00 00 00 04 00 00 00 0f 00 00 00 00 00 d4 c0 3c = |...............<| >> r6 r7 r8 r9 >>=20 >> 013e85b0 01 47 d6 c0 df 5e 56 80 14 3d 60 da 00 00 00 00 = |.G...^V..=3D`.....| >> r10 r11 r12 r13 >>=20 >> 013e85c0 00 d4 bd ec 00 cb 98 98 00 d4 c4 5c 00 d4 c4 5c = |...........\...\| >> r14 r15 r16 r17 >>=20 >> 013e85d0 00 17 99 f8 00 17 99 f8 00 00 17 99 fa 69 b8 79 = |.............i.y| >> r18 r19 r20 r21 >>=20 >> 013e85e0 00 17 9a 0a 00 00 17 99 f8 1c 2f cc 00 00 17 9a = |........../.....| >> r22 r32 r24 r25 >>=20 >> 013e85f0 06 40 c2 a8 01 43 a2 00 00 eb a7 80 01 47 d6 c0 = |.@...C.......G..| >> r26 r27 r28 r29 >>=20 >> 013e8600 00 d1 ca ac df 5e 56 40 00 53 5a d0 20 00 90 44 = |.....^V@.SZ. ..D| >> r30 r31 lr cr >>=20 >> xer ctr srr0 srr1 >> 013e8610 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| >> Note: objdump shows no code at 0x0090a030. 0x0090a030 is >> inside what objdump -x reports as the section .hash (Idx >> 2). >>=20 >> exc dar dsisr (booke dbcer0) >> 013e8620 00 00 07 00 41 eb 30 00 0a 00 00 00 01 47 c6 c0 = |....A.0......G..| >> The 0x00007000 above is the framep->exc (exception code) >> (program in this case). >>=20 >> 013e8630 00 eb a7 80 01 47 60 10 00 d1 ca ac df 5e 56 40 = |.....G`......^V@| >>=20 >> At this point the above does not match the below >> part of the stack trace. >>=20 >> The lr part of struct trapframe was: 0x00535ad0 so >> showing around that: >>=20 >> 00535ab8 stwu r1,-32(r1) >> 00535abc mflr r0 >> 00535ac0 stw r29,20(r1) >> 00535ac4 stw r30,24(r1) >> 00535ac8 stw r31,28(r1) >> 00535acc stw r0,36(r1) >> 00535ad0 mr r31,r1 >> 00535ad4 mr r29,r3 >>=20 >> Back to the stack backtrace. . . >>=20 >> 013e8640 df 5e 56 80 00 53 59 dc 00 17 99 f8 00 17 99 f8 = |.^V..SY.........| >> 0x005359c8 bl 008ea420 >> 0x005359cc mr r3,r28 >> 0x005359d0 mr r4,r27 >> 0x005359d4 mr r5,r25 >> 0x005359d8 bl 005356ec >> 0x005359dc mfsprg r9,0 >>=20 >> I show around 0x005356ec >> from the sched_add+0x19c bl that is above >> because the routine is not referenced >> in the stack tracce but the above indicates >> that it should have been called: >> 0x005356ec stwu r1,-32(r1) >> 0x005356f0 mflr r0 >> 0x005356f4 stw r28,16(r1) >> 0x005356f8 stw r29,20(r1) >> 0x005356fc stw r30,24(r1) >> 0x00535700 stw r31,28(r1) >> 0x00535704 stw r0,36(r1) >> . . . >>=20 >> Back to the stack backtrace again. . . >>=20 >> 013e8650 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 00 04 = |.....i.y........| >> 013e8660 f8 1c 2f cc 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |../......@...C..| >> 013e8670 01 47 c6 c0 01 47 60 10 00 d1 b4 30 df 5e 56 80 = |.G...G`....0.^V.| >>=20 >> 013e8680 df 5e 56 b0 00 4a 87 8c 00 d2 5b 10 00 00 00 04 = |.^V..J....[.....| >> 0x004a8780 mr r3,r28 >> 0x004a8784 li r4,4 >> 0x004a8788 bl 0053583c = >> 0x004a878c lwz r9,0(r28) >>=20 >> 013e8690 df 5e 56 b0 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.^V......@...C..| >> 013e86a0 00 00 00 00 01 46 81 c0 00 d1 b4 30 df 5e 56 b0 = |.....F.....0.^V.| >>=20 >> 013e86b0 df 5e 56 f0 00 4a 97 0c 00 00 00 00 00 00 00 04 = |.^V..J..........| >> 0x004a9700 bl 005000ec >> 0x004a9704 mr r3,r27 >> 0x004a9708 bl 004a86bc = >> 0x004a970c lwz r11,0(r1) >>=20 >> 013e86c0 df 5e 56 e0 01 47 d6 c0 01 47 d6 c0 01 45 4d 40 = |.^V..G...G...EM@| >> 013e86d0 df 5e 56 f0 00 8e a4 44 06 40 c2 a8 00 00 17 9a = |.^V....D.@......| >> 013e86e0 78 00 00 00 00 e9 56 00 00 d1 c8 20 df 5e 56 f0 = |x.....V.... .^V.| >>=20 >> 013e86f0 df 5e 57 50 00 51 79 6c df 5e 58 78 01 47 d6 c0 = |.^WP.Qyl.^Xx.G..| >> 0x00517960 lwz r3,264(r29) >> 0x00517964 li r4,0 >> 0x00517968 bl 004a965c >> 0x0051796c lwz r11,0(r1) >>=20 >> 013e8700 01 47 d7 b8 00 00 00 00 00 d1 ab 24 00 00 00 04 = |.G.........$....| >> 013e8710 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e8720 00 d0 53 00 00 eb a7 80 00 00 00 01 00 00 00 00 = |..S.............| >> 013e8730 df 5e 59 8c 00 00 00 00 df 5e 58 78 00 00 17 99 = |.^Y......^Xx....| >> 013e8740 f8 1c 2f cc d0 01 dd 00 00 d2 5b 10 df 5e 57 50 = |../.......[..^WP| >>=20 >> 013e8750 df 5e 57 a0 00 8a b2 70 df 5e 57 60 df 5e 57 60 = |.^W....p.^W`.^W`| >> 0x008ab264 mr r3,r27 >> 0x008ab268 mr r4,r28 >> 0x008ab26c bl 00517540 >> 0x008ab270 li r3,0 >>=20 >> 013e8760 df 5e 57 a0 df 5e 58 78 05 86 37 00 00 00 00 04 = |.^W..^Xx..7.....| >> 013e8770 00 00 00 00 05 9b f2 00 00 c9 66 bc 01 47 d6 c0 = |..........f..G..| >> 013e8780 df 5e 59 8c 00 f6 1d 10 00 00 17 99 f8 1c 2f cc = |.^Y.........../.| >> 013e8790 d0 01 dd 00 d0 01 dd 30 00 d2 5b 10 df 5e 57 a0 = |.......0..[..^W.| >>=20 >> 013e87a0 df 5e 58 20 00 8a d1 10 00 d2 6e 5c df 5e 57 b0 |.^X = ......n\.^W.| >> 0x008ad100 mr r3,r26 >> 0x008ad104 mr r4,r27 >> 0x008ad108 li r5,0 >> 0x008ad10c bl 008aafc0 >> 0x008ad110 lwz r11,0(r1) >>=20 >> 013e87b0 df 5e 57 e0 00 4a 96 00 00 00 17 99 00 00 00 00 = |.^W..J..........| >> 013e87c0 f8 1c 2f cc 0a 3f a0 12 df 5e 58 78 05 86 37 00 = |../..?...^Xx..7.| >> 013e87d0 01 48 b1 00 05 86 37 80 00 d4 bd ec 00 cb 98 98 = |.H....7.........| >> 013e87e0 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e87f0 df 5e 59 e0 00 eb a7 80 00 c9 66 bc 01 47 d6 c0 = |.^Y.......f..G..| >> 013e8800 df 5e 59 8c df 5e 58 78 01 47 d6 c0 00 00 00 00 = |.^Y..^Xx.G......| >> 013e8810 00 f6 1d 10 00 00 00 01 00 d2 6b c8 df 5e 58 20 = |..........k..^X | >>=20 >> 013e8820 df 5e 58 40 00 8e 1e 48 00 00 00 00 00 eb a7 80 = |.^X@...H........| >> So 0x13e8820 from vmcore.9 has start of struct trapeframe. >> The 0x8e1e48 is from: >> 0x008e1e34 lwz r0,56(r28) >> 0x008e1e38 mtctr r0 >> 0x008e1e3c mr r3,r28 >> 0x008e1e40 lwz r4,64(r28) >> 0x008e1e44 bctrl >> 0x008e1e48 addi r29,r29,-1 >>=20 >> 013e8830 01 47 d7 94 00 00 00 01 00 d2 6e 4c df 5e 58 40 = |.G........nL.^X@| >> 013e8840 df 5e 58 70 00 8e 7c 9c 00 d1 ca ac df 5e 58 50 = |.^Xp..|......^XP| >> 013e8850 00 cd f0 74 00 00 00 01 00 00 00 01 00 eb a7 80 = |...t............| >> 013e8860 41 eb 30 00 0a 00 00 00 00 00 00 00 00 00 90 32 = |A.0............2| >> 013e8870 df 5e 59 30 00 10 08 f8 00 04 90 32 df 5e 59 30 = |.^Y0.......2.^Y0| >> 013e8880 01 47 d6 c0 00 00 00 00 0d 0b bf e3 00 00 00 00 = |.G..............| >> 013e8890 0d 0b bf e3 00 19 eb 7c 00 00 00 00 00 00 00 44 = |.......|.......D| >> 013e88a0 01 fc a0 55 00 00 90 32 d0 01 dd 00 00 00 00 00 = |...U...2........| >> 013e88b0 00 d4 bd ec 00 cb 98 98 00 c9 66 bc 00 c4 5d 48 = |..........f...]H| >> 013e88c0 00 c9 66 bc 00 d4 c5 3c df 5e 59 e0 00 eb a7 80 = |..f....<.^Y.....| >> 013e88d0 00 c9 66 bc 01 47 d6 c0 df 5e 59 8c 00 00 00 01 = |..f..G...^Y.....| >> 013e88e0 00 00 00 01 00 eb a7 80 00 00 00 00 00 8e 3b f8 = |..............;.| >>=20 >> srr0 >> 013e88f0 00 d2 6b f0 df 5e 59 30 00 8e 3c 14 40 00 00 42 = |..k..^Y0..<.@..B| >> The 0x00833c14 from the trap frame is the srr0 (conceptual lr): >> 0x008e3c04 stw r31,28(r1) >> 0x008e3c08 stw r0,36(r1) >> 0x008e3c0c mr r31,r1 >> 0x008e3c10 bcl- 20,4*cr7+so,008e3c14 = >> 0x008e3c14 mflr r30 >> 0x008e3c18 lwz r0,-32(r30) >>=20 >> 013e8900 20 00 00 00 00 8e 3b f8 00 8e 3c 80 00 00 90 32 | = .....;...<....2| >>=20 >> exc >> 013e8910 00 00 09 00 41 eb 30 00 0a 00 00 00 00 00 00 00 = |....A.0.........| >> The 0x00009000 above is the framep->exc (exception code) >> (decrementer in this case). >>=20 >> 013e8920 eb 10 25 b5 55 7e c4 22 00 00 00 00 00 00 00 04 = |..%.U~."........| >>=20 >> 013e8930 df 5e 59 50 00 00 00 01 00 00 00 01 00 eb a7 80 = |.^YP............| >> (trap [above] means no lr filled in here) >>=20 >> 013e8940 00 00 00 00 00 d4 ca 34 00 d2 6b f0 df 5e 59 50 = |.......4..k..^YP| >>=20 >> 013e8950 df 5e 59 70 00 8e 31 9c 00 00 00 02 00 eb a7 80 = |.^Yp..1.........| >> 0x008e318c bl 008ad8b8 >> 0x008e3190 lwz r29,0(r29) >> 0x008e3194 mtctr r29 >> 0x008e3198 bctrl >> 0x008e319c bl 008ad7b8 >>=20 >> 013e8960 00 f2 d5 fc 00 00 00 01 00 d1 ca ac df 5e 59 70 = |.............^Yp| >>=20 >> 013e8970 df 5e 5a 50 00 53 6e 7c fa 50 05 af fa 50 05 af = |.^ZP.Sn|.P...P..| >> 0x00536e78 bl 008e3144 >> 0x00536e7c stw r28,136(r27) >>=20 >>=20 >> 013e8980 fa 50 05 af fa 50 05 af fa 50 05 af ff ff ff fe = |.P...P...P......| >> 013e8990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89a0 ff ff ff ff ff ff ff ff ff ff ff ff fa 50 05 af = |.............P..| >> 013e89b0 fa 50 05 af 00 00 00 02 ff ff ff ff 00 00 01 f0 = |.P..............| >> 013e89c0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89e0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e8a00 df 5e 5a 20 fa 50 05 af 00 00 00 00 00 00 00 00 |.^Z = .P..........| >> 013e8a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8a30 00 00 00 00 00 53 69 a8 df 5e 5a 98 00 00 00 00 = |.....Si..^Z.....| >> 013e8a40 01 47 96 e0 01 47 d6 c0 00 d1 b3 70 df 5e 5a 50 = |.G...G.....p.^ZP| >>=20 >> 013e8a50 df 5e 5a 80 00 4a 3c b4 df 5e 5a 60 df 5e 5a 60 = |.^Z..J<..^Z`.^Z`| >> 0x004a3ca0 bl 008ea420 >> 0x004a3ca4 mr r3,r27 >> 0x004a3ca8 mr r4,r26 >> 0x004a3cac mtctr r25 >> 0x004a3cb0 bctrl >> 0x004a3cb4 lwz r0,108(r28) >>=20 >> 013e8a60 df 5e 5a 80 00 00 00 00 00 00 00 00 00 00 00 00 = |.^Z.............| >> 013e8a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >>=20 >> 013e8a80 00 00 00 00 00 8f 18 d0 00 53 69 a8 00 00 00 00 = |.........Si.....| >> 0x008f18c0 lwz r3,8(r1) >> 0x008f18c4 lwz r4,12(r1) >> 0x008f18c8 lwz r5,16(r1) >> 0x008f18cc bl 004a3bbc >> 0x008f18d0 addi r1,r1,16 >> 0x008f18d4 b 001008f8 >>=20 >>=20 >> So that is an example context for the failure. >>=20 >> It has taken weeks to get this. It may be some >> time before I get another for comparison/contrast. >>=20 >> And sticking with the same build vs. trying some more >> to find a better way to get more evidence is not >> obvious to me at this point. >=20 > I should have mentioned that this is > TARGET_ARCH=3Dpowerpc but used on a so-called PowerMac > G5 "Quad Core". >=20 > I've had 2 more examples, with the same 0x0090a030 > srr0 and such (vmcore.0 and vmcore.1) (I have added > one more little block of code for detecting an earlier > problem symptom not being currently seen so the build > is slight different from the earlier report.) >=20 > Looking around in vmcore.0 I find 3 examples of > "00 90 a0 30" in areas overlapping with objdump -x's > coverage. . . >=20 > 00c77fd0 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| > [ ] >=20 > (from sorted objdump -x output:) > 00c775a8 l O .data 00000dc0 seqprog > 00c78368 l O .data 0000000c seeprom_long_ewen >=20 > [ ] > 00f65820 41 eb 30 00 0a 00 00 00 00 90 a0 30 00 08 10 32 = |A.0........0...2| > . . . > 00f65870 00 90 a0 30 00 08 10 32 00 00 00 00 df 5d 30 00 = |...0...2.....]0.| > [ ] >=20 > (from sorted objdump -x output:) > 00f64780 g O .bss 00020000 __pcpu > 00f84780 l O .bss 00000004 ap_letgo >=20 >=20 > For vmcore.1 I went ahead and tried exploring > some with ddb --and had no new panics. . . > (Hand transcriptions of pictures) >=20 > fatal kernel trap: > exception =3D 0x700 (program) > srr0 =3D 0x90a030 > srr1 =3D 0x81032 > lr =3D 0x535ad0 > curthread =3D 0x147d6c0 > pic =3D 11, comm =3D idle: cpu0 >=20 > [ thread pid 11 tid 100003 ] > Stopped at _etext+0xb8fc: illegal instruction 0 > db> bt > 0xdf5e55d0: at sched_wakeup+0xa4 > 0xdf5e55f0: at setrunnable+0x9c > 0xdf5e5610: at sleepq_resume_thread+0x17c > 0xdf5e5640: at sleepq_timeout+0xc8 > 0xdf5e5680: at softclock_call_cc+0x1f0 > 0xdf5e56f0: at callout_process+0x27c > 0xdf5e57a0: at timercb+0x4c4 > 0xdf5e5820: at decr_intr+0xf0 > 0xdf5e5840: at powerpc_interrupt_0xf4 > 0xdf5e5870: at kernel DECR trap > by cpu_idle_60x+0x88 (so: srr0) > srr1=3D0x9032 > r1 =3D0xdf5e5930 > cr =3D0x40000042 > xer =3D0x20000000 > ctr =3D0x8e3bf8 > saved LR(0xfffffffd) is invalid. >=20 > db> show reg (but reformatted > r0 =3D0x4 > r1 =3D0xdf5e5590 > r2 =3D0x147d6c0 > r3 =3D0x54 testppc64size+0x34 > r4 =3D0x591d000 > r5 =3D0 > r6 =3D0 > r7 =3D0xf > r8 =3D0 > r9 =3D0xd4c03c cold > r10 =3D0x147d6c0 > r11 =3D0xdf5e55d0 > r12 =3D0 > r13 =3D0 > r14 =3D0xd4bdec sdt_probe_func > r15 =3D0xcb9898 std_lockstat___spin__release > r16 =3D0xd4c45c callwheelmask > r17 =3D0xd4c45c callwheelmask > r18 =3D0x55925 > r19 =3D0x559a4 > r20 =3D0x559 dsmisssize+0x469 > r21 =3D0x591d000 > r22 =3D0x566430 sleeppq_timeout > r23 =3D0x114 dsmisssize+0x24 > r24 =3D0 > r25 =3D0 > r26 =3D0x1 > r27 =3D0 > r28 =3D0xeba780 tdq_cpu > r29 =3D0x147d6c0 > r30 =3D0xd1caac > r31 =3D0xdf5e5590 > srr0 =3D0x90a030 > srr1 =3D0x81032 > lr =3D0x535ad0 shed_affinity+0x18 > ctr =3D0 > cr =3D0x20009034 > xer =3D0 > dar =3D0x419df5d4 > dsisr=3D0x24000000 > _etext+0xb8fc: illegal instruction 0 >=20 >=20 > Just for completeness: acttrace also showed: >=20 > Tracing command less pid 1144 tid 100150 td 0x5bc8a20 (CPU 3) > 0xd25a59f0: at powrpc_dispatch_intr+0xc8 > 0xd25a5a20: at openpic_dispatch+0x90 > 0xd25a5a50: at powerpc_interrupt+0xc0 > 0xd25a5a80: at user EXI trap > by 0x181c68c (so: ssr0) > r1 =3D0xffffdb30 > cr =3D0x44020624 > xer=3D0 > ctr=3D0x41989570 >=20 > Tracing command idle pid 11 tid 100004 td 0x147d360 (CPU 1) > saved LR(0x4c) in invalid >=20 > Tracing command idle pid 11 tid 100005 td 0x147d360 (CPU 2) > saved LR(0x4c) in invalid _______________________________________________ freebsd-ppc@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ppc To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org" From owner-freebsd-ppc@freebsd.org Sat May 27 09:12:58 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A357D8479A for ; Sat, 27 May 2017 09:12:58 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-63.reflexion.net [208.70.210.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 24FFA1749 for ; Sat, 27 May 2017 09:12:57 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 10831 invoked from network); 27 May 2017 09:12:56 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 27 May 2017 09:12:56 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Sat, 27 May 2017 05:12:56 -0400 (EDT) Received: (qmail 13951 invoked from network); 27 May 2017 09:12:55 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 09:12:55 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 3F9D8EC7B46; Sat, 27 May 2017 02:12:55 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) [patch now fixed] Date: Sat, 27 May 2017 02:12:54 -0700 References: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> <62BF8E69-E7E6-4C4F-AB33-38B03E903CDA@dsl-only.net> <67C79408-0DDC-4A22-BFEC-6EC5DD80F076@dsl-only.net> To: Justin Hibbits , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org In-Reply-To: <67C79408-0DDC-4A22-BFEC-6EC5DD80F076@dsl-only.net> Message-Id: <9D70C580-C545-4EA0-AB76-FE757C1AF60A@dsl-only.net> X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 09:12:58 -0000 [Testing shows the prior patch makes the PowerMac G5 so-called "Quad Core" hang very early.] I forgot to deal with the prepare side of things. So, now with both prepare and restore code fixes, this code boots. # svnlite diff /usr/src/sys/powerpc/ofw/ofw_machdep.c = = Index: = /usr/src/sys/powerpc/ofw/ofw_machdep.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/ofw/ofw_machdep.c (revision 317820) +++ /usr/src/sys/powerpc/ofw/ofw_machdep.c (working copy) @@ -111,26 +111,27 @@ * Assume that interrupt are disabled at this point, or * SPRG1-3 could be trashed */ -#ifdef __powerpc64__ - __asm __volatile("mtsprg1 %0\n\t" - "mtsprg2 %1\n\t" - "mtsprg3 %2\n\t" - : - : "r"(ofmsr[2]), - "r"(ofmsr[3]), - "r"(ofmsr[4])); -#else - __asm __volatile("mfsprg0 %0\n\t" - "mtsprg0 %1\n\t" - "mtsprg1 %2\n\t" - "mtsprg2 %3\n\t" - "mtsprg3 %4\n\t" - : "=3D&r"(ofw_sprg0_save) - : "r"(ofmsr[1]), - "r"(ofmsr[2]), - "r"(ofmsr[3]), - "r"(ofmsr[4])); +#ifndef __powerpc64__ + if (!(cpu_features & PPC_FEATURE_64)) + __asm __volatile("mfsprg0 %0\n\t" + "mtsprg0 %1\n\t" + "mtsprg1 %2\n\t" + "mtsprg2 %3\n\t" + "mtsprg3 %4\n\t" + : "=3D&r"(ofw_sprg0_save) + : "r"(ofmsr[1]), + "r"(ofmsr[2]), + "r"(ofmsr[3]), + "r"(ofmsr[4])); + else #endif + __asm __volatile("mtsprg1 %0\n\t" + "mtsprg2 %1\n\t" + "mtsprg3 %2\n\t" + : + : "r"(ofmsr[2]), + "r"(ofmsr[3]), + "r"(ofmsr[4])); } =20 static __inline void @@ -147,7 +148,8 @@ * PCPU data cannot be used until this routine is called ! */ #ifndef __powerpc64__ - __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); + if (!(cpu_features & PPC_FEATURE_64)) + __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); #endif } #endif =3D=3D=3D Mark Millard markmi at dsl-only.net [The original (better) panic evidence. . .] On 2017-May-26, at 10:29 PM, Mark Millard wrote: > [Additional information that does not need to > interlace with the prior material, so see > after. A somewhat better backtrace reported > by ddb. And so on.] >=20 >=20 > On 2017-May-26, at 7:14 PM, Mark Millard = wrote: >=20 >> I lucked out and got a vmcore.9 for a random >> panic that I could manage to backtrace for >> one of my test builds of -r317820. It appears >> that not all that much happened before it got >> the panic so much context was better preserved >> this time. >>=20 >> (I do not explore from ddb as I've had that >> panic and mess up the dump just made by >> replacing it. So this is a manual backtrace >> from the debug.minidump=3D0 style vmcore.9 >> file. objdump was used on the >> /boot/kernel/kernel to find code.) >>=20 >> Being able to see the problem is very >> sensitive to kernel memory layout. This >> is why I'm sticking with -r317820 built >> production style: the kind of context >> the problem was first observed in. >> Attempting a debug kernel build simply >> did not repeat the problem for days >> (vs. the usual hours for builds like >> this). >>=20 >>=20 >> So below is the backtrace: >>=20 >> (I do not show what trap_fatal calls: >> starting with trap_fatal and going toward >> larger memory addresses. . .) >>=20 >> [vmcore.9's >> offset >> in file >> when no >> 0x prefix] >>=20 >> 013e83b0 df 5e 55 50 00 8f 34 5c fa 50 05 af fa 50 05 af = |.^UP..4\.P...P..| >> 0x008f3454 mr r3,r26 >> 0x008f3458 bl 008f2030 >> 0x008f345c b 008f34c8 >>=20 >> 013e83c0 fa 50 05 af fa 50 05 af fa 50 05 af fa 50 05 af = |.P...P...P...P..| >> * >> 013e83e0 df 5e 54 00 fa 50 05 af fa 50 05 af fa 50 05 af = |.^T..P...P...P..| >> 013e83f0 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 00 = |.P...P.......^T.| >> 013e8400 df 5e 54 20 00 53 3a d0 fa 50 05 af fa 50 05 af |.^T = .S:..P...P..| >> 013e8410 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 20 = |.P...P.......^T | >> 013e8420 df 5e 54 d0 00 53 3a d0 00 00 00 00 00 00 00 00 = |.^T..S:.........| >> 013e8430 00 00 00 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |................| >> 013e8440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8460 00 00 00 54 7f ff ff ff 00 00 00 00 ff ff ff aa = |...T............| >> 013e8470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8490 00 d4 c4 5c 00 d4 c4 5c 00 17 99 dd 00 17 9a 5c = |...\...\.......\| >> 013e84a0 00 00 17 9a 00 d4 c4 6c df 5e 54 ec 00 00 00 54 = |.......l.^T....T| >> 013e84b0 df 5e 54 d0 7f ff ff ff 05 aa 36 c0 05 aa 39 e8 = |.^T.......6...9.| >> 013e84c0 00 00 00 00 00 00 00 00 00 d1 ca ac df 5e 54 d0 = |.............^T.| >> 013e84d0 df 5e 55 80 00 53 3a d0 05 91 d0 00 05 91 d3 28 = |.^U..S:........(| >> 013e84e0 df 5e 55 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |.^U.............| >> 013e84f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> 013e8500 00 00 00 00 00 00 00 00 00 d4 bd ec 00 cb 98 98 = |................| >> 013e8510 00 d4 c4 5c 00 d4 c4 5c 00 17 99 f8 00 17 99 f8 = |...\...\........| >> 013e8520 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 17 99 = |.....i.y........| >> 013e8530 f8 1c 2f cc df 5e 55 88 01 47 d6 c0 01 43 a2 00 = |../..^U..G...C..| >> 013e8540 41 eb 30 00 0a 00 00 00 00 d2 6e 4c df 5e 55 50 = |A.0.......nL.^UP| >>=20 >> 013e8550 df 5e 55 80 00 8e 7d 40 df 5e 55 9c 00 00 00 28 = |.^U...}@.^U....(| >> 0x008e7d28 mfmsr r0 >> 0x008e7d2c or r0,r0,r9 >> 0x008e7d30 mtmsr r0 >> 0x008e7d34 isync >> 0x008e7d38 mr r3,r25 >> 0x008e7d3c bl 008f222c >> 0x008e7d40 lwz r11,0(r1) >>=20 >> 013e8560 00 00 00 00 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.........@...C..| >> 013e8570 41 eb 30 00 0a 00 00 00 00 00 00 00 00 08 10 32 = |A.0............2| >>=20 >> r0 r1 >> 013e8580 df 5e 56 40 00 10 08 f8 00 00 00 04 df 5e 56 40 = |.^V@.........^V@| >> The struct trapframe starts is 0x 013e8588 in the >> vmcore.9 file. The 0x00108f8 is as shown below: >>=20 >> 0x001008ec isync >> 0x001008f0 addi r3,r1,8 >> 0x001008f4 bl 008e7ba4 >> 0x001008f8 mfmsr r3 >> 0x001008fc andi. r3,r3,32767 >>=20 >> 013e8590 01 47 d6 c0 00 00 00 28 01 47 c6 c0 00 00 00 04 = |.G.....(.G......| >> r2 r3 r4 r5 >>=20 >> 013e85a0 00 00 00 04 00 00 00 0f 00 00 00 00 00 d4 c0 3c = |...............<| >> r6 r7 r8 r9 >>=20 >> 013e85b0 01 47 d6 c0 df 5e 56 80 14 3d 60 da 00 00 00 00 = |.G...^V..=3D`.....| >> r10 r11 r12 r13 >>=20 >> 013e85c0 00 d4 bd ec 00 cb 98 98 00 d4 c4 5c 00 d4 c4 5c = |...........\...\| >> r14 r15 r16 r17 >>=20 >> 013e85d0 00 17 99 f8 00 17 99 f8 00 00 17 99 fa 69 b8 79 = |.............i.y| >> r18 r19 r20 r21 >>=20 >> 013e85e0 00 17 9a 0a 00 00 17 99 f8 1c 2f cc 00 00 17 9a = |........../.....| >> r22 r32 r24 r25 >>=20 >> 013e85f0 06 40 c2 a8 01 43 a2 00 00 eb a7 80 01 47 d6 c0 = |.@...C.......G..| >> r26 r27 r28 r29 >>=20 >> 013e8600 00 d1 ca ac df 5e 56 40 00 53 5a d0 20 00 90 44 = |.....^V@.SZ. ..D| >> r30 r31 lr cr >>=20 >> xer ctr srr0 srr1 >> 013e8610 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| >> Note: objdump shows no code at 0x0090a030. 0x0090a030 is >> inside what objdump -x reports as the section .hash (Idx >> 2). >>=20 >> exc dar dsisr (booke dbcer0) >> 013e8620 00 00 07 00 41 eb 30 00 0a 00 00 00 01 47 c6 c0 = |....A.0......G..| >> The 0x00007000 above is the framep->exc (exception code) >> (program in this case). >>=20 >> 013e8630 00 eb a7 80 01 47 60 10 00 d1 ca ac df 5e 56 40 = |.....G`......^V@| >>=20 >> At this point the above does not match the below >> part of the stack trace. >>=20 >> The lr part of struct trapframe was: 0x00535ad0 so >> showing around that: >>=20 >> 00535ab8 stwu r1,-32(r1) >> 00535abc mflr r0 >> 00535ac0 stw r29,20(r1) >> 00535ac4 stw r30,24(r1) >> 00535ac8 stw r31,28(r1) >> 00535acc stw r0,36(r1) >> 00535ad0 mr r31,r1 >> 00535ad4 mr r29,r3 >>=20 >> Back to the stack backtrace. . . >>=20 >> 013e8640 df 5e 56 80 00 53 59 dc 00 17 99 f8 00 17 99 f8 = |.^V..SY.........| >> 0x005359c8 bl 008ea420 >> 0x005359cc mr r3,r28 >> 0x005359d0 mr r4,r27 >> 0x005359d4 mr r5,r25 >> 0x005359d8 bl 005356ec >> 0x005359dc mfsprg r9,0 >>=20 >> I show around 0x005356ec >> from the sched_add+0x19c bl that is above >> because the routine is not referenced >> in the stack tracce but the above indicates >> that it should have been called: >> 0x005356ec stwu r1,-32(r1) >> 0x005356f0 mflr r0 >> 0x005356f4 stw r28,16(r1) >> 0x005356f8 stw r29,20(r1) >> 0x005356fc stw r30,24(r1) >> 0x00535700 stw r31,28(r1) >> 0x00535704 stw r0,36(r1) >> . . . >>=20 >> Back to the stack backtrace again. . . >>=20 >> 013e8650 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 00 04 = |.....i.y........| >> 013e8660 f8 1c 2f cc 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |../......@...C..| >> 013e8670 01 47 c6 c0 01 47 60 10 00 d1 b4 30 df 5e 56 80 = |.G...G`....0.^V.| >>=20 >> 013e8680 df 5e 56 b0 00 4a 87 8c 00 d2 5b 10 00 00 00 04 = |.^V..J....[.....| >> 0x004a8780 mr r3,r28 >> 0x004a8784 li r4,4 >> 0x004a8788 bl 0053583c = >> 0x004a878c lwz r9,0(r28) >>=20 >> 013e8690 df 5e 56 b0 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.^V......@...C..| >> 013e86a0 00 00 00 00 01 46 81 c0 00 d1 b4 30 df 5e 56 b0 = |.....F.....0.^V.| >>=20 >> 013e86b0 df 5e 56 f0 00 4a 97 0c 00 00 00 00 00 00 00 04 = |.^V..J..........| >> 0x004a9700 bl 005000ec >> 0x004a9704 mr r3,r27 >> 0x004a9708 bl 004a86bc = >> 0x004a970c lwz r11,0(r1) >>=20 >> 013e86c0 df 5e 56 e0 01 47 d6 c0 01 47 d6 c0 01 45 4d 40 = |.^V..G...G...EM@| >> 013e86d0 df 5e 56 f0 00 8e a4 44 06 40 c2 a8 00 00 17 9a = |.^V....D.@......| >> 013e86e0 78 00 00 00 00 e9 56 00 00 d1 c8 20 df 5e 56 f0 = |x.....V.... .^V.| >>=20 >> 013e86f0 df 5e 57 50 00 51 79 6c df 5e 58 78 01 47 d6 c0 = |.^WP.Qyl.^Xx.G..| >> 0x00517960 lwz r3,264(r29) >> 0x00517964 li r4,0 >> 0x00517968 bl 004a965c >> 0x0051796c lwz r11,0(r1) >>=20 >> 013e8700 01 47 d7 b8 00 00 00 00 00 d1 ab 24 00 00 00 04 = |.G.........$....| >> 013e8710 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e8720 00 d0 53 00 00 eb a7 80 00 00 00 01 00 00 00 00 = |..S.............| >> 013e8730 df 5e 59 8c 00 00 00 00 df 5e 58 78 00 00 17 99 = |.^Y......^Xx....| >> 013e8740 f8 1c 2f cc d0 01 dd 00 00 d2 5b 10 df 5e 57 50 = |../.......[..^WP| >>=20 >> 013e8750 df 5e 57 a0 00 8a b2 70 df 5e 57 60 df 5e 57 60 = |.^W....p.^W`.^W`| >> 0x008ab264 mr r3,r27 >> 0x008ab268 mr r4,r28 >> 0x008ab26c bl 00517540 >> 0x008ab270 li r3,0 >>=20 >> 013e8760 df 5e 57 a0 df 5e 58 78 05 86 37 00 00 00 00 04 = |.^W..^Xx..7.....| >> 013e8770 00 00 00 00 05 9b f2 00 00 c9 66 bc 01 47 d6 c0 = |..........f..G..| >> 013e8780 df 5e 59 8c 00 f6 1d 10 00 00 17 99 f8 1c 2f cc = |.^Y.........../.| >> 013e8790 d0 01 dd 00 d0 01 dd 30 00 d2 5b 10 df 5e 57 a0 = |.......0..[..^W.| >>=20 >> 013e87a0 df 5e 58 20 00 8a d1 10 00 d2 6e 5c df 5e 57 b0 |.^X = ......n\.^W.| >> 0x008ad100 mr r3,r26 >> 0x008ad104 mr r4,r27 >> 0x008ad108 li r5,0 >> 0x008ad10c bl 008aafc0 >> 0x008ad110 lwz r11,0(r1) >>=20 >> 013e87b0 df 5e 57 e0 00 4a 96 00 00 00 17 99 00 00 00 00 = |.^W..J..........| >> 013e87c0 f8 1c 2f cc 0a 3f a0 12 df 5e 58 78 05 86 37 00 = |../..?...^Xx..7.| >> 013e87d0 01 48 b1 00 05 86 37 80 00 d4 bd ec 00 cb 98 98 = |.H....7.........| >> 013e87e0 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e87f0 df 5e 59 e0 00 eb a7 80 00 c9 66 bc 01 47 d6 c0 = |.^Y.......f..G..| >> 013e8800 df 5e 59 8c df 5e 58 78 01 47 d6 c0 00 00 00 00 = |.^Y..^Xx.G......| >> 013e8810 00 f6 1d 10 00 00 00 01 00 d2 6b c8 df 5e 58 20 = |..........k..^X | >>=20 >> 013e8820 df 5e 58 40 00 8e 1e 48 00 00 00 00 00 eb a7 80 = |.^X@...H........| >> So 0x13e8820 from vmcore.9 has start of struct trapeframe. >> The 0x8e1e48 is from: >> 0x008e1e34 lwz r0,56(r28) >> 0x008e1e38 mtctr r0 >> 0x008e1e3c mr r3,r28 >> 0x008e1e40 lwz r4,64(r28) >> 0x008e1e44 bctrl >> 0x008e1e48 addi r29,r29,-1 >>=20 >> 013e8830 01 47 d7 94 00 00 00 01 00 d2 6e 4c df 5e 58 40 = |.G........nL.^X@| >> 013e8840 df 5e 58 70 00 8e 7c 9c 00 d1 ca ac df 5e 58 50 = |.^Xp..|......^XP| >> 013e8850 00 cd f0 74 00 00 00 01 00 00 00 01 00 eb a7 80 = |...t............| >> 013e8860 41 eb 30 00 0a 00 00 00 00 00 00 00 00 00 90 32 = |A.0............2| >> 013e8870 df 5e 59 30 00 10 08 f8 00 04 90 32 df 5e 59 30 = |.^Y0.......2.^Y0| >> 013e8880 01 47 d6 c0 00 00 00 00 0d 0b bf e3 00 00 00 00 = |.G..............| >> 013e8890 0d 0b bf e3 00 19 eb 7c 00 00 00 00 00 00 00 44 = |.......|.......D| >> 013e88a0 01 fc a0 55 00 00 90 32 d0 01 dd 00 00 00 00 00 = |...U...2........| >> 013e88b0 00 d4 bd ec 00 cb 98 98 00 c9 66 bc 00 c4 5d 48 = |..........f...]H| >> 013e88c0 00 c9 66 bc 00 d4 c5 3c df 5e 59 e0 00 eb a7 80 = |..f....<.^Y.....| >> 013e88d0 00 c9 66 bc 01 47 d6 c0 df 5e 59 8c 00 00 00 01 = |..f..G...^Y.....| >> 013e88e0 00 00 00 01 00 eb a7 80 00 00 00 00 00 8e 3b f8 = |..............;.| >>=20 >> srr0 >> 013e88f0 00 d2 6b f0 df 5e 59 30 00 8e 3c 14 40 00 00 42 = |..k..^Y0..<.@..B| >> The 0x00833c14 from the trap frame is the srr0 (conceptual lr): >> 0x008e3c04 stw r31,28(r1) >> 0x008e3c08 stw r0,36(r1) >> 0x008e3c0c mr r31,r1 >> 0x008e3c10 bcl- 20,4*cr7+so,008e3c14 = >> 0x008e3c14 mflr r30 >> 0x008e3c18 lwz r0,-32(r30) >>=20 >> 013e8900 20 00 00 00 00 8e 3b f8 00 8e 3c 80 00 00 90 32 | = .....;...<....2| >>=20 >> exc >> 013e8910 00 00 09 00 41 eb 30 00 0a 00 00 00 00 00 00 00 = |....A.0.........| >> The 0x00009000 above is the framep->exc (exception code) >> (decrementer in this case). >>=20 >> 013e8920 eb 10 25 b5 55 7e c4 22 00 00 00 00 00 00 00 04 = |..%.U~."........| >>=20 >> 013e8930 df 5e 59 50 00 00 00 01 00 00 00 01 00 eb a7 80 = |.^YP............| >> (trap [above] means no lr filled in here) >>=20 >> 013e8940 00 00 00 00 00 d4 ca 34 00 d2 6b f0 df 5e 59 50 = |.......4..k..^YP| >>=20 >> 013e8950 df 5e 59 70 00 8e 31 9c 00 00 00 02 00 eb a7 80 = |.^Yp..1.........| >> 0x008e318c bl 008ad8b8 >> 0x008e3190 lwz r29,0(r29) >> 0x008e3194 mtctr r29 >> 0x008e3198 bctrl >> 0x008e319c bl 008ad7b8 >>=20 >> 013e8960 00 f2 d5 fc 00 00 00 01 00 d1 ca ac df 5e 59 70 = |.............^Yp| >>=20 >> 013e8970 df 5e 5a 50 00 53 6e 7c fa 50 05 af fa 50 05 af = |.^ZP.Sn|.P...P..| >> 0x00536e78 bl 008e3144 >> 0x00536e7c stw r28,136(r27) >>=20 >>=20 >> 013e8980 fa 50 05 af fa 50 05 af fa 50 05 af ff ff ff fe = |.P...P...P......| >> 013e8990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89a0 ff ff ff ff ff ff ff ff ff ff ff ff fa 50 05 af = |.............P..| >> 013e89b0 fa 50 05 af 00 00 00 02 ff ff ff ff 00 00 01 f0 = |.P..............| >> 013e89c0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89e0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e8a00 df 5e 5a 20 fa 50 05 af 00 00 00 00 00 00 00 00 |.^Z = .P..........| >> 013e8a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8a30 00 00 00 00 00 53 69 a8 df 5e 5a 98 00 00 00 00 = |.....Si..^Z.....| >> 013e8a40 01 47 96 e0 01 47 d6 c0 00 d1 b3 70 df 5e 5a 50 = |.G...G.....p.^ZP| >>=20 >> 013e8a50 df 5e 5a 80 00 4a 3c b4 df 5e 5a 60 df 5e 5a 60 = |.^Z..J<..^Z`.^Z`| >> 0x004a3ca0 bl 008ea420 >> 0x004a3ca4 mr r3,r27 >> 0x004a3ca8 mr r4,r26 >> 0x004a3cac mtctr r25 >> 0x004a3cb0 bctrl >> 0x004a3cb4 lwz r0,108(r28) >>=20 >> 013e8a60 df 5e 5a 80 00 00 00 00 00 00 00 00 00 00 00 00 = |.^Z.............| >> 013e8a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >>=20 >> 013e8a80 00 00 00 00 00 8f 18 d0 00 53 69 a8 00 00 00 00 = |.........Si.....| >> 0x008f18c0 lwz r3,8(r1) >> 0x008f18c4 lwz r4,12(r1) >> 0x008f18c8 lwz r5,16(r1) >> 0x008f18cc bl 004a3bbc >> 0x008f18d0 addi r1,r1,16 >> 0x008f18d4 b 001008f8 >>=20 >>=20 >> So that is an example context for the failure. >>=20 >> It has taken weeks to get this. It may be some >> time before I get another for comparison/contrast. >>=20 >> And sticking with the same build vs. trying some more >> to find a better way to get more evidence is not >> obvious to me at this point. >=20 > I should have mentioned that this is > TARGET_ARCH=3Dpowerpc but used on a so-called PowerMac > G5 "Quad Core". >=20 > I've had 2 more examples, with the same 0x0090a030 > srr0 and such (vmcore.0 and vmcore.1) (I have added > one more little block of code for detecting an earlier > problem symptom not being currently seen so the build > is slight different from the earlier report.) >=20 > Looking around in vmcore.0 I find 3 examples of > "00 90 a0 30" in areas overlapping with objdump -x's > coverage. . . >=20 > 00c77fd0 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| > [ ] >=20 > (from sorted objdump -x output:) > 00c775a8 l O .data 00000dc0 seqprog > 00c78368 l O .data 0000000c seeprom_long_ewen >=20 > [ ] > 00f65820 41 eb 30 00 0a 00 00 00 00 90 a0 30 00 08 10 32 = |A.0........0...2| > . . . > 00f65870 00 90 a0 30 00 08 10 32 00 00 00 00 df 5d 30 00 = |...0...2.....]0.| > [ ] >=20 > (from sorted objdump -x output:) > 00f64780 g O .bss 00020000 __pcpu > 00f84780 l O .bss 00000004 ap_letgo >=20 >=20 > For vmcore.1 I went ahead and tried exploring > some with ddb --and had no new panics. . . > (Hand transcriptions of pictures) >=20 > fatal kernel trap: > exception =3D 0x700 (program) > srr0 =3D 0x90a030 > srr1 =3D 0x81032 > lr =3D 0x535ad0 > curthread =3D 0x147d6c0 > pic =3D 11, comm =3D idle: cpu0 >=20 > [ thread pid 11 tid 100003 ] > Stopped at _etext+0xb8fc: illegal instruction 0 > db> bt > 0xdf5e55d0: at sched_wakeup+0xa4 > 0xdf5e55f0: at setrunnable+0x9c > 0xdf5e5610: at sleepq_resume_thread+0x17c > 0xdf5e5640: at sleepq_timeout+0xc8 > 0xdf5e5680: at softclock_call_cc+0x1f0 > 0xdf5e56f0: at callout_process+0x27c > 0xdf5e57a0: at timercb+0x4c4 > 0xdf5e5820: at decr_intr+0xf0 > 0xdf5e5840: at powerpc_interrupt_0xf4 > 0xdf5e5870: at kernel DECR trap > by cpu_idle_60x+0x88 (so: srr0) > srr1=3D0x9032 > r1 =3D0xdf5e5930 > cr =3D0x40000042 > xer =3D0x20000000 > ctr =3D0x8e3bf8 > saved LR(0xfffffffd) is invalid. >=20 > db> show reg (but reformatted > r0 =3D0x4 > r1 =3D0xdf5e5590 > r2 =3D0x147d6c0 > r3 =3D0x54 testppc64size+0x34 > r4 =3D0x591d000 > r5 =3D0 > r6 =3D0 > r7 =3D0xf > r8 =3D0 > r9 =3D0xd4c03c cold > r10 =3D0x147d6c0 > r11 =3D0xdf5e55d0 > r12 =3D0 > r13 =3D0 > r14 =3D0xd4bdec sdt_probe_func > r15 =3D0xcb9898 std_lockstat___spin__release > r16 =3D0xd4c45c callwheelmask > r17 =3D0xd4c45c callwheelmask > r18 =3D0x55925 > r19 =3D0x559a4 > r20 =3D0x559 dsmisssize+0x469 > r21 =3D0x591d000 > r22 =3D0x566430 sleeppq_timeout > r23 =3D0x114 dsmisssize+0x24 > r24 =3D0 > r25 =3D0 > r26 =3D0x1 > r27 =3D0 > r28 =3D0xeba780 tdq_cpu > r29 =3D0x147d6c0 > r30 =3D0xd1caac > r31 =3D0xdf5e5590 > srr0 =3D0x90a030 > srr1 =3D0x81032 > lr =3D0x535ad0 shed_affinity+0x18 > ctr =3D0 > cr =3D0x20009034 > xer =3D0 > dar =3D0x419df5d4 > dsisr=3D0x24000000 > _etext+0xb8fc: illegal instruction 0 >=20 >=20 > Just for completeness: acttrace also showed: >=20 > Tracing command less pid 1144 tid 100150 td 0x5bc8a20 (CPU 3) > 0xd25a59f0: at powrpc_dispatch_intr+0xc8 > 0xd25a5a20: at openpic_dispatch+0x90 > 0xd25a5a50: at powerpc_interrupt+0xc0 > 0xd25a5a80: at user EXI trap > by 0x181c68c (so: ssr0) > r1 =3D0xffffdb30 > cr =3D0x44020624 > xer=3D0 > ctr=3D0x41989570 >=20 > Tracing command idle pid 11 tid 100004 td 0x147d360 (CPU 1) > saved LR(0x4c) in invalid >=20 > Tracing command idle pid 11 tid 100005 td 0x147d360 (CPU 2) > saved LR(0x4c) in invalid From owner-freebsd-ppc@freebsd.org Sat May 27 11:03:16 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E187D8329B for ; Sat, 27 May 2017 11:03:16 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-62.reflexion.net [208.70.210.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1719619ED for ; Sat, 27 May 2017 11:03:15 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 10467 invoked from network); 27 May 2017 11:06:55 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 27 May 2017 11:06:55 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Sat, 27 May 2017 07:03:14 -0400 (EDT) Received: (qmail 28060 invoked from network); 27 May 2017 11:03:14 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 11:03:14 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 6DFA5EC8171; Sat, 27 May 2017 04:03:13 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) [patch now fixed] Date: Sat, 27 May 2017 04:03:12 -0700 References: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> <62BF8E69-E7E6-4C4F-AB33-38B03E903CDA@dsl-only.net> <67C79408-0DDC-4A22-BFEC-6EC5DD80F076@dsl-only.net> <9D70C580-C545-4EA0-AB76-FE757C1AF60A@dsl-only.net> To: Justin Hibbits , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org In-Reply-To: <9D70C580-C545-4EA0-AB76-FE757C1AF60A@dsl-only.net> Message-Id: X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 11:03:16 -0000 While the patch probably is still appropriate, testing has shown that it is not sufficient to prevent the periodic/random panics that I have been getting. =3D=3D=3D Mark Millard markmi at dsl-only.net On 2017-May-27, at 2:12 AM, Mark Millard wrote: > [Testing shows the prior patch makes the PowerMac > G5 so-called "Quad Core" hang very early.] >=20 > I forgot to deal with the prepare side of things. > So, now with both prepare and restore code fixes, > this code boots. >=20 > # svnlite diff /usr/src/sys/powerpc/ofw/ofw_machdep.c = = Index: = /usr/src/sys/powerpc/ofw/ofw_machdep.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/powerpc/ofw/ofw_machdep.c (revision 317820) > +++ /usr/src/sys/powerpc/ofw/ofw_machdep.c (working copy) > @@ -111,26 +111,27 @@ > * Assume that interrupt are disabled at this point, or > * SPRG1-3 could be trashed > */ > -#ifdef __powerpc64__ > - __asm __volatile("mtsprg1 %0\n\t" > - "mtsprg2 %1\n\t" > - "mtsprg3 %2\n\t" > - : > - : "r"(ofmsr[2]), > - "r"(ofmsr[3]), > - "r"(ofmsr[4])); > -#else > - __asm __volatile("mfsprg0 %0\n\t" > - "mtsprg0 %1\n\t" > - "mtsprg1 %2\n\t" > - "mtsprg2 %3\n\t" > - "mtsprg3 %4\n\t" > - : "=3D&r"(ofw_sprg0_save) > - : "r"(ofmsr[1]), > - "r"(ofmsr[2]), > - "r"(ofmsr[3]), > - "r"(ofmsr[4])); > +#ifndef __powerpc64__ > + if (!(cpu_features & PPC_FEATURE_64)) > + __asm __volatile("mfsprg0 %0\n\t" > + "mtsprg0 %1\n\t" > + "mtsprg1 %2\n\t" > + "mtsprg2 %3\n\t" > + "mtsprg3 %4\n\t" > + : "=3D&r"(ofw_sprg0_save) > + : "r"(ofmsr[1]), > + "r"(ofmsr[2]), > + "r"(ofmsr[3]), > + "r"(ofmsr[4])); > + else > #endif > + __asm __volatile("mtsprg1 %0\n\t" > + "mtsprg2 %1\n\t" > + "mtsprg3 %2\n\t" > + : > + : "r"(ofmsr[2]), > + "r"(ofmsr[3]), > + "r"(ofmsr[4])); > } >=20 > static __inline void > @@ -147,7 +148,8 @@ > * PCPU data cannot be used until this routine is called ! > */ > #ifndef __powerpc64__ > - __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); > + if (!(cpu_features & PPC_FEATURE_64)) > + __asm __volatile("mtsprg0 %0" :: "r"(ofw_sprg0_save)); > #endif > } > #endif From owner-freebsd-ppc@freebsd.org Sat May 27 11:40:43 2017 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0090FD845F1 for ; Sat, 27 May 2017 11:40:43 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-62.reflexion.net [208.70.210.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B9B83121F for ; Sat, 27 May 2017 11:40:42 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 13194 invoked from network); 27 May 2017 11:41:59 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 27 May 2017 11:41:59 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Sat, 27 May 2017 07:40:41 -0400 (EDT) Received: (qmail 1586 invoked from network); 27 May 2017 11:40:41 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 May 2017 11:40:41 -0000 Received: from [192.168.1.114] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 54AAFEC8171; Sat, 27 May 2017 04:40:40 -0700 (PDT) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: A good backtrace from a head -r317820 powerpc random/periodic panic: execution of garbage at 0x0090a030 (in .hash section) [compare reg values] Date: Sat, 27 May 2017 04:40:39 -0700 References: <1CE8346B-04F3-48AB-A3E9-6DF3B86B8D1A@dsl-only.net> <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> To: Justin Hibbits , Nathan Whitehorn , FreeBSD PowerPC ML , freebsd-hackers@freebsd.org In-Reply-To: <8C88BB6F-E747-42A1-9DDC-35EC6D865141@dsl-only.net> Message-Id: <83A22794-AD54-4D27-951A-986E6FA693DB@dsl-only.net> X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 11:40:43 -0000 [I compare a new panic's register values to prior panic's values.] On 2017-May-26, at 10:29 PM, Mark Millard wrote: > [Additional information that does not need to > interlace with the prior material, so see > after. A somewhat better backtrace reported > by ddb. And so on.] >=20 >=20 > On 2017-May-26, at 7:14 PM, Mark Millard = wrote: >=20 >> I lucked out and got a vmcore.9 for a random >> panic that I could manage to backtrace for >> one of my test builds of -r317820. It appears >> that not all that much happened before it got >> the panic so much context was better preserved >> this time. >>=20 >> (I do not explore from ddb as I've had that >> panic and mess up the dump just made by >> replacing it. So this is a manual backtrace >> from the debug.minidump=3D0 style vmcore.9 >> file. objdump was used on the >> /boot/kernel/kernel to find code.) >>=20 >> Being able to see the problem is very >> sensitive to kernel memory layout. This >> is why I'm sticking with -r317820 built >> production style: the kind of context >> the problem was first observed in. >> Attempting a debug kernel build simply >> did not repeat the problem for days >> (vs. the usual hours for builds like >> this). >>=20 >>=20 >> So below is the backtrace: >>=20 >> (I do not show what trap_fatal calls: >> starting with trap_fatal and going toward >> larger memory addresses. . .) >>=20 >> [vmcore.9's >> offset >> in file >> when no >> 0x prefix] >>=20 >> 013e83b0 df 5e 55 50 00 8f 34 5c fa 50 05 af fa 50 05 af = |.^UP..4\.P...P..| >> 0x008f3454 mr r3,r26 >> 0x008f3458 bl 008f2030 >> 0x008f345c b 008f34c8 >>=20 >> 013e83c0 fa 50 05 af fa 50 05 af fa 50 05 af fa 50 05 af = |.P...P...P...P..| >> * >> 013e83e0 df 5e 54 00 fa 50 05 af fa 50 05 af fa 50 05 af = |.^T..P...P...P..| >> 013e83f0 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 00 = |.P...P.......^T.| >> 013e8400 df 5e 54 20 00 53 3a d0 fa 50 05 af fa 50 05 af |.^T = .S:..P...P..| >> 013e8410 fa 50 05 af fa 50 05 af 00 d1 ca ac df 5e 54 20 = |.P...P.......^T | >> 013e8420 df 5e 54 d0 00 53 3a d0 00 00 00 00 00 00 00 00 = |.^T..S:.........| >> 013e8430 00 00 00 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |................| >> 013e8440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8460 00 00 00 54 7f ff ff ff 00 00 00 00 ff ff ff aa = |...T............| >> 013e8470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8490 00 d4 c4 5c 00 d4 c4 5c 00 17 99 dd 00 17 9a 5c = |...\...\.......\| >> 013e84a0 00 00 17 9a 00 d4 c4 6c df 5e 54 ec 00 00 00 54 = |.......l.^T....T| >> 013e84b0 df 5e 54 d0 7f ff ff ff 05 aa 36 c0 05 aa 39 e8 = |.^T.......6...9.| >> 013e84c0 00 00 00 00 00 00 00 00 00 d1 ca ac df 5e 54 d0 = |.............^T.| >> 013e84d0 df 5e 55 80 00 53 3a d0 05 91 d0 00 05 91 d3 28 = |.^U..S:........(| >> 013e84e0 df 5e 55 00 00 00 00 00 00 d1 ca ac 00 00 00 0f = |.^U.............| >> 013e84f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> 013e8500 00 00 00 00 00 00 00 00 00 d4 bd ec 00 cb 98 98 = |................| >> 013e8510 00 d4 c4 5c 00 d4 c4 5c 00 17 99 f8 00 17 99 f8 = |...\...\........| >> 013e8520 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 17 99 = |.....i.y........| >> 013e8530 f8 1c 2f cc df 5e 55 88 01 47 d6 c0 01 43 a2 00 = |../..^U..G...C..| >> 013e8540 41 eb 30 00 0a 00 00 00 00 d2 6e 4c df 5e 55 50 = |A.0.......nL.^UP| >>=20 >> 013e8550 df 5e 55 80 00 8e 7d 40 df 5e 55 9c 00 00 00 28 = |.^U...}@.^U....(| >> 0x008e7d28 mfmsr r0 >> 0x008e7d2c or r0,r0,r9 >> 0x008e7d30 mtmsr r0 >> 0x008e7d34 isync >> 0x008e7d38 mr r3,r25 >> 0x008e7d3c bl 008f222c >> 0x008e7d40 lwz r11,0(r1) >>=20 >> 013e8560 00 00 00 00 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.........@...C..| >> 013e8570 41 eb 30 00 0a 00 00 00 00 00 00 00 00 08 10 32 = |A.0............2| >>=20 >> r0 r1 >> 013e8580 df 5e 56 40 00 10 08 f8 00 00 00 04 df 5e 56 40 = |.^V@.........^V@| >> The struct trapframe starts is 0x 013e8588 in the >> vmcore.9 file. The 0x00108f8 is as shown below: >>=20 >> 0x001008ec isync >> 0x001008f0 addi r3,r1,8 >> 0x001008f4 bl 008e7ba4 >> 0x001008f8 mfmsr r3 >> 0x001008fc andi. r3,r3,32767 >>=20 >> 013e8590 01 47 d6 c0 00 00 00 28 01 47 c6 c0 00 00 00 04 = |.G.....(.G......| >> r2 r3 r4 r5 >>=20 >> 013e85a0 00 00 00 04 00 00 00 0f 00 00 00 00 00 d4 c0 3c = |...............<| >> r6 r7 r8 r9 >>=20 >> 013e85b0 01 47 d6 c0 df 5e 56 80 14 3d 60 da 00 00 00 00 = |.G...^V..=3D`.....| >> r10 r11 r12 r13 >>=20 >> 013e85c0 00 d4 bd ec 00 cb 98 98 00 d4 c4 5c 00 d4 c4 5c = |...........\...\| >> r14 r15 r16 r17 >>=20 >> 013e85d0 00 17 99 f8 00 17 99 f8 00 00 17 99 fa 69 b8 79 = |.............i.y| >> r18 r19 r20 r21 >>=20 >> 013e85e0 00 17 9a 0a 00 00 17 99 f8 1c 2f cc 00 00 17 9a = |........../.....| >> r22 r32 r24 r25 >>=20 >> 013e85f0 06 40 c2 a8 01 43 a2 00 00 eb a7 80 01 47 d6 c0 = |.@...C.......G..| >> r26 r27 r28 r29 >>=20 >> 013e8600 00 d1 ca ac df 5e 56 40 00 53 5a d0 20 00 90 44 = |.....^V@.SZ. ..D| >> r30 r31 lr cr >>=20 >> xer ctr srr0 srr1 >> 013e8610 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| >> Note: objdump shows no code at 0x0090a030. 0x0090a030 is >> inside what objdump -x reports as the section .hash (Idx >> 2). >>=20 >> exc dar dsisr (booke dbcer0) >> 013e8620 00 00 07 00 41 eb 30 00 0a 00 00 00 01 47 c6 c0 = |....A.0......G..| >> The 0x00007000 above is the framep->exc (exception code) >> (program in this case). >>=20 >> 013e8630 00 eb a7 80 01 47 60 10 00 d1 ca ac df 5e 56 40 = |.....G`......^V@| >>=20 >> At this point the above does not match the below >> part of the stack trace. >>=20 >> The lr part of struct trapframe was: 0x00535ad0 so >> showing around that: >>=20 >> 00535ab8 stwu r1,-32(r1) >> 00535abc mflr r0 >> 00535ac0 stw r29,20(r1) >> 00535ac4 stw r30,24(r1) >> 00535ac8 stw r31,28(r1) >> 00535acc stw r0,36(r1) >> 00535ad0 mr r31,r1 >> 00535ad4 mr r29,r3 >>=20 >> Back to the stack backtrace. . . >>=20 >> 013e8640 df 5e 56 80 00 53 59 dc 00 17 99 f8 00 17 99 f8 = |.^V..SY.........| >> 0x005359c8 bl 008ea420 >> 0x005359cc mr r3,r28 >> 0x005359d0 mr r4,r27 >> 0x005359d4 mr r5,r25 >> 0x005359d8 bl 005356ec >> 0x005359dc mfsprg r9,0 >>=20 >> I show around 0x005356ec >> from the sched_add+0x19c bl that is above >> because the routine is not referenced >> in the stack tracce but the above indicates >> that it should have been called: >> 0x005356ec stwu r1,-32(r1) >> 0x005356f0 mflr r0 >> 0x005356f4 stw r28,16(r1) >> 0x005356f8 stw r29,20(r1) >> 0x005356fc stw r30,24(r1) >> 0x00535700 stw r31,28(r1) >> 0x00535704 stw r0,36(r1) >> . . . >>=20 >> Back to the stack backtrace again. . . >>=20 >> 013e8650 00 00 17 99 fa 69 b8 79 00 17 9a 0a 00 00 00 04 = |.....i.y........| >> 013e8660 f8 1c 2f cc 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |../......@...C..| >> 013e8670 01 47 c6 c0 01 47 60 10 00 d1 b4 30 df 5e 56 80 = |.G...G`....0.^V.| >>=20 >> 013e8680 df 5e 56 b0 00 4a 87 8c 00 d2 5b 10 00 00 00 04 = |.^V..J....[.....| >> 0x004a8780 mr r3,r28 >> 0x004a8784 li r4,4 >> 0x004a8788 bl 0053583c = >> 0x004a878c lwz r9,0(r28) >>=20 >> 013e8690 df 5e 56 b0 00 00 17 9a 06 40 c2 a8 01 43 a2 00 = |.^V......@...C..| >> 013e86a0 00 00 00 00 01 46 81 c0 00 d1 b4 30 df 5e 56 b0 = |.....F.....0.^V.| >>=20 >> 013e86b0 df 5e 56 f0 00 4a 97 0c 00 00 00 00 00 00 00 04 = |.^V..J..........| >> 0x004a9700 bl 005000ec >> 0x004a9704 mr r3,r27 >> 0x004a9708 bl 004a86bc = >> 0x004a970c lwz r11,0(r1) >>=20 >> 013e86c0 df 5e 56 e0 01 47 d6 c0 01 47 d6 c0 01 45 4d 40 = |.^V..G...G...EM@| >> 013e86d0 df 5e 56 f0 00 8e a4 44 06 40 c2 a8 00 00 17 9a = |.^V....D.@......| >> 013e86e0 78 00 00 00 00 e9 56 00 00 d1 c8 20 df 5e 56 f0 = |x.....V.... .^V.| >>=20 >> 013e86f0 df 5e 57 50 00 51 79 6c df 5e 58 78 01 47 d6 c0 = |.^WP.Qyl.^Xx.G..| >> 0x00517960 lwz r3,264(r29) >> 0x00517964 li r4,0 >> 0x00517968 bl 004a965c >> 0x0051796c lwz r11,0(r1) >>=20 >> 013e8700 01 47 d7 b8 00 00 00 00 00 d1 ab 24 00 00 00 04 = |.G.........$....| >> 013e8710 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e8720 00 d0 53 00 00 eb a7 80 00 00 00 01 00 00 00 00 = |..S.............| >> 013e8730 df 5e 59 8c 00 00 00 00 df 5e 58 78 00 00 17 99 = |.^Y......^Xx....| >> 013e8740 f8 1c 2f cc d0 01 dd 00 00 d2 5b 10 df 5e 57 50 = |../.......[..^WP| >>=20 >> 013e8750 df 5e 57 a0 00 8a b2 70 df 5e 57 60 df 5e 57 60 = |.^W....p.^W`.^W`| >> 0x008ab264 mr r3,r27 >> 0x008ab268 mr r4,r28 >> 0x008ab26c bl 00517540 >> 0x008ab270 li r3,0 >>=20 >> 013e8760 df 5e 57 a0 df 5e 58 78 05 86 37 00 00 00 00 04 = |.^W..^Xx..7.....| >> 013e8770 00 00 00 00 05 9b f2 00 00 c9 66 bc 01 47 d6 c0 = |..........f..G..| >> 013e8780 df 5e 59 8c 00 f6 1d 10 00 00 17 99 f8 1c 2f cc = |.^Y.........../.| >> 013e8790 d0 01 dd 00 d0 01 dd 30 00 d2 5b 10 df 5e 57 a0 = |.......0..[..^W.| >>=20 >> 013e87a0 df 5e 58 20 00 8a d1 10 00 d2 6e 5c df 5e 57 b0 |.^X = ......n\.^W.| >> 0x008ad100 mr r3,r26 >> 0x008ad104 mr r4,r27 >> 0x008ad108 li r5,0 >> 0x008ad10c bl 008aafc0 >> 0x008ad110 lwz r11,0(r1) >>=20 >> 013e87b0 df 5e 57 e0 00 4a 96 00 00 00 17 99 00 00 00 00 = |.^W..J..........| >> 013e87c0 f8 1c 2f cc 0a 3f a0 12 df 5e 58 78 05 86 37 00 = |../..?...^Xx..7.| >> 013e87d0 01 48 b1 00 05 86 37 80 00 d4 bd ec 00 cb 98 98 = |.H....7.........| >> 013e87e0 00 c9 66 bc 00 c4 5d 48 00 c9 66 bc 00 d4 c5 3c = |..f...]H..f....<| >> 013e87f0 df 5e 59 e0 00 eb a7 80 00 c9 66 bc 01 47 d6 c0 = |.^Y.......f..G..| >> 013e8800 df 5e 59 8c df 5e 58 78 01 47 d6 c0 00 00 00 00 = |.^Y..^Xx.G......| >> 013e8810 00 f6 1d 10 00 00 00 01 00 d2 6b c8 df 5e 58 20 = |..........k..^X | >>=20 >> 013e8820 df 5e 58 40 00 8e 1e 48 00 00 00 00 00 eb a7 80 = |.^X@...H........| >> So 0x13e8820 from vmcore.9 has start of struct trapeframe. >> The 0x8e1e48 is from: >> 0x008e1e34 lwz r0,56(r28) >> 0x008e1e38 mtctr r0 >> 0x008e1e3c mr r3,r28 >> 0x008e1e40 lwz r4,64(r28) >> 0x008e1e44 bctrl >> 0x008e1e48 addi r29,r29,-1 >>=20 >> 013e8830 01 47 d7 94 00 00 00 01 00 d2 6e 4c df 5e 58 40 = |.G........nL.^X@| >> 013e8840 df 5e 58 70 00 8e 7c 9c 00 d1 ca ac df 5e 58 50 = |.^Xp..|......^XP| >> 013e8850 00 cd f0 74 00 00 00 01 00 00 00 01 00 eb a7 80 = |...t............| >> 013e8860 41 eb 30 00 0a 00 00 00 00 00 00 00 00 00 90 32 = |A.0............2| >> 013e8870 df 5e 59 30 00 10 08 f8 00 04 90 32 df 5e 59 30 = |.^Y0.......2.^Y0| >> 013e8880 01 47 d6 c0 00 00 00 00 0d 0b bf e3 00 00 00 00 = |.G..............| >> 013e8890 0d 0b bf e3 00 19 eb 7c 00 00 00 00 00 00 00 44 = |.......|.......D| >> 013e88a0 01 fc a0 55 00 00 90 32 d0 01 dd 00 00 00 00 00 = |...U...2........| >> 013e88b0 00 d4 bd ec 00 cb 98 98 00 c9 66 bc 00 c4 5d 48 = |..........f...]H| >> 013e88c0 00 c9 66 bc 00 d4 c5 3c df 5e 59 e0 00 eb a7 80 = |..f....<.^Y.....| >> 013e88d0 00 c9 66 bc 01 47 d6 c0 df 5e 59 8c 00 00 00 01 = |..f..G...^Y.....| >> 013e88e0 00 00 00 01 00 eb a7 80 00 00 00 00 00 8e 3b f8 = |..............;.| >>=20 >> srr0 >> 013e88f0 00 d2 6b f0 df 5e 59 30 00 8e 3c 14 40 00 00 42 = |..k..^Y0..<.@..B| >> The 0x00833c14 from the trap frame is the srr0 (conceptual lr): >> 0x008e3c04 stw r31,28(r1) >> 0x008e3c08 stw r0,36(r1) >> 0x008e3c0c mr r31,r1 >> 0x008e3c10 bcl- 20,4*cr7+so,008e3c14 = >> 0x008e3c14 mflr r30 >> 0x008e3c18 lwz r0,-32(r30) >>=20 >> 013e8900 20 00 00 00 00 8e 3b f8 00 8e 3c 80 00 00 90 32 | = .....;...<....2| >>=20 >> exc >> 013e8910 00 00 09 00 41 eb 30 00 0a 00 00 00 00 00 00 00 = |....A.0.........| >> The 0x00009000 above is the framep->exc (exception code) >> (decrementer in this case). >>=20 >> 013e8920 eb 10 25 b5 55 7e c4 22 00 00 00 00 00 00 00 04 = |..%.U~."........| >>=20 >> 013e8930 df 5e 59 50 00 00 00 01 00 00 00 01 00 eb a7 80 = |.^YP............| >> (trap [above] means no lr filled in here) >>=20 >> 013e8940 00 00 00 00 00 d4 ca 34 00 d2 6b f0 df 5e 59 50 = |.......4..k..^YP| >>=20 >> 013e8950 df 5e 59 70 00 8e 31 9c 00 00 00 02 00 eb a7 80 = |.^Yp..1.........| >> 0x008e318c bl 008ad8b8 >> 0x008e3190 lwz r29,0(r29) >> 0x008e3194 mtctr r29 >> 0x008e3198 bctrl >> 0x008e319c bl 008ad7b8 >>=20 >> 013e8960 00 f2 d5 fc 00 00 00 01 00 d1 ca ac df 5e 59 70 = |.............^Yp| >>=20 >> 013e8970 df 5e 5a 50 00 53 6e 7c fa 50 05 af fa 50 05 af = |.^ZP.Sn|.P...P..| >> 0x00536e78 bl 008e3144 >> 0x00536e7c stw r28,136(r27) >>=20 >>=20 >> 013e8980 fa 50 05 af fa 50 05 af fa 50 05 af ff ff ff fe = |.P...P...P......| >> 013e8990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89a0 ff ff ff ff ff ff ff ff ff ff ff ff fa 50 05 af = |.............P..| >> 013e89b0 fa 50 05 af 00 00 00 02 ff ff ff ff 00 00 01 f0 = |.P..............| >> 013e89c0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89e0 ff ff ff fe ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e89f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff = |................| >> 013e8a00 df 5e 5a 20 fa 50 05 af 00 00 00 00 00 00 00 00 |.^Z = .P..........| >> 013e8a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >> * >> 013e8a30 00 00 00 00 00 53 69 a8 df 5e 5a 98 00 00 00 00 = |.....Si..^Z.....| >> 013e8a40 01 47 96 e0 01 47 d6 c0 00 d1 b3 70 df 5e 5a 50 = |.G...G.....p.^ZP| >>=20 >> 013e8a50 df 5e 5a 80 00 4a 3c b4 df 5e 5a 60 df 5e 5a 60 = |.^Z..J<..^Z`.^Z`| >> 0x004a3ca0 bl 008ea420 >> 0x004a3ca4 mr r3,r27 >> 0x004a3ca8 mr r4,r26 >> 0x004a3cac mtctr r25 >> 0x004a3cb0 bctrl >> 0x004a3cb4 lwz r0,108(r28) >>=20 >> 013e8a60 df 5e 5a 80 00 00 00 00 00 00 00 00 00 00 00 00 = |.^Z.............| >> 013e8a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| >>=20 >> 013e8a80 00 00 00 00 00 8f 18 d0 00 53 69 a8 00 00 00 00 = |.........Si.....| >> 0x008f18c0 lwz r3,8(r1) >> 0x008f18c4 lwz r4,12(r1) >> 0x008f18c8 lwz r5,16(r1) >> 0x008f18cc bl 004a3bbc >> 0x008f18d0 addi r1,r1,16 >> 0x008f18d4 b 001008f8 >>=20 >>=20 >> So that is an example context for the failure. >>=20 >> It has taken weeks to get this. It may be some >> time before I get another for comparison/contrast. >>=20 >> And sticking with the same build vs. trying some more >> to find a better way to get more evidence is not >> obvious to me at this point. >=20 > I should have mentioned that this is > TARGET_ARCH=3Dpowerpc but used on a so-called PowerMac > G5 "Quad Core". >=20 > I've had 2 more examples, with the same 0x0090a030 > srr0 and such (vmcore.0 and vmcore.1) (I have added > one more little block of code for detecting an earlier > problem symptom not being currently seen so the build > is slight different from the earlier report.) >=20 > Looking around in vmcore.0 I find 3 examples of > "00 90 a0 30" in areas overlapping with objdump -x's > coverage. . . >=20 > 00c77fd0 00 00 00 00 00 00 00 00 00 90 a0 30 00 08 10 32 = |...........0...2| > [ ] >=20 > (from sorted objdump -x output:) > 00c775a8 l O .data 00000dc0 seqprog > 00c78368 l O .data 0000000c seeprom_long_ewen >=20 > [ ] > 00f65820 41 eb 30 00 0a 00 00 00 00 90 a0 30 00 08 10 32 = |A.0........0...2| > . . . > 00f65870 00 90 a0 30 00 08 10 32 00 00 00 00 df 5d 30 00 = |...0...2.....]0.| > [ ] >=20 > (from sorted objdump -x output:) > 00f64780 g O .bss 00020000 __pcpu > 00f84780 l O .bss 00000004 ap_letgo Making some comparisons across two examples of the panic, prior vs. newest. . . The srr0 was different by 0x40: > For vmcore.1 I went ahead and tried exploring > some with ddb --and had no new panics. . . > (Hand transcriptions of pictures) >=20 > fatal kernel trap: > exception =3D 0x700 (program) > srr0 =3D 0x90a030 vs.=3D 0x90a070 > srr1 =3D 0x81032 > lr =3D 0x535ad0 > curthread =3D 0x147d6c0 > pic =3D 11, comm =3D idle: cpu0 >=20 > [ thread pid 11 tid 100003 ] > Stopped at _etext+0xb8fc: illegal instruction 0 > db> bt > 0xdf5e55d0: at sched_wakeup+0xa4 > 0xdf5e55f0: at setrunnable+0x9c > 0xdf5e5610: at sleepq_resume_thread+0x17c > 0xdf5e5640: at sleepq_timeout+0xc8 > 0xdf5e5680: at softclock_call_cc+0x1f0 > 0xdf5e56f0: at callout_process+0x27c > 0xdf5e57a0: at timercb+0x4c4 > 0xdf5e5820: at decr_intr+0xf0 > 0xdf5e5840: at powerpc_interrupt+0xf4 > 0xdf5e5870: at kernel DECR trap > by cpu_idle_60x+0x88 (so: srr0) > srr1=3D0x9032 > r1 =3D0xdf5e5930 > cr =3D0x40000042 > xer =3D0x20000000 > ctr =3D0x8e3bf8 > saved LR(0xfffffffd) is invalid. Below I show an alternate example values were they are different, where the patch that I attempted was also involved for the new figures. > db> show reg (but reformatted > r0 =3D0x4 > r1 =3D0xdf5e5590 > r2 =3D0x147d6c0 > r3 =3D0x54 testppc64size+0x34 vs.=3D0x78 dlmisssize+0x8 > r4 =3D0x591d000 vs.=3D0x5aa4360 > r5 =3D0 > r6 =3D0 > r7 =3D0xf > r8 =3D0 > r9 =3D0xd4c03c cold > r10 =3D0x147d6c0 > r11 =3D0xdf5e55d0 > r12 =3D0 > r13 =3D0 > r14 =3D0xd4bdec sdt_probe_func > r15 =3D0xcb9898 std_lockstat___spin__release > r16 =3D0xd4c45c callwheelmask > r17 =3D0xd4c45c callwheelmask > r18 =3D0x55925 vs.=3D0x15c267 fdt_property+0x113 > r19 =3D0x559a4 vs.=3D0x15c2e6 fdt_property+0x192 > r20 =3D0x559 dsmisssize+0x469 vs.=3D0x15c2 dsmisssize+0x14d2 > r21 =3D0x591d000 vs.=3D0x5aa4360 > r22 =3D0x566430 sleepq_timeout > r23 =3D0x114 dsmisssize+0x24 > r24 =3D0 > r25 =3D0 > r26 =3D0x1 > r27 =3D0 > r28 =3D0xeba780 tdq_cpu > r29 =3D0x147d6c0 > r30 =3D0xd1caac > r31 =3D0xdf5e5590 srr0 =3D0x90a030 _etext+0xb8fc vs.=3D0x90c070 _etext+0xb8fc > srr1 =3D0x81032 > lr =3D0x535ad0 shed_affinity+0x18 > ctr =3D0 > cr =3D0x20009034 vs.=3D0x20009044 > xer =3D0 > dar =3D0x419df5d4 vs.=3D0x41d276f4 > dsisr=3D0x24000000 vs.=3D0xa0000000 > _etext+0xb8fc: illegal instruction 0 >=20 >=20 > Just for completeness: acttrace also showed: >=20 > Tracing command less pid 1144 tid 100150 td 0x5bc8a20 (CPU 3) > 0xd25a59f0: at powrpc_dispatch_intr+0xc8 > 0xd25a5a20: at openpic_dispatch+0x90 > 0xd25a5a50: at powerpc_interrupt+0xc0 > 0xd25a5a80: at user EXI trap > by 0x181c68c (so: ssr0) > r1 =3D0xffffdb30 > cr =3D0x44020624 > xer=3D0 > ctr=3D0x41989570 The newer example panic had CPU3 more like CPU 1 and 2 below. > Tracing command idle pid 11 tid 100004 td 0x147d360 (CPU 1) > saved LR(0x4c) in invalid >=20 > Tracing command idle pid 11 tid 100005 td 0x147d360 (CPU 2) > saved LR(0x4c) in invalid =3D=3D=3D Mark Millard markmi at dsl-only.net