From owner-freebsd-hackers@freebsd.org Wed Nov 11 13:01:54 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 73CE4A2C0B9; Wed, 11 Nov 2015 13:01:54 +0000 (UTC) (envelope-from aduane@juniper.net) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0132.outbound.protection.outlook.com [157.56.110.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C358B12E9; Wed, 11 Nov 2015 13:01:53 +0000 (UTC) (envelope-from aduane@juniper.net) Received: from BLUPR05MB723.namprd05.prod.outlook.com (10.141.207.153) by BLUPR05MB721.namprd05.prod.outlook.com (10.141.207.144) with Microsoft SMTP Server (TLS) id 15.1.318.15; Wed, 11 Nov 2015 12:46:05 +0000 Received: from BLUPR05MB723.namprd05.prod.outlook.com ([10.141.207.153]) by BLUPR05MB723.namprd05.prod.outlook.com ([10.141.207.153]) with mapi id 15.01.0318.003; Wed, 11 Nov 2015 12:46:05 +0000 From: Andrew Duane To: Andriy Gapon , John Baldwin CC: Hans Petter Selasky , FreeBSD Hackers , "freebsd-current@FreeBSD.org" Subject: RE: strange kernel crash Thread-Topic: strange kernel crash Thread-Index: AQHRHCNwVoZ/0UuTME+ghwXWxzr5bp6WdhgAgABOyyA= Date: Wed, 11 Nov 2015 12:46:05 +0000 Message-ID: References: <563C8CED.3020101@FreeBSD.org> <2278845.gkxYBUMIWE@ralph.baldwin.cx> <5641AF48.1000507@FreeBSD.org> <18887451.3zmRk4crln@ralph.baldwin.cx> <5642F5E0.4050402@FreeBSD.org> In-Reply-To: <5642F5E0.4050402@FreeBSD.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=aduane@juniper.net; x-originating-ip: [66.129.241.11] x-microsoft-exchange-diagnostics: 1; BLUPR05MB721; 5:YOMjMDcHkhFLEZ2TVQkdckkjC/1B4AGSC3I4b01Ndu9ZY3FkmA9r+kz4YyzTUM/+jZWBDSSfwA5NQmCrjR+YL5YjKuvmPCXAfK/uShdLM3b9gVpqszv9vqL+h4gx0kVLFO8g3ye+vjfEEKiqGj7HLw==; 24:13y6mwhIKvSk/F/7rVo7tebSrz6UF5RqkyAG7mlPwSqLq78hX1cWywHUEJd27IU2gJPcAKjHzm8nVHq+dyfZTvaLLzV4KbEKTpF019d9vWs=; 20:IYWdDqr9mbi1J9JA++FozGhW24F54w9cL64HOuI4gI9RmhoARQtz1cTR7k+DsZSQ+yGrcqPtJwh4WiLzRHtcVw== x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(42134001)(42139001); SRVR:BLUPR05MB721; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(10201501046)(3002001); SRVR:BLUPR05MB721; BCL:0; PCL:0; RULEID:; SRVR:BLUPR05MB721; x-forefront-prvs: 0757EEBDCA x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(189002)(479174004)(24454002)(13464003)(199003)(377454003)(76576001)(5008740100001)(5001960100002)(76176999)(50986999)(101416001)(10400500002)(5002640100001)(11100500001)(189998001)(54356999)(5003600100002)(5004730100002)(74316001)(5007970100001)(122556002)(33656002)(99286002)(40100003)(106116001)(19580395003)(19580405001)(66066001)(106356001)(93886004)(15975445007)(2950100001)(102836002)(2900100001)(97736004)(81156007)(5001770100001)(87936001)(77096005)(92566002)(86362001)(105586002); DIR:OUT; SFP:1102; SCL:1; SRVR:BLUPR05MB721; H:BLUPR05MB723.namprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: juniper.net does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Nov 2015 12:46:05.0445 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR05MB721 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Nov 2015 13:01:54 -0000 > -----Original Message----- > From: owner-freebsd-hackers@freebsd.org [mailto:owner-freebsd-hackers@fre= ebsd.org] On Behalf Of Andriy Gapon > Sent: Wednesday, November 11, 2015 3:02 AM > To: John Baldwin > Cc: Hans Petter Selasky ; FreeBSD Hackers ; freebsd-current@FreeBSD.org > Subject: Re: strange kernel crash >=20 > On 10/11/2015 20:42, John Baldwin wrote: > > On Tuesday, November 10, 2015 10:48:08 AM Andriy Gapon wrote: > >> On 09/11/2015 22:16, John Baldwin wrote: > >>> On Friday, November 06, 2015 07:02:59 PM Hans Petter Selasky wrote: > >>>> On 11/06/15 12:20, Andriy Gapon wrote: > >>>>> Now the strange part: > >>>>> > >>>>> 0xffffffff80619a18 <+744>: jne 0xffffffff80619a61 <__mtx_l= ock_flags+817> > >>>>> 0xffffffff80619a1a <+746>: mov %rbx,(%rsp) > >>>>> =3D> 0xffffffff80619a1e <+750>: movq $0x0,0x18(%rsp) > >>>>> 0xffffffff80619a27 <+759>: movq $0x0,0x10(%rsp) > >>>>> 0xffffffff80619a30 <+768>: movq $0x0,0x8(%rsp) > >>>> > >>>> Were these instructions dumped from RAM or from the kernel ELF file? > >>> > >>> Probably not from RAM. You can use 'info files' in gdb to see what > >>> is handling the address range in question (core vs executable). x/i > >>> in ddb would have been the "real" truth. > >> > >> Yes, according to the output of files it looks like gdb would read > >> that data from the text section of the kernel file. > >> > >> How about libkvm? Would kvm_read read data from the core file? > > > > kvm_read should only access the vmcore, yes. > > > >> I've written the following small program (cut down dmesg.c, actually): > >> https://people.freebsd.org/~avg/vmcore_read.c > >> > >> (kgdb) disassemble /r > >> =3D> 0xffffffff80619a1e <+750>: 48 c7 44 24 18 00 00 00 00 movq > >> $0x0,0x18(%rsp) > >> > >> $ vmcore_read -N /boot/kernel.29/kernel -M /var/crash/vmcore.29 > >> 0xffffffff80619a1e 9 > >> 48 c7 44 24 18 00 00 00 00 > >> > >> Seems like the code is intact. > >> > >> P.S. > >> 1. To correct something I said earlier, the fault is #UD, not #GP. > >> 2. The only "suspicious" activity at the time of the crash was the exe= cution of a bhyve VM. > > > > Was the crash in the guest or the host? UD# seems even more bizarre. >=20 > It was the host. This is bizarre indeed. I can think only of two possib= ilities: > - new CPU erratum > - corrupted data somehow getting into the instruction cache, but the co= rrect data being read during the crash dump (i.e. flaky memory) Or perhaps a missing memory sync operation somewhere.... >=20 > -- > Andriy Gapon > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= "