From owner-freebsd-stable@freebsd.org Thu May 25 09:41:10 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC734D81F74 for ; Thu, 25 May 2017 09:41:10 +0000 (UTC) (envelope-from royger@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id BA3051DBD for ; Thu, 25 May 2017 09:41:10 +0000 (UTC) (envelope-from royger@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B98E4D81F73; Thu, 25 May 2017 09:41:10 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B92E5D81F72 for ; Thu, 25 May 2017 09:41:10 +0000 (UTC) (envelope-from royger@gmail.com) Received: from mail-wm0-x229.google.com (mail-wm0-x229.google.com [IPv6:2a00:1450:400c:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A3AD1DBA; Thu, 25 May 2017 09:41:10 +0000 (UTC) (envelope-from royger@gmail.com) Received: by mail-wm0-x229.google.com with SMTP id d127so90179780wmf.0; Thu, 25 May 2017 02:41:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=x51/VTqfD+cjY+2d1j6IAY558d6qdmzj6yWEH8+XCBM=; b=EKFDNLgIT51sWtknKpe2bhEHLSlIVJzP8kM9QdPh5z8nzpBtLh54Dk/eF8vUXoU+fR t4CAMJw+mVgGSIVoUfJ2bIAJiAu1jATX2EAlS+0BBMEPEj2IMspBwAqrqo+S7HakY+RZ TMyTicWuEEnsZhizuiQbeaMF+EMU6DcVYmCmu6GwDToZd9JmaqTEh6dTyiicReGTh0qo k64zcmBcrFAxBYjF3YfU5GOJ6Fco5wwdu+30Zq4Ba2dkNOy/L3cVg27lRK8O1xG9GmSd 80YPoxz7zW9htNlrLJLUOExpgRAdzZVDPKTOC/ziiaE8btxQ7gDzaquE1j//NKFTFmjR Ud7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=x51/VTqfD+cjY+2d1j6IAY558d6qdmzj6yWEH8+XCBM=; b=IEWo6YMGuedVRLSnjRDIxDL2uGECUrSgGX4qTyBCC9KnhSeFI26nA72RSMZfXF+Pwj nP7D/9/UnnvJyn6lWRkzy/ubppAD896RgBsJalenn+y0ijlDtG2lMZwEKuRqnHEqSwnO QGqcks/my7sm3jRZPirQJLlf4Ii+/CVKOiNxFJwBTHDe04aaMh8YSBod79XV6Gz0TTjS WB3NEf9Kmr64cM2Os6nMAzG8sJkk8lVt+8IABDXoPZC7k0CtKqd6TU19YQjsWNyWXjNX 09HpFNhzgb98IA7X1r/AyzLd9RcwjJqi3uvnI8QRzT0Qy5LIFj8RFRYz1sG2l2c0yt5v 7+QA== X-Gm-Message-State: AODbwcBhOXBGy8OuROGe5B4HZ+MJUQi2hU8dMOAQnPJ01XT8s3oTI24/ wWJh6bfXJcxX9kcB X-Received: by 10.80.161.166 with SMTP id 35mr22101863edk.96.1495705268000; Thu, 25 May 2017 02:41:08 -0700 (PDT) Received: from localhost (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id a54sm2310740edd.68.2017.05.25.02.41.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 May 2017 02:41:07 -0700 (PDT) Sender: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Date: Thu, 25 May 2017 10:41:03 +0100 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Adam McDougall Cc: stable@freebsd.org, cperciva@freebsd.org Subject: Re: Boot hang on Xen after r318347/(310418) Message-ID: <20170525094103.iedycf2t4dy367fc@dhcp-3-128.uk.xensource.com> References: <20170524223307.GS79337@egr.msu.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170524223307.GS79337@egr.msu.edu> User-Agent: NeoMutt/20170428 (1.8.2) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 09:41:11 -0000 On Wed, May 24, 2017 at 06:33:07PM -0400, Adam McDougall wrote: > Hello, > > Recently I made a new build of 11-STABLE but encountered a boot hang > at this state: > http://www.egr.msu.edu/~mcdouga9/pics/r318347-smp-hang.png > > It is easy to reproduce, I can just boot from any 11 or 12 ISO that > contains the commit. I have just tested latest HEAD (r318861) and stable/11 (r318854) and they both work fine on my environment (a VM with 4 vCPUs and 2GB of RAM on OSS Xen 4.9). I'm also adding Colin in case he has some input, he has been doing some tests on HEAD and AFAIK he hasn't seen any issues. > I compiled various svn revisions to confirm that r318347 caused the > issue and r318346 is fine. With r318347 or later including the latest > 11-STABLE, the system will only boot with one virtual CPU in XenServer. > Any more cpus and it hangs. I also tried a 12 kernel from head this > afternoon and I have the same hang. I had this issue on XenServer 7 > (Xen 4.7) and XenServer 6.5 (Xen 4.4). I did most of my testing on 7. I > also did much of my testing with a GENERIC kernel to try to rule out > kernel configuration mistakes. When it hangs, the performance > monitoring in Xen tells me at least one CPU is pegged. r318674 boots > fine on physical hardware without Xen involved. > > Looking at r318347 which mentions EARLY_AP_STARTUP and later seeing > r318763 which enables EARLY_AP_STARTUP in GENERIC, I tried adding it to > my kernel but it turned the hang into a panic but with any number of > CPUs: > http://www.egr.msu.edu/~mcdouga9/pics/r318347-early-ap-startup-panic.png I guess this is on stable/11 right? The panic looks easier to debug that the hang, so let's start by this one. Can you enable the serial console and kernel debug options in order to get a trace? With just this it's almost impossible to know what went wrong. If you still have that kernel around (and it's debug symbols), can you do: $ addr2line -e /usr/lib/debug/boot/kernel/kernel.debug 0xffffffff80793344 (The address is the instruction pointer on the crash image, I think I got it right) In order to compile a stable/11 kernel with full debugging support you will have to add: # For full debugger support use (turn off in stable branch): options BUF_TRACKING # Track buffer history options DDB # Support DDB. options FULL_BUF_TRACKING # Track more buffer history options GDB # Support remote GDB. options DEADLKRES # Enable the deadlock resolver options INVARIANTS # Enable calls of extra sanity checking options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS options WITNESS # Enable checks to detect deadlocks and cycles options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed options MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones To your kernel config file. Just to be sure, this is an amd64 kernel right? Roger.