From owner-freebsd-stable@FreeBSD.ORG Mon Feb 8 19:42:44 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1BD51065670 for ; Mon, 8 Feb 2010 19:42:44 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 925E78FC18 for ; Mon, 8 Feb 2010 19:42:44 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 2DBDC46B35; Mon, 8 Feb 2010 14:42:44 -0500 (EST) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 685E38A01F; Mon, 8 Feb 2010 14:42:43 -0500 (EST) From: John Baldwin To: Kostik Belousov Date: Mon, 8 Feb 2010 14:42:33 -0500 User-Agent: KMail/1.12.1 (FreeBSD/7.2-CBSD-20100120; KDE/4.3.1; amd64; ; ) References: <4B6B89E7.8030002@sdf.lonestar.org> <201002081315.06445.jhb@freebsd.org> <20100208185146.GO9991@deviant.kiev.zoral.com.ua> In-Reply-To: <20100208185146.GO9991@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201002081442.33579.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Mon, 08 Feb 2010 14:42:43 -0500 (EST) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-1.4 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: Tom McLaughlin , freebsd-stable@freebsd.org Subject: Re: Recent MFC to 7 causes crash on VMware ESXi X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Feb 2010 19:42:44 -0000 On Monday 08 February 2010 1:51:46 pm Kostik Belousov wrote: > On Mon, Feb 08, 2010 at 01:15:06PM -0500, John Baldwin wrote: > > On Monday 08 February 2010 11:06:00 am Kostik Belousov wrote: > > > On Mon, Feb 08, 2010 at 10:32:37AM -0500, John Baldwin wrote: > > > > On Monday 08 February 2010 9:56:36 am Kostik Belousov wrote: > > > > > On Mon, Feb 08, 2010 at 09:49:00AM -0500, John Baldwin wrote: > > > > > > On Saturday 06 February 2010 4:47:16 pm Tom McLaughlin wrote: > > > > > > > John Baldwin wrote, On 02/05/2010 08:27 AM: > > > > > > > > On Thursday 04 February 2010 10:00:55 pm Tom McLaughlin wrote: > > > > > > > >> Hi all, a recent MFC to 7-STABLE has started to cause issues for > > my VMs > > > > > > > >> on VMware ESXi 3.5u4. After loading the mpt driver for the LSI > > disk > > > > > > > >> controller the VM just shuts off. The workaround is to change > > the disk > > > > > > > >> controller to the BusLogic type. Still, it used to work up until > > last > > > > > > > >> week. The change was made around January 26th and based on the > > commits > > > > > > > >> that day I'm guessing it's either r203047 or r203073 > > > > > > > >> > > > > > > > >> I have the same issue with both amd64 and i386 VMs. This affects > > HEAD > > > > > > > >> and 8-STABLE as well and first affected HEAD over the summer. (I > > just > > > > > > > >> worked around it and went about my business at the time. :-/) > > I've > > > > > > > >> attached a dmesg from a kernel before the problem and one from > > after it > > > > > > > >> started. > > > > > > > > > > > > > > > > What if you set 'hw.clfush_disable=1' from the loader? > > > > > > > > > > > > > > > > > > > > > > Yes, that corrected it on all my VMs. I've talked to people on ESXi > > 4 > > > > > > > and they do not see the problem. I have yet to try 3.5u5 to see if > > this > > > > > > > is a non-issue. 3.5 will be supported for awhile longer from > > VMware. > > > > > > > I'm going to try upgrading the box during the week. > > > > > > > > > > > > I believe folks had to do this on HEAD/8.x as well. Perhaps we can > > > > > > automatically disable clflush if we are executing under VMware or Xen: > > > > > > > > > > > > Index: amd64/amd64/initcpu.c > > > > > > =================================================================== > > > > > > --- amd64/amd64/initcpu.c (revision 203430) > > > > > > +++ amd64/amd64/initcpu.c (working copy) > > > > > > @@ -177,17 +177,16 @@ > > > > > > if ((cpu_feature & CPUID_CLFSH) != 0) > > > > > > cpu_clflush_line_size = ((cpu_procinfo >> 8) & 0xff) * 8; > > > > > > /* > > > > > > - * XXXKIB: (temporary) hack to work around traps generated when > > > > > > - * CLFLUSHing APIC registers window. > > > > > > + * XXXKIB: (temporary) hack to work around traps generated > > > > > > + * when CLFLUSHing APIC registers window under virtualization > > > > > > + * environments. > > > > > > */ > > > > > > TUNABLE_INT_FETCH("hw.clflush_disable", &hw_clflush_disable); > > > > > > - if (cpu_vendor_id == CPU_VENDOR_INTEL && !(cpu_feature & CPUID_SS) > > && > > > > > > - hw_clflush_disable == -1) > > > > > > + if (vm_guest != 0 /* VM_GUEST_NO */ && hw_clflush_disable == -1) > > > > > > cpu_feature &= ~CPUID_CLFSH; > > > > > > /* > > > > > > * Allow to disable CLFLUSH feature manually by > > > > > > - * hw.clflush_disable tunable. This may help Xen guest on some AMD > > > > > > - * CPUs. > > > > > > + * hw.clflush_disable tunable. > > > > > > */ > > > > > > if (hw_clflush_disable == 1) > > > > > > cpu_feature &= ~CPUID_CLFSH; > > > > > > Index: i386/i386/initcpu.c > > > > > > =================================================================== > > > > > > --- i386/i386/initcpu.c (revision 203430) > > > > > > +++ i386/i386/initcpu.c (working copy) > > > > > > @@ -724,17 +724,16 @@ > > > > > > if ((cpu_feature & CPUID_CLFSH) != 0) > > > > > > cpu_clflush_line_size = ((cpu_procinfo >> 8) & 0xff) * 8; > > > > > > /* > > > > > > - * XXXKIB: (temporary) hack to work around traps generated when > > > > > > - * CLFLUSHing APIC registers window. > > > > > > + * XXXKIB: (temporary) hack to work around traps generated > > > > > > + * when CLFLUSHing APIC registers window under virtualization > > > > > > + * environments. > > > > > > */ > > > > > > TUNABLE_INT_FETCH("hw.clflush_disable", &hw_clflush_disable); > > > > > > - if (cpu_vendor_id == CPU_VENDOR_INTEL && !(cpu_feature & CPUID_SS) > > && > > > > > > - hw_clflush_disable == -1) > > > > > > + if (vm_guest != 0 /* VM_GUEST_NO */ && hw_clflush_disable == -1) > > > > > > cpu_feature &= ~CPUID_CLFSH; > > > > > > /* > > > > > > * Allow to disable CLFLUSH feature manually by > > > > > > - * hw.clflush_disable tunable. This may help Xen guest on some AMD > > > > > > - * CPUs. > > > > > > + * hw.clflush_disable tunable. > > > > > > */ > > > > > > if (hw_clflush_disable == 1) > > > > > > cpu_feature &= ~CPUID_CLFSH; > > > > > > > > > > It might be better to "or" old condition, i.e. Intel without SS, and > > > > > new one, vm_guest != 0, instead of replacing the old ? > > > > > > > > I thought the old condition only happened under VMware? > > > > > > Reports I got where from XEN. > > > > Ok. Those would also be covered under the vm_guest test as it is > > non-zero for Xen, VMware, Parallels, etc. > > What I said was suggestion and not objection. Ignore me. Were there any reports of problems with Intel CPUs that weren't under a virtualization system? If so, we should keep the test, but my understanding was that the test was only true under specific virtualization environments. -- John Baldwin