From owner-freebsd-stable@FreeBSD.ORG Wed Feb 10 13:54:04 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DACEF106571E; Wed, 10 Feb 2010 13:54:04 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 9B0FC8FC18; Wed, 10 Feb 2010 13:54:04 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 2C8EE46B53; Wed, 10 Feb 2010 08:54:04 -0500 (EST) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 102068A025; Wed, 10 Feb 2010 08:54:03 -0500 (EST) From: John Baldwin To: Tom McLaughlin Date: Wed, 10 Feb 2010 08:43:14 -0500 User-Agent: KMail/1.12.1 (FreeBSD/7.2-CBSD-20100120; KDE/4.3.1; amd64; ; ) References: <4B6B89E7.8030002@sdf.lonestar.org> <201002091723.44625.jhb@freebsd.org> <4B71FC5A.9080601@sdf.lonestar.org> In-Reply-To: <4B71FC5A.9080601@sdf.lonestar.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201002100843.14432.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Wed, 10 Feb 2010 08:54:03 -0500 (EST) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-1.4 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: kib@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Recent MFC to 7 causes crash on VMware ESXi X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2010 13:54:04 -0000 On Tuesday 09 February 2010 7:22:50 pm Tom McLaughlin wrote: > John Baldwin wrote, On 02/09/2010 05:23 PM: > > On Tuesday 09 February 2010 5:17:32 pm Tom McLaughlin wrote: > >> John Baldwin wrote, On 02/09/2010 01:52 PM: > >>> On Monday 08 February 2010 9:49:00 am John Baldwin wrote: > >>>> On Saturday 06 February 2010 4:47:16 pm Tom McLaughlin wrote: > >>>>> John Baldwin wrote, On 02/05/2010 08:27 AM: > >>>>>> On Thursday 04 February 2010 10:00:55 pm Tom McLaughlin wrote: > >>>>>>> Hi all, a recent MFC to 7-STABLE has started to cause issues for my > > VMs > >>>>>>> on VMware ESXi 3.5u4. After loading the mpt driver for the LSI disk > >>>>>>> controller the VM just shuts off. The workaround is to change the > > disk > >>>>>>> controller to the BusLogic type. Still, it used to work up until last > >>>>>>> week. The change was made around January 26th and based on the > > commits > >>>>>>> that day I'm guessing it's either r203047 or r203073 > >>>>>>> > >>>>>>> I have the same issue with both amd64 and i386 VMs. This affects HEAD > >>>>>>> and 8-STABLE as well and first affected HEAD over the summer. (I just > >>>>>>> worked around it and went about my business at the time. :-/) I've > >>>>>>> attached a dmesg from a kernel before the problem and one from after > > it > >>>>>>> started. > >>>>>> > >>>>>> What if you set 'hw.clfush_disable=1' from the loader? > >>>>>> > >>>>> > >>>>> Yes, that corrected it on all my VMs. I've talked to people on ESXi 4 > >>>>> and they do not see the problem. I have yet to try 3.5u5 to see if this > >>>>> is a non-issue. 3.5 will be supported for awhile longer from VMware. > >>>>> I'm going to try upgrading the box during the week. > >>>> > >>>> I believe folks had to do this on HEAD/8.x as well. Perhaps we can > >>>> automatically disable clflush if we are executing under VMware or Xen: > >>> > >>> Tom, were you able to verify that this patch fixes the problem for you > >>> without requiring you to set the hw.clflush_disable tunable? > >>> > >> > >> John, I'm getting the following build error on all branches: > >> > >> /usr/src/sys/amd64/amd64/initcpu.c: In function 'initializecpucache': > >> /usr/src/sys/amd64/amd64/initcpu.c:184: error: 'vm_guest' undeclared > >> (first use in this function) > >> /usr/src/sys/amd64/amd64/initcpu.c:184: error: (Each undeclared > >> identifier is reported only once > >> /usr/src/sys/amd64/amd64/initcpu.c:184: error: for each function it > >> appears in.) > > > > Oh foo. Can you add 'extern int vm_guest;' to that file near the top? > > Adding that to the top fixed the build on 8.x and HEAD. Both of those > are also now working just fine. However, 7.x still does not build. Ok, I will work on a more formal 7.x patch. > cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -g > -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes > -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef > -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys > -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS > -include opt_global.h -fno-common -finline-limit=8000 --param > inline-unit-growth=100 --param large-function-growth=1000 > -mcmodel=kernel -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx > -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding > -Werror vers.c > linking kernel.debug > initcpu.o(.text+0x272): In function `initializecpucache': > /usr/src/sys/amd64/amd64/initcpu.c:186: undefined reference to `vm_guest' > *** Error code 1 > > > > > >> *** Error code 1 > >> > >> > >> tom > >> > >>>> Index: amd64/amd64/initcpu.c > >>>> =================================================================== > >>>> --- amd64/amd64/initcpu.c (revision 203430) > >>>> +++ amd64/amd64/initcpu.c (working copy) > >>>> @@ -177,17 +177,16 @@ > >>>> if ((cpu_feature & CPUID_CLFSH) != 0) > >>>> cpu_clflush_line_size = ((cpu_procinfo >> 8) & 0xff) * 8; > >>>> /* > >>>> - * XXXKIB: (temporary) hack to work around traps generated when > >>>> - * CLFLUSHing APIC registers window. > >>>> + * XXXKIB: (temporary) hack to work around traps generated > >>>> + * when CLFLUSHing APIC registers window under virtualization > >>>> + * environments. > >>>> */ > >>>> TUNABLE_INT_FETCH("hw.clflush_disable", &hw_clflush_disable); > >>>> - if (cpu_vendor_id == CPU_VENDOR_INTEL && !(cpu_feature & CPUID_SS) && > >>>> - hw_clflush_disable == -1) > >>>> + if (vm_guest != 0 /* VM_GUEST_NO */ && hw_clflush_disable == -1) > >>>> cpu_feature &= ~CPUID_CLFSH; > >>>> /* > >>>> * Allow to disable CLFLUSH feature manually by > >>>> - * hw.clflush_disable tunable. This may help Xen guest on some AMD > >>>> - * CPUs. > >>>> + * hw.clflush_disable tunable. > >>>> */ > >>>> if (hw_clflush_disable == 1) > >>>> cpu_feature &= ~CPUID_CLFSH; > >>>> Index: i386/i386/initcpu.c > >>>> =================================================================== > >>>> --- i386/i386/initcpu.c (revision 203430) > >>>> +++ i386/i386/initcpu.c (working copy) > >>>> @@ -724,17 +724,16 @@ > >>>> if ((cpu_feature & CPUID_CLFSH) != 0) > >>>> cpu_clflush_line_size = ((cpu_procinfo >> 8) & 0xff) * 8; > >>>> /* > >>>> - * XXXKIB: (temporary) hack to work around traps generated when > >>>> - * CLFLUSHing APIC registers window. > >>>> + * XXXKIB: (temporary) hack to work around traps generated > >>>> + * when CLFLUSHing APIC registers window under virtualization > >>>> + * environments. > >>>> */ > >>>> TUNABLE_INT_FETCH("hw.clflush_disable", &hw_clflush_disable); > >>>> - if (cpu_vendor_id == CPU_VENDOR_INTEL && !(cpu_feature & CPUID_SS) && > >>>> - hw_clflush_disable == -1) > >>>> + if (vm_guest != 0 /* VM_GUEST_NO */ && hw_clflush_disable == -1) > >>>> cpu_feature &= ~CPUID_CLFSH; > >>>> /* > >>>> * Allow to disable CLFLUSH feature manually by > >>>> - * hw.clflush_disable tunable. This may help Xen guest on some AMD > >>>> - * CPUs. > >>>> + * hw.clflush_disable tunable. > >>>> */ > >>>> if (hw_clflush_disable == 1) > >>>> cpu_feature &= ~CPUID_CLFSH; > >>>> > >>>> -- > >>>> John Baldwin > >>>> _______________________________________________ > >>>> freebsd-stable@freebsd.org mailing list > >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable > >>>> To unsubscribe, send any mail to "freebsd-stable- unsubscribe@freebsd.org" > >>>> > >>> > >> > >> > >> -- > >> | tmclaugh at sdf.lonestar.org tmclaugh at FreeBSD.org | > >> | FreeBSD http://www.FreeBSD.org | > >> > >> > > > > > -- > | tmclaugh at sdf.lonestar.org tmclaugh at FreeBSD.org | > | FreeBSD http://www.FreeBSD.org | > > -- John Baldwin