From owner-freebsd-current@FreeBSD.ORG Mon Sep 7 21:06:19 2009 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CAF61065672 for ; Mon, 7 Sep 2009 21:06:19 +0000 (UTC) (envelope-from nox@jelal.kn-bremen.de) Received: from smtp.kn-bremen.de (gelbbaer.kn-bremen.de [78.46.108.116]) by mx1.freebsd.org (Postfix) with ESMTP id DDC508FC1E for ; Mon, 7 Sep 2009 21:06:18 +0000 (UTC) Received: by smtp.kn-bremen.de (Postfix, from userid 10) id 227B21E0037D; Mon, 7 Sep 2009 23:06:17 +0200 (CEST) Received: from triton8.kn-bremen.de (noident@localhost [127.0.0.1]) by triton8.kn-bremen.de (8.14.3/8.14.3) with ESMTP id n87Kxu1x093121; Mon, 7 Sep 2009 22:59:56 +0200 (CEST) (envelope-from nox@triton8.kn-bremen.de) Received: (from nox@localhost) by triton8.kn-bremen.de (8.14.3/8.14.3/Submit) id n87Kxtic093120; Mon, 7 Sep 2009 22:59:55 +0200 (CEST) (envelope-from nox) From: Juergen Lock Date: Mon, 7 Sep 2009 22:59:55 +0200 To: Juergen Lock Message-ID: <20090907205955.GA91866@triton8.kn-bremen.de> References: <4A93AFF9.1060201@web.de> <52d4a3890908250316l4de68725xa9d780e7d5b37205@mail.gmail.com> <52d4a3890908250321u746e5757u136030bcbc19208d@mail.gmail.com> <4A93BF0C.8040601@web.de> <20090826221001.GA1070@triton8.kn-bremen.de> <4A96C8D9.6070804@web.de> <20090829211848.GA59305@triton8.kn-bremen.de> <4A9B800F.1040209@web.de> <20090831212723.GA32448@triton8.kn-bremen.de> <20090901201248.GA60123@triton8.kn-bremen.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090901201248.GA60123@triton8.kn-bremen.de> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: Mohammed Gamal , freebsd-current@FreeBSD.org, Jan Kiszka , qemu-devel@nongnu.org, Avi Kivity Subject: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2009 21:06:19 -0000 [I'm copying freebsd-current@FreeBSD.org because ppl there might know more about this...] qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) guest with the same HZ as the host (like, 1000) with (mostly) proper timing once, but no longer. :( It seems there are two problems involved: a) use of apic seems to cause the clock irq rate to be doubled to 2 * HZ (can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host only gets proper timing after setting hint.apic.0.disabled=1 via the loader. (as can be verified by `vmstat -i' and `time sleep 2' in an installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs or dvd1 iso.) b) qemu running on FreeBSD 8 hosts (and most likely head) has the additional problem of running its timers only at HZ/2 when using setitimer(2) (called `-clock unix' in qemu), as seen below. (as also seen below, timer_settime(2) aka `-clock dynticks' in qemu behaves even worse, but that is similarly true on FreeBSD 7 which is why I removed the patch that enabled that from our qemu port(s) a few days ago.) And the only reason FreeBSD 8 guests are usually less affected by these problems is they now reduce their HZ to 100 when they detect being run in a VM. (which makes sense for other reasons as well, don't get me wrong... but of course doesn't help when the host is running with HZ=100 too.) On Tue, Sep 01, 2009 at 10:12:48PM +0200, Juergen Lock wrote: > On Mon, Aug 31, 2009 at 11:27:23PM +0200, Juergen Lock wrote: > > On Mon, Aug 31, 2009 at 09:47:27AM +0200, Jan Kiszka wrote: > > > Juergen Lock wrote: > > > > On Thu, Aug 27, 2009 at 07:56:41PM +0200, Jan Kiszka wrote: > > > >> Juergen Lock wrote: > > > >>> On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote: > > > >>>> Mohammed Gamal wrote: > > > >>>>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal wrote: > > > >>>>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka wrote: > > > >>>>>>> Mohammed Gamal wrote: > > > >>>>>>>> qemu-system-x86_64 -hda /dev/null -cdrom > > > >>>>>>>> > > > >>>>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that > > > >>>>>>> image I'm unable to reproduce. Will download and check standard ubuntu > > > >>>>>>> later today. > > > >>>>>>> > > > >>>>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be > > > >>>>>>>> equivalent to using plain qemu, no? > > > >>>>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So > > > >>>>>>> it's better to check such issues also against "clean" qemu, specifically > > > >>>>>>> as we are on qemu-devel here. > > > >>>>>>> > > > >>>>>>> Jan > > > >>>>>>> > > > >>>>>>> > > > >>>>>> Just tested this now on a vanilla qemu, I am still able to reproduce > > > >>>>>> the same issue. > > > >>>>>> > > > >>>>> This bug might be related to the same problem > > > >>>>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000 > > > >>>> I think at least those issues should be solved with recent qemu and > > > >>>> bioses. Note that this report refers to a fairly old qemu version > > > >>>> (0.10.0-derived). > > > >>> Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8 > > > >>> hosts both with 0.10.6 and 0.11.0rc1 before: > > > >>> http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html > > > >>> As mentioned in that post I was able to work around the issue by passig > > > >>> the linux guest kernels `no_timer_check' after which they seemed to boot up > > > >>> just fine, so I suspect in that case its not actually an apic routing > > > >>> problem but just guest timer irqs arriving late/irregularly which cause > > > >>> the guest kernel timer checks to time out and fail. > > > >> Does this happen with git head and its corresponding bios, too? I cannot > > > >> imagine that the FreeBSD platform is so irregularly generating timer > > > >> events for qemu that Linux gets unhappy during this test loop (I think > > > >> to remember it needs 3 out of 10 ticks or so to be satisfied). > > > > > > > > Alright so I testwise updated the FreeBSD qemu-devel port to git head > > > > (db3c9e08e0d6eaf83f9d7a2c87dc45af3ac8f4dd, update at > > > > http://people.freebsd.org/~nox/qemu/qemu-devel-20090829.patch > > > > > > You will have to help me isolating the reason as I don't have any BSD > > > host. And running recent qemu/-kvm on Linux hosts does not trigger > > > problems around booting ubuntu here. > > > > Well I don't quite know where to start looking, of course I'd be > > happy to test patches... :) > > Ok I now found the #if 0 in vl.c:host_alarm_handler(), enabled it and got > this with a FreeBSD 7.2 iso (which defaults to HZ=1000) and -clock dynticks: > ..and this was on a FreeBSD 8 host: > timer: min=2555 us max=32004 us avg=5150 us avg_freq=194.149 Hz > timer: min=2451 us max=7205 us avg=3012 us avg_freq=331.960 Hz > timer: min=2855 us max=7994 us avg=3019 us avg_freq=331.191 Hz > timer: min=2099 us max=20003 us avg=3057 us avg_freq=327.073 Hz > timer: min=2143 us max=11897 us avg=3081 us avg_freq=324.527 Hz > timer: min=2110 us max=80014 us avg=3165 us avg_freq=315.913 Hz > timer: min=2088 us max=51999 us avg=3206 us avg_freq=311.873 Hz > > and this with the same guest and -clock unix: > > timer: min=1614 us max=7121 us avg=2009 us avg_freq=497.692 Hz > timer: min=876 us max=11134 us avg=2016 us avg_freq=495.967 Hz > timer: min=753 us max=9245 us avg=2014 us avg_freq=496.455 Hz > timer: min=1798 us max=6211 us avg=2008 us avg_freq=497.941 Hz > timer: min=1772 us max=6142 us avg=2008 us avg_freq=497.943 Hz > timer: min=1767 us max=6157 us avg=2008 us avg_freq=497.942 Hz > timer: min=14 us max=6139 us avg=2008 us avg_freq=497.939 Hz > timer: min=736 us max=9969 us avg=2016 us avg_freq=495.962 Hz > timer: min=176 us max=11427 us avg=2076 us avg_freq=481.631 Hz > timer: min=229 us max=9827 us avg=2030 us avg_freq=492.546 Hz > timer: min=430 us max=11082 us avg=2078 us avg_freq=481.166 Hz > timer: min=226 us max=9871 us avg=2026 us avg_freq=493.517 Hz > timer: min=605 us max=10829 us avg=2082 us avg_freq=480.243 Hz > timer: min=57 us max=10976 us avg=2028 us avg_freq=493.031 Hz > timer: min=398 us max=11036 us avg=2080 us avg_freq=480.704 Hz > > `vmstat -i' in the guest (available e.g. from the fixit->cdrom/dvd shell > on a livefs or dvd1 iso) gives similar values as avg_freq above for the > `cpu0: timer' irq rate. (Btw, FreeBSD 8 reduces its HZ to 100 when > running in a vm so the `vmstat -i' values will be a little lower if you > try that as a guest.) > > And `vmstat -i' on the host (also running with HZ=1000) reports a > rate of 1999 for each cpu's timer irq, so there are clearly quite a few > irqs missing there... > > (Btw I'm wondering if the poor usb performance I reported a long time > ago already could be related to this also...) And here is the same guest with -clock unix on a FreeBSD 7 host: (same with and without apic, except of course that only without apic timing in the guest is ok.) timer: min=847 us max=6146 us avg=1005 us avg_freq=994.950 Hz timer: min=866 us max=6130 us avg=1005 us avg_freq=994.960 Hz timer: min=868 us max=6133 us avg=1005 us avg_freq=994.958 Hz timer: min=287 us max=35199 us avg=1089 us avg_freq=918.216 Hz timer: min=201 us max=13551 us avg=1029 us avg_freq=971.757 Hz timer: min=792 us max=6211 us avg=1005 us avg_freq=994.961 Hz timer: min=145 us max=6185 us avg=1007 us avg_freq=992.984 Hz timer: min=852 us max=6152 us avg=1005 us avg_freq=994.961 Hz timer: min=820 us max=6175 us avg=1005 us avg_freq=994.960 Hz timer: min=654 us max=6242 us avg=1005 us avg_freq=994.966 Hz timer: min=675 us max=6196 us avg=1005 us avg_freq=994.952 Hz timer: min=829 us max=6162 us avg=1005 us avg_freq=994.960 Hz timer: min=503 us max=6136 us avg=1005 us avg_freq=994.960 Hz timer: min=859 us max=6138 us avg=1005 us avg_freq=994.960 Hz timer: min=856 us max=6149 us avg=1005 us avg_freq=994.960 Hz timer: min=832 us max=6164 us avg=1005 us avg_freq=994.961 Hz timer: min=829 us max=6174 us avg=1005 us avg_freq=994.958 Hz timer: min=826 us max=6173 us avg=1005 us avg_freq=994.960 Hz timer: min=828 us max=6168 us avg=1005 us avg_freq=994.961 Hz timer: min=841 us max=6160 us avg=1005 us avg_freq=994.960 Hz timer: min=857 us max=6135 us avg=1005 us avg_freq=994.959 Hz timer: min=857 us max=6139 us avg=1005 us avg_freq=994.959 Hz timer: min=845 us max=6147 us avg=1005 us avg_freq=994.960 Hz timer: min=854 us max=6143 us avg=1005 us avg_freq=994.960 Hz timer: min=854 us max=6142 us avg=1005 us avg_freq=994.960 Hz timer: min=857 us max=6142 us avg=1005 us avg_freq=994.960 Hz timer: min=859 us max=6141 us avg=1005 us avg_freq=994.961 Hz timer: min=849 us max=6141 us avg=1005 us avg_freq=994.962 Hz timer: min=857 us max=6143 us avg=1005 us avg_freq=994.962 Hz timer: min=864 us max=6138 us avg=1005 us avg_freq=994.958 Hz timer: min=863 us max=6139 us avg=1005 us avg_freq=994.961 Hz timer: min=869 us max=6136 us avg=1005 us avg_freq=994.960 Hz timer: min=857 us max=6143 us avg=1005 us avg_freq=994.959 Hz timer: min=865 us max=6133 us avg=1005 us avg_freq=994.960 Hz And the same guest/host with -clock dynticks: timer: min=1657 us max=32164 us avg=6732 us avg_freq=148.534 Hz timer: min=1675 us max=7158 us avg=2010 us avg_freq=497.480 Hz timer: min=1773 us max=7200 us avg=2010 us avg_freq=497.480 Hz timer: min=1798 us max=7148 us avg=2010 us avg_freq=497.480 Hz timer: min=1851 us max=7144 us avg=2013 us avg_freq=496.739 Hz timer: min=1821 us max=7136 us avg=2020 us avg_freq=495.017 Hz timer: min=1823 us max=7173 us avg=2010 us avg_freq=497.480 Hz timer: min=1825 us max=39002 us avg=2076 us avg_freq=481.664 Hz timer: min=1851 us max=7145 us avg=2012 us avg_freq=496.985 Hz timer: min=1855 us max=16998 us avg=2106 us avg_freq=474.803 Hz timer: min=1855 us max=7138 us avg=2010 us avg_freq=497.480 Hz timer: min=1858 us max=7136 us avg=2010 us avg_freq=497.480 Hz timer: min=1856 us max=7138 us avg=2010 us avg_freq=497.480 Hz timer: min=1855 us max=7138 us avg=2017 us avg_freq=495.754 Hz timer: min=1818 us max=7143 us avg=2009 us avg_freq=497.728 Hz timer: min=1851 us max=7141 us avg=2010 us avg_freq=497.480 Hz timer: min=1859 us max=7140 us avg=2010 us avg_freq=497.480 Hz timer: min=1851 us max=7138 us avg=2010 us avg_freq=497.480 Hz For people wanting to experiment, you can escape to the loader prompt from the FreeBSD boot menu and then type: set hint.apic.0.disabled=1 and optionally: set kern.hz="1000" and then: boot -v (for verbose dmesg, otherwise `boot' without the -v.) Any enlightnig comments appreciated... :) Juergen