From owner-freebsd-xen@freebsd.org Mon Nov 2 15:20:03 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6D94A03126 for ; Mon, 2 Nov 2015 15:20:03 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8955A1259 for ; Mon, 2 Nov 2015 15:20:02 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213] (may be forged)) (authenticated bits=0) by smtp.krpservers.com (8.14.9/8.14.9) with ESMTP id tA2F9Pnw028293 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 2 Nov 2015 15:09:26 GMT (envelope-from kpielorz_lst@tdx.co.uk) Date: Mon, 02 Nov 2015 15:09:25 +0000 From: Karl Pielorz To: freebsd-xen@freebsd.org Subject: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: <151F73F1EF071C3C48F17866@[10.12.30.106]> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 15:20:04 -0000 Hi, We run a number of 10.x boxes under XenServer 6.5. On those boxes we run NTP as a client with a few local time servers on the network to keep the time in sync on that VM. There's a number of big threads on the Citrix forums about whether you should, or shouldn't run NTP in domU's - but the consensus seems to be you need to in order to keep the domU's clock accurately in sync. This works fine. However if we run a live migrate (to another XenServer node) or Xen Storage Motion migration - the domU's ntp gets messed up. The time on the domU is generally "ok" - but ntpq gives output like: " ntpq> pe remote refid st t when poll reach delay offset jitter =========================================================================== ntp0 193.67.79.202 2 u 931 1024 377 0.232 -4977.5 4207.29 ntp1 66.228.38.73 3 u 890 1024 377 0.267 -4988.5 4616.94 ntp2 64.246.132.14 2 u 990 1024 377 0.324 -4978.0 4207.98 " It *never* recovers from this (you have to log into the box, and restart ntpd). I realise this may be more a 'ntpd' than Xen question - but does anyone have any suggestions for avoiding this happening? - i.e. Some option for ntp.conf or something? Thanks, -Karl From owner-freebsd-xen@freebsd.org Mon Nov 2 17:35:20 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80B1DA242A6 for ; Mon, 2 Nov 2015 17:35:20 +0000 (UTC) (envelope-from prvs=741c8dfce=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 336F813C3 for ; Mon, 2 Nov 2015 17:35:19 +0000 (UTC) (envelope-from prvs=741c8dfce=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,234,1444694400"; d="scan'208";a="315100099" Subject: Re: Does a Xen Dom0 require X to function To: , References: <5633A2B0.22403.45280A@lausts.acm.org> From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Message-ID: <56379E8A.3030701@citrix.com> Date: Mon, 2 Nov 2015 18:34:02 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <5633A2B0.22403.45280A@lausts.acm.org> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 17:35:20 -0000 El 30/10/15 a les 18.02, Thomas Laus ha escrit: > List: > > I finally found a PC that had all of the necessary CPU instructions. It is > my company issued i7 laptop. I loaded FreeBSD Current on a spare hard drive > and Xen is able to start and the 'xl list' shows that my DomU is loaded and > running. I just get a black screen when attempting a vncviewer connection to > the Dom0 address from another computer. So the VNC client attaches successfully to the server, but the output is just a black screen? IMHO, that looks more like a guest or VNC issue rather than a Xen/Qemu one. Have you tried different guest OSes and different VNC clients? (TigerVNC Viewer has always worked fine for me). But regarding your question: no, Xen Dom0 doesn't require X at all in order to run. Roger. From owner-freebsd-xen@freebsd.org Mon Nov 2 17:40:54 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9AB7AA24328 for ; Mon, 2 Nov 2015 17:40:54 +0000 (UTC) (envelope-from prvs=741c8dfce=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D7CF160F for ; Mon, 2 Nov 2015 17:40:53 +0000 (UTC) (envelope-from prvs=741c8dfce=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,234,1444694400"; d="scan'208";a="315101778" Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? To: Karl Pielorz , References: <151F73F1EF071C3C48F17866@[10.12.30.106]> From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 Message-ID: <5637A01B.1010307@citrix.com> Date: Mon, 2 Nov 2015 18:40:43 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <151F73F1EF071C3C48F17866@[10.12.30.106]> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 17:40:54 -0000 El 02/11/15 a les 16.09, Karl Pielorz ha escrit: > > Hi, > > We run a number of 10.x boxes under XenServer 6.5. On those boxes we run > NTP as a client with a few local time servers on the network to keep > the time in sync on that VM. > > There's a number of big threads on the Citrix forums about whether you > should, or shouldn't run NTP in domU's - but the consensus seems to be > you need to in order to keep the domU's clock accurately in sync. > > This works fine. > > However if we run a live migrate (to another XenServer node) or Xen > Storage Motion migration - the domU's ntp gets messed up. The time on > the domU is generally "ok" - but ntpq gives output like: > > " > ntpq> pe > remote refid st t when poll reach delay offset jitter > =========================================================================== > ntp0 193.67.79.202 2 u 931 1024 377 0.232 -4977.5 4207.29 > ntp1 66.228.38.73 3 u 890 1024 377 0.267 -4988.5 4616.94 > ntp2 64.246.132.14 2 u 990 1024 377 0.324 -4978.0 4207.98 > " I'm sorry, but I have no idea about ntpd, what does the above mean? The offset is too big with other peers and ntpd simply disconnects? There were some issues that I fixed some time ago regarding migration and PVHVM guests: http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f8e8fd56bd7d5675e8331b4ec74bae76c9dbf24e http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=32c864a35ece2c24a336d183869a546798a4b241 You should make sure XenServer has both of this commits. Roger. From owner-freebsd-xen@freebsd.org Mon Nov 2 18:56:42 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCA78A24276 for ; Mon, 2 Nov 2015 18:56:42 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 40F141B1B for ; Mon, 2 Nov 2015 18:56:41 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213] (may be forged)) (authenticated bits=0) by smtp.krpservers.com (8.14.9/8.14.9) with ESMTP id tA2Iud3x043253 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 2 Nov 2015 18:56:40 GMT (envelope-from kpielorz_lst@tdx.co.uk) Date: Mon, 02 Nov 2015 18:56:30 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: <84EFB0608568EE976A48039C@Mac-mini.local> In-Reply-To: <5637A01B.1010307@citrix.com> References: <151F73F1EF071C3C48F17866@[10.12.30.106]> <5637A01B.1010307@citrix.com> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 18:56:42 -0000 --On 2 November 2015 at 18:40:43 +0100 Roger Pau Monn=C3=A9=20 wrote: >> remote refid st t when poll reach delay offset = jitter >> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> =3D=3D=3D ntp0 193.67.79.202 2 u 931 1024 377 0.232 = -4977.5 >> 4207.29 ntp1 66.228.38.73 3 u 890 1024 377 0.267 -4988.5 >> 4616.94 ntp2 64.246.132.14 2 u 990 1024 377 0.324 -4978.0 >> 4207.98 " > > I'm sorry, but I have no idea about ntpd, what does the above mean? The > offset is too big with other peers and ntpd simply disconnects? Sorry for being so vague (I half hoped someone must have run into this=20 before and posted a "Yes, we noticed the same - this fixes it" reply :) The above seems to show that time moved forward on the VM by a few seconds. = This doesn't always happen - I just moved another VM and that shows 'offset = =3D 8000'. Yet another VM moved at the same time is fine. Comparing the output of 'date' for that offset 8000 VM with a known 'good'=20 clock - it shows the VM is indeed 8 seconds in the future after the move. ntp is fussy - and kind of understandably decides that's too much time=20 difference to slowly ebb away - so just gives up and leaves the VM clock=20 'as is'. The XenServer the VM was residing on (and the new one it's resident on)=20 both on their server consoles show the correct time (i.e. neither are 8=20 seconds 'ahead') - so it looks like the move somehow gained us 8 seconds=20 (or 5 seconds in the original example). > There were some issues that I fixed some time ago regarding migration > and PVHVM guests: > > = http://xenbits.xen.org/gitweb/?p=3Dxen.git;a=3Dcommit;h=3Df8e8fd56bd7d5675e8= 331 > b4ec74bae76c9dbf24e > = http://xenbits.xen.org/gitweb/?p=3Dxen.git;a=3Dcommit;h=3D32c864a35ece2c24a3= 36d > 183869a546798a4b241 > > You should make sure XenServer has both of this commits. I'll have a look at those - we might be a bit stuck for implementing them=20 if they're XenServer changes, as we run the XenServer ISO's (6.5 SP1=20 w/hotfixes). -Karl From owner-freebsd-xen@freebsd.org Mon Nov 2 19:43:42 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FD9EA24C23 for ; Mon, 2 Nov 2015 19:43:42 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C439C1115 for ; Mon, 2 Nov 2015 19:43:41 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213] (may be forged)) (authenticated bits=0) by smtp.krpservers.com (8.14.9/8.14.9) with ESMTP id tA2JhgNt046356 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 2 Nov 2015 19:43:43 GMT (envelope-from kpielorz_lst@tdx.co.uk) Date: Mon, 02 Nov 2015 19:43:35 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: <54CB1E09ECFC7250EB674E5D@Mac-mini.local> In-Reply-To: <5637A01B.1010307@citrix.com> References: <151F73F1EF071C3C48F17866@[10.12.30.106]> <5637A01B.1010307@citrix.com> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 19:43:42 -0000 --On 2 November 2015 at 18:40:43 +0100 Roger Pau Monn=C3=A9=20 wrote: > I'm sorry, but I have no idea about ntpd, what does the above mean? The > offset is too big with other peers and ntpd simply disconnects? > > There were some issues that I fixed some time ago regarding migration > and PVHVM guests: I'll work out (more scientifically) what's actually happening tomorrow -=20 rather than now, while I'm trying to get the servers all patched / updated=20 out of hours. It's pretty reproducible - so I'll run some proper tests that = show what's happening (to the VM's time, and to ntpd) and post back as it=20 doesn't there's a known fix / workaround etc. -Karl From owner-freebsd-xen@freebsd.org Tue Nov 3 10:04:02 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C0AFAA2349D for ; Tue, 3 Nov 2015 10:04:02 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5E5551B38 for ; Tue, 3 Nov 2015 10:04:01 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213] (may be forged)) (authenticated bits=0) by smtp.krpservers.com (8.14.9/8.14.9) with ESMTP id tA3A3oFA003675 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 Nov 2015 10:03:52 GMT (envelope-from kpielorz_lst@tdx.co.uk) Date: Tue, 03 Nov 2015 10:03:50 +0000 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: <8448B4CE863936ADEF762C0F@[10.12.30.106]> In-Reply-To: <5637A01B.1010307@citrix.com> References: <151F73F1EF071C3C48F17866@[10.12.30.106]> <5637A01B.1010307@citrix.com> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 10:04:02 -0000 Hi, Ok, I've done some proper research into this now... When Migrating a domU FreeBSD (10.1-R p6) between hosts, it "more often than not" gains time. This breaks NTP on the domU. I setup a script on the domU being migrated that read out it's clock - and called out to an external machine to read it's clock - i.e. " Me: Tue Nov 3 09:55:51 GMT 2015 Them: Tue Nov 3 09:55:51 GMT 2015 Me: Tue Nov 3 09:55:52 GMT 2015 Them: Tue Nov 3 09:55:52 GMT 2015 " As you can see 'Me' and 'Them' is in sync. At this point I live migrated the FreeBSD box from one XenServer (6.5-SP1 w/hotfixes) to another in the pool (same hardware spec / XenServer etc.) The domU's clock got advanced by a number of seconds: " Me: Tue Nov 3 09:55:52 GMT 2015 Them: Tue Nov 3 09:55:52 GMT 2015 [Migrate 'Me' Happens Now] Me: Tue Nov 3 09:56:02 GMT 2015 Them: Tue Nov 3 09:55:58 GMT 2015 Me: Tue Nov 3 09:56:06 GMT 2015 Them: Tue Nov 3 09:55:59 GMT 2015 Me: Tue Nov 3 09:56:07 GMT 2015 Them: Tue Nov 3 09:56:00 GMT 2015 Me: Tue Nov 3 09:56:08 GMT 2015 Them: Tue Nov 3 09:56:01 GMT 2015 " So you can see during the migrate the 'Me' (domU being moved) clock picks up 3 seconds, then another 4 seconds to make it 'ahead' 7 seconds in total - at which point it stabilises). It stays this way then - which is enough to cause NTP to "give up" trying to bring that back into alignment. > http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f8e8fd56bd7d5675e8331 > b4ec74bae76c9dbf24e > http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=32c864a35ece2c24a336d > 183869a546798a4b241 Are either of those fixes likely to cure this? Thanks, -Karl From owner-freebsd-xen@freebsd.org Tue Nov 3 11:21:30 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC029A24E3A for ; Tue, 3 Nov 2015 11:21:30 +0000 (UTC) (envelope-from prvs=742bbf7d9=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A14B1569 for ; Tue, 3 Nov 2015 11:21:29 +0000 (UTC) (envelope-from prvs=742bbf7d9=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,238,1444694400"; d="scan'208";a="315267955" Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? To: Karl Pielorz , References: <151F73F1EF071C3C48F17866@[10.12.30.106]> <5637A01B.1010307@citrix.com> <8448B4CE863936ADEF762C0F@[10.12.30.106]> From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 Message-ID: <563898B0.5040100@citrix.com> Date: Tue, 3 Nov 2015 12:21:20 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <8448B4CE863936ADEF762C0F@[10.12.30.106]> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 11:21:30 -0000 El 03/11/15 a les 11.03, Karl Pielorz ha escrit: > > Hi, > > Ok, I've done some proper research into this now... > > When Migrating a domU FreeBSD (10.1-R p6) between hosts, it "more often > than not" gains time. > > This breaks NTP on the domU. > > I setup a script on the domU being migrated that read out it's clock - > and called out to an external machine to read it's clock - i.e. > > " > Me: Tue Nov 3 09:55:51 GMT 2015 > Them: Tue Nov 3 09:55:51 GMT 2015 > > Me: Tue Nov 3 09:55:52 GMT 2015 > Them: Tue Nov 3 09:55:52 GMT 2015 > " > > As you can see 'Me' and 'Them' is in sync. > > At this point I live migrated the FreeBSD box from one XenServer > (6.5-SP1 w/hotfixes) to another in the pool (same hardware spec / > XenServer etc.) > > The domU's clock got advanced by a number of seconds: > > " > Me: Tue Nov 3 09:55:52 GMT 2015 > Them: Tue Nov 3 09:55:52 GMT 2015 > > [Migrate 'Me' Happens Now] > > Me: Tue Nov 3 09:56:02 GMT 2015 > Them: Tue Nov 3 09:55:58 GMT 2015 > > Me: Tue Nov 3 09:56:06 GMT 2015 > Them: Tue Nov 3 09:55:59 GMT 2015 > > Me: Tue Nov 3 09:56:07 GMT 2015 > Them: Tue Nov 3 09:56:00 GMT 2015 > > Me: Tue Nov 3 09:56:08 GMT 2015 > Them: Tue Nov 3 09:56:01 GMT 2015 > " > > So you can see during the migrate the 'Me' (domU being moved) clock > picks up 3 seconds, then another 4 seconds to make it 'ahead' 7 seconds > in total - at which point it stabilises). Can't you tell ntpd to just resynchronize no matter what skew it detects? > It stays this way then - which is enough to cause NTP to "give up" > trying to bring that back into alignment. > >> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f8e8fd56bd7d5675e8331 >> b4ec74bae76c9dbf24e >> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=32c864a35ece2c24a336d >> 183869a546798a4b241 > > Are either of those fixes likely to cure this? Hm, I don't think so. Both commits are present from 4.3 onwards, and IIRC XenServer 6.5 is based on Xen 4.4, which should contain both fixes. Is XenServer also synchronized using ntpd? When a guest resumes from migration is updates it's current time based on the time provided by the hypervisor, so if the hosts themselves are not synchronized you could see skews like that. Roger. From owner-freebsd-xen@freebsd.org Tue Nov 3 19:15:56 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8701CA252F7 for ; Tue, 3 Nov 2015 19:15:56 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0BD621DCE for ; Tue, 3 Nov 2015 19:15:55 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213] (may be forged)) (authenticated bits=0) by smtp.krpservers.com (8.14.9/8.14.9) with ESMTP id tA3JFfb8039935 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 Nov 2015 19:15:42 GMT (envelope-from kpielorz_lst@tdx.co.uk) Date: Tue, 03 Nov 2015 19:15:41 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , freebsd-xen@freebsd.org Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: In-Reply-To: <563898B0.5040100@citrix.com> References: <151F73F1EF071C3C48F17866@[10.12.30.106]> <5637A01B.1010307@citrix.com> <8448B4CE863936ADEF762C0F@[10.12.30.106]> <563898B0.5040100@citrix.com> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 19:15:56 -0000 --On 3 November 2015 at 12:21:20 +0100 Roger Pau Monn=C3=A9=20 wrote: > Can't you tell ntpd to just resynchronize no matter what skew it detects? Probably - I'll have to do some digging - I had hoped someone had hit this=20 already, though I'm also a little concerned about throwing a 7 second 'back = in time' jump on the system. >> Are either of those fixes likely to cure this? > > Hm, I don't think so. Both commits are present from 4.3 onwards, and > IIRC XenServer 6.5 is based on Xen 4.4, which should contain both fixes. > > Is XenServer also synchronized using ntpd? When a guest resumes from > migration is updates it's current time based on the time provided by the > hypervisor, so if the hosts themselves are not synchronized you could > see skews like that. All the clocks on the XenServers are NTP sync'd - and are in sync=20 (confirmed from the consoles). When the domU moves - it's clock moves >ahead< in time of both the server=20 it's leaving, and the server it's moved to. I re-ran the test adding checks for the source (Xen1) and destination=20 (Xen2) XenServer clocks - and I get: Me: Tue Nov 3 18:56:29 GMT 2015 Them: Tue Nov 3 18:56:29 GMT 2015 Xen1: Tue Nov 3 18:56:29 GMT 2015 Xen2: Tue Nov 3 18:56:29 GMT 2015 (All in sync) [moving FreeBSD from Xen1 to Xen2] Me: Tue Nov 3 18:56:39 GMT 2015 (Gained 4 seconds) Them: Tue Nov 3 18:56:35 GMT 2015 Xen1: Tue Nov 3 18:56:35 GMT 2015 Xen2: Tue Nov 3 18:56:35 GMT 2015 Me: Tue Nov 3 18:56:43 GMT 2015 (Completed - gained 7 seconds total) Them: Tue Nov 3 18:56:36 GMT 2015 Xen1: Tue Nov 3 18:56:36 GMT 2015 Xen2: Tue Nov 3 18:56:36 GMT 2015 Me: Tue Nov 3 18:56:44 GMT 2015 Them: Tue Nov 3 18:56:37 GMT 2015 Xen1: Tue Nov 3 18:56:37 GMT 2015 Xen2: Tue Nov 3 18:56:37 GMT 2015 So you can see it's now 7 seconds ahead of both 'Them' (external bare metal = machine running ntp) and Xen1/Xen2. I re-ran the test without ntp loaded on the domU - and the same thing=20 happens - so it's not ntp causing it. We've kind of 'lived with it' until now - but as the quantity of VM's=20 increases - it becomes more of a pain :( -Karl From owner-freebsd-xen@freebsd.org Tue Nov 3 20:12:54 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7567CA25DD5 for ; Tue, 3 Nov 2015 20:12:54 +0000 (UTC) (envelope-from lab@gta.com) Received: from mailgate.gta.com (gb-ha.orl.gta.com [209.208.105.194]) by mx1.freebsd.org (Postfix) with ESMTP id 21D2B14E6 for ; Tue, 3 Nov 2015 20:12:53 +0000 (UTC) (envelope-from lab@gta.com) Received: (qmail 94262 invoked by uid 1000); 3 Nov 2015 20:12:50 -0000 Date: Tue, 3 Nov 2015 15:12:50 -0500 From: Larry Baird To: freebsd-xen@freebsd.org Subject: Checksum forwarding issue on XEN Message-ID: <20151103201250.GA92469@gta.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 20:12:54 -0000 Has anybody made any progress on "Bug 188261 - [xen] FreeBSD DomU PVHVM guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host." (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188261)? The code for checksum calculation in the function xnb_add_mbuf_cksum() looks suspect. switch (iph->ip_p) { case IPPROTO_TCP: if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { size_t tcplen = ntohs(iph->ip_len) - sizeof(struct ip); struct tcphdr *th = (struct tcphdr*)(iph + 1); th->th_sum = in_pseudo(iph->ip_src.s_addr, iph->ip_dst.s_addr, htons(IPPROTO_TCP + tcplen)); th->th_sum = in_cksum_skip(mbufc, sizeof(struct ether_header) + ntohs(iph->ip_len), sizeof(struct ether_header) + (iph->ip_hl << 2)); } break; case IPPROTO_UDP: if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { size_t udplen = ntohs(iph->ip_len) - sizeof(struct ip); struct udphdr *uh = (struct udphdr*)(iph + 1); uh->uh_sum = in_pseudo(iph->ip_src.s_addr, iph->ip_dst.s_addr, htons(IPPROTO_UDP + udplen)); uh->uh_sum = in_cksum_skip(mbufc, sizeof(struct ether_header) + ntohs(iph->ip_len), sizeof(struct ether_header) + (iph->ip_hl << 2)); } break; default: break; } Both in_pseudo() and in_cksum_skip() set the same checksum. Does this make since to anybody? Larry -- ------------------------------------------------------------------------ Larry Baird Global Technology Associates, Inc. 1992-2012 | http://www.gta.com Celebrating Twenty Years of Software Innovation | Orlando, FL Email: lab@gta.com | TEL 407-380-0220 From owner-freebsd-xen@freebsd.org Thu Nov 5 14:10:47 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57725A26416 for ; Thu, 5 Nov 2015 14:10:47 +0000 (UTC) (envelope-from lausts@acm.org) Received: from cdptpa-oedge-vip.email.rr.com (cdptpa-outbound-snat.email.rr.com [107.14.166.232]) by mx1.freebsd.org (Postfix) with ESMTP id 1270A1E14 for ; Thu, 5 Nov 2015 14:10:46 +0000 (UTC) (envelope-from lausts@acm.org) Received: from [173.88.10.122] ([173.88.10.122:28153] helo=mail.laus.org) by cdptpa-oedge02 (envelope-from ) (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP id 72/9C-15038-2236B365; Thu, 05 Nov 2015 14:09:39 +0000 Received: from [192.168.1.100] (laust2 [192.168.1.100]) by mail.laus.org (8.15.2/8.15.2) with ESMTPS id tA5E9c2Y003980 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 5 Nov 2015 09:09:38 -0500 (EST) (envelope-from lausts@acm.org) X-Authentication-Warning: mail.laus.org: Host laust2 [192.168.1.100] claimed to be [192.168.1.100] From: "Thomas Laus" Organization: ABB To: freebsd-xen@freebsd.org Date: Thu, 05 Nov 2015 09:09:33 -0500 Subject: Re: Does a Xen Dom0 require X to function Reply-to: lausts@acm.org Message-ID: <563B631D.22122.FDB9F@lausts.acm.org> Priority: normal In-reply-to: <56379E8A.3030701@citrix.com> References: <5633A2B0.22403.45280A@lausts.acm.org>, <56379E8A.3030701@citrix.com> X-mailer: Pegasus Mail for Windows (4.70) X-RR-Connecting-IP: 107.14.168.130:25 X-Cloudmark-Score: 0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 14:10:47 -0000 > So the VNC client attaches successfully to the server, but the output is > just a black screen? IMHO, that looks more like a guest or VNC issue > rather than a Xen/Qemu one. Have you tried different guest OSes and > different VNC clients? (TigerVNC Viewer has always worked fine for me). > > But regarding your question: no, Xen Dom0 doesn't require X at all in > order to run. > I tried other guest OS's and VNC clients with the same result and came to the conclusion that all of my guest OS's were not booting. All of my VNC clients displayed the name of the guest OS in the margin, but had a black screen. I looked at the 'xl list' billboard and the guest OS did not have a 'r' in the column. The startup log for all of the guests had this line: 'xen be core: can't open gnttab device' My guess is that the 'gnttab' device is critical to a guest startup. I could not find any reference to this item. What am I missing? Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF From owner-freebsd-xen@freebsd.org Thu Nov 5 15:16:15 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B2DCA272F6 for ; Thu, 5 Nov 2015 15:16:15 +0000 (UTC) (envelope-from prvs=7445e0bd4=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA79D11BB; Thu, 5 Nov 2015 15:16:14 +0000 (UTC) (envelope-from prvs=7445e0bd4=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,248,1444694400"; d="scan'208";a="315910971" Subject: Re: Checksum forwarding issue on XEN To: Larry Baird , References: <20151103201250.GA92469@gta.com> From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 CC: , , Message-ID: <563B72B2.6060308@citrix.com> Date: Thu, 5 Nov 2015 16:16:02 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151103201250.GA92469@gta.com> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 15:16:15 -0000 Hello, Adding the persons that contributed that code in case they can shed some light. El 03/11/15 a les 21.12, Larry Baird ha escrit: > Has anybody made any progress on "Bug 188261 - [xen] FreeBSD DomU PVHVM > guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host." > (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188261)? > > The code for checksum calculation in the function xnb_add_mbuf_cksum() looks > suspect. > > switch (iph->ip_p) { > case IPPROTO_TCP: > if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { > size_t tcplen = ntohs(iph->ip_len) - sizeof(struct ip); > struct tcphdr *th = (struct tcphdr*)(iph + 1); > th->th_sum = in_pseudo(iph->ip_src.s_addr, > iph->ip_dst.s_addr, htons(IPPROTO_TCP + tcplen)); > th->th_sum = in_cksum_skip(mbufc, > sizeof(struct ether_header) + ntohs(iph->ip_len), > sizeof(struct ether_header) + (iph->ip_hl << 2)); > } > break; > case IPPROTO_UDP: > if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { > size_t udplen = ntohs(iph->ip_len) - sizeof(struct ip); > struct udphdr *uh = (struct udphdr*)(iph + 1); > uh->uh_sum = in_pseudo(iph->ip_src.s_addr, > iph->ip_dst.s_addr, htons(IPPROTO_UDP + udplen)); > uh->uh_sum = in_cksum_skip(mbufc, > sizeof(struct ether_header) + ntohs(iph->ip_len), > sizeof(struct ether_header) + (iph->ip_hl << 2)); > } > break; > default: > break; > } > > > Both in_pseudo() and in_cksum_skip() set the same checksum. Does this > make since to anybody? The bug you are referring to affects FreeBSD when running as a guest using xen-netfront, but the code snipped above and the function referenced (xnb_add_mbuf_cksum) is only used on FreeBSD when running as a host (AKA Dom0) by xen-netback. TBH, I don't know that much about FreeBSD network subsystem to have an opinion, but it certainly looks weird. Patches are welcome :). Roger. From owner-freebsd-xen@freebsd.org Thu Nov 5 15:19:03 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 138D2A2733C for ; Thu, 5 Nov 2015 15:19:03 +0000 (UTC) (envelope-from prvs=7445e0bd4=roger.pau@citrix.com) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C04651235 for ; Thu, 5 Nov 2015 15:19:02 +0000 (UTC) (envelope-from prvs=7445e0bd4=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,248,1444694400"; d="scan'208";a="310585453" Subject: Re: Does a Xen Dom0 require X to function To: , References: <5633A2B0.22403.45280A@lausts.acm.org> <56379E8A.3030701@citrix.com> <563B631D.22122.FDB9F@lausts.acm.org> From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Message-ID: <563B7357.3040002@citrix.com> Date: Thu, 5 Nov 2015 16:18:47 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <563B631D.22122.FDB9F@lausts.acm.org> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 15:19:03 -0000 Hello, El 05/11/15 a les 15.09, Thomas Laus ha escrit: >> So the VNC client attaches successfully to the server, but the output is >> just a black screen? IMHO, that looks more like a guest or VNC issue >> rather than a Xen/Qemu one. Have you tried different guest OSes and >> different VNC clients? (TigerVNC Viewer has always worked fine for me). >> >> But regarding your question: no, Xen Dom0 doesn't require X at all in >> order to run. >> > I tried other guest OS's and VNC clients with the same result and came to the > conclusion that all of my guest OS's were not booting. All of my VNC clients > displayed the name of the guest OS in the margin, but had a black screen. I > looked at the 'xl list' billboard and the guest OS did not have a 'r' in the > column. The startup log for all of the guests had this line: > > 'xen be core: can't open gnttab device' Is there anything else in the Qemu logs? Also, can you post the output of `xl -vvv create ` and `xl list` while the domain is running? > My guess is that the 'gnttab' device is critical to a guest startup. I could > not find any reference to this item. What am I missing? No, gnttab is not critical, Qemu should work without it. Roger. From owner-freebsd-xen@freebsd.org Thu Nov 5 16:01:06 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BB00A27E5A for ; Thu, 5 Nov 2015 16:01:06 +0000 (UTC) (envelope-from lab@gta.com) Received: from mailgate.gta.com (gb-ha.orl.gta.com [209.208.105.194]) by mx1.freebsd.org (Postfix) with ESMTP id 369E21A31 for ; Thu, 5 Nov 2015 16:01:05 +0000 (UTC) (envelope-from lab@gta.com) Received: (qmail 6792 invoked by uid 1000); 5 Nov 2015 16:00:57 -0000 Date: Thu, 5 Nov 2015 11:00:57 -0500 From: Larry Baird To: Roger Pau Monn?? Cc: freebsd-xen@freebsd.org, alans@spectralogic.com, johns@spectralogic.com, ken@FreeBSD.org Subject: Re: Checksum forwarding issue on XEN Message-ID: <20151105160057.GA2268@gta.com> References: <20151103201250.GA92469@gta.com> <563B72B2.6060308@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <563B72B2.6060308@citrix.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 16:01:06 -0000 Roger, > Adding the persons that contributed that code in case they can shed some > light. > > El 03/11/15 a les 21.12, Larry Baird ha escrit: > > Has anybody made any progress on "Bug 188261 - [xen] FreeBSD DomU PVHVM > > guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host." > > (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188261)? > > > > The code for checksum calculation in the function xnb_add_mbuf_cksum() looks > > suspect. > > > > switch (iph->ip_p) { > > case IPPROTO_TCP: > > if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { > > size_t tcplen = ntohs(iph->ip_len) - sizeof(struct ip); > > struct tcphdr *th = (struct tcphdr*)(iph + 1); > > th->th_sum = in_pseudo(iph->ip_src.s_addr, > > iph->ip_dst.s_addr, htons(IPPROTO_TCP + tcplen)); > > th->th_sum = in_cksum_skip(mbufc, > > sizeof(struct ether_header) + ntohs(iph->ip_len), > > sizeof(struct ether_header) + (iph->ip_hl << 2)); > > } > > break; > > case IPPROTO_UDP: > > if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { > > size_t udplen = ntohs(iph->ip_len) - sizeof(struct ip); > > struct udphdr *uh = (struct udphdr*)(iph + 1); > > uh->uh_sum = in_pseudo(iph->ip_src.s_addr, > > iph->ip_dst.s_addr, htons(IPPROTO_UDP + udplen)); > > uh->uh_sum = in_cksum_skip(mbufc, > > sizeof(struct ether_header) + ntohs(iph->ip_len), > > sizeof(struct ether_header) + (iph->ip_hl << 2)); > > } > > break; > > default: > > break; > > } > > > > > > Both in_pseudo() and in_cksum_skip() set the same checksum. Does this > > make since to anybody? > > The bug you are referring to affects FreeBSD when running as a guest > using xen-netfront, but the code snipped above and the function > referenced (xnb_add_mbuf_cksum) is only used on FreeBSD when running as > a host (AKA Dom0) by xen-netback. > > TBH, I don't know that much about FreeBSD network subsystem to have an > opinion, but it certainly looks weird. Patches are welcome :). Xyper-V has a similar forward issue. I found they were misusing csum_flags and were always attempting to do checksum offloading if CSUM_IP_VALID was set. I have given them a patch that fixes the issue. I was hoping that Xen's issue was similar. I found the issue above by looking at all uses of csum_flags in sys/dev/xen. It is hard to tell what the correct fix is, without fulling understand the protocal used when communicating between backend and frontend of Xen. I am sure issue with XEN guest forwarding has to with checksum offloading. If I am not misinterpreting your comments, I can ignore code in netback and concentrate on code in netfront when trying to understand what is going wrong. Thank you for some insite, Larry -- ------------------------------------------------------------------------ Larry Baird Global Technology Associates, Inc. 1992-2012 | http://www.gta.com Celebrating Twenty Years of Software Innovation | Orlando, FL Email: lab@gta.com | TEL 407-380-0220 From owner-freebsd-xen@freebsd.org Thu Nov 5 16:30:13 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 501E5A2650F for ; Thu, 5 Nov 2015 16:30:13 +0000 (UTC) (envelope-from prvs=7445e0bd4=roger.pau@citrix.com) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DDECB1C4C; Thu, 5 Nov 2015 16:30:12 +0000 (UTC) (envelope-from prvs=7445e0bd4=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,248,1444694400"; d="scan'208";a="310609872" Subject: Re: Checksum forwarding issue on XEN To: Larry Baird References: <20151103201250.GA92469@gta.com> <563B72B2.6060308@citrix.com> <20151105160057.GA2268@gta.com> CC: , , Wei Liu , From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 Message-ID: <563B840E.1050205@citrix.com> Date: Thu, 5 Nov 2015 17:30:06 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151105160057.GA2268@gta.com> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 16:30:13 -0000 El 05/11/15 a les 17.00, Larry Baird ha escrit: > Roger, > >> Adding the persons that contributed that code in case they can shed some >> light. >> >> El 03/11/15 a les 21.12, Larry Baird ha escrit: >>> Has anybody made any progress on "Bug 188261 - [xen] FreeBSD DomU PVHVM >>> guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host." >>> (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188261)? >>> >>> The code for checksum calculation in the function xnb_add_mbuf_cksum() looks >>> suspect. >>> >>> switch (iph->ip_p) { >>> case IPPROTO_TCP: >>> if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { >>> size_t tcplen = ntohs(iph->ip_len) - sizeof(struct ip); >>> struct tcphdr *th = (struct tcphdr*)(iph + 1); >>> th->th_sum = in_pseudo(iph->ip_src.s_addr, >>> iph->ip_dst.s_addr, htons(IPPROTO_TCP + tcplen)); >>> th->th_sum = in_cksum_skip(mbufc, >>> sizeof(struct ether_header) + ntohs(iph->ip_len), >>> sizeof(struct ether_header) + (iph->ip_hl << 2)); >>> } >>> break; >>> case IPPROTO_UDP: >>> if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { >>> size_t udplen = ntohs(iph->ip_len) - sizeof(struct ip); >>> struct udphdr *uh = (struct udphdr*)(iph + 1); >>> uh->uh_sum = in_pseudo(iph->ip_src.s_addr, >>> iph->ip_dst.s_addr, htons(IPPROTO_UDP + udplen)); >>> uh->uh_sum = in_cksum_skip(mbufc, >>> sizeof(struct ether_header) + ntohs(iph->ip_len), >>> sizeof(struct ether_header) + (iph->ip_hl << 2)); >>> } >>> break; >>> default: >>> break; >>> } >>> >>> >>> Both in_pseudo() and in_cksum_skip() set the same checksum. Does this >>> make since to anybody? >> >> The bug you are referring to affects FreeBSD when running as a guest >> using xen-netfront, but the code snipped above and the function >> referenced (xnb_add_mbuf_cksum) is only used on FreeBSD when running as >> a host (AKA Dom0) by xen-netback. >> >> TBH, I don't know that much about FreeBSD network subsystem to have an >> opinion, but it certainly looks weird. Patches are welcome :). > > Xyper-V has a similar forward issue. I found they were misusing csum_flags > and were always attempting to do checksum offloading if CSUM_IP_VALID was > set. I have given them a patch that fixes the issue. I was hoping that > Xen's issue was similar. I found the issue above by looking at all uses > of csum_flags in sys/dev/xen. It is hard to tell what the correct fix > is, without fulling understand the protocal used when communicating between > backend and frontend of Xen. > I am sure issue with XEN guest forwarding has to with checksum offloading. > If I am not misinterpreting your comments, I can ignore code in netback and > concentrate on code in netfront when trying to understand what is going wrong. Yes, this issue is related to netfront (sys/dev/xen/netfront/netfront.c) only, netback code is not involved. The code related to the checksum stuff is on line ~1409 for the TX side, and around line 872 for the RX side AFAICT. You can find more information about the protocol itself in sys/xen/interface/io/netif.h. Adding Wei Liu who is also doing some work to improve netfront, and knows more about the protocol than myself. Roger. From owner-freebsd-xen@freebsd.org Thu Nov 5 17:08:36 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9FFBA2700D for ; Thu, 5 Nov 2015 17:08:36 +0000 (UTC) (envelope-from alans@spectralogic.com) Received: from mail1.bemta7.messagelabs.com (mail1.bemta7.messagelabs.com [216.82.254.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail1.bemta7.messagelabs.com", Issuer "VeriSign Class 3 International Server CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A59161867; Thu, 5 Nov 2015 17:08:36 +0000 (UTC) (envelope-from alans@spectralogic.com) Received: from [216.82.253.227] by server-11.bemta-7.messagelabs.com id 54/47-15271-09B8B365; Thu, 05 Nov 2015 17:02:08 +0000 X-Env-Sender: alans@spectralogic.com X-Msg-Ref: server-14.tower-170.messagelabs.com!1446742928!3453230!1 X-Originating-IP: [192.30.190.14] X-StarScan-Received: X-StarScan-Version: 7.19.2; banners=-,-,- X-VirusChecked: Checked Received: (qmail 30103 invoked from network); 5 Nov 2015 17:02:08 -0000 Received: from outmx1.spectralogic.com (HELO mail.spectralogic.com) (192.30.190.14) by server-14.tower-170.messagelabs.com with AES256-SHA encrypted SMTP; 5 Nov 2015 17:02:08 -0000 From: Alan Somers To: Larry Baird , Roger Pau Monn?? CC: "freebsd-xen@freebsd.org" , "ken@FreeBSD.org" Subject: Re: Checksum forwarding issue on XEN Thread-Topic: Checksum forwarding issue on XEN Thread-Index: AQHRF9zm9zZvkZi2QEuPJaWrDdnj/J6OC++AgAAQ9AA= Date: Thu, 5 Nov 2015 17:02:07 +0000 Message-ID: <563B8B72.5050800@spectralogic.com> References: <20151103201250.GA92469@gta.com> <563B72B2.6060308@citrix.com> <20151105160057.GA2268@gta.com> In-Reply-To: <20151105160057.GA2268@gta.com> Reply-To: Alan Somers Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 x-ms-exchange-transport-fromentityheader: Hosted Content-Type: text/plain; charset="Windows-1252" Content-ID: <389D9AE248AF5C4CB427288AA6767528@spectralogic.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 17:08:36 -0000 On 11/05/2015 09:00 AM, Larry Baird wrote: > Roger, > >> Adding the persons that contributed that code in case they can shed some >> light. Removing John Suykerbuyk's outdated address. I don't have a current=20 address for him. And replacing my spectralogic email address with my=20 FreeBSD one. >> >> El 03/11/15 a les 21.12, Larry Baird ha escrit: >>> Has anybody made any progress on "Bug 188261 - [xen] FreeBSD DomU PVHVM >>> guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host= ." >>> (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D188261)? >>> >>> The code for checksum calculation in the function xnb_add_mbuf_cksum() = looks >>> suspect. >>> >>> switch (iph->ip_p) { >>> case IPPROTO_TCP: >>> if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { >>> size_t tcplen =3D ntohs(iph->ip_len) - sizeof(= struct ip); >>> struct tcphdr *th =3D (struct tcphdr*)(iph + 1= ); >>> th->th_sum =3D in_pseudo(iph->ip_src.s_addr, >>> iph->ip_dst.s_addr, htons(IPPROTO_TCP + tc= plen)); >>> th->th_sum =3D in_cksum_skip(mbufc, >>> sizeof(struct ether_header) + ntohs(iph->i= p_len), >>> sizeof(struct ether_header) + (iph->ip_hl = << 2)); >>> } >>> break; >>> case IPPROTO_UDP: >>> if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { >>> size_t udplen =3D ntohs(iph->ip_len) - sizeof(= struct ip); >>> struct udphdr *uh =3D (struct udphdr*)(iph + 1= ); >>> uh->uh_sum =3D in_pseudo(iph->ip_src.s_addr, >>> iph->ip_dst.s_addr, htons(IPPROTO_UDP + ud= plen)); >>> uh->uh_sum =3D in_cksum_skip(mbufc, >>> sizeof(struct ether_header) + ntohs(iph->i= p_len), >>> sizeof(struct ether_header) + (iph->ip_hl = << 2)); >>> } >>> break; >>> default: >>> break; >>> } >>> >>> >>> Both in_pseudo() and in_cksum_skip() set the same checksum. Does this >>> make since to anybody? Though it looks weird, I think it's actually correct. th->th_sum lies=20 within the packet that it inspected by in_cksum_skip. in_cksum_skip=20 skips the ethernet and IP headers, but not the TCP header. So it=20 consumes whatever value was previously set in th_sum. In any case, it=20 will be easy to check this code by running the builtin unit tests. Just=20 build the kernel with XNB_DEBUG defined, instantiate an xnb interface,=20 and do "sysctl dev.xnb.0.unit_test_results". >> The bug you are referring to affects FreeBSD when running as a guest >> using xen-netfront, but the code snipped above and the function >> referenced (xnb_add_mbuf_cksum) is only used on FreeBSD when running as >> a host (AKA Dom0) by xen-netback. >> >> TBH, I don't know that much about FreeBSD network subsystem to have an >> opinion, but it certainly looks weird. Patches are welcome :). > Xyper-V has a similar forward issue. I found they were misusing csum_flag= s > and were always attempting to do checksum offloading if CSUM_IP_VALID was > set. I have given them a patch that fixes the issue. I was hoping that > Xen's issue was similar. I found the issue above by looking at all uses > of csum_flags in sys/dev/xen. It is hard to tell what the correct fix > is, without fulling understand the protocal used when communicating betwe= en > backend and frontend of Xen. > > I am sure issue with XEN guest forwarding has to with checksum offloading= . > If I am not misinterpreting your comments, I can ignore code in netback a= nd > concentrate on code in netfront when trying to understand what is going w= rong. > > Thank you for some insite, > Larry > > FWIW, there is a performance problem with Xen and checksums. Whenever=20 FreeBSD is used as a dom0, checksum offloading should be disabled on the=20 netfronts. I don't know if the same problem exists on other OSes, but=20 it might. See xnb(4) for more details about this problem. -Alan= From owner-freebsd-xen@freebsd.org Thu Nov 5 17:53:30 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16D93A27A64 for ; Thu, 5 Nov 2015 17:53:30 +0000 (UTC) (envelope-from lab@gta.com) Received: from mailgate.gta.com (gb-ha.orl.gta.com [209.208.105.194]) by mx1.freebsd.org (Postfix) with ESMTP id C484F19BC for ; Thu, 5 Nov 2015 17:53:29 +0000 (UTC) (envelope-from lab@gta.com) Received: (qmail 27061 invoked by uid 1000); 5 Nov 2015 17:53:25 -0000 Date: Thu, 5 Nov 2015 12:53:25 -0500 From: Larry Baird To: Alan Somers Cc: Roger Pau Monn?? , "freebsd-xen@freebsd.org" , "ken@FreeBSD.org" Subject: Re: Checksum forwarding issue on XEN Message-ID: <20151105175325.GA13746@gta.com> References: <20151103201250.GA92469@gta.com> <563B72B2.6060308@citrix.com> <20151105160057.GA2268@gta.com> <563B8B72.5050800@spectralogic.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <563B8B72.5050800@spectralogic.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 17:53:30 -0000 Alan, > Though it looks weird, I think it's actually correct. th->th_sum lies > within the packet that it inspected by in_cksum_skip. in_cksum_skip > skips the ethernet and IP headers, but not the TCP header. So it > consumes whatever value was previously set in th_sum. In any case, it > will be easy to check this code by running the builtin unit tests. Just > build the kernel with XNB_DEBUG defined, instantiate an xnb interface, > and do "sysctl dev.xnb.0.unit_test_results". I agress the code looks strange but is probably correct as far as it goes. The code assumes that there aren't any IP options. I am not sure if this is a valid assumption or not. The code also assumes packet is IPv4. > FWIW, there is a performance problem with Xen and checksums. Whenever > FreeBSD is used as a dom0, checksum offloading should be disabled on the > netfronts. I don't know if the same problem exists on other OSes, but > it might. See xnb(4) for more details about this problem. Ok I read the CAVEATS on xnb(4). Let me play with that for a bit. I have an ugly code patch, that works for me with out disabling checksum offloading. Before I share, let me do a few more tests. Larry -- ------------------------------------------------------------------------ Larry Baird Global Technology Associates, Inc. 1992-2012 | http://www.gta.com Celebrating Twenty Years of Software Innovation | Orlando, FL Email: lab@gta.com | TEL 407-380-0220 From owner-freebsd-xen@freebsd.org Thu Nov 5 20:37:36 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 270B9A27C45 for ; Thu, 5 Nov 2015 20:37:36 +0000 (UTC) (envelope-from lausts@acm.org) Received: from cdptpa-oedge-vip.email.rr.com (cdptpa-outbound-snat.email.rr.com [107.14.166.232]) by mx1.freebsd.org (Postfix) with ESMTP id E04151AE8 for ; Thu, 5 Nov 2015 20:37:35 +0000 (UTC) (envelope-from lausts@acm.org) Received: from [173.88.10.122] ([173.88.10.122:34820] helo=mail.laus.org) by cdptpa-oedge01 (envelope-from ) (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP id B0/D7-24909-E0EBB365; Thu, 05 Nov 2015 20:37:34 +0000 Received: from [192.168.1.100] (laust2 [192.168.1.100]) by mail.laus.org (8.15.2/8.15.2) with ESMTPS id tA5KbXNO004720 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 5 Nov 2015 15:37:33 -0500 (EST) (envelope-from lausts@acm.org) X-Authentication-Warning: mail.laus.org: Host laust2 [192.168.1.100] claimed to be [192.168.1.100] From: "Thomas Laus" Organization: ABB To: Roger Pau Monné Date: Thu, 05 Nov 2015 15:37:28 -0500 Subject: Re: Does a Xen Dom0 require X to function Reply-to: lausts@acm.org CC: freebsd-xen@freebsd.org Message-ID: <563BBE08.32089.ECACD@lausts.acm.org> Priority: normal In-reply-to: <563B7357.3040002@citrix.com> References: <5633A2B0.22403.45280A@lausts.acm.org>, <563B631D.22122.FDB9F@lausts.acm.org>, <563B7357.3040002@citrix.com> X-mailer: Pegasus Mail for Windows (4.70) X-RR-Connecting-IP: 107.14.168.118:25 X-Cloudmark-Score: 0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 20:37:36 -0000 > Is there anything else in the Qemu logs? > > Also, can you post the output of `xl -vvv create ` and `xl > list` while the domain is running? > I can't capture the 'xl -vvv create freebsd.cfg output to a file. This computer doesn't have a functioning 'X' installation. The redirect to a file '>' does not capture anything. ======== The output of 'xl list': Name ID Mem VCPUs Stat Time(s) Domain-0 0 2048 4 r----- 23.5 FreeBSD 1 512 1 ------ 0.0 ======== The qemu-dm-FreeBSD.log: xen be core: can't open gnttab device qemu: terminating on signal 1 from pid 727 The xl-FreeBSD.log: Waiting for domain FreeBSD (domid 1) to die [pid 708] libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch w=0x8028193f8 wpath=@releaseDomain token=3/0: register slotnum=3 libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch w=0x802818440 wpath=/local/domain/1/device/vbd/5632/eject token=2/1: register slotnum=2 libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x8028193f8 wpath=@releaseDomain token=3/0: event epath=@releaseDomain libxl: debug: libxl.c:1187:domain_death_xswatch_callback: [evg=0x802842a80:1] nentries=1 rc=1 1..1 libxl: debug: libxl.c:1198:domain_death_xswatch_callback: [evg=0x802842a80:1] got=domaininfos[0] got->domain=1 libxl: debug: libxl.c:1225:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0002 libxl: debug: libxl.c:1191:domain_death_xswatch_callback: [evg=0] all reported libxl: debug: libxl.c:1254:domain_death_xswatch_callback: domain death search done libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x802818440 wpath=/local/domain/1/device/vbd/5632/eject token=2/1: event epath=/local/domain/1/device/vbd/5632/eject libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x802818440 wpath=/local/domain/1/device/vbd/5632/eject token=2/1: event epath=/local/domain/1/device/vbd/5632/eject libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x802818440 wpath=/local/domain/1/device/vbd/5632/eject token=2/1: event epath=/local/domain/1/device/vbd/5632/eject libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x802818440 wpath=/local/domain/1/device/vbd/5632/eject token=2/1: event epath=/local/domain/1/device/vbd/5632/eject libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x8028193f8 wpath=@releaseDomain token=3/0: event epath=@releaseDomain libxl: debug: libxl.c:1187:domain_death_xswatch_callback: [evg=0x802842a80:1] nentries=1 rc=1 1..1 libxl: debug: libxl.c:1198:domain_death_xswatch_callback: [evg=0x802842a80:1] got=domaininfos[0] got->domain=1 libxl: debug: libxl.c:1225:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff000b libxl: debug: libxl.c:1143:domain_death_occurred: dying libxl: debug: libxl.c:1191:domain_death_xswatch_callback: [evg=0] all reported libxl: debug: libxl.c:1254:domain_death_xswatch_callback: domain death search done Domain 1 has been destroyed. libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch w=0x8028193f8 wpath=@releaseDomain token=3/0: deregister slotnum=3 libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch w=0x802818440 wpath=/local/domain/1/device/vbd/5632/eject token=2/1: deregister slotnum=2 xc: debug: hypercall buffer: total allocations:4 total releases:4 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:2 misses:2 toobig:0 I hope that this information is helpful. Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF From owner-freebsd-xen@freebsd.org Fri Nov 6 08:08:45 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1892DA26BC2 for ; Fri, 6 Nov 2015 08:08:45 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C3E391930 for ; Fri, 6 Nov 2015 08:08:44 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,251,1444694400"; d="scan'208";a="310788217" Subject: Re: Does a Xen Dom0 require X to function To: References: <5633A2B0.22403.45280A@lausts.acm.org> <563B631D.22122.FDB9F@lausts.acm.org> <563B7357.3040002@citrix.com> <563BBE08.32089.ECACD@lausts.acm.org> CC: From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 Message-ID: <563C6006.2030206@citrix.com> Date: Fri, 6 Nov 2015 09:08:38 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <563BBE08.32089.ECACD@lausts.acm.org> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2015 08:08:45 -0000 El 05/11/15 a les 21.37, Thomas Laus ha escrit: >> Is there anything else in the Qemu logs? >> >> Also, can you post the output of `xl -vvv create ` and `xl >> list` while the domain is running? >> > I can't capture the 'xl -vvv create freebsd.cfg output to a file. This > computer doesn't have a functioning 'X' installation. The redirect to a file > '>' does not capture anything. xl debug messages go to stderr, not stdout, so you have to use "2>" instead of ">" in order to do the redirection. Setting up a ssh server on the Dom0 and using it remotely will also help ;). > ======== > The output of 'xl list': > > Name ID Mem VCPUs Stat Time(s) > Domain-0 0 2048 4 r----- 23.5 > FreeBSD 1 512 1 ------ 0.0 It looks like the guest called FreeBSD is not running at all. Can you paste the configuration file of your domain also? Can you also paste the output of `xl dmesg` after trying to launch the guest? Which revision of FreeBSD are you running in the Dom0? Roger. From owner-freebsd-xen@freebsd.org Fri Nov 6 12:32:43 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87755A27E4D for ; Fri, 6 Nov 2015 12:32:43 +0000 (UTC) (envelope-from mike@reifenberger.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 6EB6812BB for ; Fri, 6 Nov 2015 12:32:43 +0000 (UTC) (envelope-from mike@reifenberger.com) Received: by mailman.ysv.freebsd.org (Postfix) id 6CDD5A27E4C; Fri, 6 Nov 2015 12:32:43 +0000 (UTC) Delivered-To: xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C74BA27E4B for ; Fri, 6 Nov 2015 12:32:43 +0000 (UTC) (envelope-from mike@reifenberger.com) Received: from mail.eeeit.de (mail.eeeit.de [37.120.160.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 356F912BA for ; Fri, 6 Nov 2015 12:32:39 +0000 (UTC) (envelope-from mike@reifenberger.com) Received: from localhost (localhost [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: mike@reifenberger.com) by mail.eeeit.de (Postfix) with ESMTPSA id 1B6504A7C; Fri, 6 Nov 2015 13:23:09 +0100 (CET) Received: from ppp-93-104-2-183.dynamic.mnet-online.de (ppp-93-104-2-183.dynamic.mnet-online.de [93.104.2.183]) by mail.eeeit.de (Horde Framework) with HTTP; Fri, 06 Nov 2015 13:23:08 +0100 Date: Fri, 06 Nov 2015 13:23:08 +0100 Message-ID: <20151106132308.Horde.wqlInItwP9c68Ih_23aoBbQ@mail.eeeit.de> From: Michael Reifenberger To: Roger Pau =?utf-8?b?TW9ubsOp?= Cc: xen@freebsd.org Subject: Current shortcomings (XEN dom0 & FreeBSD) References: <20150918194154.Horde.PQcchwucJFPQY4U0K75MgpW@mail.eeeit.de> <56027D61.70207@citrix.com> <20150923163642.Horde.C2gq8tfwkC45mOK8NwCIAj-@mail.eeeit.de> <5602BD02.7050004@citrix.com> In-Reply-To: <5602BD02.7050004@citrix.com> User-Agent: Horde Application Framework 5 Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2015 12:32:43 -0000 Hi, before analyzing deeper I just wanted to ask if the following symptoms are already known: - Often / Mostly the FreeBSD dom0 hangs without further output during poweroff (reboot works though) - The same happens mostly for the Centos7 guests I'm currently testing. - Sometimes (usually after some uptime and/or many guest reboots) the blockback (ZFS Zvols in my case) seems to get wedged (guest hang during startup)... Besides that the whole system is quite usable (For Windows, SUSE and Centos guests)! Greetings --- Michael Gruß --- Michael Reifenberger From owner-freebsd-xen@freebsd.org Fri Nov 6 14:56:45 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38CD4A28DDA for ; Fri, 6 Nov 2015 14:56:45 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 20CA61136 for ; Fri, 6 Nov 2015 14:56:45 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) Received: by mailman.ysv.freebsd.org (Postfix) id 207B8A28DD9; Fri, 6 Nov 2015 14:56:45 +0000 (UTC) Delivered-To: xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20197A28DD8 for ; Fri, 6 Nov 2015 14:56:45 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA5BA1135 for ; Fri, 6 Nov 2015 14:56:44 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,252,1444694400"; d="scan'208";a="316195242" Subject: Re: Current shortcomings (XEN dom0 & FreeBSD) To: Michael Reifenberger References: <20150918194154.Horde.PQcchwucJFPQY4U0K75MgpW@mail.eeeit.de> <56027D61.70207@citrix.com> <20150923163642.Horde.C2gq8tfwkC45mOK8NwCIAj-@mail.eeeit.de> <5602BD02.7050004@citrix.com> <20151106132308.Horde.wqlInItwP9c68Ih_23aoBbQ@mail.eeeit.de> CC: From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 Message-ID: <563CBFA7.30502@citrix.com> Date: Fri, 6 Nov 2015 15:56:39 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151106132308.Horde.wqlInItwP9c68Ih_23aoBbQ@mail.eeeit.de> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2015 14:56:45 -0000 Hello, El 06/11/15 a les 13.23, Michael Reifenberger ha escrit: > Hi, > before analyzing deeper I just wanted to ask if the following symptoms > are already known: > > - Often / Mostly the FreeBSD dom0 hangs without further output during > poweroff (reboot works though) Yes, I'm aware of this issue, unfortunately the way to solve it it's not clear. In order to do a proper power off when running as Dom0 we would have to modify ACPICA code, which is a separate upstream project that's used by a bunch of different OSes. Linux does it this way, but then they have to keep all this local modifications on top of upstream ACPICA, which increases the maintainership burden. > - The same happens mostly for the Centos7 guests I'm currently testing. Do you mean that CentOS 7 guests do not poweroff properly? Are those PV or HVM guests? I usually use Debian guests when I have to test Linux and have never experienced this. > - Sometimes (usually after some uptime and/or many guest reboots) the > blockback > (ZFS Zvols in my case) seems to get wedged (guest hang during startup)... Yes, I've also experienced this once or twice. The problem is that the blkback error path can get deadlocked depending on the situation. I have to look into cleaning it when I have some time. > Besides that the whole system is quite usable (For Windows, SUSE and > Centos guests)! Thanks for the testing and the list of issues! Roger. From owner-freebsd-xen@freebsd.org Fri Nov 6 15:23:06 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B69EA27256 for ; Fri, 6 Nov 2015 15:23:06 +0000 (UTC) (envelope-from lausts@acm.org) Received: from cdptpa-oedge-vip.email.rr.com (cdptpa-outbound-snat.email.rr.com [107.14.166.229]) by mx1.freebsd.org (Postfix) with ESMTP id 1D40B1D22 for ; Fri, 6 Nov 2015 15:23:05 +0000 (UTC) (envelope-from lausts@acm.org) Received: from [173.88.10.122] ([173.88.10.122:38534] helo=mail.laus.org) by cdptpa-oedge03 (envelope-from ) (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP id 6A/1A-05699-3D5CC365; Fri, 06 Nov 2015 15:22:59 +0000 Received: from [192.168.1.100] (laust2 [192.168.1.100]) by mail.laus.org (8.15.2/8.15.2) with ESMTPS id tA6FMveQ009029 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 6 Nov 2015 10:22:58 -0500 (EST) (envelope-from lausts@acm.org) X-Authentication-Warning: mail.laus.org: Host laust2 [192.168.1.100] claimed to be [192.168.1.100] From: "Thomas Laus" Organization: ABB To: Roger Pau Monné , Date: Fri, 06 Nov 2015 10:22:52 -0500 Subject: Re: Does a Xen Dom0 require X to function Reply-to: lausts@acm.org Message-ID: <563CC5CC.19454.173726@lausts.acm.org> Priority: normal In-reply-to: <563C6006.2030206@citrix.com> References: <5633A2B0.22403.45280A@lausts.acm.org>, <563BBE08.32089.ECACD@lausts.acm.org>, <563C6006.2030206@citrix.com> X-mailer: Pegasus Mail for Windows (4.70) X-RR-Connecting-IP: 107.14.168.142:25 X-Cloudmark-Score: 0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2015 15:23:06 -0000 > xl debug messages go to stderr, not stdout, so you have to use "2>" > instead of ">" in order to do the redirection. Setting up a ssh server > on the Dom0 and using it remotely will also help ;). > Still nothing was sent to a file for 2>. Here is my xl dmesg: (XEN) ERROR: 16550-compatible serial UART not present Xen 4.5.1 (XEN) Xen version 4.5.1 (root@) (gcc48 (FreeBSD Ports Collection) 4.8.5) debug=n Sat Oct 24 18:29:18 UTC 2015 (XEN) Latest ChangeSet: (XEN) Bootloader: FreeBSD Loader (XEN) Command line: dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009d000 (usable) (XEN) 000000000009d000 - 00000000000a0000 (reserved) (XEN) 00000000000e0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 000000006a39b000 (usable) (XEN) 000000006a39b000 - 000000007cd2f000 (reserved) (XEN) 000000007cd2f000 - 000000007ce7f000 (ACPI NVS) (XEN) 000000007ce7f000 - 000000007ceff000 (ACPI data) (XEN) 000000007ceff000 - 000000007fa00000 (reserved) (XEN) 00000000b8000000 - 00000000fc000000 (reserved) (XEN) 00000000fec00000 - 00000000fec01000 (reserved) (XEN) 00000000fed08000 - 00000000fed09000 (reserved) (XEN) 00000000fed10000 - 00000000fed1a000 (reserved) (XEN) 00000000fed1c000 - 00000000fed20000 (reserved) (XEN) 00000000fee00000 - 00000000fee01000 (reserved) (XEN) 00000000ffc00000 - 0000000100000000 (reserved) (XEN) 0000000100000000 - 000000027e600000 (usable) (XEN) ACPI: RSDP 000F0120, 0024 (r2 LENOVO) (XEN) ACPI: XSDT 7CEFE170, 00EC (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: FACP 7CEF8000, 010C (r5 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: DSDT 7CEE0000, 120D4 (r1 LENOVO TP-GN 2100 INTL 20120711) (XEN) ACPI: FACS 7CE42000, 0040 (XEN) ACPI: SLIC 7CEFD000, 0176 (r1 LENOVO TP-GN 2100 PTEC 1) (XEN) ACPI: DBGP 7CEFB000, 0034 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: ECDT 7CEFA000, 0052 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: HPET 7CEF7000, 0038 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: APIC 7CEF6000, 0098 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: MCFG 7CEF5000, 003C (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: SSDT 7CEF4000, 0033 (r1 LENOVO TP-SSDT1 100 INTL 20120711) (XEN) ACPI: SSDT 7CEF3000, 044F (r1 LENOVO TP-SSDT2 200 INTL 20120711) (XEN) ACPI: SSDT 7CEDF000, 0B78 (r1 LENOVO SataAhci 1000 INTL 20120711) (XEN) ACPI: SSDT 7CEDE000, 07E7 (r1 LENOVO Cpu0Ist 3000 INTL 20120711) (XEN) ACPI: SSDT 7CEDD000, 0AD8 (r1 LENOVO CpuPm 3000 INTL 20120711) (XEN) ACPI: SSDT 7CEDB000, 1215 (r1 LENOVO SaSsdt 3000 INTL 20120711) (XEN) ACPI: SSDT 7CEDA000, 0379 (r1 LENOVO CppcTabl 1000 INTL 20120711) (XEN) ACPI: PCCT 7CED9000, 006E (r5 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: SSDT 7CED8000, 0AC4 (r1 LENOVO Cpc_Tabl 1000 INTL 20120711) (XEN) ACPI: TCPA 7CED7000, 0032 (r2 PTL LENOVO 6040000 LNVO 1) (XEN) ACPI: UEFI 7CED6000, 0042 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: MSDM 7CDAA000, 0055 (r3 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: ASF! 7CEFC000, 00A5 (r32 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: BATB 7CED5000, 0046 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: FPDT 7CED4000, 0064 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: UEFI 7CED3000, 02E2 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) ACPI: SSDT 7CED2000, 047F (r1 LENOVO IsctTabl 1000 INTL 20120711) (XEN) ACPI: DMAR 7CED1000, 00B8 (r1 LENOVO TP-GN 2100 PTEC 2) (XEN) System RAM: 7817MB (8004832kB) (XEN) No NUMA configuration found (XEN) Faking a node at 0000000000000000-000000027e600000 (XEN) Domain heap initialised (XEN) found SMP MP-table at 000f0100 (XEN) DMI 2.7 present. (XEN) Using APIC driver default (XEN) ACPI: PM-Timer IO Port: 0x1808 (XEN) ACPI: v5 SLEEP INFO: control[1:0], status[1:0] (XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1804,1:0], pm1x_evt[1:1800,1:0] (XEN) ACPI: 32/64X FACS address mismatch in FADT - 7ce42000/0000000000000000, using 32 (XEN) ACPI: wakeup_vec[7ce4200c], vec_size[20] (XEN) ACPI: Local APIC address 0xfee00000 (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) (XEN) Processor #0 7:12 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) (XEN) Processor #1 7:12 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) (XEN) Processor #2 7:12 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) (XEN) Processor #3 7:12 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled) (XEN) Processor #4 7:12 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled) (XEN) Processor #5 7:12 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] enabled) (XEN) Processor #6 7:12 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled) (XEN) Processor #7 7:12 APIC version 21 (XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) (XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: IRQ0 used by override. (XEN) ACPI: IRQ2 used by override. (XEN) ACPI: IRQ9 used by override. (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000 (XEN) ERST table was not found (XEN) Using ACPI (MADT) for SMP configuration information (XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs) (XEN) IRQ limits: 24 GSI, 1528 MSI/MSI-X (XEN) Switched to APIC driver x2apic_cluster. (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 2693.807 MHz processor. (XEN) Initing memory sharing. (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7 (XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0 (XEN) Intel machine check reporting enabled (XEN) alt table ffff82d0802b6cf0 -> ffff82d0802b7e60 (XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f (XEN) PCI: MCFG area at f8000000 reserved in E820 (XEN) PCI: Using MCFG for segment 0000 bus 00-3f (XEN) Intel VT-d iommu 0 supported page sizes: 4kB. (XEN) Intel VT-d iommu 1 supported page sizes: 4kB. (XEN) Intel VT-d Snoop Control not enabled. (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. (XEN) Intel VT-d Queued Invalidation enabled. (XEN) Intel VT-d Interrupt Remapping enabled. (XEN) Intel VT-d Shared EPT tables not enabled. (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) TSC deadline timer enabled (XEN) Platform timer is 14.318MHz HPET (XEN) Allocated console ring of 64 KiB. (XEN) mwait-idle: MWAIT substates: 0x42120 (XEN) mwait-idle: v0.4 model 0x3c (XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff (XEN) VMX: Supported advanced features: (XEN) - APIC MMIO access virtualisation (XEN) - APIC TPR shadow (XEN) - Extended Page Tables (EPT) (XEN) - Virtual-Processor Identifiers (VPID) (XEN) - Virtual NMI (XEN) - MSR direct-access bitmap (XEN) - Unrestricted Guest (XEN) - VMCS shadowing (XEN) HVM: ASIDs enabled. (XEN) HVM: VMX enabled (XEN) HVM: Hardware Assisted Paging (HAP) detected (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) Brought up 8 CPUs (XEN) ACPI sleep modes: S3 (XEN) mcheck_poll: Machine check polling timer started. (XEN) Dom0 has maximum 792 PIRQs (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x200000 -> 0x1da6878 (XEN) Dom0 symbol map 0x1da6878 -> 0x20d6830 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000270000000->0000000274000000 (507901 pages to be allocated) (XEN) Init. ramdisk: 000000027e5fd000->000000027e600000 (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff80200000->ffffffff820d6830 (XEN) Init. ramdisk: ffffffff820d7000->ffffffff820da000 (XEN) Phys-Mach map: ffffffff820da000->ffffffff824da000 (XEN) Start info: ffffffff824da000->ffffffff824db4b4 (XEN) Page tables: ffffffff824dc000->ffffffff824f3000 (XEN) Boot stack: ffffffff824f3000->ffffffff824f4000 (XEN) TOTAL: ffffffff80000000->ffffffff82800000 (XEN) ENTRY ADDRESS: ffffffff80f09000 (XEN) Dom0 has maximum 4 VCPUs (XEN) Masked UR signaling on 0000:00:00.0 (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:1d.0] fault addr 7af37000, iommu reg = ffff82c000203000 (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set (XEN) ....................done. (XEN) Initial low memory virq threshold set at 0x4000 pages. (XEN) Std. Loglevel: All (XEN) Guest Loglevel: All (XEN) Xen is relinquishing VGA console. (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 304kB init memory. (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:14.0] fault addr 7aec4000, iommu reg = ffff82c000203000 (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set (XEN) irq.c:380: Dom0 callback via changed to Direct Vector 0x93 (XEN) Masked UR signaling on 0000:00:00.0 (XEN) PCI add device 0000:00:00.0 (XEN) PCI add device 0000:00:01.0 (XEN) PCI add device 0000:00:02.0 (XEN) PCI add device 0000:00:03.0 (XEN) PCI add device 0000:00:14.0 (XEN) PCI add device 0000:00:16.0 (XEN) PCI add device 0000:00:16.3 (XEN) PCI add device 0000:00:19.0 (XEN) PCI add device 0000:00:1a.0 (XEN) PCI add device 0000:00:1b.0 (XEN) PCI add device 0000:00:1c.0 (XEN) PCI add device 0000:00:1c.1 (XEN) PCI add device 0000:00:1c.2 (XEN) PCI add device 0000:00:1c.4 (XEN) PCI add device 0000:00:1d.0 (XEN) PCI add device 0000:00:1f.0 (XEN) PCI add device 0000:00:1f.2 (XEN) PCI add device 0000:00:1f.3 (XEN) PCI add device 0000:01:00.0 (XEN) PCI add device 0000:02:00.0 (XEN) PCI add device 0000:03:00.0 (d1) HVM Loader (d1) Detected Xen v4.5.1 (d1) Xenbus rings @0xfeffc000, event channel 1 (d1) System requested SeaBIOS (d1) CPU speed is 2694 MHz (d1) Relocating guest memory for lowmem MMIO space disabled I see the error about the UART here is the section of my device probes during the boot: Nov 6 09:12:40 xenserver kernel: Copyright (c) 1992-2015 The FreeBSD Project. Nov 6 09:12:40 xenserver kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Nov 6 09:12:40 xenserver kernel: The Regents of the University of California. All rights reserved. Nov 6 09:12:40 xenserver kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Nov 6 09:12:40 xenserver kernel: FreeBSD 11.0-CURRENT #0: Wed Nov 4 14:39:53 EST 2015 Nov 6 09:12:40 xenserver kernel: root@xenserver:/usr/obj/usr/src/sys/GENERIC-NODEBUG amd64 Nov 6 09:12:40 xenserver kernel: FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906 Nov 6 09:12:40 xenserver kernel: VT(vga): resolution 640x480 Nov 6 09:12:40 xenserver kernel: CPU: Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz (2693.76-MHz K8-class CPU) Nov 6 09:12:40 xenserver kernel: Origin="GenuineIntel" Id=0x306c3 Family=0x6 Model=0x3c Stepping=3 Nov 6 09:12:40 xenserver kernel: Features=0x1fc3ebff Nov 6 09:12:40 xenserver kernel: Features2=0xfff83a83 Nov 6 09:12:40 xenserver kernel: AMD Features=0x20100800 Nov 6 09:12:40 xenserver kernel: AMD Features2=0x21 Nov 6 09:12:40 xenserver kernel: Structured Extended Features=0xb38 Nov 6 09:12:40 xenserver kernel: XSAVE Features=0x1 Nov 6 09:12:40 xenserver kernel: TSC: P-state invariant, performance statistics Nov 6 09:12:40 xenserver kernel: Hypervisor: Origin = "XenVMMXenVMM" Nov 6 09:12:40 xenserver kernel: real memory = 4660690944 (4444 MB) Nov 6 09:12:40 xenserver kernel: avail memory = 1970360320 (1879 MB) Nov 6 09:12:40 xenserver kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Nov 6 09:12:40 xenserver kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) Nov 6 09:12:40 xenserver kernel: cpu0 (BSP): APIC ID: 0 Nov 6 09:12:40 xenserver kernel: cpu1 (AP): APIC ID: 2 Nov 6 09:12:40 xenserver kernel: cpu2 (AP): APIC ID: 4 Nov 6 09:12:40 xenserver kernel: cpu3 (AP): APIC ID: 6 Nov 6 09:12:40 xenserver kernel: uart2: port 0x50b0-0x50b7 mem 0xb2a40000-0xb2a40fff irq 17 at device 22.3 on pci0 > Can you also paste the output of `xl dmesg` after trying to launch the > guest? > My FreeBSD guest file: builder = "hvm" memory = 512 vcpus = 1 name = "FreeBSD" disk = [ '/root/xen/freebsd.img,raw,hda,rw', '/root/bhyve/freebsd11.iso,raw,hdc:cdrom,r' ] #boot = "c" #boot to hard drive image boot = "d" #boot to ISO image vnc = 1 vnclisten = "0.0.0.0" > Which revision of FreeBSD are you running in the Dom0? > FreeBSD current that was built yesterday from source. I am a contract programmer working on an off-shore natural gas platform project. I have 5 computers running at home in my office and was thinking about making all of them guests on a single hypervisor as a winter project. I brought along a small ITX sized PC for to help me get acquaited with hypervisors. It's CPU is missing Intel VT-d instructions, so I went to the local Best Buy store and bought a SSD upgrade kit for this company laptop. The laptop has VT-d features and is an i7. It is running on a spare partition but I will need to go back to the original hard drive soon because the Windows OS is complaining about needing re-activation. I just work on my hypervisor upgrade in my hotel room in the evenings and on the days that the helicopter doesn't fly because of weather. There isn't any way to grant remote SSH access until I get back home in January. Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF From owner-freebsd-xen@freebsd.org Fri Nov 6 17:09:23 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2EBBAA26635 for ; Fri, 6 Nov 2015 17:09:23 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ADA0B1BFC for ; Fri, 6 Nov 2015 17:09:22 +0000 (UTC) (envelope-from prvs=7454c2131=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.20,252,1444694400"; d="scan'208";a="310895179" Subject: Re: Does a Xen Dom0 require X to function To: , References: <5633A2B0.22403.45280A@lausts.acm.org> <563BBE08.32089.ECACD@lausts.acm.org> <563C6006.2030206@citrix.com> <563CC5CC.19454.173726@lausts.acm.org> From: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= X-Enigmail-Draft-Status: N1110 Message-ID: <563CDEB0.5010405@citrix.com> Date: Fri, 6 Nov 2015 18:09:04 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <563CC5CC.19454.173726@lausts.acm.org> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 8bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2015 17:09:23 -0000 El 06/11/15 a les 16.22, Thomas Laus ha escrit: >> xl debug messages go to stderr, not stdout, so you have to use "2>" >> instead of ">" in order to do the redirection. Setting up a ssh server >> on the Dom0 and using it remotely will also help ;). >> > Still nothing was sent to a file for 2>. Here is my xl dmesg: Then you will have to check how to redirect stderr to a file with the shell you are using. It works fine for me when using bash. # xl -vvv create freebsd_nodm.cfg 2> test.out # cat test.out Parsing config from freebsd_nodm.cfg libxl: debug: libxl_create.c:1575:do_domain_create: ao 0x1c7a8e0: create: how=(nil) callback=(nil) poller=0x1c6fb20 libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=unknown libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk vdev=hda, using backend phy [...] > (XEN) ERROR: 16550-compatible serial UART not present > Xen 4.5.1 > (XEN) Xen version 4.5.1 (root@) (gcc48 (FreeBSD Ports Collection) 4.8.5) > debug=n Sat Oct 24 18:29:18 UTC 2015 > (XEN) Latest ChangeSet: > (XEN) Bootloader: FreeBSD Loader > (XEN) Command line: dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 > guest_loglvl=all loglvl=all > (XEN) Video information: > (XEN) VGA is text mode 80x25, font 8x16 > (XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds > (XEN) Disc information: > (XEN) Found 1 MBR signatures > (XEN) Found 1 EDD information structures > (XEN) Xen-e820 RAM map: > (XEN) 0000000000000000 - 000000000009d000 (usable) > (XEN) 000000000009d000 - 00000000000a0000 (reserved) > (XEN) 00000000000e0000 - 0000000000100000 (reserved) > (XEN) 0000000000100000 - 000000006a39b000 (usable) > (XEN) 000000006a39b000 - 000000007cd2f000 (reserved) > (XEN) 000000007cd2f000 - 000000007ce7f000 (ACPI NVS) > (XEN) 000000007ce7f000 - 000000007ceff000 (ACPI data) > (XEN) 000000007ceff000 - 000000007fa00000 (reserved) > (XEN) 00000000b8000000 - 00000000fc000000 (reserved) > (XEN) 00000000fec00000 - 00000000fec01000 (reserved) > (XEN) 00000000fed08000 - 00000000fed09000 (reserved) > (XEN) 00000000fed10000 - 00000000fed1a000 (reserved) > (XEN) 00000000fed1c000 - 00000000fed20000 (reserved) > (XEN) 00000000fee00000 - 00000000fee01000 (reserved) > (XEN) 00000000ffc00000 - 0000000100000000 (reserved) > (XEN) 0000000100000000 - 000000027e600000 (usable) > (XEN) ACPI: RSDP 000F0120, 0024 (r2 LENOVO) > (XEN) ACPI: XSDT 7CEFE170, 00EC (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: FACP 7CEF8000, 010C (r5 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: DSDT 7CEE0000, 120D4 (r1 LENOVO TP-GN 2100 INTL 20120711) > (XEN) ACPI: FACS 7CE42000, 0040 > (XEN) ACPI: SLIC 7CEFD000, 0176 (r1 LENOVO TP-GN 2100 PTEC 1) > (XEN) ACPI: DBGP 7CEFB000, 0034 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: ECDT 7CEFA000, 0052 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: HPET 7CEF7000, 0038 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: APIC 7CEF6000, 0098 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: MCFG 7CEF5000, 003C (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: SSDT 7CEF4000, 0033 (r1 LENOVO TP-SSDT1 100 INTL 20120711) > (XEN) ACPI: SSDT 7CEF3000, 044F (r1 LENOVO TP-SSDT2 200 INTL 20120711) > (XEN) ACPI: SSDT 7CEDF000, 0B78 (r1 LENOVO SataAhci 1000 INTL 20120711) > (XEN) ACPI: SSDT 7CEDE000, 07E7 (r1 LENOVO Cpu0Ist 3000 INTL 20120711) > (XEN) ACPI: SSDT 7CEDD000, 0AD8 (r1 LENOVO CpuPm 3000 INTL 20120711) > (XEN) ACPI: SSDT 7CEDB000, 1215 (r1 LENOVO SaSsdt 3000 INTL 20120711) > (XEN) ACPI: SSDT 7CEDA000, 0379 (r1 LENOVO CppcTabl 1000 INTL 20120711) > (XEN) ACPI: PCCT 7CED9000, 006E (r5 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: SSDT 7CED8000, 0AC4 (r1 LENOVO Cpc_Tabl 1000 INTL 20120711) > (XEN) ACPI: TCPA 7CED7000, 0032 (r2 PTL LENOVO 6040000 LNVO 1) > (XEN) ACPI: UEFI 7CED6000, 0042 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: MSDM 7CDAA000, 0055 (r3 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: ASF! 7CEFC000, 00A5 (r32 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: BATB 7CED5000, 0046 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: FPDT 7CED4000, 0064 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: UEFI 7CED3000, 02E2 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) ACPI: SSDT 7CED2000, 047F (r1 LENOVO IsctTabl 1000 INTL 20120711) > (XEN) ACPI: DMAR 7CED1000, 00B8 (r1 LENOVO TP-GN 2100 PTEC 2) > (XEN) System RAM: 7817MB (8004832kB) > (XEN) No NUMA configuration found > (XEN) Faking a node at 0000000000000000-000000027e600000 > (XEN) Domain heap initialised > (XEN) found SMP MP-table at 000f0100 > (XEN) DMI 2.7 present. > (XEN) Using APIC driver default > (XEN) ACPI: PM-Timer IO Port: 0x1808 > (XEN) ACPI: v5 SLEEP INFO: control[1:0], status[1:0] > (XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1804,1:0], pm1x_evt[1:1800,1:0] > (XEN) ACPI: 32/64X FACS address mismatch in FADT - 7ce42000/0000000000000000, > using 32 > (XEN) ACPI: wakeup_vec[7ce4200c], vec_size[20] > (XEN) ACPI: Local APIC address 0xfee00000 > (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) > (XEN) Processor #0 7:12 APIC version 21 > (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) > (XEN) Processor #1 7:12 APIC version 21 > (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) > (XEN) Processor #2 7:12 APIC version 21 > (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) > (XEN) Processor #3 7:12 APIC version 21 > (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled) > (XEN) Processor #4 7:12 APIC version 21 > (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled) > (XEN) Processor #5 7:12 APIC version 21 > (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] enabled) > (XEN) Processor #6 7:12 APIC version 21 > (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled) > (XEN) Processor #7 7:12 APIC version 21 > (XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) > (XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) > (XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) > (XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23 > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) > (XEN) ACPI: IRQ0 used by override. > (XEN) ACPI: IRQ2 used by override. > (XEN) ACPI: IRQ9 used by override. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000 > (XEN) ERST table was not found > (XEN) Using ACPI (MADT) for SMP configuration information > (XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs) > (XEN) IRQ limits: 24 GSI, 1528 MSI/MSI-X > (XEN) Switched to APIC driver x2apic_cluster. > (XEN) Using scheduler: SMP Credit Scheduler (credit) > (XEN) Detected 2693.807 MHz processor. > (XEN) Initing memory sharing. > (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7 > (XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 > extended MCE MSR 0 > (XEN) Intel machine check reporting enabled > (XEN) alt table ffff82d0802b6cf0 -> ffff82d0802b7e60 > (XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f > (XEN) PCI: MCFG area at f8000000 reserved in E820 > (XEN) PCI: Using MCFG for segment 0000 bus 00-3f > (XEN) Intel VT-d iommu 0 supported page sizes: 4kB. > (XEN) Intel VT-d iommu 1 supported page sizes: 4kB. > (XEN) Intel VT-d Snoop Control not enabled. > (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. > (XEN) Intel VT-d Queued Invalidation enabled. > (XEN) Intel VT-d Interrupt Remapping enabled. > (XEN) Intel VT-d Shared EPT tables not enabled. > (XEN) I/O virtualisation enabled > (XEN) - Dom0 mode: Relaxed > (XEN) Interrupt remapping enabled > (XEN) Enabled directed EOI with ioapic_ack_old on! > (XEN) ENABLING IO-APIC IRQs > (XEN) -> Using old ACK method > (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 > (XEN) TSC deadline timer enabled > (XEN) Platform timer is 14.318MHz HPET > (XEN) Allocated console ring of 64 KiB. > (XEN) mwait-idle: MWAIT substates: 0x42120 > (XEN) mwait-idle: v0.4 model 0x3c > (XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff > (XEN) VMX: Supported advanced features: > (XEN) - APIC MMIO access virtualisation > (XEN) - APIC TPR shadow > (XEN) - Extended Page Tables (EPT) > (XEN) - Virtual-Processor Identifiers (VPID) > (XEN) - Virtual NMI > (XEN) - MSR direct-access bitmap > (XEN) - Unrestricted Guest > (XEN) - VMCS shadowing > (XEN) HVM: ASIDs enabled. > (XEN) HVM: VMX enabled > (XEN) HVM: Hardware Assisted Paging (HAP) detected > (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB > (XEN) Brought up 8 CPUs > (XEN) ACPI sleep modes: S3 > (XEN) mcheck_poll: Machine check polling timer started. > (XEN) Dom0 has maximum 792 PIRQs > (XEN) *** LOADING DOMAIN 0 *** > (XEN) Xen kernel: 64-bit, lsb, compat32 > (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x200000 -> 0x1da6878 > (XEN) Dom0 symbol map 0x1da6878 -> 0x20d6830 > (XEN) PHYSICAL MEMORY ARRANGEMENT: > (XEN) Dom0 alloc.: 0000000270000000->0000000274000000 (507901 pages to be > allocated) > (XEN) Init. ramdisk: 000000027e5fd000->000000027e600000 > (XEN) VIRTUAL MEMORY ARRANGEMENT: > (XEN) Loaded kernel: ffffffff80200000->ffffffff820d6830 > (XEN) Init. ramdisk: ffffffff820d7000->ffffffff820da000 > (XEN) Phys-Mach map: ffffffff820da000->ffffffff824da000 > (XEN) Start info: ffffffff824da000->ffffffff824db4b4 > (XEN) Page tables: ffffffff824dc000->ffffffff824f3000 > (XEN) Boot stack: ffffffff824f3000->ffffffff824f4000 > (XEN) TOTAL: ffffffff80000000->ffffffff82800000 > (XEN) ENTRY ADDRESS: ffffffff80f09000 > (XEN) Dom0 has maximum 4 VCPUs > (XEN) Masked UR signaling on 0000:00:00.0 > (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs > (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:1d.0] fault addr > 7af37000, iommu reg = ffff82c000203000 > (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set > (XEN) ....................done. > (XEN) Initial low memory virq threshold set at 0x4000 pages. > (XEN) Std. Loglevel: All > (XEN) Guest Loglevel: All > (XEN) Xen is relinquishing VGA console. > (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to > Xen) > (XEN) Freed 304kB init memory. > (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:14.0] fault addr > 7aec4000, iommu reg = ffff82c000203000 > (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set > (XEN) irq.c:380: Dom0 callback via changed to Direct Vector 0x93 > (XEN) Masked UR signaling on 0000:00:00.0 > (XEN) PCI add device 0000:00:00.0 > (XEN) PCI add device 0000:00:01.0 > (XEN) PCI add device 0000:00:02.0 > (XEN) PCI add device 0000:00:03.0 > (XEN) PCI add device 0000:00:14.0 > (XEN) PCI add device 0000:00:16.0 > (XEN) PCI add device 0000:00:16.3 > (XEN) PCI add device 0000:00:19.0 > (XEN) PCI add device 0000:00:1a.0 > (XEN) PCI add device 0000:00:1b.0 > (XEN) PCI add device 0000:00:1c.0 > (XEN) PCI add device 0000:00:1c.1 > (XEN) PCI add device 0000:00:1c.2 > (XEN) PCI add device 0000:00:1c.4 > (XEN) PCI add device 0000:00:1d.0 > (XEN) PCI add device 0000:00:1f.0 > (XEN) PCI add device 0000:00:1f.2 > (XEN) PCI add device 0000:00:1f.3 > (XEN) PCI add device 0000:01:00.0 > (XEN) PCI add device 0000:02:00.0 > (XEN) PCI add device 0000:03:00.0 > (d1) HVM Loader > (d1) Detected Xen v4.5.1 > (d1) Xenbus rings @0xfeffc000, event channel 1 > (d1) System requested SeaBIOS > (d1) CPU speed is 2694 MHz > (d1) Relocating guest memory for lowmem MMIO space disabled That's weird, you should generally see more output here, this is what I typically see: (d451) HVM Loader (d451) Detected Xen v4.7-unstable (d451) Xenbus rings @0xfeffc000, event channel 1 (d451) System requested SeaBIOS (d451) CPU speed is 2095 MHz (d451) Relocating guest memory for lowmem MMIO space disabled (XEN) irq.c:275: Dom451 PCI link 0 changed 0 -> 5 (d451) PCI-ISA link 0 routed to IRQ5 (XEN) irq.c:275: Dom451 PCI link 1 changed 0 -> 10 (d451) PCI-ISA link 1 routed to IRQ10 (XEN) irq.c:275: Dom451 PCI link 2 changed 0 -> 11 (d451) PCI-ISA link 2 routed to IRQ11 (XEN) irq.c:275: Dom451 PCI link 3 changed 0 -> 5 (d451) PCI-ISA link 3 routed to IRQ5 (d451) pci dev 01:3 INTA->IRQ10 (d451) pci dev 02:0 INTA->IRQ11 (d451) pci dev 04:0 INTA->IRQ5 (d451) RAM in high memory; setting high_mem resource base to 186800000 (d451) pci dev 03:0 bar 10 size 002000000: 0f0000008 (d451) pci dev 02:0 bar 14 size 001000000: 0f2000008 (d451) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d451) pci dev 03:0 bar 30 size 000010000: 0f3040000 (d451) pci dev 03:0 bar 14 size 000001000: 0f3050000 (d451) pci dev 02:0 bar 10 size 000000100: 00000c001 (d451) pci dev 04:0 bar 10 size 000000100: 00000c101 (d451) pci dev 04:0 bar 14 size 000000100: 0f3051000 (d451) pci dev 01:1 bar 20 size 000000010: 00000c201 [...] > I see the error about the UART here is the section of my device probes during > the boot: Yes, if you want to get rid of that just change the Xen command line to: "dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 guest_loglvl=all loglvl=all" > Nov 6 09:12:40 xenserver kernel: Copyright (c) 1992-2015 The FreeBSD > Project. > Nov 6 09:12:40 xenserver kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, > 1989, 1991, 1992, 1993, 1994 > Nov 6 09:12:40 xenserver kernel: The Regents of the University of > California. All rights reserved. > Nov 6 09:12:40 xenserver kernel: FreeBSD is a registered trademark of The > FreeBSD Foundation. > Nov 6 09:12:40 xenserver kernel: FreeBSD 11.0-CURRENT #0: Wed Nov 4 > 14:39:53 EST 2015 > Nov 6 09:12:40 xenserver kernel: > root@xenserver:/usr/obj/usr/src/sys/GENERIC-NODEBUG amd64 > Nov 6 09:12:40 xenserver kernel: FreeBSD clang version 3.7.0 > (tags/RELEASE_370/final 246257) 20150906 > Nov 6 09:12:40 xenserver kernel: VT(vga): resolution 640x480 > Nov 6 09:12:40 xenserver kernel: CPU: Intel(R) Core(TM) i7-4800MQ CPU @ > 2.70GHz (2693.76-MHz K8-class CPU) > Nov 6 09:12:40 xenserver kernel: Origin="GenuineIntel" Id=0x306c3 > Family=0x6 Model=0x3c Stepping=3 > Nov 6 09:12:40 xenserver kernel: > Features=0x1fc3ebff AT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT> > Nov 6 09:12:40 xenserver kernel: > Features2=0xfff83a83 IC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> > Nov 6 09:12:40 xenserver kernel: AMD Features=0x20100800 > Nov 6 09:12:40 xenserver kernel: AMD Features2=0x21 > Nov 6 09:12:40 xenserver kernel: Structured Extended > Features=0xb38 > Nov 6 09:12:40 xenserver kernel: XSAVE Features=0x1 > Nov 6 09:12:40 xenserver kernel: TSC: P-state invariant, performance > statistics > Nov 6 09:12:40 xenserver kernel: Hypervisor: Origin = "XenVMMXenVMM" > Nov 6 09:12:40 xenserver kernel: real memory = 4660690944 (4444 MB) > Nov 6 09:12:40 xenserver kernel: avail memory = 1970360320 (1879 MB) > Nov 6 09:12:40 xenserver kernel: FreeBSD/SMP: Multiprocessor System > Detected: 4 CPUs > Nov 6 09:12:40 xenserver kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) > Nov 6 09:12:40 xenserver kernel: cpu0 (BSP): APIC ID: 0 > Nov 6 09:12:40 xenserver kernel: cpu1 (AP): APIC ID: 2 > Nov 6 09:12:40 xenserver kernel: cpu2 (AP): APIC ID: 4 > Nov 6 09:12:40 xenserver kernel: cpu3 (AP): APIC ID: 6 > Nov 6 09:12:40 xenserver kernel: uart2: > port 0x50b0-0x50b7 mem 0xb2a40000-0xb2a40fff irq 17 at device 22.3 on pci0 > >> Can you also paste the output of `xl dmesg` after trying to launch the >> guest? >> > My FreeBSD guest file: > > builder = "hvm" > memory = 512 > vcpus = 1 > name = "FreeBSD" > disk = [ > '/root/xen/freebsd.img,raw,hda,rw', > '/root/bhyve/freebsd11.iso,raw,hdc:cdrom,r' > ] > #boot = "c" #boot to hard drive image > boot = "d" #boot to ISO image > vnc = 1 > vnclisten = "0.0.0.0" Config file looks fine (you don't have any nic added, but I guess that's on purpose). >> Which revision of FreeBSD are you running in the Dom0? >> > FreeBSD current that was built yesterday from source. Can you make sure you are using a version of FreeBSD equal or greater than r290392? This is a fix for a bug that could cause guests to stop booting. > I am a contract programmer working on an off-shore natural gas platform > project. I have 5 computers running at home in my office and was thinking > about making all of them guests on a single hypervisor as a winter project. > I brought along a small ITX sized PC for to help me get acquaited with > hypervisors. It's CPU is missing Intel VT-d instructions, so I went to the > local Best Buy store and bought a SSD upgrade kit for this company laptop. > The laptop has VT-d features and is an i7. It is running on a spare > partition but I will need to go back to the original hard drive soon because > the Windows OS is complaining about needing re-activation. I just work on my > hypervisor upgrade in my hotel room in the evenings and on the days that the > helicopter doesn't fly because of weather. There isn't any way to grant > remote SSH access until I get back home in January. FWIW, I also tent to do development on a small box when possible (I don't like to waste power). What I use is a NUC5i3MYHE which has room for a serial bracket (that's the only suitable small x86 box I've found that comes with a serial bracket option). With 8GB of memory and a hard drive it's price should be around 400-450€. Roger. From owner-freebsd-xen@freebsd.org Sat Nov 7 01:56:00 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54AA8A258CC for ; Sat, 7 Nov 2015 01:56:00 +0000 (UTC) (envelope-from lausts@acm.org) Received: from cdptpa-oedge-vip.email.rr.com (cdptpa-outbound-snat.email.rr.com [107.14.166.226]) by mx1.freebsd.org (Postfix) with ESMTP id 17D8A1325 for ; Sat, 7 Nov 2015 01:55:59 +0000 (UTC) (envelope-from lausts@acm.org) Received: from [173.88.10.122] ([173.88.10.122:25137] helo=mail.laus.org) by cdptpa-oedge02 (envelope-from ) (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP id 33/44-08703-82A5D365; Sat, 07 Nov 2015 01:55:53 +0000 Received: from [192.168.1.100] (laust2 [192.168.1.100]) by mail.laus.org (8.15.2/8.15.2) with ESMTPS id tA71tqut010205 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 6 Nov 2015 20:55:52 -0500 (EST) (envelope-from lausts@acm.org) X-Authentication-Warning: mail.laus.org: Host laust2 [192.168.1.100] claimed to be [192.168.1.100] From: "Thomas Laus" Organization: ABB To: Roger Pau Monné Date: Fri, 06 Nov 2015 20:55:47 -0500 Subject: Re: Does a Xen Dom0 require X to function - WORKS NOW Reply-to: lausts@acm.org CC: freebsd-xen@freebsd.org Message-ID: <563D5A23.25666.B1515@lausts.acm.org> Priority: normal In-reply-to: <563CDEB0.5010405@citrix.com> References: <5633A2B0.22403.45280A@lausts.acm.org>, <563CC5CC.19454.173726@lausts.acm.org>, <563CDEB0.5010405@citrix.com> X-mailer: Pegasus Mail for Windows (4.70) X-RR-Connecting-IP: 107.14.168.130:25 X-Cloudmark-Score: 0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2015 01:56:00 -0000 > # xl -vvv create freebsd_nodm.cfg 2> test.out > # cat test.out > Parsing config from freebsd_nodm.cfg > libxl: debug: libxl_create.c:1575:do_domain_create: ao 0x1c7a8e0: > create: how=(nil) callback=(nil) poller=0x1c6fb20 > libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk > vdev=hda spec.backend=unknown > libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk > vdev=hda, using backend phy > [...] > Paul: I made a temporary change to the FreeBSD default 'csh' shell for the root account and captured the 'xl create' file: Parsing config from freebsd.cfg libxl: debug: libxl_create.c:1504:do_domain_create: ao 0x802849060: create: how=0x0 callback=0x0 poller=0x802824060 libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=unknown libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk vdev=hda, using backend phy libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=hdc spec.backend=unknown libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk vdev=hdc, using backend phy libxl: debug: libxl_create.c:907:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:323:libxl__bootloader_run: not a PV domain, skipping bootloader libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x802858770: deregister unregistered xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x59b0c xc: detail: elf_parse_binary: memory: 0x100000 -> 0x159b0c xc: detail: VIRTUAL MEMORY ARRANGEMENT: xc: detail: Loader: 0000000000100000->0000000000159b0c xc: detail: Modules: 0000000000000000->0000000000000000 xc: detail: TOTAL: 0000000000000000->000000001f800000 xc: detail: ENTRY: 0000000000100000 xc: detail: PHYSICAL MEMORY ALLOCATION: xc: detail: 4KB PAGES: 0x0000000000000200 xc: detail: 2MB PAGES: 0x00000000000000fb xc: detail: 1GB PAGES: 0x0000000000000000 xc: detail: elf_load_binary: phdr 0 at 0x80067a000 -> 0x8006caab1 domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000 libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=phy libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch w=0x802b9e208 wpath=/local/domain/0/backend/vbd/2/768/state token=3/0: register slotnum=3 libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=hdc spec.backend=phy libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=hdc spec.backend=phy libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch w=0x802b9e3c8 wpath=/local/domain/0/backend/vbd/2/5632/state token=2/1: register slotnum=2 libxl: debug: libxl_create.c:1520:do_domain_create: ao 0x802849060: inprogress: poller=0x802824060, flags=i libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x802b9e208 wpath=/local/domain/0/backend/vbd/2/768/state token=3/0: event epath=/local/domain/0/backend/vbd/2/768/state libxl: debug: libxl_event.c:830:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/768/state wanted state 2 ok libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch w=0x802b9e208 wpath=/local/domain/0/backend/vbd/2/768/state token=3/0: deregister slotnum=3 libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x802b9e208: deregister unregistered libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x802b9e290: deregister unregistered libxl: debug: libxl_event.c:483:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/768/state token=3/0: empty slot libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x802b9e3c8 wpath=/local/domain/0/backend/vbd/2/5632/state token=2/1: event epath=/local/domain/0/backend/vbd/2/5632/state libxl: debug: libxl_event.c:834:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/5632/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x802b9e3c8 wpath=/local/domain/0/backend/vbd/2/5632/state token=2/1: event epath=/local/domain/0/backend/vbd/2/5632/state libxl: debug: libxl_event.c:830:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/5632/state wanted state 2 ok libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch w=0x802b9e3c8 wpath=/local/domain/0/backend/vbd/2/5632/state token=2/1: deregister slotnum=2 libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x802b9e3c8: deregister unregistered libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x802b9e450: deregister unregistered libxl: debug: libxl_dm.c:1435:libxl__spawn_local_dm: Spawning device-model /usr/local/lib/xen/bin/qemu-system-i386 with arguments: libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: /usr/local/lib/xen/bin/qemu-system-i386 libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -xen-domid libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: 2 libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -chardev libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-2,server,nowait libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -mon libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: chardev=libxl-cmd,mode=control libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -nodefaults libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -name libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: FreeBSD libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -vnc libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: 0.0.0.0:0,to=99 libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -display libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -device libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: cirrus-vga,vgamem_mb=8 libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -boot libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: order=d libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -net libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: none libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -machine libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: xenfv libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -m libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: 504 libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -drive libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: file=/root/xen/freebsd.img,if=ide,index=0,media=disk,format=raw,cache=writebac k libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: -drive libxl: debug: libxl_dm.c:1437:libxl__spawn_local_dm: file=/root/bhyve/freebsd11.iso,if=ide,index=2,readonly=on,media=cdrom,format=r aw,cache=writeback,id=ide-5632 libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch w=0x8028589d0 wpath=/local/domain/0/device-model/2/state token=2/2: register slotnum=2 libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x8028589d0 wpath=/local/domain/0/device-model/2/state token=2/2: event epath=/local/domain/0/device-model/2/state libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x8028589d0 wpath=/local/domain/0/device-model/2/state token=2/2: event epath=/local/domain/0/device-model/2/state libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch w=0x8028589d0 wpath=/local/domain/0/device-model/2/state token=2/2: deregister slotnum=2 libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch w=0x8028589d0: deregister unregistered libxl: debug: libxl_qmp.c:691:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-2 libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp libxl: debug: libxl_qmp.c:541:qmp_send_prepare: next qmp command: '{ "execute": "qmp_capabilities", "id": 1 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:541:qmp_send_prepare: next qmp command: '{ "execute": "query-chardev", "id": 2 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_qmp.c:541:qmp_send_prepare: next qmp command: '{ "execute": "query-vnc", "id": 3 } ' libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return libxl: debug: libxl_event.c:1941:libxl__ao_progress_report: ao 0x802849060: progress report: ignored libxl: debug: libxl_event.c:1765:libxl__ao_complete: ao 0x802849060: complete, rc=0 libxl: debug: libxl_event.c:1737:libxl__ao__destroy: ao 0x802849060: destroy xc: debug: hypercall buffer: total allocations:620 total releases:620 xc: debug: hypercall buffer: current allocations:0 maximum allocations:4 xc: debug: hypercall buffer: cache current size:4 xc: debug: hypercall buffer: cache hits:607 misses:4 toobig:9 I also upgraded my FreeBSD-CURRENT to r290462 as well as upgraded the xen port to the most recent. After all of that, everything appears to work and I can install both an OpenBSD and a FreeBSD instance. Both of them boot OK and are fully functional. Thank you for all of the help in pointing me down the right path. Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF