From owner-freebsd-xen@FreeBSD.ORG Mon Dec 15 01:26:39 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CE9D9618 for ; Mon, 15 Dec 2014 01:26:39 +0000 (UTC) Received: from mail-wg0-x235.google.com (mail-wg0-x235.google.com [IPv6:2a00:1450:400c:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48C90BC7 for ; Mon, 15 Dec 2014 01:26:39 +0000 (UTC) Received: by mail-wg0-f53.google.com with SMTP id l18so13135187wgh.26 for ; Sun, 14 Dec 2014 17:26:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=hS1/DmSGoqnyNe/Q+pSOJAvhMIyncvWvXtQb8P4AObo=; b=ImKTtCklEyfSWeWT7IUAaxnMp1CnYWaQ86wMFBed7mt7GSNFfARKJdytdp7u0918is a5C3SZSIoQFGowa2AO1ivu5XfsxQTdhzKlAM/h6n44VqK7Z7cM637X9m5s1w+XBjgRMJ w4/oq//6WJhR0qVsV7Gr2VYVGwvBeB3y2C9ofrcNOIBYPNEPhzSwWorER6VzRf9egEe8 JH+/vFYad3qYeLEPHurcxoKPQ1v+OBIDgBudrHzt0o9okxcRgi4mhvJdfBitw/yhLBu/ my8jXyXYsVoFj9afRputTezH9cKaT0YMqlAo2PNC2fD4MHPxQZWwMTb6WB7vbUcjV/zo qDmg== X-Received: by 10.180.218.74 with SMTP id pe10mr27311354wic.48.1418606797662; Sun, 14 Dec 2014 17:26:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.194.16.98 with HTTP; Sun, 14 Dec 2014 17:26:17 -0800 (PST) In-Reply-To: References: <481F7D02-BFE9-4E35-A475-5A8A05A801CE@dpdtech.com> <547DFCC0.6030003@citrix.com> <547F1476.8080305@citrix.com> <29437DB9-7DC8-47A8-8FC4-2BFDE736B5BC@dpdtech.com> <547F59B4.1010105@citrix.com> <2DDCAA68-3B11-4E3C-AE61-EAD8CEEF1E2D@dpdtech.com> <547F6AC1.9060709@citrix.com> <7C356027-01D8-4800-B211-282566BC9871@dpdtech.com> <54801F77.5050700@citrix.com> <54819D6A.4050603@citrix.com> <7F90454D-16D5-4871-A155-5ADD01014976@dpdtech.com> <5486B287.7070302@citrix.com> <55E2C419-E537-4324-B7AB-A21E94863E17@dpdtech.com> From: Miguel Clara Date: Mon, 15 Dec 2014 01:26:17 +0000 Message-ID: Subject: Re: Attempting to Get Xen FreeBSD Dom0 working To: Miguel Clara Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2014 01:26:39 -0000 On Mon, Dec 15, 2014 at 1:24 AM, Miguel Clara wrote: > > > On Mon, Dec 15, 2014 at 12:30 AM, Miguel Clara > wrote: >> >> >> On Sun, Dec 14, 2014 at 11:34 PM, David P. Discher >> wrote: >>> >>> First make sure you CPU supports both EPT and has a IOMMU. >>> >> >> Yes it does. >> >> I should note theat I've already tested with Xen in this laptop using >> Debian. >> >> >>> >>> It sounds like you have Xen to use the serial console. >>> >>> To /etc/ttys, you need to add: >>> >>> xc0 "/usr/libexec/getty Pc" vt100 on secure >>> >>> Also, make sure all the ttyu[0-4] are off. Recent versions of FreeBSD >>> maybe putting onifconsole =E2=80=A6 and I can't remember all the combin= ations I >>> used, but I had a lot of issues with Xen and FreeBSD fighting for the >>> serial ports. >>> >> >> Those this make sense: >> [miguelc@hpbsd]~% cat /etc/ttys|eg 'ttyuv|xc0' >> xc0 "/usr/libexec/getty Pc" vt100 on secure >> [miguelc@hpbsd]~% cat /etc/ttys | eg 'ttyu|xc0' >> ttyu0 "/usr/libexec/getty 3wire" vt100 onifconsole secure >> ttyu1 "/usr/libexec/getty std.9600" dialup off secure >> ttyu2 "/usr/libexec/getty std.9600" dialup off secure >> ttyu3 "/usr/libexec/getty std.9600" dialup off secure >> xc0 "/usr/libexec/getty Pc" vt100 on secure >> >> >> (I've just added xc0 following the guide, not sure if ttyu0 onifconsole >> is correct or should just be off) >> >>> >>> Also, make sure /boot/loader.conf is configured for console=3D=E2=80=9C= vidconsole" >>> on the freebsd side, and that the xen_cmdline should have =E2=80=9Ccons= ole=3Dvga=E2=80=9D and >>> not =3Dcom1. It=E2=80=99s not explicit, but the Xen kernel creates the= =E2=80=9Cconsole=E2=80=9D. >>> The xc0 device (and Xen itself) seems to take care of piping the Dom0 >>> console over to whatever Xen is using as console. >>> >> >> Oh wait... com1 and vga ... I copy pasted from the guide and left it to >> "com1".. this must be the issue, let me retry and I'll post some feedbac= k. >> >>> >>> >>> Other tips, if you are running ZFS, you will probably need to add >>> "vm.max_wired=3D-1=E2=80=9D to /etc/sysctl.conf (I=E2=80=99m actually n= ot sure this is valid, >>> but if you don=E2=80=99t you=E2=80=99ll run out of wired memory and all= the =E2=80=9Cxl=E2=80=9D commands >>> will fail. Or limit the size of ARC. ) >>> >> >> My arc_max is limitted to 2G atm but in any case I've set the >> vm.max_wired to -1 and as you say we will see. >> >> >>> >>> You might need to turn off MSI interrupts on AHCI, "hint.ahci.0.msi=3D0= =E2=80=9D >>> in /boot/loader.conf. However, try both ways (default I believe is =3D1= ). >>> I=E2=80=99m running Intel ICH10 and have to disable MSI. Roger has ICH= 8 and >>> doesn=E2=80=99t seem to have the issue. >>> >>> Do I have a way to see what ICH is it in freebsd... (I can ofc lookit u= p >> online) its a I7 so I think its ICH10, but not sure. >> >> >>> The latest issue is very poor network performance (with Intel 82574L) >>> from the DomU (guests) over the bridge and to the network. However, Gu= ests >>> -> Dom0 seem ok. >>> >>> >>> Console issue solved, however I'm getting a panic message with > > Presently, iommu must be enabled for pvh > > Looking the model up online I see: > Intel=C2=AE Virtualization Technology (VT-x) =E2=80=A1 Yes > Intel=C2=AE Virtualization Technology for Directed I/O (VT-d) =E2=80=A1 N= o > Intel=C2=AE VT-x with Extended Page Tables (EPT) =E2=80=A1 Yes > > So I guess it does have EPT and vtx but not vtd, so I don't have ioemu :( > *IOMMU > > > - >>> David P. Discher >>> http://davidpdischer.com/ >>> AIM: DavidDPD | Y!M: daviddpdz >>> >>> >>> >>> On Dec 13, 2014, at 10:51 PM, Miguel Clara >>> wrote: >>> >>> > I was just trying too boot Freebsd Xen dom0 on a laptop but I just >>> get a black screen after the boot process.... >>> > >>> > any idea what it could be? >>> > >>> > the system boots fine wihtout loading "boot/xen", I'm not sure how to >>> get more info with the black scren! >>> > >>> > thanks >>> > >>> > >>> > Melhores Cumprimentos // Best Regards >>> > ----------------------------------------------- >>> > Miguel Clara >>> > IT - Sys Admin & Developer >>> > E-mail: miguelmclara@gmail.com >>> > www.linkedin.com/in/miguelmclara/ >>> > >>> > On Tue, Dec 9, 2014 at 7:17 PM, David P. Discher >>> wrote: >>> > ah, sorry missed that. Will try that today. >>> > >>> > AHCI lasted over night with MSI off. Something I noticed, is that >>> when the AHCI bus was timing out, it looked like the Xen Kernel was >>> re-scanning the PCI bus. (Sorry, didn=E2=80=99t save these logs). I=E2= =80=99ve love to dig >>> into this further. >>> > >>> > Please let me know what/where to add some debugging code. >>> > >>> > - >>> > David P. Discher >>> > http://davidpdischer.com/ >>> > AIM: DavidDPD | Y!M: daviddpdz >>> > Mobile: 408.368.3725 >>> > >>> > >>> > >>> > On Dec 9, 2014, at 12:27 AM, Roger Pau Monn=C3=A9 >>> wrote: >>> > >>> > > Hello, >>> > > >>> > > El 08/12/14 a les 23.45, David P. Discher ha escrit: >>> > > >>> > >> >>> > >> Sent SIGTERM to all processes >>> =E2=94=82 >>> > >> Sent SIGKILL to all >>> processes=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=98 >>> > >> Requesting system reboot >>> > >> [ 1157.299205] Restarting system. >>> > >> root@borg:/zdata/debian # >>> > >> root@borg:/zdata/debian # >>> > >> root@borg:/zdata/debian # xl create -c debian.cfg >>> > >> root@borg:/zdata/debian # xl destroy debian >>> > >> xc: error: Could not bounce buffer for version hypercall (35 = =3D >>> Resource temporarily unavailabl): Internal error >>> > >> xc: error: Could not bounce buffer for version hypercall (35 = =3D >>> Resource temporarily unavailabl): Internal error >>> > >> xc: error: Could not bounce buffer for version hypercall (35 = =3D >>> Resource temporarily unavailabl): Internal error >>> > >> xc: error: Could not bounce buffer for version hypercall (35 = =3D >>> Resource temporarily unavailabl): Internal error >>> > >> xc: error: Could not bounce buffer for version hypercall (35 = =3D >>> Resource temporarily unavailabl): Internal error >>> > >> xc: error: Could not bounce buffer for version hypercall (35 = =3D >>> Resource temporarily unavailabl): Internal error >>> > >> libxl: error: libxl.c:658:libxl_list_domain: getting domain >>> info list: Resource temporarily unavailable >>> > >> debian is an invalid domain identifier (rc=3D-5) >>> > >> root@borg:/zdata/debian # >>> > >> >>> > >> I=E2=80=99m running AHCI with MSI off in the FreeBSD kernel, and s= o far, so >>> good on that front. The great thing is now I got the Xen console worki= ng, >>> so can get the debug output. However the bounce buffer/hypercall issu= e i >>> would think is far more important than MSI interrupts at the monument. >>> > > >>> > > Glad to know you got it working at the end! I've already pointed th= is >>> > > out in my last email, but did you try to increase vm.max_wired even >>> further? >>> > > >>> > > This usually happens when mlock in >>> > > freebsd_privcmd_alloc_hypercall_buffer (xc_freebsd_osdep.c) fails t= o >>> > > wire down the memory that would be used by the hypercalls. >>> > > >>> > > Roger. >>> > > >>> > >>> >>>