From owner-freebsd-xen@FreeBSD.ORG Mon Dec 8 22:45:18 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 98BA14EA for ; Mon, 8 Dec 2014 22:45:18 +0000 (UTC) Received: from mail-pa0-x232.google.com (mail-pa0-x232.google.com [IPv6:2607:f8b0:400e:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B306C3A for ; Mon, 8 Dec 2014 22:45:18 +0000 (UTC) Received: by mail-pa0-f50.google.com with SMTP id bj1so6135801pad.9 for ; Mon, 08 Dec 2014 14:45:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dpdtech.com; s=google; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=XjktEFc5pfz7aSAwoYMKSr904TuV8lQDbWzjuHkMUCI=; b=cOGcVOZ0y2yvFgHiEGUapFlALjJNLimUhWAsZ9hIlrdSOTuGEj5Ofp9FjbjG6Yue97 WCO0V7ZtCX99gHsI6thMmH0aYiOpBu8egiGJwZJ1z4c8d8trbgwxbagVFIU5dAyrb+AA gJVaMfer0hEg7pG2oGfSc/SWRKqJE/OIcvJVU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=XjktEFc5pfz7aSAwoYMKSr904TuV8lQDbWzjuHkMUCI=; b=YVR6u4fruiJYhwXSZ8K8IKXSDz2n8IVHhcwTa5CPt4+TS3Scek3rg1Bnk645Bn1/7r weAQD7hulOXol7M4RaYzpa0K/xbJJm7xah8rzhCqfBnmVXourgm24uN6FNa7I+iDmdJF tqBcP/sTLLa/QpDMa7CwtKqxXx1j8avQZMw/gBjaF4IW6Ff4Oo6CfP0BeFZvrEly8Lol lhCmKjbGjA41OBz6F5g9jflwHwbMbpoeBhlbe17cGlrWxJL/EOCKn4m1Ka9Q94k0GFw3 HqjtYfuTnbN6aMRlRngrgVUOxqeW7yRa20ctGIiv9rRF5ARBNTGpq9snhRKhRFTt6S4O hlww== X-Gm-Message-State: ALoCoQnxBj7tt6trLcfJoUv08Q9hmqF0Tr4V0rGlERTu4NNNfblDqPPvHf4JE9xUrJ6qimnN88II X-Received: by 10.70.128.80 with SMTP id nm16mr25184368pdb.1.1418078717605; Mon, 08 Dec 2014 14:45:17 -0800 (PST) Received: from 173-13-188-41-sfba.hfc.comcastbusiness.net (173-13-188-41-sfba.hfc.comcastbusiness.net. [173.13.188.41]) by mx.google.com with ESMTPSA id z3sm13489450pdp.17.2014.12.08.14.45.16 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 08 Dec 2014 14:45:17 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_01F670E0-CE32-40AC-A367-DE072E2D22C9"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Attempting to Get Xen FreeBSD Dom0 working From: "David P. Discher" In-Reply-To: <54819D6A.4050603@citrix.com> Date: Mon, 8 Dec 2014 14:45:21 -0800 Message-Id: <7F90454D-16D5-4871-A155-5ADD01014976@dpdtech.com> References: <481F7D02-BFE9-4E35-A475-5A8A05A801CE@dpdtech.com> <547DFCC0.6030003@citrix.com> <547F1476.8080305@citrix.com> <29437DB9-7DC8-47A8-8FC4-2BFDE736B5BC@dpdtech.com> <547F59B4.1010105@citrix.com> <2DDCAA68-3B11-4E3C-AE61-EAD8CEEF1E2D@dpdtech.com> <547F6AC1.9060709@citrix.com> <7C356027-01D8-4800-B211-282566BC9871@dpdtech.com> <54801F77.5050700@citrix.com> <54819D6A.4050603@citrix.com> To: =?iso-8859-1?Q?Roger_Pau_Monn=E9?= X-Mailer: Apple Mail (2.1878.6) Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2014 22:45:18 -0000 --Apple-Mail=_01F670E0-CE32-40AC-A367-DE072E2D22C9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I tried all the combinations of things =E2=80=A6 and still couldn=E2=80=99= t get the xen console switch to work. I finally googled "xen X8STi = console=E2=80=9D, and found - = http://wiki.xen.org/wiki/Xen_Serial_Console - which calls out the super = micro board X8STi - with the options: com1=3D115200,8n1,0x3e8,5 I don=E2=80=99t know what that =E2=80=9C,5=E2=80=9D does =E2=80=A6 but = it fixed it. I have also disabled all the ttys in /etc/ttys for the = serial ports. (haven=E2=80=99t tried the trailing 5 with the ttyu[012] = =E2=80=9Con=E2=80=9D. ) Ok, back to it =E2=80=A6 re-started the debian install, and had = =E2=80=9Cxentop=E2=80=9D running and =E2=80=9Cxl info=E2=80=9D in a = loop, and eventually, within in minutes, the hyper call fails.=20 xc: error: Could not bounce buffer for sysctl hypercall (35 =3D = Resource temporarily unavailabl): Internal error libxl: error: libxl.c:5024:libxl_get_physinfo: getting physinfo: = Resource temporarily unavailable libxl_physinfo failed. xc: error: Could not bounce buffer for sysctl hypercall (35 =3D = Resource temporarily unavailabl): Internal error libxl: error: libxl.c:5507:libxl_get_scheduler: getting domain = info list: Resource temporarily unavailable The debian install is still going, but makes it impossible to launch the = installed system, since no call for xl will work. Sent SIGTERM to all processes = =E2=94=82 Sent SIGKILL to all = processes=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=98 Requesting system reboot [ 1157.299205] Restarting system. root@borg:/zdata/debian # root@borg:/zdata/debian # root@borg:/zdata/debian # xl create -c debian.cfg root@borg:/zdata/debian # xl destroy debian xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error libxl: error: libxl.c:658:libxl_list_domain: getting domain info = list: Resource temporarily unavailable debian is an invalid domain identifier (rc=3D-5) root@borg:/zdata/debian # I=E2=80=99m running AHCI with MSI off in the FreeBSD kernel, and so far, = so good on that front. The great thing is now I got the Xen console = working, so can get the debug output. However the bounce = buffer/hypercall issue i would think is far more important than MSI = interrupts at the monument.=20 - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz=20 On Dec 5, 2014, at 3:56 AM, Roger Pau Monn=C3=A9 = wrote: > El 04/12/14 a les 22.28, David P. Discher ha escrit: >> Update: So, I changed a few things, and stuff is working better. >>=20 >> The Xen kernel line is now: >>=20 >> dom0_mem=3D2048M dom0_max_vcpus=3D4 dom0pvh=3D1 sync_console = com1=3D115200,8n1,0x3e8 console=3Dvga,com1 iommu=3Ddebug >=20 > I would advise against sync_console, it can easily cause delays in > interrupt delivery which can cause timeouts in FreeBSD. >=20 >>=20 >> Also note, I have these set in FreeBSD: >>=20 >> console=3D"comconsole vidconsole" >> comconsole_speed=3D"115200" >> comconsole_port=3D"0x3e8" >> boot_multicons=3D"yes" >> vm.max_wired=3D2097152 >> verbose_loading=3D"YES" >> boot_verbose=3D"" # -v: Causes extra debugging information to be = printed >>=20 >> hint.ahci.0.msi=3D0 >> hw.acpi.verbose=3D1 >> debug.acpi.enable_debug_objects=3D1 >>=20 >> So far, no AHCI timeouts. I=E2=80=99v gotten completely through an = install of Debian =E2=80=A6 granted it failed, but for a linux reasons - = couldn=E2=80=99t find/download a package. But is still going. >>=20 >> The change to the console lines also help =E2=80=A6 console=3Dvga,com1 = & sync_console to xen allowed the IPMI SOL COM3 to fully complete the = boot under freebsd. And the tty/login ran and displayed on xc0 : >>=20 >> FreeBSD/amd64 (borg.dpdtech.com) (xc0) >=20 > I will try to find a system similar to yours with an IPMI SOL console > and see if I can figure out what's going on. As a test, could you try = to > disable the comconsole from FreeBSD and see if that makes a = difference? >=20 > console=3D"vidconsole" >=20 > And remove all the comconsole_* and boot_multicons options. This will > have the side effect of removing serial output from the bootloader, = but > it might prevent FreeBSD from screwing the Xen serial console. >=20 >> However, this console will not take any input. I still can=E2=80=99t = get switched into the Xen console (Ctrl-A x3) on either the serial of = VGA consoles.=20 >>=20 >> Another troubling item, em0 flaps when debian is starting up: >>=20 >> xnb(xnb_probe:1144): Claiming device 0, xnb >> xnb1.0: bpf attached >> xnb(xnb_attach:1292): Attaching to backend/vif/1/0 >> xnb(xnb_frontend_changed:1416): frontend_state=3DInitialising, = xnb_state=3DInitWait >> em0: Link is Down >> xnb1.0: 2 link states coalesced >> (d1) mapping kernel into physical memory >> (d1) about to get started... >> xnb(xnb_frontend_changed:1416): frontend_state=3DConnected, = xnb_state=3DInitWait >> xnb(xnb_connect_comms:796): rings connected! >> em0: Link is up 1000 Mbps Full Duplex >>=20 >> em0 is in bridge0, which is what the debian.cfg is using. >=20 > I certainly don't see this kind of flipping on my network card, but I > would focus on fixing the console first, so we can get debug info. >=20 >>=20 >> Also, something really odd =E2=80=A6 hyper calls aren=E2=80=99t = working after launching the debian guest - which also means I can=E2=80=99= t launch any more guests.=20 >>=20 >> root@borg:~ # xl list >> xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error >> xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error >> xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error >> xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error >> xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error >> xc: error: Could not bounce buffer for version hypercall (35 =3D = Resource temporarily unavailabl): Internal error >> libxl: error: libxl.c:658:libxl_list_domain: getting domain info = list: Resource temporarily unavailable >> libxl_list_domain failed. >=20 > It seems like you are running out of wired memory, you should increase > vm.max_wired, this can be changed at runtime with sysctl without = problems. >=20 >>=20 >> I=E2=80=99m heading out for the afternoon shortly, but it seems the = next thing to do is to get the consoles working correctly so I can get = debugging info from the hypervisor. Will hopefully bang on this this = evening.=20 >=20 > Thanks for the efforts, please keep me posted on how it goes. >=20 > Roger. >=20 --Apple-Mail=_01F670E0-CE32-40AC-A367-DE072E2D22C9 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUhioCAAoJEEmwU6XuhYWO7WsIAJbOnNA2PKi8s7LY/T+1OGs4 vkq4jsETqONfAa/ScvW6s6PqtLeiE2vliAQD2gCp+nQpGIzoOUQgH9owbB1RNQkU UmuyPDXrDbJH75s7eP/Vl35kUJGYcjhPi8gjpEAt+olqiIwqQngvipJt8vG5gqW5 hMVx7S+CaAhQHdZCjk9LPly8+5Ykk9RdBWodp+T02ZsnWjgYRR5C7OcYmPtUyC6f bP08XcZajjt5nHS6rbWwjVb4MLpeFlSipSkdnvjDuNY5DF6v6w5NTLE2S7YSFd97 nBVgzA4aheq8EpPucGmKxZsndpUzMzFDHNjRr+cTwIA+IGgClham4cexQ7N+cLk= =7T5o -----END PGP SIGNATURE----- --Apple-Mail=_01F670E0-CE32-40AC-A367-DE072E2D22C9--