From owner-freebsd-xen@freebsd.org Mon Dec 19 12:27:40 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 588F8C86A2E for ; Mon, 19 Dec 2016 12:27:40 +0000 (UTC) (envelope-from prvs=154abc353=roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.ctxuk.citrix.com [185.25.65.24]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF9541DC1 for ; Mon, 19 Dec 2016 12:27:38 +0000 (UTC) (envelope-from prvs=154abc353=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.33,373,1477958400"; d="scan'208";a="37066361" Date: Mon, 19 Dec 2016 12:27:27 +0000 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Nathan Friess CC: Subject: Re: 11-RELEASE acting as vbd backend Message-ID: <20161219122727.pd4dzo2hb2m6ld7d@MacBook-Pro-de-Roger.local> References: <247e5b6c-2713-27cf-f8fa-61e55c9e2025@gmail.com> <20161206100414.pi7ep2zbkduhuol7@dhcp-3-221.uk.xensource.com> <39e43ae4-6388-c698-c3c2-43cbc1f7b93c@gmail.com> <20161209144630.dfga5mozh72veo4g@dhcp-3-221.uk.xensource.com> <584ADC91.8050909@gmail.com> <20161212103257.tg46q2s3j5riixp5@dhcp-3-221.uk.xensource.com> <20161215173432.3qjzilzot7lnddhm@dhcp-3-221.uk.xensource.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: NeoMutt/20161126 (1.7.1) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2016 12:27:40 -0000 On Sun, Dec 18, 2016 at 12:36:32PM -0700, Nathan Friess wrote: > On 12/15/2016 10:34 AM, Roger Pau Monné wrote: > > OK, I finally have this ready, you will need the following patches for FreeBSD, > > which I pushed to my git repo on top of current HEAD: > > > > git://xenbits.xen.org/people/royger/freebsd.git add_watches_xenstore > > > > You can either compile a kernel from this branch or pickup the 4 top patches > > and apply them to your local tree (I don't think you are going to find any > > conflicts even if the tree is stable/11 or 11.0-RELEASE). > > > > You should also update your xen-tools package to the latest version, I've just > > updated it to contain the xenstore device path fix: > > > > https://svnweb.freebsd.org/ports?view=revision&revision=428628 > > > > After this is done, it should just be a matter of launching xendriverdomain at > > system boot and it should work out of the box. Let me know how this goes. > > Hi Roger, > > Thank you! With your patches I've made some progress but not quite there > yet. I've been trying to trace through xl devd but I haven't been able to > pinpoint the underlying issue. Here is where I'm at. > > I applied the 4 patches from your add_watches_xenstore branch to the > 11.0-RELEASE kernel and built xen-tools from svn. > > The xendriverdomain rc script now starts xl devd successfully. However, > when the other domU starts and the backend should be set up in FreeBSD, xl > devd attempts to run /etc/xen/scripts/block, which is in /usr/local instead. > My guess is that dom0 is generating that path and because dom0 is Linux the > paths don't match. (I'm not specifying the script in the domU config > directly.) Adding a symlink in /etc to /usr/local/etc/xen gets around this > for now. That's expected, this path is set by the Dom0, and it uses the Linux default value. It might make more sense to simply don't set it when the backend is a driver domain, but in any case you can easily solve it by using the "script" disk option[0]. You should add script=/usr/local/etc/xen/scripts/block to your disk configuration line. > Secondly, when /etc/xen/scripts/block runs, it doesn't have /usr/local/bin > or sbin in its PATH, so it cannot execute xenstore-read or write. That was > also easy enough to fix. Hm, I guess I haven't seen this because I was running xl devd manually from the command line. AFAICT the patch I'm attaching at the bottom of this email should solve it. > So, now the block script runs but after that nothing seems to happen. This > is where I started to trace the code in libxl. I think what needs to happen > is that xl devd (or the block script?) needs to set > local/domain/(freebsd)/backend/vbd/(linuxpv)/51713/sectors, info, and > sector-size. The .../state variable is also still set to 3 after the block > script runs. Backend in state 3 means it's ready and waiting for the frontend to connect. That's quite weird because I've just tested a FreeBSD driver domain from a Linux Dom0 and it seems to work fine, other guests can properly see and attach the devices served by the FreeBSD driver domain. Do you have driver_domain=1 set in your FreeBSD xl config file? > It looks like the Linux hotplug scripts don't set those variables directly > either, so I'm not sure where that should be happening. (Aside from that, I > did see that Linux sets .../physical-device to a major:minor and some > feature- variables that FreeBSD does not set.) The xldevd.log doesn't show > anything interesting and running with -vvv only shows some cleanup being > done after the script has finished. This should be set by blkback itself IIRC. Roger. [0] https://xenbits.xen.org/docs/unstable/misc/xl-disk-configuration.txt ---8<--- >From 0e2e6b6f3892d83548dd1d5c3644972ee5fe9847 Mon Sep 17 00:00:00 2001 From: Roger Pau Monne Date: Mon, 19 Dec 2016 12:16:33 +0000 Subject: [PATCH] init/FreeBSD: set correct PATH for xl devd MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FreeBSD init scripts don't have /usr/local/{bin/sbin} in it's PATH, which prevents `xl devd` from working properly since hotplug scripts require the set of xenstore cli tools to be in PATH. While there also fix the usage of --pidfile, which according to the xl help doesn't use "=", and add braces around XLDEVD_PIDFILE. Signed-off-by: Roger Pau Monné --- Cc: Ian Jackson Cc: Wei Liu --- tools/hotplug/FreeBSD/rc.d/xendriverdomain.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in b/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in index 8ece7c3..3917de2 100644 --- a/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in +++ b/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in @@ -29,7 +29,7 @@ xendriverdomain_startcmd() { printf "Starting xenservices: xl devd." - ${sbindir}/xl devd --pidfile=$XLDEVD_PIDFILE ${XLDEVD_ARGS} + PATH="${bindir}:${sbindir}:$PATH" ${sbindir}/xl devd --pidfile ${XLDEVD_PIDFILE} ${XLDEVD_ARGS} printf "\n" } -- 2.10.1 (Apple Git-78)