From owner-freebsd-xen@freebsd.org Tue Dec 20 04:16:37 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBE16C874F2 for ; Tue, 20 Dec 2016 04:16:37 +0000 (UTC) (envelope-from nathan.friess@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7EDDB1455 for ; Tue, 20 Dec 2016 04:16:37 +0000 (UTC) (envelope-from nathan.friess@gmail.com) Received: by mail-io0-x22e.google.com with SMTP id h30so170186198iod.2 for ; Mon, 19 Dec 2016 20:16:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=oTwim4y2ZtDvd1jgUTs5MLprcKSuj0zkx3ZnTv7qwVQ=; b=BVcxjffKSWV1N/5tVX1J+KvN+bJsGE0BD0UDnT27Yp9+VOMAk6sL6JZ2/gINwwGjLi KCnC0G108NOvwWmCfmSmddq+/jVU6KHxWuONKQwT48KicheN6qcGA4IMw6v7fbLKnC4A xuh1HomSQd/oeCaaEo10tyTEXir5CC6wc8X7wQlqxlrRqjC8hMfCPxFpBjVtL6o77stL +qugf+dbONmokVdcbGy5liD3dCMt4ZBzieweI5v2rmmnziytUh4F4hrtPQm7erheMnrV O+1/AdiwWVb/WIVA6Lb9cfMSrVYcMJmb+aIpjT14kePxLPqNNd/RMz2hZ8d/Ok8rjFY8 cEpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=oTwim4y2ZtDvd1jgUTs5MLprcKSuj0zkx3ZnTv7qwVQ=; b=Wup57TMxy5zbxo8A8aZIsd/1yUxPsxnHxJCNpAbL44Zlcd8C6enffvwgM3Ks18xAh8 7+5y4Uwe1c7LhmCstxNBM6/02Afh6niCRsd+jJUliJjRXNWxSrzS4io5fCwMahlg7H2T WmvT2zXV1xZFj+NJLWbL63XuwakIYl6C6mzdZX1TnwVyZvdBZGyWcKPHwBxHjLRdJGFt MpcUFripLmZeeKlQqzr96vUpq0vhdpyfu5aCbISqRwBe2Nx+gBd5PFVYC6Pw/hcFT+WZ OwGCbf0w+cWZ42JQnBUvSfYlEobzReGpCXbxYmvKGeNGRIULq6f9enK8KhpPZEia600o IFgA== X-Gm-Message-State: AIkVDXJGVr+8JGCo0V2/qKQRtgZARmUPKTU0s3zw3Oa4a13h42dllUCW2MbUJF7z93SyUQ== X-Received: by 10.107.137.165 with SMTP id t37mr17675015ioi.233.1482207396568; Mon, 19 Dec 2016 20:16:36 -0800 (PST) Received: from [10.2.1.1] (S01060018e7c4b870.cg.shawcable.net. [50.66.15.125]) by smtp.gmail.com with ESMTPSA id i62sm8310070itb.12.2016.12.19.20.16.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Dec 2016 20:16:35 -0800 (PST) Subject: Re: 11-RELEASE acting as vbd backend To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= References: <247e5b6c-2713-27cf-f8fa-61e55c9e2025@gmail.com> <20161206100414.pi7ep2zbkduhuol7@dhcp-3-221.uk.xensource.com> <39e43ae4-6388-c698-c3c2-43cbc1f7b93c@gmail.com> <20161209144630.dfga5mozh72veo4g@dhcp-3-221.uk.xensource.com> <584ADC91.8050909@gmail.com> <20161212103257.tg46q2s3j5riixp5@dhcp-3-221.uk.xensource.com> <20161215173432.3qjzilzot7lnddhm@dhcp-3-221.uk.xensource.com> <20161219122727.pd4dzo2hb2m6ld7d@MacBook-Pro-de-Roger.local> Cc: freebsd-xen@freebsd.org From: Nathan Friess Message-ID: <9d8eee5c-1e37-869a-95c2-b340881d9875@gmail.com> Date: Mon, 19 Dec 2016 21:16:32 -0700 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20161219122727.pd4dzo2hb2m6ld7d@MacBook-Pro-de-Roger.local> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Dec 2016 04:16:37 -0000 On 12/19/2016 5:27 AM, Roger Pau Monné wrote: >> Secondly, when /etc/xen/scripts/block runs, it doesn't have /usr/local/bin >> or sbin in its PATH, so it cannot execute xenstore-read or write. That was >> also easy enough to fix. > > Hm, I guess I haven't seen this because I was running xl devd manually from the > command line. AFAICT the patch I'm attaching at the bottom of this email should > solve it. > I should have mentioned earlier that also in the rc script there is a typo in the stop function. rc_pid is set to the pid of xl devd but the next lines refer to rc_pids. Not that I ever want to stop xl devd anyway. :) >> So, now the block script runs but after that nothing seems to happen. This >> is where I started to trace the code in libxl. I think what needs to happen >> is that xl devd (or the block script?) needs to set >> local/domain/(freebsd)/backend/vbd/(linuxpv)/51713/sectors, info, and >> sector-size. The .../state variable is also still set to 3 after the block >> script runs. > > Backend in state 3 means it's ready and waiting for the frontend to connect. > That's quite weird because I've just tested a FreeBSD driver domain from a > Linux Dom0 and it seems to work fine, other guests can properly see and attach > the devices served by the FreeBSD driver domain. > > Do you have driver_domain=1 set in your FreeBSD xl config file? I did not have that before but I added it now. >> It looks like the Linux hotplug scripts don't set those variables directly >> either, so I'm not sure where that should be happening. (Aside from that, I >> did see that Linux sets .../physical-device to a major:minor and some >> feature- variables that FreeBSD does not set.) The xldevd.log doesn't show >> anything interesting and running with -vvv only shows some cleanup being >> done after the script has finished. > > This should be set by blkback itself IIRC. Yes, now I see the missing connection. What is happening is that blkback is setting the state to XenbusStateInitWait (2) in xbb_attach(). Once the block script runs the state is set to XenbusStateInitialised in xbb_attach_disk(). Now it waits for the frontend to update some variables. In the Linux kernel (4.4 that I'm using, and 4.9 from what I see), drivers/block/xen-blkfront.c, the blkback_changed() function waits for state XenbusStateInitWait in the backend after which it will set the variables that the backend is waiting for. It ignores state XenbusStateInitialised entirely. I think what is happening in my case is that FreeBSD is too fast and is setting the state twice before Linux notices, so the frontend isn't continuing on as it should. If I manually set the state of the backend backwards to 2 then the frontend connects up and the block device seems to work. Now both the frontend and backend transition to the connected state as they should. I don't know what the protocol should be, but it seems that the two ends are not agreeing on those states. Maybe on your systems the timing is different so Linux sees the InitWait state before FreeBSD moves to Initalised and everything works? (Aside from that, I wonder why Linux's watch on the state isn't firing once for each update?) Nathan