Date: Sun, 14 Jul 2024 10:02:48 -0400 From: Emil Tsalapatis <freebsd-lists@etsalapatis.com> To: David Chisnall <theraven@freebsd.org> Cc: Warner Losh <imp@bsdimp.com>, Alan Somers <asomers@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: Is anyone working on VirtFS (FUSE over VirtIO) Message-ID: <CABFh=a4t=73NLyJFqBOs1pRuo8B_d8wOH_mavnD-Da9dU-3k8Q@mail.gmail.com> In-Reply-To: <B926C3DA-A9F0-47EC-96BD-68EAA7A5A3B9@FreeBSD.org> References: <CAOtMX2gmc6L4H8L9107D84xofmd-idDgtVg8nkFkXPaPX1E8wg@mail.gmail.com> <A14C40DA-15EE-4777-B47F-2B342CE787EA@freebsd.org> <CANCZdfr7kKxTgBJ_LSKxAGsMUN9%2B=fiw1Fwy7Oxrc4G2mdSdYQ@mail.gmail.com> <E46B9396-7679-4C03-B144-BF4F9431F649@FreeBSD.org> <CANCZdfp0h%2BXhGoTk43e3ycZSg2Qh-6Fih9y5zfssGjRpoJaS-A@mail.gmail.com> <B926C3DA-A9F0-47EC-96BD-68EAA7A5A3B9@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000d7b5ab061d35a11e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi David, Warner,
I'm glad you find this approach interesting! I've been meaning to
update the virtio-dbg patch for a while but unfortunately haven't found the
time in the last month since I uploaded it... I'll update it soon to
address the reviews and split off the userspace device emulation code out
of the patch to make reviewing easier (thanks Alan for the suggestion). If
you have any questions or feedback please let me know.
WRT virtiofs itself, I've been working on it too but I haven't found the
time to clean it up and upload it. I have a messy but working
implementation here
<https://github.com/etsal/freebsd-src/tree/virtiofs-head>. The changes to
FUSE itself are indeed minimal because it is enough to redirect the
messages into a virtiofs device instead of sending them to a local FUSE
device. The virtiofs device and the FUSE device are both simple
bidirectional queues. Not sure on how to deal with directly mapping files
between host and guest just yet, because the Linux driver uses their DAX
interface for that, but it should be possible.
Emil
On Sun, Jul 14, 2024 at 3:11=E2=80=AFAM David Chisnall <theraven@freebsd.or=
g> wrote:
> Wow, that looks incredibly useful. Not needing bhyve / qemu (nested, if
> your main development is a VM) to test virtio drivers would be a huge
> productivity win.
>
> David
>
> On 13 Jul 2024, at 23:06, Warner Losh <imp@bsdimp.com> wrote:
>
> Hey David,
>
> You might want to check out https://reviews.freebsd.org/D45370 which has
> the testing framework as well as hints at other work that's been done for
> virtiofs by Emil Tsalapatis. It looks quite interesting. Anything he's do=
ne
> that's at odds with what I've said just shows where my analysis was flawe=
d
> :) This looks quite promising, but I've not had the time to look at it in
> detail yet.
>
> Warner
>
> On Sat, Jul 13, 2024 at 2:44=E2=80=AFAM David Chisnall <theraven@freebsd.=
org>
> wrote:
>
>> On 31 Dec 2023, at 16:19, Warner Losh <imp@bsdimp.com> wrote:
>>
>>
>> Yea. The FUSE protocol is going to be the challenge here. For this to be
>> useful, the VirtioFS support on the FreeBSD needs to be 100% in the
>> kernel, since you can't have userland in the loop. This isn't so terribl=
e,
>> though, since our VFS interface provides a natural breaking point for
>> converting the requests into FUSE requests. The trouble, I fear, is a
>> mismatch between FreeBSD's VFS abstraction layer and Linux's will cause
>> issues (many years ago, the weakness of FreeBSD VFS caused problems for =
a
>> company doing caching, though things have no doubt improved from those
>> days). Second, there's a KVM tie-in for the direct mapped pages between =
the
>> VM and the hypervisor. I'm not sure how that works on the client (FreeBS=
D)
>> side (though the description also says it's mapped via a PCI bar, so may=
be
>> the VM OS doesn't care).
>>
>>
>> From what I can tell from a little bit of looking at the code, our FUSE
>> implementation has a fairly cleanly abstracted layer (in fuse_ipc.c) for
>> handling the message queue. For VirtioFS, it would 'just' be necessary =
to
>> factor out the bits here that do uio into something that talked to a Vir=
tIO
>> ring. I don=E2=80=99t know what the VFS limitations are, but since the =
protocol
>> for VirtioFS is the kernel <-> userspace protocol for FUSE, it seems tha=
t
>> any functionality that works with FUSE filesystems in userspace would wo=
rk
>> with VirtioFS filesystems.
>>
>> The shared buffer cache bits are nice, but are optional, so could be don=
e
>> in a later version once the basic functionality worked.
>>
>> David
>>
>>
>
--000000000000d7b5ab061d35a11e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Hi David, Warner,</div><div><br></div><div>=C2=A0=C2=
=A0=C2=A0 I'm glad you find this approach interesting! I've been me=
aning to update the virtio-dbg patch for a while but unfortunately haven=
9;t found the time in the last month since I uploaded it... I'll update=
it soon to address the reviews and split off the=20
userspace device emulation code out of the patch to make reviewing=20
easier (thanks Alan for the suggestion). If you have any questions or feedb=
ack please let me know.<br></div><div><br></div><div>WRT virtiofs itself, I=
've been working on it too but I haven't found the time to clean it=
up and upload it. I have a messy but working implementation <a href=3D"htt=
ps://github.com/etsal/freebsd-src/tree/virtiofs-head">here</a>. The changes=
to FUSE itself are indeed minimal because it is enough to redirect the mes=
sages into a virtiofs device instead of sending them to a local FUSE device=
. The virtiofs device and the FUSE device are both simple bidirectional que=
ues. Not sure on how to deal with directly mapping files between host and g=
uest just yet, because the Linux driver uses their DAX interface for that, =
but it should be possible.<br></div><div><br></div><div>Emil<br></div></div=
><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Su=
n, Jul 14, 2024 at 3:11=E2=80=AFAM David Chisnall <<a href=3D"mailto:the=
raven@freebsd.org">theraven@freebsd.org</a>> wrote:<br></div><blockquote=
class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px so=
lid rgb(204,204,204);padding-left:1ex"><div>Wow, that looks incredibly usef=
ul.=C2=A0 Not needing bhyve / qemu (nested, if your main development is a V=
M) to test virtio drivers would be a huge productivity win. =C2=A0<div><br>=
</div><div>David<br id=3D"m_2432313125591762966lineBreakAtBeginningOfMessag=
e"><div><br><blockquote type=3D"cite"><div>On 13 Jul 2024, at 23:06, Warner=
Losh <<a href=3D"mailto:imp@bsdimp.com" target=3D"_blank">imp@bsdimp.co=
m</a>> wrote:</div><br><div><div dir=3D"ltr"><div>Hey David,</div><div><=
br></div><div>You might want to check out=C2=A0 <a href=3D"https://reviews.=
freebsd.org/D45370" target=3D"_blank">https://reviews.freebsd.org/D45370</a=
> which has the testing framework as well as hints at other work that's=
been done for virtiofs=C2=A0by Emil=C2=A0Tsalapatis. It looks quite intere=
sting. Anything he's done that's at odds with what I've said ju=
st shows where my analysis was flawed :) This looks quite promising, but I&=
#39;ve not had the time to look at it in detail yet.</div><div><br></div><d=
iv>Warner</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=
=3D"gmail_attr">On Sat, Jul 13, 2024 at 2:44=E2=80=AFAM David Chisnall <=
<a href=3D"mailto:theraven@freebsd.org" target=3D"_blank">theraven@freebsd.=
org</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x"><div>On 31 Dec 2023, at 16:19, Warner Losh <<a href=3D"mailto:imp@bsd=
imp.com" target=3D"_blank">imp@bsdimp.com</a>> wrote:<br><div><blockquot=
e type=3D"cite"><br><div><div style=3D"font-family:Helvetica;font-size:12px=
;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:=
normal;text-align:start;text-indent:0px;text-transform:none;white-space:nor=
mal;word-spacing:0px;text-decoration:none">Yea. The FUSE protocol is going =
to be the challenge here. For this to be useful, the VirtioFS=C2=A0support =
on=C2=A0the FreeBSD=C2=A0 needs to be 100% in the kernel, since you can'=
;t have userland in the loop. This isn't so terrible, though, since our=
VFS interface provides a natural breaking point for converting the request=
s into FUSE requests. The trouble, I fear, is a mismatch between FreeBSD=
9;s VFS abstraction layer and Linux's will cause issues (many years ago=
, the weakness of FreeBSD VFS caused problems for a company doing caching, =
though things have no doubt improved from those days). Second, there's =
a KVM tie-in for the direct mapped pages between the VM and the hypervisor.=
I'm not sure how that works on the client (FreeBSD) side (though the d=
escription also says it's mapped via a PCI bar, so maybe the VM OS does=
n't care).</div></div></blockquote><br></div><div>From what I can tell =
from a little bit of looking at the code, our FUSE implementation has a fai=
rly cleanly abstracted layer (in fuse_ipc.c) for handling the message queue=
.=C2=A0 For VirtioFS, it would 'just' be necessary to factor out th=
e bits here that do uio into something that talked to a VirtIO ring.=C2=A0 =
I don=E2=80=99t know what the VFS limitations are, but since the protocol f=
or VirtioFS is the kernel <-> userspace protocol for FUSE, it seems t=
hat any functionality that works with FUSE filesystems in userspace would w=
ork with VirtioFS filesystems.</div><div><br></div><div>The shared buffer c=
ache bits are nice, but are optional, so could be done in a later version o=
nce the basic functionality worked. =C2=A0</div><div><br></div><div>David</=
div><div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>
--000000000000d7b5ab061d35a11e--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABFh=a4t=73NLyJFqBOs1pRuo8B_d8wOH_mavnD-Da9dU-3k8Q>