Date: Tue, 29 Nov 2016 15:44:00 +0000 From: Steven Crangle <Steven@stream-technologies.com> To: Vincenzo Maffione <v.maffione@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Looking for some help with netmap/bhyve Message-ID: <DB5PR07MB1685470E0EB0B93A48E7AA229B8D0@DB5PR07MB1685.eurprd07.prod.outlook.com> In-Reply-To: <CA%2B_eA9jW_O_a5uRBAA9XSPspnwATrGhXM3NYusyXxemftn3uZw@mail.gmail.com> References: <DB5PR07MB1685DAA40193595950A464FB9BB40@DB5PR07MB1685.eurprd07.prod.outlook.com> <CA%2B_eA9gggyo_ncSZDribOr%2BsWoFELbPKEeLdaZ8AZwgHAYjcRA@mail.gmail.com> <DB5PR07MB168582129D05D52878D3DCE59B8D0@DB5PR07MB1685.eurprd07.prod.outlook.com>, <CA%2B_eA9jW_O_a5uRBAA9XSPspnwATrGhXM3NYusyXxemftn3uZw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Vincenzo,
No problem! We've decided that we will try testing with 11.0 too, as it wou=
ld help us to have our test system locked to a specific version like 11.0 a=
nd can hopefully help you with testing too. We'll compile netmap from sourc=
e as advised, but I was wondering what steps we would take in order to port=
the changes to bhyve/ptnetmap from HEAD into 11.0?
I've currently only tested with pkt-gen as our bhyve instances are pretty m=
inimal right now, below are a few samples of the speeds seen in each of the=
different configurations:
Using netmap on top of standard tap devices with no vale:
vm1:
197.259100 main_thread [2325] 30.977 Kpps (32.435 Kpkts 15.569 Mbps in =
1047062 usec) 1.93 avg_batch 826 min_space
203.306126 main_thread [2325] 32.149 Kpps (32.411 Kpkts 15.557 Mbps in =
1008156 usec) 1.88 avg_batch 866 min_space
204.313055 main_thread [2325] 33.095 Kpps (33.324 Kpkts 15.996 Mbps in =
1006929 usec) 1.98 avg_batch 911 min_space
vm2:
245.397418 main_thread [2325] 31.422 Kpps (33.262 Kpkts 15.966 Mbps in =
1058559 usec) 313.79 avg_batch 99999 min_space
246.429810 main_thread [2325] 31.254 Kpps (32.266 Kpkts 15.488 Mbps in =
1032392 usec) 319.47 avg_batch 99999 min_space
251.621436 main_thread [2325] 31.606 Kpps (33.329 Kpkts 15.998 Mbps in =
1054531 usec) 314.42 avg_batch 99999 min_space
After then trying to switch to the netmap in userspace/netmap in kernel spa=
ce with ptnetmap, I realised I made a silly mistake and built from head ins=
tead of the branch ptnet-head... haha So I'm currently doing a quick rebuil=
d on the correct branch and will report back with the results from pkt-gen =
using just vale, and also with the new ptnetmap changes!
Kind Regards
________________________________
[Stream Logo]
Steven Crangle
Systems Developer | Stream Technologies | Glasgow, UK
[http://www.stream-technologies.com/img/phone.png]
+44 (0)844 800 8520
|
[http://www.stream-technologies.com/img/mouse.png]
www.stream-technologies.com<http://www.stream-technologies.com/>;
________________________________
From: Vincenzo Maffione <v.maffione@gmail.com>
Sent: 29 November 2016 11:52:18
To: Steven Crangle
Cc: FreeBSD Net
Subject: Re: Looking for some help with netmap/bhyve
Hi Steven,
Thanks for your testing this in HEAD, good news to know that it works ok!
I think there is not going to be much difference between HEAD and 11.0, bec=
ause in the end bhyve code didn't really change (for the parts I touched), =
and you need to use the latest netmap version (github) anyway, irrespective=
of whether you are testing on 10.3, 11.0, head, etc. In other words, you w=
ould be testing approximately the same code. Anyway, if you happen to test =
11.0 by chance please let me know.
What kind of tests did you perform? netmap applications (e.g. pkt-gen), or =
netperf/iperf standard TCP/IP tools?
Could you please share the performance number you got for configurations (A=
) and (B) (or just the one you tried)?
I think it is useful to compare your results with the ones I collected here=
(see the performance evaluation section):
https://wiki.freebsd.org/DevSummit/201609?action=3DAttachFile&do=3Dview&tar=
get=3D20160923-freebsd-summit-ptnet.pdf
Regarding the crashes, thanks for reporting, I would be nice if you could d=
escribe in more detail how you trigger them (with some higher likelyhood); =
also, compiling the kernel with debug symbols should help in understanding =
which part of the code is involved in the crash.
Cheers,
Vincenzo
2016-11-29 12:26 GMT+01:00 Steven Crangle <Steven@stream-technologies.com<m=
ailto:Steven@stream-technologies.com>>:
Hi Vincenzo!
Thank you so much for your help! With your instructions I've managed to get=
a set of bhyves up and running with netmap working properly within them! I=
chose to go with the recent version of head instead of 10.3. I will also =
try and build a version on 11.0 if I get chance!
I will make sure and ask any further questions on the github page for netma=
p, but while I'm emailing, I've attached a text file containing a few kerne=
l page fault errors I've ran into. They don't happen repeatedly. They seem =
to just happen randomly if I try to shut the process down and possibly catc=
h netmap in an incorrect state! Figured they might be useful to you. If I f=
igure out what the error is myself I will try and help out!
And thanks again for your help!
Kind Regards
________________________________
[Stream Logo]
Steven Crangle
Systems Developer | Stream Technologies | Glasgow, UK
[http://www.stream-technologies.com/img/phone.png]
+44 (0)844 800 8520<tel:%2B44%20%280%29844%20800%208520>
|
[http://www.stream-technologies.com/img/mouse.png]
www.stream-technologies.com<http://www.stream-technologies.com/>;
________________________________
From: Vincenzo Maffione <v.maffione@gmail.com<mailto:v.maffione@gmail.com>>
Sent: 23 November 2016 08:33:11
To: Steven Crangle
Cc: FreeBSD Net
Subject: Re: Looking for some help with netmap/bhyve
2016-11-22 11:31 GMT+01:00 Steven Crangle <Steven@stream-technologies.com<m=
ailto:Steven@stream-technologies.com>>:
Hi,
I've recently been trying to boot up several bhyves so that I can test netm=
ap communication between instances. The problem is, no matter what configur=
ation I try, the guest vm running in bhyve completely hangs and becomes unu=
sable as soon as a packet hits the netmap interface. When testing with pkt-=
gen, the TX side successfully starts sending packets, but the RX side will =
reliable freeze with the only option being killing the bhyve process.
The bhyve command used for the above test was:
bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip=
er1vol -s 3,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 4,virtio-net,tap4=
-l com1,/dev/nmdm0A -A -H -P -m 6g viper1 &
bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip=
er2vol -s 3,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 4,virtio-net,tap5=
-l com1,/dev/nmdm1A -A -H -P -m 6g viper2
For this test the host OS was FreeBSD-11.0-p3 and the guest OS was FreeBSD-=
11.0-p3.
After failing to get this solution working, I pulled down the source from t=
he following url and installed it on the host box:
https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/
I then ran the following commands to try and bring up the machines using th=
e ptnetmap interface (the guest still running 11.0p3) :
bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v=
iper1vol -s 2:0,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 2:1,ptnetmap-=
memdev -s 2:2,ptnet,vale0:0 -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 &
bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v=
iper2vol -s 2:0,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 2:1,ptnetmap-=
memdev -s 2:2,ptnet,vale0:1 -l com1,/dev/nmdm1A -A -H -P -m 6g viper2
With the above commands the vm's fail to boot with the following message:
ptnet_init: failed to get ptnetmap
Output in /var/log/messages seems to just show the ptnetmap driver allocati=
ng one RX/TX ring for each vm, while bringing the device up, the device the=
n goes down and the above error is seen in the console.
Is there something I'm doing wrong with regards to running netmap or ptnetm=
ap within a bhyve? Any pointers in the right direction will be much appreci=
ated!
Kind Regards
Steven
Hi Steven,
The code you are looking at is the final code released by my gsoc 2016 pr=
oject at the end of August 2016. However, I've been working on that for a w=
hile after the gsoc, so that code is not updated anymore. My modification b=
asically involves two subsystems: netmap and bhyve.
The updates to netmap are already available in HEAD. The updates to bhyve a=
re not yet upstream, as we are in the process to review that with the bhyve=
maintainers.
Anyway, you need two sources to get the latest code:
1) https://github.com/luigirizzo/netmap for the latest netmap code, that yo=
u could also compile as an external kernel module.
2) https://github.com/vmaffione/freebsd to get the updates to bhyve (and vm=
m.ko). You can use this as the host system. There are two branches here: pt=
net-10.3, and ptnet-head. My original code was developed under FreeBSD 10.3=
, so a first possibility is to try this as the host system (using the ptnet=
-10.3 branch). The other branch (ptnet-head) contains a porting of the work=
to a recent version of HEAD. I would be very glad if you could test the co=
de also on FreeBSD 11.
We support two combinations of bhyve networking with netmap:
(A) virtio-net + netmap: that is something like "-s 2:0,virtio-net,vale0:=
0" in the bhyve command line.
(B) ptnet + ptnetmap: that is something like "-s 2:1,ptnetmap-memdev -s 2=
:2,ptnet,vale0:0" in the bhyve command line.
so you may want to try A first (netmap backend in user-space, slower) and t=
hen B (netmap backend in kernel-space, faster).
Sorry about the confusion on the code repositories, I'll try also to update=
the wiki page (and/or the gsoc svn repo) to reflect these updates. It's pe=
rfectly ok for me to discuss these issues here in the ML, however for more =
detailed/low-level discussion and support about problems you are running in=
to, feel free to open Github issues here https://github.com/luigirizzo/netm=
ap.
Cheers,
Vincenzo
________________________________
[Stream Logo]
Steven Crangle
Systems Developer | Stream Technologies | Glasgow, UK
[http://www.stream-technologies.com/img/phone.png]
+44 (0)844 800 8520<tel:%2B44%20%280%29844%20800%208520>
|
[http://www.stream-technologies.com/img/mouse.png]
www.stream-technologies.com<http://www.stream-technologies.com><http://www.=
stream-technologies.com/>
_______________________________________________
freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org<mailt=
o:freebsd-net-unsubscribe@freebsd.org>"
--
Vincenzo Maffione
--
Vincenzo Maffione
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DB5PR07MB1685470E0EB0B93A48E7AA229B8D0>