Date: Fri, 2 Dec 2016 19:41:35 +0100 From: Vincenzo Maffione <v.maffione@gmail.com> To: Steven Crangle <Steven@stream-technologies.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Looking for some help with netmap/bhyve Message-ID: <CA%2B_eA9h=wTKANiS_oGcBz881pC2ppy7uiPQPYsYq_dtHvjmWYA@mail.gmail.com> In-Reply-To: <DB5PR07MB16857B8F7A2B4783640074979B8E0@DB5PR07MB1685.eurprd07.prod.outlook.com> References: <DB5PR07MB1685DAA40193595950A464FB9BB40@DB5PR07MB1685.eurprd07.prod.outlook.com> <CA%2B_eA9gggyo_ncSZDribOr%2BsWoFELbPKEeLdaZ8AZwgHAYjcRA@mail.gmail.com> <DB5PR07MB168582129D05D52878D3DCE59B8D0@DB5PR07MB1685.eurprd07.prod.outlook.com> <CA%2B_eA9jW_O_a5uRBAA9XSPspnwATrGhXM3NYusyXxemftn3uZw@mail.gmail.com> <DB5PR07MB1685470E0EB0B93A48E7AA229B8D0@DB5PR07MB1685.eurprd07.prod.outlook.com> <CA%2B_eA9jfQc2B18fktgSt_JB4cRqsVLKA_11eWPfsbAKBMiJmUA@mail.gmail.com> <DB5PR07MB16857B8F7A2B4783640074979B8E0@DB5PR07MB1685.eurprd07.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, Thanks for testing, the numbers you report make sense, 7-15 Mpps is good for an old machine. On my machine I get 25 Mpps, so probably you could get something similar with newer hardware. The crash dumps you reported are not related to ptnetmap, but to the emulated (aka generic) netmap mode driver, that you have used over virtio-net VM interfaces. When you use ptnetmap, instead, inside the VM you see "ptnetX" interfaces. In any case, thanks for reporting. Hopefully we should get rid of these patches soon, once all the code will be pushed into HEAD. We are still progressing with this item, together with the bhyve maintainers. Cheers, Vincenzo 2016-12-02 15:22 GMT+01:00 Steven Crangle <Steven@stream-technologies.com>: > Hi Vincenzo, > > > Figured I would reply to give you some more testing feedback! Ptnetmap is > successfully working for us now on 11.0 RELEASE (stock src.txz from > FreeBSD.org) with the help of your patch! There were a few files missing > from the patch though, so I've attached a revised one which should work > incase it's of use to you! (might be a bit messy) > > I've attached a txt file with some testing results too , but in summary: > > > 1) netmap with no ptnet or vale: between 65-70Kpps. > > > 2) netmap with vale but no ptnet: between 121-160Kpps. > > > 3) ptnetmap with vale: seems to vary quite alot between 7-15Mpps. > > > All the tests were performed using pkt-gen with 60 byte packets. It's > worth noting that currently we're running the tests on a very old tired Sun > Micro server(Sun Fire X2200 M2 with Quad Core Processor). Currently in > the process of testing out our patch file on a clean 11.0 source on a much > faster box, so can run more detailed testing there using iperf etc if it's > useful to you! > > > I have also attached a few kernel traces, I tried to turn symbols on, but > i'm not sure it worked 100% Will try and fix it for the next install so > that I can provide better crash dumps if we run into any bugs! The crashes > attached only ever occured in setup number 2 above. > > > Thanks again for your help! > > > Kind Regards > > > Steven > > > > > > > > > > > > > ------------------------------ > > > > [image: Stream Logo] > > > > > > *Steven Crangle* > > Systems Developer | Stream Technologies | Glasgow, UK > > [image: http://www.stream-technologies.com/img/phone.png] > > +44 (0)844 800 8520 <+44%20844%20800%208520> > > | > > [image: http://www.stream-technologies.com/img/mouse.png] > > www.stream-technologies.com > > > > > > ------------------------------ > *From:* Vincenzo Maffione <v.maffione@gmail.com> > *Sent:* 29 November 2016 18:42:04 > > *To:* Steven Crangle > *Cc:* FreeBSD Net > *Subject:* Re: Looking for some help with netmap/bhyve > > Hi Steven, > > 2016-11-29 16:44 GMT+01:00 Steven Crangle <Steven@stream-technologies.com> > : > >> >> Hi Vincenzo, >> >> >> No problem! We've decided that we will try testing with 11.0 too, as it >> would help us to have our test system locked to a specific version like >> 11.0 and can hopefully help you with testing too. We'll compile netmap from >> source as advised, but I was wondering what steps we would take in order to >> port the changes to bhyve/ptnetmap from HEAD into 11.0? >> > > I think you can get the patch containing my modifications to HEAD for > bhyve+vmm.ko+libvmmapi, that you can extract with the following command in > my freebsd git repository > > $ git diff 9e10acc0303003d8733a546bca60f242b7b0aa64 ptnet-head > lib/libvmmapi/ sys/amd64/ sys/modules/vmm/ usr.sbin/bhyve > > and apply that patch in your 11.0 FreeBSD tree. > > >> >> I've currently only tested with pkt-gen as our bhyve instances are pretty >> minimal right now, below are a few samples of the speeds seen in each of >> the different configurations: >> > > which configurations? > >> >> Using netmap on top of standard tap devices with no vale: >> >> >> vm1: >> 197.259100 main_thread [2325] 30.977 Kpps (32.435 Kpkts 15.569 Mbps >> in 1047062 usec) 1.93 avg_batch 826 min_space >> 203.306126 main_thread [2325] 32.149 Kpps (32.411 Kpkts 15.557 Mbps >> in 1008156 usec) 1.88 avg_batch 866 min_space >> 204.313055 main_thread [2325] 33.095 Kpps (33.324 Kpkts 15.996 Mbps >> in 1006929 usec) 1.98 avg_batch 911 min_space >> vm2: >> 245.397418 main_thread [2325] 31.422 Kpps (33.262 Kpkts 15.966 Mbps >> in 1058559 usec) 313.79 avg_batch 99999 min_space >> 246.429810 main_thread [2325] 31.254 Kpps (32.266 Kpkts 15.488 Mbps >> in 1032392 usec) 319.47 avg_batch 99999 min_space >> 251.621436 main_thread [2325] 31.606 Kpps (33.329 Kpkts 15.998 Mbps >> in 1054531 usec) 314.42 avg_batch 99999 min_space >> > > Be aware that you typically want to use netmap both in the VM and in the > host (to use VALE and ptnet). Here you are probably using netmap just in > the VM. Performance are however extremely low. > When using netmap in both guest and host, with ptnet, you should be able > to get at least 10-30 Mpps from vm1 to vm2. > >> >> After then trying to switch to the netmap in userspace/netmap in kernel >> space with ptnetmap, I realised I made a silly mistake and built from head >> instead of the branch ptnet-head... haha So I'm currently doing a quick >> rebuild on the correct branch and will report back with the results from >> pkt-gen using just vale, and also with the new ptnetmap changes! >> > > Ok, also consider that you can use the patch I indicated above against > HEAD. > > Cheers, > Vincenzo > >> >> >> Kind Regards >> >> >> >> >> >> >> >> ------------------------------ >> >> >> >> [image: Stream Logo] >> >> >> >> >> >> *Steven Crangle* >> >> Systems Developer | Stream Technologies | Glasgow, UK >> >> [image: http://www.stream-technologies.com/img/phone.png] >> >> +44 (0)844 800 8520 >> >> | >> >> [image: http://www.stream-technologies.com/img/mouse.png] >> >> www.stream-technologies.com >> >> >> >> >> >> ------------------------------ >> *From:* Vincenzo Maffione <v.maffione@gmail.com> >> *Sent:* 29 November 2016 11:52:18 >> >> *To:* Steven Crangle >> *Cc:* FreeBSD Net >> *Subject:* Re: Looking for some help with netmap/bhyve >> >> Hi Steven, >> Thanks for your testing this in HEAD, good news to know that it works >> ok! >> >> I think there is not going to be much difference between HEAD and 11.0, >> because in the end bhyve code didn't really change (for the parts I >> touched), and you need to use the latest netmap version (github) anyway, >> irrespective of whether you are testing on 10.3, 11.0, head, etc. In other >> words, you would be testing approximately the same code. Anyway, if you >> happen to test 11.0 by chance please let me know. >> >> What kind of tests did you perform? netmap applications (e.g. pkt-gen), >> or netperf/iperf standard TCP/IP tools? >> Could you please share the performance number you got for configurations >> (A) and (B) (or just the one you tried)? >> >> I think it is useful to compare your results with the ones I collected >> here (see the performance evaluation section): >> >> https://wiki.freebsd.org/DevSummit/201609?action=AttachFile&; >> do=view&target=20160923-freebsd-summit-ptnet.pdf >> >> Regarding the crashes, thanks for reporting, I would be nice if you could >> describe in more detail how you trigger them (with some higher likelyhood); >> also, compiling the kernel with debug symbols should help in understanding >> which part of the code is involved in the crash. >> >> Cheers, >> Vincenzo >> >> >> 2016-11-29 12:26 GMT+01:00 Steven Crangle <Steven@stream-technologies.com >> >: >> >>> Hi Vincenzo! >>> >>> >>> Thank you so much for your help! With your instructions I've managed to >>> get a set of bhyves up and running with netmap working properly within >>> them! I chose to go with the recent version of head instead of 10.3. I >>> will also try and build a version on 11.0 if I get chance! >>> >>> >>> I will make sure and ask any further questions on the github page for >>> netmap, but while I'm emailing, I've attached a text file containing a few >>> kernel page fault errors I've ran into. They don't happen repeatedly. They >>> seem to just happen randomly if I try to shut the process down and possibly >>> catch netmap in an incorrect state! Figured they might be useful to you. If >>> I figure out what the error is myself I will try and help out! >>> >>> >>> And thanks again for your help! >>> >>> >>> Kind Regards >>> >>> >>> >>> >>> >>> >>> >>> ------------------------------ >>> >>> >>> >>> [image: Stream Logo] >>> >>> >>> >>> >>> >>> *Steven Crangle* >>> >>> Systems Developer | Stream Technologies | Glasgow, UK >>> >>> [image: http://www.stream-technologies.com/img/phone.png] >>> >>> +44 (0)844 800 8520 >>> >>> | >>> >>> [image: http://www.stream-technologies.com/img/mouse.png] >>> >>> www.stream-technologies.com >>> >>> >>> >>> >>> >>> ------------------------------ >>> *From:* Vincenzo Maffione <v.maffione@gmail.com> >>> *Sent:* 23 November 2016 08:33:11 >>> *To:* Steven Crangle >>> *Cc:* FreeBSD Net >>> *Subject:* Re: Looking for some help with netmap/bhyve >>> >>> >>> >>> 2016-11-22 11:31 GMT+01:00 Steven Crangle <Steven@stream-technologies.co >>> m>: >>> >>>> >>>> Hi, >>>> >>>> >>>> I've recently been trying to boot up several bhyves so that I can test >>>> netmap communication between instances. The problem is, no matter what >>>> configuration I try, the guest vm running in bhyve completely hangs and >>>> becomes unusable as soon as a packet hits the netmap interface. When >>>> testing with pkt-gen, the TX side successfully starts sending packets, but >>>> the RX side will reliable freeze with the only option being killing the >>>> bhyve process. >>>> >>>> >>>> The bhyve command used for the above test was: >>>> >>>> >>>> bhyve -c 1 -s 0,hostbridge -s 1,lpc -s >>>> 2,virtio-blk,/dev/zvol/zroot/viper1vol -s >>>> 3,virtio-net,tap0,mac=00:01:23:45:67:83 -s 4,virtio-net,tap4 -l >>>> com1,/dev/nmdm0A -A -H -P -m 6g viper1 & >>>> bhyve -c 1 -s 0,hostbridge -s 1,lpc -s >>>> 2,virtio-blk,/dev/zvol/zroot/viper2vol -s >>>> 3,virtio-net,tap1,mac=00:01:23:45:67:84 -s 4,virtio-net,tap5 -l >>>> com1,/dev/nmdm1A -A -H -P -m 6g viper2 >>>> >>>> For this test the host OS was FreeBSD-11.0-p3 and the guest OS was >>>> FreeBSD-11.0-p3. >>>> >>>> After failing to get this solution working, I pulled down the source >>>> from the following url and installed it on the host box: >>>> >>>> https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/ >>>> >>>> I then ran the following commands to try and bring up the machines >>>> using the ptnetmap interface (the guest still running 11.0p3) : >>>> >>>> bhyve -c 1 -s 0,hostbridge -s 1,lpc -s >>>> 1:1,virtio-blk,/dev/zvol/zroot/viper1vol -s >>>> 2:0,virtio-net,tap0,mac=00:01:23:45:67:83 -s 2:1,ptnetmap-memdev -s >>>> 2:2,ptnet,vale0:0 -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 & >>>> bhyve -c 1 -s 0,hostbridge -s 1,lpc -s >>>> 1:1,virtio-blk,/dev/zvol/zroot/viper2vol -s >>>> 2:0,virtio-net,tap1,mac=00:01:23:45:67:84 -s 2:1,ptnetmap-memdev -s >>>> 2:2,ptnet,vale0:1 -l com1,/dev/nmdm1A -A -H -P -m 6g viper2 >>>> >>>> With the above commands the vm's fail to boot with the following >>>> message: >>>> >>>> ptnet_init: failed to get ptnetmap >>>> >>>> Output in /var/log/messages seems to just show the ptnetmap driver >>>> allocating one RX/TX ring for each vm, while bringing the device up, the >>>> device then goes down and the above error is seen in the console. >>>> >>>> Is there something I'm doing wrong with regards to running netmap or >>>> ptnetmap within a bhyve? Any pointers in the right direction will be much >>>> appreciated! >>>> >>>> Kind Regards >>>> >>>> Steven >>>> >>>> >>> Hi Steven, >>> >>> The code you are looking at is the final code released by my gsoc 2016 >>> project at the end of August 2016. However, I've been working on that for a >>> while after the gsoc, so that code is not updated anymore. My modification >>> basically involves two subsystems: netmap and bhyve. >>> >>> The updates to netmap are already available in HEAD. The updates to >>> bhyve are not yet upstream, as we are in the process to review that with >>> the bhyve maintainers. >>> >>> Anyway, you need two sources to get the latest code: >>> 1) https://github.com/luigirizzo/netmap for the latest netmap code, >>> that you could also compile as an external kernel module. >>> 2) https://github.com/vmaffione/freebsd to get the updates to bhyve >>> (and vmm.ko). You can use this as the host system. There are two branches >>> here: ptnet-10.3, and ptnet-head. My original code was developed under >>> FreeBSD 10.3, so a first possibility is to try this as the host system >>> (using the ptnet-10.3 branch). The other branch (ptnet-head) contains a >>> porting of the work to a recent version of HEAD. I would be very glad if >>> you could test the code also on FreeBSD 11. >>> >>> We support two combinations of bhyve networking with netmap: >>> >>> (A) virtio-net + netmap: that is something like "-s >>> 2:0,virtio-net,vale0:0" in the bhyve command line. >>> (B) ptnet + ptnetmap: that is something like "-s 2:1,ptnetmap-memdev >>> -s 2:2,ptnet,vale0:0" in the bhyve command line. >>> >>> so you may want to try A first (netmap backend in user-space, slower) >>> and then B (netmap backend in kernel-space, faster). >>> >>> Sorry about the confusion on the code repositories, I'll try also to >>> update the wiki page (and/or the gsoc svn repo) to reflect these updates. >>> It's perfectly ok for me to discuss these issues here in the ML, however >>> for more detailed/low-level discussion and support about problems you are >>> running into, feel free to open Github issues here >>> https://github.com/luigirizzo/netmap. >>> >>> Cheers, >>> Vincenzo >>> >>> >>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> ________________________________ >>>> >>>> >>>> >>>> [Stream Logo] >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Steven Crangle >>>> >>>> >>>> Systems Developer | Stream Technologies | Glasgow, UK >>>> >>>> >>>> [http://www.stream-technologies.com/img/phone.png] >>>> >>>> >>>> +44 (0)844 800 8520 >>>> >>>> >>>> | >>>> >>>> >>>> [http://www.stream-technologies.com/img/mouse.png] >>>> >>>> >>>> www.stream-technologies.com<http://www.stream-technologies.com/>; >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>>> >>> >>> >>> >>> -- >>> Vincenzo Maffione >>> >> >> >> >> -- >> Vincenzo Maffione >> > > > > -- > Vincenzo Maffione > -- Vincenzo Maffione
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B_eA9h=wTKANiS_oGcBz881pC2ppy7uiPQPYsYq_dtHvjmWYA>