From owner-freebsd-net@freebsd.org Tue Nov 29 18:17:55 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8B69C5C980 for ; Tue, 29 Nov 2016 18:17:55 +0000 (UTC) (envelope-from Steven@stream-technologies.com) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0049.outbound.protection.outlook.com [104.47.0.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8FF112F8 for ; Tue, 29 Nov 2016 18:17:54 +0000 (UTC) (envelope-from Steven@stream-technologies.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamtechnologiesuk.onmicrosoft.com; s=selector1-streamtechnologies-com01e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=vVmM2XJYZTvs3vg3deT0MAa1DNXKg/snQn7gqsO5omQ=; b=Q2tMtV+7NiQZ8zYveJUrHKNMXa0i4VWrt8/2SPJ7TRHUMnnHEFo16dRQFA15li5e/4piIClhiHnw+F50CUqFaa3uJkav0oF8QKPULZdVn+BfflOoEXDU7oZgSUd1eOBbnKIx3WJcdra/QGPiPZTBA2H8F0C0BgiNq4jPzE5zwPk= Received: from DB5PR07MB1685.eurprd07.prod.outlook.com (10.166.13.14) by DB5PR07MB1685.eurprd07.prod.outlook.com (10.166.13.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.761.5; Tue, 29 Nov 2016 15:44:01 +0000 Received: from DB5PR07MB1685.eurprd07.prod.outlook.com ([10.166.13.14]) by DB5PR07MB1685.eurprd07.prod.outlook.com ([10.166.13.14]) with mapi id 15.01.0761.009; Tue, 29 Nov 2016 15:44:01 +0000 From: Steven Crangle To: Vincenzo Maffione CC: FreeBSD Net Subject: Re: Looking for some help with netmap/bhyve Thread-Topic: Looking for some help with netmap/bhyve Thread-Index: AQHSRKuiOapzmKvhI0y8Pcsl7N08/KDmPxuAgAmeBLSAAAebAIAAJnrz Date: Tue, 29 Nov 2016 15:44:00 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Steven@stream-technologies.com; x-originating-ip: [212.20.240.118] x-ms-office365-filtering-correlation-id: 70ba034b-ae79-4e50-0bd9-08d4186e8a42 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DB5PR07MB1685; x-microsoft-exchange-diagnostics: 1; DB5PR07MB1685; 7:zktX2SGbovP/fabzgVlMZ02UDOk4/+0WhIgAa6bxDHB4dPWqLP2KKvaUseK13V/T6RpSbBPlneDpapc9or6V975Wm8cOzbDvW1E+RkuGpL2dg5sjkxr+EoKwSP4qLKqIZE6dOpY45zuWn16RFU2gEI0ZQITkPc5KGWtarGtj6OOWsBYRJ+suY5Ac6s8sTYpf/UpZAf0hixuCaHZgP9Tn+ptInDeX1ywglcirLODXagS8GNkG95g5yjnkr3BvYuLk7CIl5sS4dOqBOrhyL7I1NFN28xsVQKjPIIy8tInz6QHsd3pqwIdg8KKh3/1c/ZG4oExe6rXWp9rmYBSO1/OBHEmAnrq3XhWI9IbxPVctPm+N2Jg9Y53ktS+Inn0kfFWiexBnYMzDNNouh03aWRaxGWTPhGiYRHmpnTpE97cIwOMtrAogeGRQSi8ikYlVLPiP6ooFQHk94nZwto5GagQTTQ== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(56005881305849)(278428928389397)(166708455590820)(203790808753936)(266611908612381)(75325880899374)(211171220733660)(83272782819144); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040361)(6045199)(6060326)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6061324)(6041248)(2016111802025)(20161123562025)(20161123560025)(20161123555025)(20161123564025)(6043046); SRVR:DB5PR07MB1685; BCL:0; PCL:0; RULEID:; SRVR:DB5PR07MB1685; x-forefront-prvs: 01415BB535 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916002)(189002)(199003)(377424004)(52314003)(43784003)(110136003)(4326007)(38730400001)(8936002)(3846002)(39060400001)(606004)(2906002)(3660700001)(5890100001)(6916009)(68736007)(3280700002)(5660300001)(101416001)(7696004)(229853002)(6506003)(105586002)(189998001)(33656002)(39380400001)(39410400001)(39450400002)(9686002)(106116001)(77096006)(39400400001)(6116002)(102836003)(2950100002)(97736004)(80792005)(4001150100001)(106356001)(81156014)(54356999)(76576001)(16601075003)(2900100001)(122556002)(76176999)(50986999)(7736002)(86362001)(575784001)(81166006)(93886004)(66066001)(7906003)(8676002)(7846002)(19273905006)(19627405001)(74316002)(92566002)(7099028)(16866105001)(473944003); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR07MB1685; H:DB5PR07MB1685.eurprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: stream-technologies.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: stream-technologies.com X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Nov 2016 15:44:00.9264 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 720fa073-5781-43bf-bc14-7bef2603ed21 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR07MB1685 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Nov 2016 18:17:55 -0000 Hi Vincenzo, No problem! We've decided that we will try testing with 11.0 too, as it wou= ld help us to have our test system locked to a specific version like 11.0 a= nd can hopefully help you with testing too. We'll compile netmap from sourc= e as advised, but I was wondering what steps we would take in order to port= the changes to bhyve/ptnetmap from HEAD into 11.0? I've currently only tested with pkt-gen as our bhyve instances are pretty m= inimal right now, below are a few samples of the speeds seen in each of the= different configurations: Using netmap on top of standard tap devices with no vale: vm1: 197.259100 main_thread [2325] 30.977 Kpps (32.435 Kpkts 15.569 Mbps in = 1047062 usec) 1.93 avg_batch 826 min_space 203.306126 main_thread [2325] 32.149 Kpps (32.411 Kpkts 15.557 Mbps in = 1008156 usec) 1.88 avg_batch 866 min_space 204.313055 main_thread [2325] 33.095 Kpps (33.324 Kpkts 15.996 Mbps in = 1006929 usec) 1.98 avg_batch 911 min_space vm2: 245.397418 main_thread [2325] 31.422 Kpps (33.262 Kpkts 15.966 Mbps in = 1058559 usec) 313.79 avg_batch 99999 min_space 246.429810 main_thread [2325] 31.254 Kpps (32.266 Kpkts 15.488 Mbps in = 1032392 usec) 319.47 avg_batch 99999 min_space 251.621436 main_thread [2325] 31.606 Kpps (33.329 Kpkts 15.998 Mbps in = 1054531 usec) 314.42 avg_batch 99999 min_space After then trying to switch to the netmap in userspace/netmap in kernel spa= ce with ptnetmap, I realised I made a silly mistake and built from head ins= tead of the branch ptnet-head... haha So I'm currently doing a quick rebuil= d on the correct branch and will report back with the results from pkt-gen = using just vale, and also with the new ptnetmap changes! Kind Regards ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520 | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com ________________________________ From: Vincenzo Maffione Sent: 29 November 2016 11:52:18 To: Steven Crangle Cc: FreeBSD Net Subject: Re: Looking for some help with netmap/bhyve Hi Steven, Thanks for your testing this in HEAD, good news to know that it works ok! I think there is not going to be much difference between HEAD and 11.0, bec= ause in the end bhyve code didn't really change (for the parts I touched), = and you need to use the latest netmap version (github) anyway, irrespective= of whether you are testing on 10.3, 11.0, head, etc. In other words, you w= ould be testing approximately the same code. Anyway, if you happen to test = 11.0 by chance please let me know. What kind of tests did you perform? netmap applications (e.g. pkt-gen), or = netperf/iperf standard TCP/IP tools? Could you please share the performance number you got for configurations (A= ) and (B) (or just the one you tried)? I think it is useful to compare your results with the ones I collected here= (see the performance evaluation section): https://wiki.freebsd.org/DevSummit/201609?action=3DAttachFile&do=3Dview&tar= get=3D20160923-freebsd-summit-ptnet.pdf Regarding the crashes, thanks for reporting, I would be nice if you could d= escribe in more detail how you trigger them (with some higher likelyhood); = also, compiling the kernel with debug symbols should help in understanding = which part of the code is involved in the crash. Cheers, Vincenzo 2016-11-29 12:26 GMT+01:00 Steven Crangle >: Hi Vincenzo! Thank you so much for your help! With your instructions I've managed to get= a set of bhyves up and running with netmap working properly within them! I= chose to go with the recent version of head instead of 10.3. I will also = try and build a version on 11.0 if I get chance! I will make sure and ask any further questions on the github page for netma= p, but while I'm emailing, I've attached a text file containing a few kerne= l page fault errors I've ran into. They don't happen repeatedly. They seem = to just happen randomly if I try to shut the process down and possibly catc= h netmap in an incorrect state! Figured they might be useful to you. If I f= igure out what the error is myself I will try and help out! And thanks again for your help! Kind Regards ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520 | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com ________________________________ From: Vincenzo Maffione > Sent: 23 November 2016 08:33:11 To: Steven Crangle Cc: FreeBSD Net Subject: Re: Looking for some help with netmap/bhyve 2016-11-22 11:31 GMT+01:00 Steven Crangle >: Hi, I've recently been trying to boot up several bhyves so that I can test netm= ap communication between instances. The problem is, no matter what configur= ation I try, the guest vm running in bhyve completely hangs and becomes unu= sable as soon as a packet hits the netmap interface. When testing with pkt-= gen, the TX side successfully starts sending packets, but the RX side will = reliable freeze with the only option being killing the bhyve process. The bhyve command used for the above test was: bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip= er1vol -s 3,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 4,virtio-net,tap4= -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 & bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip= er2vol -s 3,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 4,virtio-net,tap5= -l com1,/dev/nmdm1A -A -H -P -m 6g viper2 For this test the host OS was FreeBSD-11.0-p3 and the guest OS was FreeBSD-= 11.0-p3. After failing to get this solution working, I pulled down the source from t= he following url and installed it on the host box: https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/ I then ran the following commands to try and bring up the machines using th= e ptnetmap interface (the guest still running 11.0p3) : bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v= iper1vol -s 2:0,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 2:1,ptnetmap-= memdev -s 2:2,ptnet,vale0:0 -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 & bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v= iper2vol -s 2:0,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 2:1,ptnetmap-= memdev -s 2:2,ptnet,vale0:1 -l com1,/dev/nmdm1A -A -H -P -m 6g viper2 With the above commands the vm's fail to boot with the following message: ptnet_init: failed to get ptnetmap Output in /var/log/messages seems to just show the ptnetmap driver allocati= ng one RX/TX ring for each vm, while bringing the device up, the device the= n goes down and the above error is seen in the console. Is there something I'm doing wrong with regards to running netmap or ptnetm= ap within a bhyve? Any pointers in the right direction will be much appreci= ated! Kind Regards Steven Hi Steven, The code you are looking at is the final code released by my gsoc 2016 pr= oject at the end of August 2016. However, I've been working on that for a w= hile after the gsoc, so that code is not updated anymore. My modification b= asically involves two subsystems: netmap and bhyve. The updates to netmap are already available in HEAD. The updates to bhyve a= re not yet upstream, as we are in the process to review that with the bhyve= maintainers. Anyway, you need two sources to get the latest code: 1) https://github.com/luigirizzo/netmap for the latest netmap code, that yo= u could also compile as an external kernel module. 2) https://github.com/vmaffione/freebsd to get the updates to bhyve (and vm= m.ko). You can use this as the host system. There are two branches here: pt= net-10.3, and ptnet-head. My original code was developed under FreeBSD 10.3= , so a first possibility is to try this as the host system (using the ptnet= -10.3 branch). The other branch (ptnet-head) contains a porting of the work= to a recent version of HEAD. I would be very glad if you could test the co= de also on FreeBSD 11. We support two combinations of bhyve networking with netmap: (A) virtio-net + netmap: that is something like "-s 2:0,virtio-net,vale0:= 0" in the bhyve command line. (B) ptnet + ptnetmap: that is something like "-s 2:1,ptnetmap-memdev -s 2= :2,ptnet,vale0:0" in the bhyve command line. so you may want to try A first (netmap backend in user-space, slower) and t= hen B (netmap backend in kernel-space, faster). Sorry about the confusion on the code repositories, I'll try also to update= the wiki page (and/or the gsoc svn repo) to reflect these updates. It's pe= rfectly ok for me to discuss these issues here in the ML, however for more = detailed/low-level discussion and support about problems you are running in= to, feel free to open Github issues here https://github.com/luigirizzo/netm= ap. Cheers, Vincenzo ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520 | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Vincenzo Maffione -- Vincenzo Maffione