From owner-freebsd-net@freebsd.org Tue Nov 29 11:27:00 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83E7FC4B3D5 for ; Tue, 29 Nov 2016 11:27:00 +0000 (UTC) (envelope-from Steven@stream-technologies.com) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0073.outbound.protection.outlook.com [104.47.1.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B37A417BE for ; Tue, 29 Nov 2016 11:26:59 +0000 (UTC) (envelope-from Steven@stream-technologies.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamtechnologiesuk.onmicrosoft.com; s=selector1-streamtechnologies-com01e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=8OrJ+g/5to+tXXHXUvAVQhLuPLDiE6eY9wRlEHDxFdw=; b=qDSYB0sX4mMMUXtm/3xGtuOaMqwYfrH44gICFA5GOJA072dpBZlrwMg1RxvwENEWGd6zPOpdsV+tPY6aVd4SiVyT6EbgVV51WkSKO/ldUFReA+QYYYw73Ed1RGFmGzS3flSFFPpsGZgcCnYuJZ9A7A8eqI2CC+H4IXYX0YCyiKQ= Received: from DB5PR07MB1685.eurprd07.prod.outlook.com (10.166.13.14) by DB5PR07MB1687.eurprd07.prod.outlook.com (10.166.13.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.747.5; Tue, 29 Nov 2016 11:26:50 +0000 Received: from DB5PR07MB1685.eurprd07.prod.outlook.com ([10.166.13.14]) by DB5PR07MB1685.eurprd07.prod.outlook.com ([10.166.13.14]) with mapi id 15.01.0761.009; Tue, 29 Nov 2016 11:26:50 +0000 From: Steven Crangle To: Vincenzo Maffione CC: FreeBSD Net Subject: Re: Looking for some help with netmap/bhyve Thread-Topic: Looking for some help with netmap/bhyve Thread-Index: AQHSRKuiOapzmKvhI0y8Pcsl7N08/KDmPxuAgAmeBLQ= Date: Tue, 29 Nov 2016 11:26:50 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Steven@stream-technologies.com; x-originating-ip: [212.20.240.118] x-ms-office365-filtering-correlation-id: d7934d21-a836-48bc-0731-08d4184a9cd2 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DB5PR07MB1687; x-microsoft-exchange-diagnostics: 1; DB5PR07MB1687; 7:3GGSE+bWQjFO9AR5zWbH1HNAG7HIJ+QIUsbDtuclc7YQ0XAoZuQGJDbarx69XUKJUivEzDravcKPFWdIeXJY6UvH/1WgeBcps8jWkFzPbpgN496MzAvfXtzMFjj3Z9dRmYAiZsmaDZNof1mePVWWNphKxS83XWAHZwCb6CDPqr90PCHTxDs6A+ErpszoVFD6w+IvKRD2Ma1pUBsFpSA6nc3Gp1r9tFMJAhaIG9ovGGZXlplx3uWTzQnsOeKiPKOsrVs1OH0AY6m/NIsk1wmNdWozETNkfer622ZgYsbsjNZTQqp8XjqaJQHvnMc3pilpJ5v19FApWYdalkwbxMUCP6cQ/NBsvbawKjvfBqcdyBCFMKhl6N0BDrHi7sExikaXXZjO8l06NoQLaJOncNhXq7/aAdXkrcOTDPOuhGAadnwE72Ymx7L12U8S2WPR1rKFM1AZlg+ftX7/DCtji4T84Q== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(56005881305849)(166708455590820)(203790808753936)(266611908612381)(75325880899374)(83272782819144); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415395)(6060326)(6045199)(6040361)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6061324)(6041248)(20161123555025)(2016111802025)(20161123560025)(20161123564025)(20161123562025)(6043046); SRVR:DB5PR07MB1687; BCL:0; PCL:0; RULEID:; SRVR:DB5PR07MB1687; x-forefront-prvs: 01415BB535 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916002)(52314003)(43784003)(377424004)(189002)(199003)(575784001)(4001150100001)(80792005)(97736004)(101416001)(76176999)(54356999)(189998001)(6116002)(50986999)(102836003)(3846002)(99936001)(66066001)(33656002)(16601075003)(606004)(6506003)(229853002)(39450400002)(5890100001)(122556002)(9686002)(76576001)(77096006)(7906003)(19273905006)(81156014)(7736002)(8936002)(7696004)(81166006)(110136003)(5660300001)(39410400001)(39400400001)(8676002)(68736007)(2900100001)(74316002)(105586002)(106116001)(39380400001)(3280700002)(2950100002)(6916009)(7846002)(106356001)(2906002)(86362001)(39060400001)(3660700001)(4326007)(38730400001)(92566002)(19627405001)(7099028)(16866105001)(473944003); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR07MB1687; H:DB5PR07MB1685.eurprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: stream-technologies.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/mixed; boundary="_004_DB5PR07MB168582129D05D52878D3DCE59B8D0DB5PR07MB1685eurp_" MIME-Version: 1.0 X-OriginatorOrg: stream-technologies.com X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Nov 2016 11:26:50.2107 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 720fa073-5781-43bf-bc14-7bef2603ed21 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR07MB1687 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Nov 2016 11:27:00 -0000 --_004_DB5PR07MB168582129D05D52878D3DCE59B8D0DB5PR07MB1685eurp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Vincenzo! Thank you so much for your help! With your instructions I've managed to get= a set of bhyves up and running with netmap working properly within them! I= chose to go with the recent version of head instead of 10.3. I will also = try and build a version on 11.0 if I get chance! I will make sure and ask any further questions on the github page for netma= p, but while I'm emailing, I've attached a text file containing a few kerne= l page fault errors I've ran into. They don't happen repeatedly. They seem = to just happen randomly if I try to shut the process down and possibly catc= h netmap in an incorrect state! Figured they might be useful to you. If I f= igure out what the error is myself I will try and help out! And thanks again for your help! Kind Regards ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520 | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com ________________________________ From: Vincenzo Maffione Sent: 23 November 2016 08:33:11 To: Steven Crangle Cc: FreeBSD Net Subject: Re: Looking for some help with netmap/bhyve 2016-11-22 11:31 GMT+01:00 Steven Crangle >: Hi, I've recently been trying to boot up several bhyves so that I can test netm= ap communication between instances. The problem is, no matter what configur= ation I try, the guest vm running in bhyve completely hangs and becomes unu= sable as soon as a packet hits the netmap interface. When testing with pkt-= gen, the TX side successfully starts sending packets, but the RX side will = reliable freeze with the only option being killing the bhyve process. The bhyve command used for the above test was: bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip= er1vol -s 3,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 4,virtio-net,tap4= -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 & bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/zvol/zroot/vip= er2vol -s 3,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 4,virtio-net,tap5= -l com1,/dev/nmdm1A -A -H -P -m 6g viper2 For this test the host OS was FreeBSD-11.0-p3 and the guest OS was FreeBSD-= 11.0-p3. After failing to get this solution working, I pulled down the source from t= he following url and installed it on the host box: https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/ I then ran the following commands to try and bring up the machines using th= e ptnetmap interface (the guest still running 11.0p3) : bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v= iper1vol -s 2:0,virtio-net,tap0,mac=3D00:01:23:45:67:83 -s 2:1,ptnetmap-= memdev -s 2:2,ptnet,vale0:0 -l com1,/dev/nmdm0A -A -H -P -m 6g viper1 & bhyve -c 1 -s 0,hostbridge -s 1,lpc -s 1:1,virtio-blk,/dev/zvol/zroot/v= iper2vol -s 2:0,virtio-net,tap1,mac=3D00:01:23:45:67:84 -s 2:1,ptnetmap-= memdev -s 2:2,ptnet,vale0:1 -l com1,/dev/nmdm1A -A -H -P -m 6g viper2 With the above commands the vm's fail to boot with the following message: ptnet_init: failed to get ptnetmap Output in /var/log/messages seems to just show the ptnetmap driver allocati= ng one RX/TX ring for each vm, while bringing the device up, the device the= n goes down and the above error is seen in the console. Is there something I'm doing wrong with regards to running netmap or ptnetm= ap within a bhyve? Any pointers in the right direction will be much appreci= ated! Kind Regards Steven Hi Steven, The code you are looking at is the final code released by my gsoc 2016 pr= oject at the end of August 2016. However, I've been working on that for a w= hile after the gsoc, so that code is not updated anymore. My modification b= asically involves two subsystems: netmap and bhyve. The updates to netmap are already available in HEAD. The updates to bhyve a= re not yet upstream, as we are in the process to review that with the bhyve= maintainers. Anyway, you need two sources to get the latest code: 1) https://github.com/luigirizzo/netmap for the latest netmap code, that yo= u could also compile as an external kernel module. 2) https://github.com/vmaffione/freebsd to get the updates to bhyve (and vm= m.ko). You can use this as the host system. There are two branches here: pt= net-10.3, and ptnet-head. My original code was developed under FreeBSD 10.3= , so a first possibility is to try this as the host system (using the ptnet= -10.3 branch). The other branch (ptnet-head) contains a porting of the work= to a recent version of HEAD. I would be very glad if you could test the co= de also on FreeBSD 11. We support two combinations of bhyve networking with netmap: (A) virtio-net + netmap: that is something like "-s 2:0,virtio-net,vale0:= 0" in the bhyve command line. (B) ptnet + ptnetmap: that is something like "-s 2:1,ptnetmap-memdev -s 2= :2,ptnet,vale0:0" in the bhyve command line. so you may want to try A first (netmap backend in user-space, slower) and t= hen B (netmap backend in kernel-space, faster). Sorry about the confusion on the code repositories, I'll try also to update= the wiki page (and/or the gsoc svn repo) to reflect these updates. It's pe= rfectly ok for me to discuss these issues here in the ML, however for more = detailed/low-level discussion and support about problems you are running in= to, feel free to open Github issues here https://github.com/luigirizzo/netm= ap. Cheers, Vincenzo ________________________________ [Stream Logo] Steven Crangle Systems Developer | Stream Technologies | Glasgow, UK [http://www.stream-technologies.com/img/phone.png] +44 (0)844 800 8520 | [http://www.stream-technologies.com/img/mouse.png] www.stream-technologies.com _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Vincenzo Maffione --_004_DB5PR07MB168582129D05D52878D3DCE59B8D0DB5PR07MB1685eurp_ Content-Type: text/plain; name="kernel_stacktraces.txt" Content-Description: kernel_stacktraces.txt Content-Disposition: attachment; filename="kernel_stacktraces.txt"; size=1049; creation-date="Tue, 29 Nov 2016 11:25:17 GMT"; modification-date="Tue, 29 Nov 2016 11:25:17 GMT" Content-Transfer-Encoding: base64 CkZhdGFsIHRyYXAgMTI6IHBhZ2UgZmF1bHQgd2hpbGUgaW4ga2VybmVsIG1vZGUKY3B1aWQgPSAw OyBhcGljIGlkID0gMDAKZmF1bHQgdmlydHVhbCBhZGRyZXNzCT0gMHgwCmZhdWx0IGNvZGUJCT0g c3VwZXJ2aXNvciByZWFkIGluc3RydWN0aW9uLCBwYWdlIG5vdCBwcmVzZW50Cmluc3RydWN0aW9u IHBvaW50ZXIJPSAweDIwOjB4MApzdGFjayBwb2ludGVyCSAgICAgICAgPSAweDI4OjB4ZmZmZmZl MDFhNGQ2ZDkzOApmcmFtZSBwb2ludGVyCSAgICAgICAgPSAweDI4OjB4ZmZmZmZlMDFhNGQ2ZDk2 MApjb2RlIHNlZ21lbnQJCT0gYmFzZSAweDAsIGxpbWl0IDB4ZmZmZmYsIHR5cGUgMHgxYgoJCQk9 IERQTCAwLCBwcmVzIDEsIGxvbmcgMSwgZGVmMzIgMCwgZ3JhbiAxCnByb2Nlc3NvciBlZmxhZ3MJ PSBpbnRlcnJ1cHQgZW5hYmxlZCwgcmVzdW1lLCBJT1BMID0gMApjdXJyZW50IHByb2Nlc3MJCT0g MTIgKHN3aTQ6IGNsb2NrICgwKSkKWyB0aHJlYWQgcGlkIDEyIHRpZCAxMDAwMDUgXQpTdG9wcGVk IGF0ICAgICAgMAoKCkZhdGFsIHRyYXAgMTI6IHBhZ2UgZmF1bHQgd2hpbGUgaW4ga2VybmVsIG1v ZGUKY3B1aWQgPSAwOyBhcGljIGlkID0gMDAKZmF1bHQgdmlydHVhbCBhZGRyZXNzCT0gMHgwCmZh dWx0IGNvZGUJCT0gc3VwZXJ2aXNvciByZWFkIGluc3RydWN0aW9uLCBwYWdlIG5vdCBwcmVzZW50 Cmluc3RydWN0aW9uIHBvaW50ZXIJPSAweDIwOjB4MApzdGFjayBwb2ludGVyCSAgICAgICAgPSAw eDA6MHhmZmZmZmUwMWE0ZDZkOTM4CmZyYW1lIHBvaW50ZXIJICAgICAgICA9IDB4MDoweGZmZmZm ZTAxYTRkNmQ5NjAKY29kZSBzZWdtZW50CQk9IGJhc2UgMHgwLCBsaW1pdCAweGZmZmZmLCB0eXBl IDB4MWIKCQkJPSBEUEwgMCwgcHJlcyAxLCBsb25nIDEsIGRlZjMyIDAsIGdyYW4gMQpwcm9jZXNz b3IgZWZsYWdzCT0gaW50ZXJydXB0IGVuYWJsZWQsIHJlc3VtZSwgSU9QTCA9IDAKY3VycmVudCBw cm9jZXNzCQk9IDEyIChzd2k0OiBjbG9jayAoMCkpClsgdGhyZWFkIHBpZCAxMiB0aWQgMTAwMDA1 IF0KU3RvcHBlZCBhdCAgICAgIDAKCgo= --_004_DB5PR07MB168582129D05D52878D3DCE59B8D0DB5PR07MB1685eurp_--