From owner-freebsd-virtualization@freebsd.org Thu Apr 26 13:32:25 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8B86FAD5AF for ; Thu, 26 Apr 2018 13:32:24 +0000 (UTC) (envelope-from paul.esson@redstor.com) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0042.outbound.protection.outlook.com [104.47.2.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 30BC6837B5 for ; Thu, 26 Apr 2018 13:32:23 +0000 (UTC) (envelope-from paul.esson@redstor.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redstorltd.onmicrosoft.com; s=selector1-redstor-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=H+FLOruqXX6cwY+1ncXo8sd4d9kWAOueMM5pYbAKrjs=; b=mHGEP1rmRba4hYI1vxqZrmb5QlpVxhL1IeZLQMFtaGaSL6RGPQxI/gkxA7Cn61Wrmm8bczKXa+U7iggagYlxiksh2etoG/nCU+KvB+O55tkhta9gbVy+M0ukJM0A4spx3QhXOKeggyXgCdUQPHn/DcVMZ0YO63LWz+xPt4CZ8wY= Received: from HE1PR0102MB2588.eurprd01.prod.exchangelabs.com (10.170.251.141) by HE1PR0102MB3321.eurprd01.prod.exchangelabs.com (10.167.124.158) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.15; Thu, 26 Apr 2018 13:32:20 +0000 Received: from HE1PR0102MB2588.eurprd01.prod.exchangelabs.com ([fe80::dd48:cd01:a067:e152]) by HE1PR0102MB2588.eurprd01.prod.exchangelabs.com ([fe80::dd48:cd01:a067:e152%13]) with mapi id 15.20.0715.018; Thu, 26 Apr 2018 13:32:20 +0000 From: Paul Esson To: Harry Schmalzbauer CC: "freebsd-virtualization@freebsd.org" Subject: RE: bhyve networking Thread-Topic: bhyve networking Thread-Index: AdPcwrhbbh0fzB+SRv2CNcsNb6p8OQABdzsAAAJH2bQAAMHYgAABdrkYABXFYAAABzXgsAAE0VUw Date: Thu, 26 Apr 2018 13:32:20 +0000 Message-ID: References: , <201804252033.w3PKXv8M097827@pdx.rh.CN85.dnsmgr.net> <5AE18222.6000900@omnilan.de> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=paul.esson@redstor.com; x-originating-ip: [81.136.184.115] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR0102MB3321; 7:cNsKcYQhLVtSI93upkwel96gKdtwGowGJwJvofCAnUy5vgKnvdLhUyyRLxvR8jc12b04GokGvGUOvCkp/HkSpTvgatsxWbdumn3TR5HEhqkHHYSdm+itWYbosvOTjzRP9KRPjsJl/A93fTon33SIw9AE7yG4sbKL60MCY8SW4E3hW/P//zCyjDlrSVn1huEMzoWO7JztgGNgSzXm1n75mB5vKWz7jj6Dpja0N1Q2NuvPVYMgdUhaVjXsHrG04m/0 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:HE1PR0102MB3321; x-ms-traffictypediagnostic: HE1PR0102MB3321: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(788757137089)(75325880899374)(127952516941037)(21532816269658)(17755550239193); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3231232)(944501410)(52105095)(93006095)(93001095)(10201501046)(3002001)(6041310)(20161123564045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(6072148)(201708071742011); SRVR:HE1PR0102MB3321; BCL:0; PCL:0; RULEID:; SRVR:HE1PR0102MB3321; x-forefront-prvs: 0654257CF5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(376002)(39380400002)(39850400004)(346002)(396003)(199004)(189003)(13464003)(229853002)(4326008)(486006)(7736002)(53546011)(7696005)(966005)(59450400001)(76176011)(33656002)(8676002)(186003)(99286004)(6506007)(6116002)(3480700004)(3846002)(53936002)(74316002)(11346002)(68736007)(305945005)(6916009)(81156014)(81166006)(221733001)(6246003)(6436002)(8936002)(53946003)(3280700002)(478600001)(446003)(6306002)(7116003)(5660300001)(97736004)(55016002)(2906002)(106356001)(44832011)(26005)(86362001)(14454004)(2940100002)(5250100002)(476003)(316002)(66066001)(2900100001)(3660700001)(15974865002)(102836004)(93886005)(105586002)(25786009)(9686003)(111123002)(21314002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0102MB3321; H:HE1PR0102MB2588.eurprd01.prod.exchangelabs.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: redstor.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: KQjOg67hEbA3jrs7wisvJ8OOa3a8DQlXq+3m6pYGBFBeM/Xr1x6jpffehBZGY7eBIbJlEdLPF+WG2d2wA173gOFKNYf09Geuj/g0pUwgr140HNI9itf71wR5MCUidRVV9dmYMS+23vazwaGlZhUh8ciZBGi4pCNB5xaqnuX+EKoa0FBNAVY6nXuXK2KcIYZA spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 5fa5e8db-87f7-45a4-117a-08d5ab7a22fc X-OriginatorOrg: redstor.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5fa5e8db-87f7-45a4-117a-08d5ab7a22fc X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Apr 2018 13:32:20.3629 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 24ac53ae-15a7-4211-afef-61d8f34e2571 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0102MB3321 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 13:32:25 -0000 Hi Folks, I've just realised that the igb1 interface is not up in any of the output I= shared. So I took the switch out of the equation and created tap and brid= ge interfaces manually, then added igb1 and tap0 to bridge0 and brought the= bridge up. Finally, I brought igb1 and tap0 up. Once all the interfaces = were up I amended the guest configuration to replace network0_switch=3D"pub= lic" with network0_device=3D"tap0". Now when I start my guest I have netwo= rk connectivity on the guest VLAN. I'd really like to try and use the switch approach if possible and had thou= ght that creating the switch and adding the igb1 interface would have broug= ht igb1 up automatically. Is that the expected behaviour? Regards, Paul Esson=A0=A0|=A0=A0Redstor Limited t=A0=A0+44 (0)118 951 5235=A0=A0|=A0=A0=A0m=A0=A0+44 (0)776 690 6514 e=A0=A0paul.esson@redstor.com www.redstor.com -----Original Message----- From: Paul Esson=20 Sent: 26 April 2018 13:41 To: Harry Schmalzbauer Cc: freebsd-virtualization@freebsd.org Subject: RE: bhyve networking Hi Folks, Apologies for the lack of detail on my first post. To recap, I am attempti= ng to set-up a guest using vm-bhyve. I have a Dell PER730xd server with qu= ad-port INTEL 350 NIC. The first two ports have been configured on a) a ma= nagement LAN for the host and b) an application LAN for the guests. FreeBSD 11.1-RELEASE-p9 Dell PowerEdge R730xd - INTEL i350 NICs NIC-1 igb0 24:6E:96:B4:61:CC VLAN92 ge-6/0/11 (Host) NIC-2 igb1 24:6E:96:B4:61:CD VLAN101 ge-6/0/18 (Guests) - not a trunk Both interfaces are active as viewed from the host, but I have only assigne= d an ipv4 address to igb0 for management of the host igb0: flags=3D8843 metric 0 mtu 150= 0 options=3D6403bb ether 24:6e:96:b4:61:cc hwaddr 24:6e:96:b4:61:cc inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255 nd6 options=3D29 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=3D8c02 metric 0 mtu 1500 = options=3D6403bb ether 24:6e:96:b4:61:cd hwaddr 24:6e:96:b4:61:cd nd6 options=3D29 media: Ethernet autoselect (1000baseT ) status: active =20 If I assign a temporary address to igb1 I can then ping other computers on = the guests subnet - I've had to hide the address as the network is restrict= ed. # ifconfig igb1 inet xx.xxx.xxx.xx/25 up # ping xx.xxx.xxx.xx PING xx.xxx.x= xx.xx (xx.xxx.xxx.xx): 56 data bytes 64 bytes from xx.xxx.xxx.xx: icmp_seq=3D0 ttl=3D64 time=3D0.145 ms 64 bytes from xx.xxx.xxx.xx: icmp_seq=3D1 ttl=3D64 time=3D0.080 ms 64 bytes from xx.xxx.xxx.xx: icmp_seq=3D2 ttl=3D64 time=3D0.078 ms 64 bytes from xx.xxx.xxx.xx: icmp_seq=3D3 ttl=3D64 time=3D0.077 ms 64 bytes from xx.xxx.xxx.xx: icmp_seq=3D4 ttl=3D64 time=3D0.076 ms I then used the "vm" command to create a virtual switch and add interface i= gb1 to it. This automatically created the bridge interface. root@dc1-olbp-hn-01:~ # vm switch create public root@dc1-olbp-hn-01:~ # vm = switch add public igb1 root@dc1-olbp-hn-01:~ # vm switch info public ------------------------ Virtual Switch: public ------------------------ type: auto ident: bridge0 vlan: - nat: - physical-ports: igb1 bytes-in: 0 (0.000B) bytes-out: 0 (0.000B) Finally, I created a guest VM and gave its NIC the same ipv4 address detail= s I used previously to test igb1 from the host. This automatically created= the tap interface. igb0: flags=3D8843 metric 0 mtu 150= 0 options=3D6403bb ether 24:6e:96:b4:61:cc hwaddr 24:6e:96:b4:61:cc inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255 nd6 options=3D29 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=3D8d02 metric 0 mt= u 1500 options=3D6403bb ether 24:6e:96:b4:61:cd hwaddr 24:6e:96:b4:61:cd nd6 options=3D29 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 nd6 options=3D21 groups: lo bridge0: flags=3D8843 metric 0 mtu = 1500 description: vm-public ether 02:ee:ce:b0:6a:00 nd6 options=3D1 groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=3D143 ifmaxaddr 0 port 7 priority 128 path cost 2000000 member: igb1 flags=3D143 ifmaxaddr 0 port 2 priority 128 path cost 20000 tap0: flags=3D8943 metric 0= mtu 1500 description: vmnet-testvm-0-public options=3D80000 ether 00:bd:dd:51:0a:00 hwaddr 00:bd:dd:51:0a:00 nd6 options=3D29 media: Ethernet autoselect status: active groups: tap Opened by PID 1791 >From the guest VM I can see that the interface vtnet0 is up and has the rel= evant ipv4 address information. However, I cannot communicate with any oth= er computer on the guest subnet or beyond. vtnet0: flags=3D8943 metric= 0 mtu 1500 options=3D80028 ether 58:9c:fc:08:4a:20 hwaddr 58:9c:fc:08:4a:20 inet xx.xxx.xxx.xx netmask 0xffffff80 broadcast xx.xxx.xxx.xx nd6 options=3D29 media: Ethernet 10Gbase-T status: active lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=3D21 groups: lo The guest configuration file has the following network details network0_typ= e=3D"virtio-net" network0_switch=3D"public" >From the vm-bhyve.log I see the following Apr 26 07:59:23: initialising Apr= 26 07:59:23: [loader: bhyveload] Apr 26 07:59:23: [uefi: no] Apr 26 07:5= 9:23: [cpu: 1] Apr 26 07:59:23: [memory: 256M] Apr 26 07:59:23: [hostbri= dge: standard] Apr 26 07:59:23: [com ports: com1] Apr 26 07:59:23: [uuid:= 417cfb63-491f-11e8-949b-246e96b461cc] Apr 26 07:59:23: [utctime: no] Apr 26 07:59:23: [debug mode: no] Apr 26 07:59:23: [primary disk: disk0] Apr 26 07:59:23: [primary disk dev: sparse-zvol] Apr 26 07:59:23: generate= d static mac 58:9c:fc:08:4a:20 (based on 'testvm:0:1524725963:0') Apr 26 07= :59:23: initialising network device tap0 Apr 26 07:59:23: adding tap0 -> br= idge0 (public) Apr 26 07:59:23: booting Should I have to supply ipv4 details anywhere other than the guest's own vt= net0 interface? If I re-configure the switch to remove the igb1 interface = and add igb0 instead, then change the guest ipv4 address details to the man= agement network (172.16.92.0/24), I can connect to other computers on that = subnet and beyond. =20 vtnet0: flags=3D8943 metric= 0 mtu 1500 options=3D80028 ether 58:9c:fc:08:4a:20 hwaddr 58:9c:fc:08:4a:20 inet 172.16.92.21 netmask 0xffffff80 broadcast 172.16.92.127 nd6 options=3D29 media: Ethernet 10Gbase-T status: active lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=3D21 groups: lo $ ping 172.16.92.11 PING 172.16.92.11 (172.16.92.11): 56 data bytes 64 bytes from 172.16.92.11: icmp_seq=3D0 ttl=3D64 time=3D0.416 ms 64 bytes from 172.16.92.11: icmp_seq=3D1 ttl=3D64 time=3D0.371 ms 64 bytes from 172.16.92.11: icmp_seq=3D2 ttl=3D64 time=3D0.369 ms --- 172.16.92.11 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/= avg/max/stddev =3D 0.369/0.385/0.416/0.022 ms $ telnet bbc.co.uk 443 Trying 151.101.192.81... Connected to bbc.co.uk. Escape character is '^]'. Regards, Paul Esson | Redstor Limited t +44 (0)118 951 5235 | m +44 (0)776 690 6514 e paul.esson@redstor.com www.redstor.com -----Original Message----- From: Rodney W. Grimes Sent: 25 April 2018 22:31 To: Paul Esson Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve networking > Hi Rod, > Can you share a command line for that?=20 Its not just a single command, but you want these in /etc/rc.conf of the GU= EST: network_interfaces=3D"lo0" cloned_interfaces=3D"vlan48" ifconfig_lo0=3D" inet 127.0.0.1 netmask 0xff000000" ifconfig_vtnet0=3D" up" ifconfig_vlan48=3D" inet 192.168.48.38 netmask 0xffffff00 vlan 4= 8 vlandev vtnet0" That may be your issue... is your vtnet0 "up" in the guest. It would help a whole lot to share more of the info about your system, from= commands, not from "vm-bhyve" settings. Like ifconfig -a on the host and the guest would be a starting point. > I also tried presenting an access port from my switch on a specific VLAN = - not trimmed. Trimmed? You mean you set the switch port to untagged mode, and had the sw= itch tag/untag the packets to a specific vlan. Be sure you also set the default incoming tag at the switch if you did this= , some switches do not follow the vlan setting. > Would I still have to tag the interface on the guest in that scenario? No. If I understand what I think you meant by trimmed. > Regards, >=20 >=20 > Paul Esson | Redstor Limited > t +44 (0)118 951 5235 > m +44 (0)776 690 6514 > e paul.esson@redstor.com >=20 > ________________________________ > From: Rodney W. Grimes > Sent: Wednesday, April 25, 2018 9:33:57 PM > To: Paul Esson > Cc: Harry Schmalzbauer; freebsd-virtualization@freebsd.org > Subject: Re: bhyve networking >=20 > [ Charset windows-1252 unsupported, converting... ] > > Hi Harry, > > I?m simply using the ?vm? utility as in > > > > vm switch create public > > vm switch add public igb1 > > > > That must make underlying calls to if config or equivalent as the bridg= e and tap interfaces are created automatically. > > > > The vm template file has these relevant parameters > > > > network0_type=3D?virtio-net? > > network0_switch=3D?public? > > > > I?ve done nothing to the igb1 interface other than connect it to a phys= ical switch on the appropriate VLAN. >=20 > How have you configured your vtnet devices inside the guest? If your=20 > pass a "trunked" ethernet device to a guest the guest is going to need to= run vlan decapuslation. I do this here, and it works fine. >=20 > vtnet0: flags=3D8943 metr= ic 0 mtu 1500 > options=3D80028 > ether 58:9c:fc:0e:8b:ec > nd6 options=3D29 > media: Ethernet 10Gbase-T > status: active > lo0: flags=3D8049 metric 0 mtu 16384 > options=3D600003 > inet 127.0.0.1 netmask 0xff000000 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > nd6 options=3D21 > groups: lo > vlan48: flags=3D8843 metric 0 mtu= 1500 > ether 58:9c:fc:0e:8b:ec > inet 192.168.48.38 netmask 0xffffff00 broadcast 192.168.48.255 > nd6 options=3D29 > media: Ethernet 10Gbase-T > status: active > vlan: 48 vlanpcp: 0 parent interface: vtnet0 > groups: vlan >=20 > ... > > > > Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime): > > > Hi Folks, > > > > > > I'm struggling with networking when using vm-bhyve on FreeBSD=20 > > > 11.1-RELEASE. I have two NICs and have configured the first (igb0)=20 > > > on a management network and want to use the second (igb1) for VMs. > > > However, I can't get any VM to communicate through the virtual=20 > > > switch if I have igb1 added to it. If I take the NIC out of the=20 > > > switch and configure an ipv4 address on it I can reach other hosts=20 > > > on the relevant subnet so I believe the poet set-up is valid. If=20 > > > I replace igb1 in the switch with igb0, I can then configure VMs=20 > > > on my management network and they have network connectivity. Can=20 > > > I only use > > > > Hello, > > > > a example of the command you used was nice. > > I guess you're using if_bridge(4) ? the example would clarify. > > But there's ng_bridge(4) and vale(4) also, and others are using=20 > > OpenVSwitch... > > > > > an interface that has an IP address configured at the host level bef= ore adding it to the switch? I've used other > > > > No. > > But the interface has to be in promisc mode. And some offloading=20 > > functions must be disabled, but in case of if_bridge(4), it's done=20 > > automagically (and reverted if you remove the interface again). > > > > Hth, > > > > -harry > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list=20 > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr= eebsd.org" > > > > >=20 > -- > Rod Grimes rgrimes@freebs= d.org > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list=20 > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org" >=20 --=20 Rod Grimes rgrimes@freebsd.= org Paul Esson=A0=A0|=A0=A0Redstor Limited t=A0=A0+44 (0)118 951 5235=A0=A0|=A0=A0=A0m=A0=A0+44 (0)776 690 6514 e=A0= =A0paul.esson@redstor.com www.redstor.com -----Original Message----- From: Harry Schmalzbauer Sent: 26 April 2018 08:39 To: Paul Esson Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve networking Bez=FCglich Paul Esson's Nachricht vom 25.04.2018 23:15 (localtime): > Hi Rod, > Can you share a command line for that? I also tried presenting an=20 > access port from my switch on a specific VLAN - not trimmed. Would I=20 > still have to tag the interface on the guest in that scenario? Hmm, I lost the overview - I'm not familar with 'vm'. To filter a specific id (tag/untag frames) inside the guest: 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnn' 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnm' At boot time by rc(8): vlans_vtnet0=3D"vtnet_dmz vtnet_dmz2" create_args_vtnet_dmz=3D"vlan nnnn" create_args_vtnet_dmz2=3D"vlan nnnm" [To optionally also rename the vlan interfaces after manually creating clon= ed vlan interfaces, which is what the rc.conf(5) example does: ifconfig rename vlan0 vtnet_dmz; ifconfig rename vlan0 vtnet_dmz2; ] Hth, -harry