Date: Thu, 26 Apr 2018 12:40:38 +0000 From: Paul Esson <paul.esson@redstor.com> To: Harry Schmalzbauer <freebsd@omnilan.de> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: RE: bhyve networking Message-ID: <HE1PR0102MB258808D79971776BD0EF7D219E8E0@HE1PR0102MB2588.eurprd01.prod.exchangelabs.com> In-Reply-To: <5AE18222.6000900@omnilan.de> References: <HE1PR0102MB25884F467EE8019D3D298E419E8F0@HE1PR0102MB2588.eurprd01.prod.exchangelabs.com>, <201804252033.w3PKXv8M097827@pdx.rh.CN85.dnsmgr.net> <HE1PR0102MB2588F1B078AB3A5D89DD6D1A9E8F0@HE1PR0102MB2588.eurprd01.prod.exchangelabs.com> <5AE18222.6000900@omnilan.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Folks,
Apologies for the lack of detail on my first post. To recap, I am attempti=
ng to set-up a guest using vm-bhyve. I have a Dell PER730xd server with qu=
ad-port INTEL 350 NIC. The first two ports have been configured on a) a ma=
nagement LAN for the host and b) an application LAN for the guests.
FreeBSD 11.1-RELEASE-p9
Dell PowerEdge R730xd - INTEL i350 NICs
NIC-1 igb0 24:6E:96:B4:61:CC VLAN92 ge-6/0/11 (Host)
NIC-2 igb1 24:6E:96:B4:61:CD VLAN101 ge-6/0/18 (Guests) - not a trunk
Both interfaces are active as viewed from the host, but I have only assigne=
d an ipv4 address to igb0 for management of the host
igb0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 150=
0 options=3D6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWC=
SUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cc
hwaddr 24:6e:96:b4:61:cc
inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=3D8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 =
options=3D6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN=
_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cd
hwaddr 24:6e:96:b4:61:cd
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
=20
If I assign a temporary address to igb1 I can then ping other computers on =
the guests subnet - I've had to hide the address as the network is restrict=
ed.
# ifconfig igb1 inet xx.xxx.xxx.xx/25 up
# ping xx.xxx.xxx.xx
PING xx.xxx.xxx.xx (xx.xxx.xxx.xx): 56 data bytes
64 bytes from xx.xxx.xxx.xx: icmp_seq=3D0 ttl=3D64 time=3D0.145 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=3D1 ttl=3D64 time=3D0.080 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=3D2 ttl=3D64 time=3D0.078 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=3D3 ttl=3D64 time=3D0.077 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=3D4 ttl=3D64 time=3D0.076 ms
I then used the "vm" command to create a virtual switch and add interface i=
gb1 to it. This automatically created the bridge interface.
root@dc1-olbp-hn-01:~ # vm switch create public
root@dc1-olbp-hn-01:~ # vm switch add public igb1
root@dc1-olbp-hn-01:~ # vm switch info public
------------------------
Virtual Switch: public
------------------------
type: auto
ident: bridge0
vlan: -
nat: -
physical-ports: igb1
bytes-in: 0 (0.000B)
bytes-out: 0 (0.000B)
Finally, I created a guest VM and gave its NIC the same ipv4 address detail=
s I used previously to test igb1 from the host. This automatically created=
the tap interface.
igb0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 150=
0 options=3D6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLA=
N_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cc
hwaddr 24:6e:96:b4:61:cc
inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=3D8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mt=
u 1500 options=3D6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,V=
LAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cd
hwaddr 24:6e:96:b4:61:cd
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu =
1500
description: vm-public
ether 02:ee:ce:b0:6a:00
nd6 options=3D1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
member: igb1 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 20000
tap0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0=
mtu 1500
description: vmnet-testvm-0-public
options=3D80000<LINKSTATE>
ether 00:bd:dd:51:0a:00
hwaddr 00:bd:dd:51:0a:00
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: tap
Opened by PID 1791
>From the guest VM I can see that the interface vtnet0 is up and has the rel=
evant ipv4 address information. However, I cannot communicate with any oth=
er computer on the guest subnet or beyond.
vtnet0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric=
0 mtu 1500
options=3D80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:08:4a:20
hwaddr 58:9c:fc:08:4a:20
inet xx.xxx.xxx.xx netmask 0xffffff80 broadcast xx.xxx.xxx.xx
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
The guest configuration file has the following network details
network0_type=3D"virtio-net"
network0_switch=3D"public"
>From the vm-bhyve.log I see the following
Apr 26 07:59:23: initialising
Apr 26 07:59:23: [loader: bhyveload]
Apr 26 07:59:23: [uefi: no]
Apr 26 07:59:23: [cpu: 1]
Apr 26 07:59:23: [memory: 256M]
Apr 26 07:59:23: [hostbridge: standard]
Apr 26 07:59:23: [com ports: com1]
Apr 26 07:59:23: [uuid: 417cfb63-491f-11e8-949b-246e96b461cc]
Apr 26 07:59:23: [utctime: no]
Apr 26 07:59:23: [debug mode: no]
Apr 26 07:59:23: [primary disk: disk0]
Apr 26 07:59:23: [primary disk dev: sparse-zvol]
Apr 26 07:59:23: generated static mac 58:9c:fc:08:4a:20 (based on 'testvm:0=
:1524725963:0')
Apr 26 07:59:23: initialising network device tap0
Apr 26 07:59:23: adding tap0 -> bridge0 (public)
Apr 26 07:59:23: booting
Should I have to supply ipv4 details anywhere other than the guest's own vt=
net0 interface? If I re-configure the switch to remove the igb1 interface =
and add igb0 instead, then change the guest ipv4 address details to the man=
agement network (172.16.92.0/24), I can connect to other computers on that =
subnet and beyond. =20
vtnet0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric=
0 mtu 1500
options=3D80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:08:4a:20
hwaddr 58:9c:fc:08:4a:20
inet 172.16.92.21 netmask 0xffffff80 broadcast 172.16.92.127
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
$ ping 172.16.92.11
PING 172.16.92.11 (172.16.92.11): 56 data bytes
64 bytes from 172.16.92.11: icmp_seq=3D0 ttl=3D64 time=3D0.416 ms
64 bytes from 172.16.92.11: icmp_seq=3D1 ttl=3D64 time=3D0.371 ms
64 bytes from 172.16.92.11: icmp_seq=3D2 ttl=3D64 time=3D0.369 ms
--- 172.16.92.11 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev =3D 0.369/0.385/0.416/0.022 ms
$ telnet bbc.co.uk 443
Trying 151.101.192.81...
Connected to bbc.co.uk.
Escape character is '^]'.
Regards,
Paul Esson | Redstor Limited
t +44 (0)118 951 5235 | m +44 (0)776 690 6514
e paul.esson@redstor.com
www.redstor.com
-----Original Message-----
From: Rodney W. Grimes <freebsd-rwg@pdx.rh.CN85.dnsmgr.net>=20
Sent: 25 April 2018 22:31
To: Paul Esson <paul.esson@redstor.com>
Cc: freebsd-virtualization@freebsd.org
Subject: Re: bhyve networking
> Hi Rod,
> Can you share a command line for that?=20
Its not just a single command, but you want these in /etc/rc.conf of the GU=
EST:
network_interfaces=3D"lo0"
cloned_interfaces=3D"vlan48"
ifconfig_lo0=3D" inet 127.0.0.1 netmask 0xff000000"
ifconfig_vtnet0=3D" up"
ifconfig_vlan48=3D" inet 192.168.48.38 netmask 0xffffff00 vlan 4=
8 vlandev vtnet0"
That may be your issue... is your vtnet0 "up" in the guest.
It would help a whole lot to share more of the info about your system, from=
commands, not from "vm-bhyve" settings.
Like
ifconfig -a
on the host and the guest would be a starting point.
> I also tried presenting an access port from my switch on a specific VLAN =
- not trimmed.
Trimmed? You mean you set the switch port to untagged mode, and had the sw=
itch tag/untag the packets to a specific vlan.
Be sure you also set the default incoming tag at the switch if you did this=
, some switches do not follow the vlan setting.
> Would I still have to tag the interface on the guest in that scenario?
No. If I understand what I think you meant by trimmed.
> Regards,
>=20
>=20
> Paul Esson | Redstor Limited
> t +44 (0)118 951 5235
> m +44 (0)776 690 6514
> e paul.esson@redstor.com
>=20
> ________________________________
> From: Rodney W. Grimes <freebsd-rwg@pdx.rh.CN85.dnsmgr.net>
> Sent: Wednesday, April 25, 2018 9:33:57 PM
> To: Paul Esson
> Cc: Harry Schmalzbauer; freebsd-virtualization@freebsd.org
> Subject: Re: bhyve networking
>=20
> [ Charset windows-1252 unsupported, converting... ]
> > Hi Harry,
> > I?m simply using the ?vm? utility as in
> >
> > vm switch create public
> > vm switch add public igb1
> >
> > That must make underlying calls to if config or equivalent as the bridg=
e and tap interfaces are created automatically.
> >
> > The vm template file has these relevant parameters
> >
> > network0_type=3D?virtio-net?
> > network0_switch=3D?public?
> >
> > I?ve done nothing to the igb1 interface other than connect it to a phys=
ical switch on the appropriate VLAN.
>=20
> How have you configured your vtnet devices inside the guest? If your=20
> pass a "trunked" ethernet device to a guest the guest is going to need to=
run vlan decapuslation. I do this here, and it works fine.
>=20
> vtnet0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metr=
ic 0 mtu 1500
> options=3D80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> ether 58:9c:fc:0e:8b:ec
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T <full-duplex>
> status: active
> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet 127.0.0.1 netmask 0xff000000
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
> vlan48: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu=
1500
> ether 58:9c:fc:0e:8b:ec
> inet 192.168.48.38 netmask 0xffffff00 broadcast 192.168.48.255
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T <full-duplex>
> status: active
> vlan: 48 vlanpcp: 0 parent interface: vtnet0
> groups: vlan
>=20
> ...
> >
> > Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime):
> > > Hi Folks,
> > >
> > > I'm struggling with networking when using vm-bhyve on FreeBSD=20
> > > 11.1-RELEASE. I have two NICs and have configured the first (igb0)=20
> > > on a management network and want to use the second (igb1) for VMs. =20
> > > However, I can't get any VM to communicate through the virtual=20
> > > switch if I have igb1 added to it. If I take the NIC out of the=20
> > > switch and configure an ipv4 address on it I can reach other hosts=20
> > > on the relevant subnet so I believe the poet set-up is valid. If=20
> > > I replace igb1 in the switch with igb0, I can then configure VMs=20
> > > on my management network and they have network connectivity. Can=20
> > > I only use
> >
> > Hello,
> >
> > a example of the command you used was nice.
> > I guess you're using if_bridge(4) ? the example would clarify.
> > But there's ng_bridge(4) and vale(4) also, and others are using=20
> > OpenVSwitch...
> >
> > > an interface that has an IP address configured at the host level bef=
ore adding it to the switch? I've used other
> >
> > No.
> > But the interface has to be in promisc mode. And some offloading=20
> > functions must be disabled, but in case of if_bridge(4), it's done=20
> > automagically (and reverted if you remove the interface again).
> >
> > Hth,
> >
> > -harry
> > _______________________________________________
> > freebsd-virtualization@freebsd.org mailing list=20
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr=
eebsd.org"
> >
> >
>=20
> --
> Rod Grimes rgrimes@freebs=
d.org
> _______________________________________________
> freebsd-virtualization@freebsd.org mailing list=20
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free=
bsd.org"
>=20
--=20
Rod Grimes rgrimes@freebsd.=
org
Paul Esson=A0=A0|=A0=A0Redstor Limited
t=A0=A0+44 (0)118 951 5235=A0=A0|=A0=A0=A0m=A0=A0+44 (0)776 690 6514
e=A0=A0paul.esson@redstor.com
www.redstor.com
-----Original Message-----
From: Harry Schmalzbauer <freebsd@omnilan.de>=20
Sent: 26 April 2018 08:39
To: Paul Esson <paul.esson@redstor.com>
Cc: freebsd-virtualization@freebsd.org
Subject: Re: bhyve networking
Bez=FCglich Paul Esson's Nachricht vom 25.04.2018 23:15 (localtime):
> Hi Rod,
> Can you share a command line for that? I also tried presenting an=20
> access port from my switch on a specific VLAN - not trimmed. Would I=20
> still have to tag the interface on the guest in that scenario?
Hmm, I lost the overview - I'm not familar with 'vm'.
To filter a specific id (tag/untag frames) inside the guest:
'ifconfig vlan[N] create vlandev vtnet0 vlan nnnn'
'ifconfig vlan[N] create vlandev vtnet0 vlan nnnm'
At boot time by rc(8):
vlans_vtnet0=3D"vtnet_dmz vtnet_dmz2"
create_args_vtnet_dmz=3D"vlan nnnn"
create_args_vtnet_dmz2=3D"vlan nnnm"
[To optionally also rename the vlan interfaces after manually creating clon=
ed vlan interfaces, which is what the rc.conf(5) example does:
ifconfig rename vlan0 vtnet_dmz; ifconfig rename vlan0 vtnet_dmz2; ]
Hth,
-harry
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?HE1PR0102MB258808D79971776BD0EF7D219E8E0>