From owner-freebsd-virtualization@FreeBSD.ORG Sun Jan 11 22:11:09 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5F81930D for ; Sun, 11 Jan 2015 22:11:09 +0000 (UTC) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "Self-signed" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 438A78B3 for ; Sun, 11 Jan 2015 22:11:08 +0000 (UTC) Received: from smtp3.hushmail.com (smtp3a.hushmail.com [65.39.178.201]) by smtp3.hushmail.com (Postfix) with SMTP id B990DE0144 for ; Sun, 11 Jan 2015 21:32:03 +0000 (UTC) Received: from smtp.hushmail.com (w7.hushmail.com [65.39.178.32]) by smtp3.hushmail.com (Postfix) with ESMTP; Sun, 11 Jan 2015 21:32:03 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 60018E00B1; Sun, 11 Jan 2015 21:32:03 +0000 (UTC) MIME-Version: 1.0 Date: Sun, 11 Jan 2015 21:32:03 +0000 To: freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org Subject: ipv4 routing from bhyve From: williamecowell@hush.ai Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20150111213203.60018E00B1@smtp.hushmail.com> X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jan 2015 22:11:09 -0000 Hello, I hope I can have some assistance. I am trying to get networking via wlan0 but without NAT or bridging (doesn't work on wifi unless WDS). say my my main network is 10.10.2.0/24, gateway/internet is 10.10.2.1, my ip is 10.10.2.252. I started to config my bhyve network on 172.16.32.0/24 I added a bridge interface with an ip of 172.16.32.1 enable forwarding and fastforwarding. from my understanding of the handbook chapter things should work when I type: # route add -net 172.16.32.0/24 10.10.2.252 route: writing to routing socket: File exists add net 172.16.32.0: gateway 10.10.2.252 fib 0: route already in table # # netstat -4nr Routing tables Internet: Destination Gateway Flags Netif Expire default 10.10.2.1 UGS lagg0 127.0.0.1 link#3 UH lo0 10.10.2.0/24 link#5 U lagg0 10.10.2.252 link#5 UHS lo0 172.16.32.0/24 link#4 U bridge0 172.16.32.1 link#4 UHS lo0 # bridge0: flags=8843 metric 0 mtu 1500 ether 00:bd:0f:fc:01:10 inet 172.16.32.1 netmask 0xffffff00 broadcast 172.16.32.255 nd6 options=9 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=143 ifmaxaddr 0 port 6 priority 128 path cost 2000000 lagg0: flags=8843 metric 0 mtu 1500 ... inet 10.10.2.252 netmask 0xffffff00 broadcast 10.10.2.255 nd6 options=9 media: Ethernet autoselect status: active laggproto failover lagghash l2,l3,l4 laggport: alc0 flags=1 laggport: wlan0 flags=4 tap0: flags=8903 metric 0 mtu 1500 options=80000 ether 00:bd:8f:62:67:10 nd6 options=9 media: Ethernet autoselect status: no carrier wlan0: flags=8843 metric 0 mtu 1500 ... pflog0: flags=141 metric 0 mtu 33160 tap9: flags=8802 metric 0 mtu 1500 options=80000 ether 00:bd:cb:46:02:09 nd6 options=1 media: Ethernet autoselect status: no carrier tap1: flags=8802 metric 0 mtu 1500 options=80000 ether 00:bd:58:61:02:01 nd6 options=1 media: Ethernet autoselect status: no carrier Willy, PS. sorry for the x post as wasn't sure which list.. From owner-freebsd-virtualization@FreeBSD.ORG Sun Jan 11 22:47:02 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E4FEC876; Sun, 11 Jan 2015 22:47:02 +0000 (UTC) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8FAA3BD5; Sun, 11 Jan 2015 22:47:01 +0000 (UTC) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 3272716A403; Sun, 11 Jan 2015 23:46:53 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hNsD98Uk7GBQ; Sun, 11 Jan 2015 23:46:51 +0100 (CET) Received: from [IPv6:2001:4cb8:3:1:a07a:8688:5c0:df34] (unknown [IPv6:2001:4cb8:3:1:a07a:8688:5c0:df34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 2E7B016A401; Sun, 11 Jan 2015 23:46:51 +0100 (CET) Message-ID: <54B2FD59.9000407@digiware.nl> Date: Sun, 11 Jan 2015 23:46:49 +0100 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: williamecowell@hush.ai, freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: ipv4 routing from bhyve References: <20150111213203.60018E00B1@smtp.hushmail.com> In-Reply-To: <20150111213203.60018E00B1@smtp.hushmail.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jan 2015 22:47:03 -0000 On 11-1-2015 22:32, williamecowell@hush.ai wrote: > Hello, I hope I can have some assistance. > > I am trying to get networking via wlan0 but without NAT or bridging (doesn't work on wifi unless WDS). > > say my my main network is 10.10.2.0/24, gateway/internet is 10.10.2.1, my ip is 10.10.2.252. > > I started to config my bhyve network on 172.16.32.0/24 > > I added a bridge interface with an ip of 172.16.32.1 > > enable forwarding and fastforwarding. from my understanding of the handbook chapter things should work when I type: > > # route add -net 172.16.32.0/24 10.10.2.252 > route: writing to routing socket: File exists > add net 172.16.32.0: gateway 10.10.2.252 fib 0: route already in table > # > > # netstat -4nr > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 10.10.2.1 UGS lagg0 > 127.0.0.1 link#3 UH lo0 > 10.10.2.0/24 link#5 U lagg0 > 10.10.2.252 link#5 UHS lo0 > 172.16.32.0/24 link#4 U bridge0 > 172.16.32.1 link#4 UHS lo0 > # > > bridge0: flags=8843 metric 0 mtu 1500 > ether 00:bd:0f:fc:01:10 > inet 172.16.32.1 netmask 0xffffff00 broadcast 172.16.32.255 > nd6 options=9 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 6 priority 128 path cost 2000000 > lagg0: flags=8843 metric 0 mtu 1500 > ... > inet netmask 0xffffff00 broadcast 10.10.2.255 > nd6 options=9 > media: Ethernet autoselect > status: active > laggproto failover lagghash l2,l3,l4 > laggport: alc0 flags=1 > laggport: wlan0 flags=4 > tap0: flags=8903 metric 0 mtu 1500 > options=80000 > ether 00:bd:8f:62:67:10 > nd6 options=9 > media: Ethernet autoselect > status: no carrier > wlan0: flags=8843 metric 0 mtu 1500 > ... > pflog0: flags=141 metric 0 mtu 33160 > tap9: flags=8802 metric 0 mtu 1500 > options=80000 > ether 00:bd:cb:46:02:09 > nd6 options=1 > media: Ethernet autoselect > status: no carrier > tap1: flags=8802 metric 0 mtu 1500 > options=80000 > ether 00:bd:58:61:02:01 > nd6 options=1 > media: Ethernet autoselect > status: no carrier Well one of the things of concern is the fact that your tap interfaces have: status: no carrier My connected bhyve vm's have, amongst others: status: active groups: tap Opened by PID 20763 And my bridge device tells me: bridge0: flags=8843 metric 0 mtu 1500 ether 02:76:2d:3d:9c:00 inet xxx.xxx.xxx.xxx netmask 0xff000000 broadcast 37.255.255.255 nd6 options=9 groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap651 flags=143 ifmaxaddr 0 port 11 priority 128 path cost 2000000 member: tap6 flags=143 ifmaxaddr 0 port 10 priority 128 path cost 55 member: tap14041 flags=143 ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: tap13101 flags=143 ifmaxaddr 0 port 8 priority 128 path cost 2000000 member: tap12041 flags=143 ifmaxaddr 0 port 6 priority 128 path cost 2000000 member: tap13 flags=143 ifmaxaddr 0 port 4 priority 128 path cost 2000000 member: em0 flags=143 ifmaxaddr 0 port 1 priority 128 path cost 20000 So I think you first need to connect your VM's, before anything else will start to work. Like adding the tap-ifs to the bridge. And on the host itself you don't really need to add routing for the VM's because everything is actually already connected. Which is what the netstat output tells you. The routing table tells you that traffic for 172.16.32.0/24 link#4 U bridge0 is send into the the bridge0 devices, which is directly connected. And ip-nrs in that range should appear in the the arp table. And the host then knows how to get to them directly. Routing for 172.16.32.0/24, if any needed, will be required on other hosts on you network on lagg0. Unless all hosts there have 10.10.2.252 as their default route. Regards, --WjW From owner-freebsd-virtualization@FreeBSD.ORG Sun Jan 11 22:59:53 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 93E6AB06; Sun, 11 Jan 2015 22:59:53 +0000 (UTC) Received: from mail-qg0-x22a.google.com (mail-qg0-x22a.google.com [IPv6:2607:f8b0:400d:c04::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C01DCAB; Sun, 11 Jan 2015 22:59:53 +0000 (UTC) Received: by mail-qg0-f42.google.com with SMTP id q108so15614237qgd.1; Sun, 11 Jan 2015 14:59:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=jO4qqRqiAf84By1LZZ/WryI32LSxWUVSn+abzMqjUwk=; b=eK2tZAiRqw7WrSon6ZRkTqgzEFsyvJPKxBRDcafH9XAZb9iENEWc5drP6jmmGHYLFz FiqPFWi54/VmKJINIOkybThP6xSiXKk90p5ozdDsCbXYL1RyIFej44LQMWv6GvHgbVKi 71pl+cdTpObRxd9iv8H54s8xQ1U2QaRb/QsEjJiT/gaXNsf0UADFHkNykPgWAJpqQ4Qo /QpeFqCzwXibmDMW9QtUsVdyvlZAoJ7Btc9Bxb8/WL29gELgNu4sFiVtHHpUyO3YMs3O o4ixip3wUHOnGdLQGHa6ThtAhEJWA5HICzlk6WVfDGgJHNgmkWvSSIW5fKDRcIYx0OdQ INUA== MIME-Version: 1.0 X-Received: by 10.140.20.50 with SMTP id 47mr42240029qgi.61.1421017192396; Sun, 11 Jan 2015 14:59:52 -0800 (PST) Received: by 10.96.218.162 with HTTP; Sun, 11 Jan 2015 14:59:52 -0800 (PST) In-Reply-To: <20150111213203.60018E00B1@smtp.hushmail.com> References: <20150111213203.60018E00B1@smtp.hushmail.com> Date: Sun, 11 Jan 2015 14:59:52 -0800 Message-ID: Subject: Re: ipv4 routing from bhyve From: Jason Cox To: williamecowell@hush.ai Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jan 2015 22:59:53 -0000 I am in no way an expert on bhyve (have not used it) so I am going to look at this from a pure network standpoint. A bridge is a layer 2 connection and is used to bridge two separate networks over layer 2. That means they share the same IP subnet, but have no other means of connecting together physically (note: to connect two networks with different IP Subnets IE: 10.10.1.0/24 and 172.16.32.0/24 you use a router since that is layer 3). So with an example like using VirtualHost, when you add a VM and set it to bridge, you are telling VirtualHost you want the VM to be on the same network as your physical machine. Thus from the standpoint of the rest of you network, it "looks" like it is a separate machine. It could even DHCP from your LANs DHCP server and get a LAN IP if you wanted. So your bridge needs to contain the physical interface you want to pass traffic out on (like your wlan0 or lagg0). Then you add your bhyve guests as taps to that bridge to get them access to the same physical network over the bridge. You really do not need to set an IP on the bridge interface, unless say for example you where going to bridge two physical networks together on like em0 and em1. You would assign an IP to bridge0, not em0 or em1 to be able to access/admin the server from either LAN. I hope this helps some... On Sun, Jan 11, 2015 at 1:32 PM, wrote: > Hello, I hope I can have some assistance. > > I am trying to get networking via wlan0 but without NAT or bridging > (doesn't work on wifi unless WDS). > > say my my main network is 10.10.2.0/24, gateway/internet is 10.10.2.1, my > ip is 10.10.2.252. > > I started to config my bhyve network on 172.16.32.0/24 > > I added a bridge interface with an ip of 172.16.32.1 > > enable forwarding and fastforwarding. from my understanding of the > handbook chapter things should work when I type: > > # route add -net 172.16.32.0/24 10.10.2.252 > route: writing to routing socket: File exists > add net 172.16.32.0: gateway 10.10.2.252 fib 0: route already in table > # > > # netstat -4nr > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 10.10.2.1 UGS lagg0 > 127.0.0.1 link#3 UH lo0 > 10.10.2.0/24 link#5 U lagg0 > 10.10.2.252 link#5 UHS lo0 > 172.16.32.0/24 link#4 U bridge0 > 172.16.32.1 link#4 UHS lo0 > # > > bridge0: flags=8843 metric 0 mtu > 1500 > ether 00:bd:0f:fc:01:10 > inet 172.16.32.1 netmask 0xffffff00 broadcast 172.16.32.255 > nd6 options=9 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 6 priority 128 path cost 2000000 > lagg0: flags=8843 metric 0 mtu 1500 > ... > inet 10.10.2.252 netmask 0xffffff00 broadcast 10.10.2.255 > nd6 options=9 > media: Ethernet autoselect > status: active > laggproto failover lagghash l2,l3,l4 > laggport: alc0 flags=1 > laggport: wlan0 flags=4 > tap0: flags=8903 metric 0 mtu 1500 > options=80000 > ether 00:bd:8f:62:67:10 > nd6 options=9 > media: Ethernet autoselect > status: no carrier > wlan0: flags=8843 metric 0 mtu 1500 > ... > pflog0: flags=141 metric 0 mtu 33160 > tap9: flags=8802 metric 0 mtu 1500 > options=80000 > ether 00:bd:cb:46:02:09 > nd6 options=1 > media: Ethernet autoselect > status: no carrier > tap1: flags=8802 metric 0 mtu 1500 > options=80000 > ether 00:bd:58:61:02:01 > nd6 options=1 > media: Ethernet autoselect > status: no carrier > > Willy, > > PS. sorry for the x post as wasn't sure which list.. > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > -- Jason Cox From owner-freebsd-virtualization@FreeBSD.ORG Mon Jan 12 08:42:48 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 00E246C8 for ; Mon, 12 Jan 2015 08:42:47 +0000 (UTC) Received: from smtp2.hushmail.com (smtp2a.hushmail.com [65.39.178.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "Self-signed" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C959EA8A for ; Mon, 12 Jan 2015 08:42:47 +0000 (UTC) Received: from smtp2.hushmail.com (smtp2a.hushmail.com [65.39.178.237]) by smtp2.hushmail.com (Postfix) with SMTP id C4E94A020A for ; Mon, 12 Jan 2015 08:11:20 +0000 (UTC) Received: from smtp.hushmail.com (w7.hushmail.com [65.39.178.32]) by smtp2.hushmail.com (Postfix) with ESMTP; Mon, 12 Jan 2015 08:11:19 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id E7779E00B1; Mon, 12 Jan 2015 08:11:19 +0000 (UTC) MIME-Version: 1.0 Date: Mon, 12 Jan 2015 08:11:19 +0000 To: "Willem Jan Withagen" , freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: ipv4 routing from bhyve From: williamecowell@hush.ai In-Reply-To: <54B2FD59.9000407@digiware.nl> References: <20150111213203.60018E00B1@smtp.hushmail.com> <54B2FD59.9000407@digiware.nl> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20150112081119.E7779E00B1@smtp.hushmail.com> X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2015 08:42:48 -0000 >Well one of the things of concern is the fact that your tap >interfaces >have: > status: no carrier > >My connected bhyve vm's have, amongst others: > status: active > groups: tap > Opened by PID 20763 > I had no running bhyve vms when I ran the commands. >And my bridge device tells me: >bridge0: flags=8843 metric >0 mtu >1500 > ether 02:76:2d:3d:9c:00 > inet xxx.xxx.xxx.xxx netmask 0xff000000 broadcast >37.255.255.255 > nd6 options=9 > groups: bridge > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap651 >flags=143 > ifmaxaddr 0 port 11 priority 128 path cost 2000000 > member: tap6 flags=143 > ifmaxaddr 0 port 10 priority 128 path cost 55 > member: tap14041 >flags=143 > ifmaxaddr 0 port 9 priority 128 path cost 2000000 > member: tap13101 >flags=143 > ifmaxaddr 0 port 8 priority 128 path cost 2000000 > member: tap12041 >flags=143 > ifmaxaddr 0 port 6 priority 128 path cost 2000000 > member: tap13 flags=143 > ifmaxaddr 0 port 4 priority 128 path cost 2000000 > member: em0 flags=143 > ifmaxaddr 0 port 1 priority 128 path cost 20000 > > >So I think you first need to connect your VM's, before anything >else >will start to work. Like adding the tap-ifs to the bridge. > >And on the host itself you don't really need to add routing for >the VM's >because everything is actually already connected. Which is what the >netstat output tells you. The routing table tells you that traffic >for > 172.16.32.0/24 link#4 U bridge0 >is send into the the bridge0 devices, which is directly connected. >And ip-nrs in that range should appear in the the arp table. >And the host then knows how to get to them directly. > >Routing for 172.16.32.0/24, if any needed, will be required on >other >hosts on you network on lagg0. Unless all hosts there have >10.10.2.252 >as their default route. > On my gateway/router/internet connection, I added a static route via its web interface: 172.16.32.0 255.255.255.0 10.10.2.252 think maybe I am mis-understanding something. Basically, Internet connected laptop via wifi on a 10.10.2.0/24 network, laptops IP 10.10.2.252, gateway IP is 10.10.2.1. I want to put my VMs on a separate, but internet connected subnet 172.16.32.0/24 in the bhyve vms: # cat /etc/rc.conf: ifconfig_vtnet0="172.16.32.11/24" defaultrouter="172.16.32.1" # cat /etc/resolv.conf nameserver 127.0.0.1 #unbound can this sort of network config be done without NAT or a bridge, by pure routing? >Regards, >--WjW From owner-freebsd-virtualization@FreeBSD.ORG Mon Jan 12 09:40:28 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4D07AE7 for ; Mon, 12 Jan 2015 09:40:28 +0000 (UTC) Received: from smtp5.hushmail.com (smtp5a.hushmail.com [65.39.178.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.hushmail.com", Issuer "Self-signed" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 30739F7F for ; Mon, 12 Jan 2015 09:40:27 +0000 (UTC) Received: from smtp5.hushmail.com (smtp5a.hushmail.com [65.39.178.235]) by smtp5.hushmail.com (Postfix) with SMTP id 5EB46602A9 for ; Mon, 12 Jan 2015 09:03:03 +0000 (UTC) Received: from smtp.hushmail.com (w7.hushmail.com [65.39.178.32]) by smtp5.hushmail.com (Postfix) with ESMTP; Mon, 12 Jan 2015 09:03:02 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 89310E00B3; Mon, 12 Jan 2015 09:03:02 +0000 (UTC) MIME-Version: 1.0 Date: Mon, 12 Jan 2015 09:03:02 +0000 To: "Willem Jan Withagen" , freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: ipv4 routing from bhyve From: williamecowell@hush.ai In-Reply-To: <20150112081119.E7779E00B1@smtp.hushmail.com> References: <20150111213203.60018E00B1@smtp.hushmail.com> <54B2FD59.9000407@digiware.nl> <20150112081119.E7779E00B1@smtp.hushmail.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20150112090302.89310E00B3@smtp.hushmail.com> X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2015 09:40:28 -0000 please ignore, for some reason autobridge wasn't picking up newly create taps From owner-freebsd-virtualization@FreeBSD.ORG Mon Jan 12 12:43:02 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0CD60B05; Mon, 12 Jan 2015 12:43:02 +0000 (UTC) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DFD11894; Mon, 12 Jan 2015 12:43:01 +0000 (UTC) Received: from jre-mbp.elischer.org (ppp121-45-233-252.lns20.per1.internode.on.net [121.45.233.252]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t0CCgnh1030737 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 12 Jan 2015 04:42:52 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <54B3C143.70505@freebsd.org> Date: Mon, 12 Jan 2015 20:42:43 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: williamecowell@hush.ai, freebsd-net@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: ipv4 routing from bhyve References: <20150111213203.60018E00B1@smtp.hushmail.com> In-Reply-To: <20150111213203.60018E00B1@smtp.hushmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2015 12:43:02 -0000 On 1/12/15 5:32 AM, williamecowell@hush.ai wrote: > Hello, I hope I can have some assistance. > > I am trying to get networking via wlan0 but without NAT or bridging (doesn't work on wifi unless WDS). > > say my my main network is 10.10.2.0/24, gateway/internet is 10.10.2.1, my ip is 10.10.2.252. > > I started to config my bhyve network on 172.16.32.0/24 > > I added a bridge interface with an ip of 172.16.32.1 > > enable forwarding and fastforwarding. from my understanding of the handbook chapter things should work when I type: > > # route add -net 172.16.32.0/24 10.10.2.252 > route: writing to routing socket: File exists > add net 172.16.32.0: gateway 10.10.2.252 fib 0: route already in table > # > > # netstat -4nr > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 10.10.2.1 UGS lagg0 > 127.0.0.1 link#3 UH lo0 > 10.10.2.0/24 link#5 U lagg0 > 10.10.2.252 link#5 UHS lo0 > 172.16.32.0/24 link#4 U bridge0 > 172.16.32.1 link#4 UHS lo0 > # > > bridge0: flags=8843 metric 0 mtu 1500 > ether 00:bd:0f:fc:01:10 > inet 172.16.32.1 netmask 0xffffff00 broadcast 172.16.32.255 > nd6 options=9 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 6 priority 128 path cost 2000000 > lagg0: flags=8843 metric 0 mtu 1500 > ... > inet 10.10.2.252 netmask 0xffffff00 broadcast 10.10.2.255 > nd6 options=9 > media: Ethernet autoselect > status: active > laggproto failover lagghash l2,l3,l4 > laggport: alc0 flags=1 > laggport: wlan0 flags=4 > tap0: flags=8903 metric 0 mtu 1500 > options=80000 > ether 00:bd:8f:62:67:10 > nd6 options=9 > media: Ethernet autoselect > status: no carrier > wlan0: flags=8843 metric 0 mtu 1500 > ... > pflog0: flags=141 metric 0 mtu 33160 > tap9: flags=8802 metric 0 mtu 1500 > options=80000 > ether 00:bd:cb:46:02:09 > nd6 options=1 > media: Ethernet autoselect > status: no carrier > tap1: flags=8802 metric 0 mtu 1500 > options=80000 > ether 00:bd:58:61:02:01 > nd6 options=1 > media: Ethernet autoselect > status: no carrier > > Willy, > > PS. sorry for the x post as wasn't sure which list.. I may be misunderstanding something here, but are you routing or bridging? or ar eyou trying to have a virtual VM-network (using bridging) to hook VMS together, and then using routing from that to reach the outside? obviously that will require some NAT somewhere. > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > From owner-freebsd-virtualization@FreeBSD.ORG Mon Jan 12 16:54:42 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6FCCC28F for ; Mon, 12 Jan 2015 16:54:42 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id 489EE871 for ; Mon, 12 Jan 2015 16:54:41 +0000 (UTC) Received: from [192.168.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 705CF8DD5C for ; Mon, 12 Jan 2015 16:54:39 +0000 (UTC) Message-ID: <54B3FC53.3080900@freebsd.org> Date: Mon, 12 Jan 2015 11:54:43 -0500 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: Re: ipv4 routing from bhyve References: <20150111213203.60018E00B1@smtp.hushmail.com> In-Reply-To: <20150111213203.60018E00B1@smtp.hushmail.com> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GpLfWkt1VkxDEghiGsKD4BimKimlghjeW" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2015 16:54:42 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --GpLfWkt1VkxDEghiGsKD4BimKimlghjeW Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-01-11 16:32, williamecowell@hush.ai wrote: > Hello, I hope I can have some assistance. >=20 > I am trying to get networking via wlan0 but without NAT or bridging (do= esn't work on wifi unless WDS). >=20 > say my my main network is 10.10.2.0/24, gateway/internet is 10.10.2.1, = my ip is 10.10.2.252. >=20 > I started to config my bhyve network on 172.16.32.0/24 >=20 > I added a bridge interface with an ip of 172.16.32.1 >=20 > enable forwarding and fastforwarding. from my understanding of the hand= book chapter things should work when I type: >=20 > # route add -net 172.16.32.0/24 10.10.2.252 > route: writing to routing socket: File exists > add net 172.16.32.0: gateway 10.10.2.252 fib 0: route already in table > #=20 >=20 > # netstat -4nr > Routing tables >=20 > Internet: > Destination Gateway Flags Netif Expire > default 10.10.2.1 UGS lagg0 > 127.0.0.1 link#3 UH lo0 > 10.10.2.0/24 link#5 U lagg0 > 10.10.2.252 link#5 UHS lo0 > 172.16.32.0/24 link#4 U bridge0 > 172.16.32.1 link#4 UHS lo0 > #=20 >=20 > bridge0: flags=3D8843 metric 0 = mtu 1500 > ether 00:bd:0f:fc:01:10 > inet 172.16.32.1 netmask 0xffffff00 broadcast 172.16.32.255=20 > nd6 options=3D9 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=3D143 > ifmaxaddr 0 port 6 priority 128 path cost 2000000 > lagg0: flags=3D8843 metric 0 mt= u 1500 > ... > inet 10.10.2.252 netmask 0xffffff00 broadcast 10.10.2.255=20 > nd6 options=3D9 > media: Ethernet autoselect > status: active > laggproto failover lagghash l2,l3,l4 > laggport: alc0 flags=3D1 > laggport: wlan0 flags=3D4 > tap0: flags=3D8903 metric 0 mtu= 1500 > options=3D80000 > ether 00:bd:8f:62:67:10 > nd6 options=3D9 > media: Ethernet autoselect > status: no carrier > wlan0: flags=3D8843 metric 0 mt= u 1500 > ... > pflog0: flags=3D141 metric 0 mtu 33160 > tap9: flags=3D8802 metric 0 mtu 1500 > options=3D80000 > ether 00:bd:cb:46:02:09 > nd6 options=3D1 > media: Ethernet autoselect > status: no carrier > tap1: flags=3D8802 metric 0 mtu 1500 > options=3D80000 > ether 00:bd:58:61:02:01 > nd6 options=3D1 > media: Ethernet autoselect > status: no carrier >=20 > Willy, >=20 > PS. sorry for the x post as wasn't sure which list.. >=20 > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr= eebsd.org" >=20 You can use ip forwarding (or fast forwarding) to send traffic generated on the new subnet (172.16.32.0/24) out to your network. But unless you configure a static route (or setup a routing protocol like RIP), the other hosts on your network will not know how to reach 172.16.32.0/24 to reply. So the static route you were trying to add, would need to be added to every machine EXCEPT the VM Host, which already has an IP in that subnet. --=20 Allan Jude --GpLfWkt1VkxDEghiGsKD4BimKimlghjeW Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJUs/xWAAoJEJrBFpNRJZKfeMkP/irmCtZPYndktcNBrmpsTcYB iaDzLOy+k1+hk3POyBYTgeqhtu2ttEcuRvO4szPTQ4uwNQDMFqYlKO73iwy/tgTn uRZwenP9yE4IJfwC+fUKvKs2puUA/U8Aay1FOptC5/4xZH/m/f8IbsO+dI7ql1si OLTaN0q1vikECp8mZsbImojRRYMvHIF3VqLshByYWRDSNxG+m9NVJXNPkpm9YLZW Vlo2+0EhdLKSPyqRCh4lLT/QhIUMsg2u4ISBN6kLXt9qzvdNR1xZC8JaP9JttxRN poJt3St48oWAEmPlmT2OWl3bLcw1RXHg/GOcN/JBnJ33ZiGBv+72nHYCv65YVE/r 3YQM3rGtG8+W3qhmUZpnV9lCZjZxd4NcrxvvQ2hVa90YYZhKfMmkM/BSVTln0t1g 2JuHgYV9V13tlItXbaboxq3HyLdS/2Za/5ROi8748MutjH2dvZXa+g90Ld/wYBCT fL1q3XJ2sNCNPCNtNkj2G++dASdKtReqLJ9iseutd7c4UVpIkpUcBbRnHD6M8w6d MYrw8f6VRODjPCpVQlqZFfCL7yLikDicL0qgI1xUgd4fU+4LRxoY/LAz8L+nXhM2 9p9bZIZBrUewxIm7JmXNfhXJSXkCSHxaEW/q+W61HkWJU1IQ2di57xWHVqPbNxGG zudWQK0rK0QP7b3olAlG =HhdC -----END PGP SIGNATURE----- --GpLfWkt1VkxDEghiGsKD4BimKimlghjeW-- From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 13 21:45:34 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82771D23 for ; Tue, 13 Jan 2015 21:45:34 +0000 (UTC) Received: from mail-qg0-x233.google.com (mail-qg0-x233.google.com [IPv6:2607:f8b0:400d:c04::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C7998D for ; Tue, 13 Jan 2015 21:45:34 +0000 (UTC) Received: by mail-qg0-f51.google.com with SMTP id i50so4351795qgf.10 for ; Tue, 13 Jan 2015 13:45:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=tgIZ5Q3EUEXYxZS76VB5t4UqEut4auD3MQFl2MtOgKs=; b=lfl5Pu+K+BcbmucliugHpkm2T5NsVzCtX+U6wZDOWy1nI3EIxcznPaahjZcPFLiiFz +FmJLsEEkHujfm27h3uXdVOu0GJ+rfYD/Es6x/qIIAPp4oCrncidQ66o04YmgJciu51k bQQrw5KwjHUxVXAVNoDX5I3FzEhBMdFYa90ED8BUxMET5xCBTUu7mUnONFMwjA2UogDC ZBznK0/80+iPnX92phJFATsMy6mzI/TwsGILPfW+I4411sRAsSJrbNdwCrNgZjrlfEHX zV1G0a1J2gA0eZ+dwjD+n4/CMxh4m/jvftR0EsPvyIS+o7YYZY7aXmVeUgM6AEAU4Ce9 xiyg== MIME-Version: 1.0 X-Received: by 10.229.219.74 with SMTP id ht10mr1236122qcb.25.1421185533192; Tue, 13 Jan 2015 13:45:33 -0800 (PST) Received: by 10.140.96.13 with HTTP; Tue, 13 Jan 2015 13:45:33 -0800 (PST) Received: by 10.140.96.13 with HTTP; Tue, 13 Jan 2015 13:45:33 -0800 (PST) Date: Tue, 13 Jan 2015 13:45:33 -0800 Message-ID: Subject: Attaching block devices to a VM in bhyve From: Jonathan Wong To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 21:45:34 -0000 Hi. I was looking around and it seems is possible to attach zvols to a bhyve vm, at least according to the vmrc script. Is it possible to attach a block device to a VM after it has been launched? Thanks, Jon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 13 21:47:03 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B58DFD74 for ; Tue, 13 Jan 2015 21:47:03 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id 8EC7F9B for ; Tue, 13 Jan 2015 21:47:03 +0000 (UTC) Received: from [192.168.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id B28CC8EBF5 for ; Tue, 13 Jan 2015 21:47:01 +0000 (UTC) Message-ID: <54B5925F.8010308@freebsd.org> Date: Tue, 13 Jan 2015 16:47:11 -0500 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: Re: Attaching block devices to a VM in bhyve References: In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OpENoO2lA96kAODoFkMqKcbonlsvb6ioF" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 21:47:03 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --OpENoO2lA96kAODoFkMqKcbonlsvb6ioF Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-01-13 16:45, Jonathan Wong wrote: > Hi. >=20 > I was looking around and it seems is possible to attach zvols to a bhyv= e > vm, at least according to the vmrc script. Is it possible to attach a b= lock > device to a VM after it has been launched? >=20 > Thanks, > Jon > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr= eebsd.org" >=20 You can attach a block device, just use the path to the device in the place of a filename, same way you do with a zvol. However, it is not currently possible to attach a block device after bhyve has started. --=20 Allan Jude --OpENoO2lA96kAODoFkMqKcbonlsvb6ioF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJUtZJiAAoJEJrBFpNRJZKfuU4P/jFyvi3s556b+jdazueUATS+ 010gVh3QzLDsQ6Qy0WNVvNu7jaAdpK+k8GkHC4V4UvleFu0YcpDx1Nqz39fUvzLj TsD8g3+n1RM/u6XTMGJgUpeXtC7NL9hbRxbgG7xx3LjnXXLETXhbx/EqyO+dwrck A5+YwEFjjWXrZH5VOVZ1LuHh2I8xG5oiGluq/2bG/2GLo0O2rNL6Y9m0kqOILvHR lg450PRrU3rEqF3raE97CMweHWQhepVDRkQLUWzQZ9vgsgfWtBzRcsWRwmjrDip+ d8yWMqEQXHmIp0nihFJTKUsqqEfp7fYzfOCFwY3g/AtsEV0x32PH0FdPWKaKZ2ph GJPAQ4+1ri2HkqiiDKWkiOnE0daBLLVGZhQB+sc3Nlog1BbZOFJO2Artxxfq38YO kqEezxGb+Z9Xx1dFmf1M92TD5bFe/MG+7IS6ITpxOsfb7OdMgtukyodZSiROP0RX jPhbNAAxWr5R/K4M7/ZbZSAXsSgCbxBKDYtN2IMDRIj6Re9hTW42iTaehxqlZvyw VFEBXuVPSd2PU0N7FwefNABH48MV4KhBwYmb+hNNf/Gud0GXyKnSwE9U4PoAeh5n Fhixe0s106xpPXAwIibfSe8ad+m8nSH4hiteJkDKmtmM66ZGLf4hr5x8hFXcacgF 8KI3RR+V3GEA/UrF3fod =RB1c -----END PGP SIGNATURE----- --OpENoO2lA96kAODoFkMqKcbonlsvb6ioF-- From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 13 22:16:21 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A2129844; Tue, 13 Jan 2015 22:16:21 +0000 (UTC) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 62B223DA; Tue, 13 Jan 2015 22:16:21 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id l89so4451438qgf.12; Tue, 13 Jan 2015 14:16:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=SpHbalUvXClPxHt0e7Zei7ItsSfb01WroY8weZCLML0=; b=e2iq4uK32OE5tRzMctILJnsDmg9vpa+oe+GAo0SMFJx1nf0tBV7lDJTktng4cIou0E /G8gckfdxVGo458nq+wfClanogcsdvtbiGQ+osMWwFzSjHIWKOzfwvk0/qfAcNDo+m5a 10/sRobDodPSmUfcPiiTv+E2muPeh09FPJrCwcYDMcuCSK4/PHVfYddRMa0GxS4+n6W/ rh7nRtUkZ6ZX+RekK2m9vaIpfBBE6mxkp98qT6XQe6NyeOT1v19fDD36J68fUGXYMoo6 UMTk5iktAiv/wg2SgpXDp9DhGI5u+RX7URUo7SyeZfvl8cWfucPqlZjNEXKsTqzwaeHj 7WIQ== MIME-Version: 1.0 X-Received: by 10.229.211.193 with SMTP id gp1mr1493755qcb.19.1421187378837; Tue, 13 Jan 2015 14:16:18 -0800 (PST) Received: by 10.140.96.13 with HTTP; Tue, 13 Jan 2015 14:16:18 -0800 (PST) Received: by 10.140.96.13 with HTTP; Tue, 13 Jan 2015 14:16:18 -0800 (PST) In-Reply-To: <54B5925F.8010308@freebsd.org> References: <54B5925F.8010308@freebsd.org> Date: Tue, 13 Jan 2015 14:16:18 -0800 Message-ID: Subject: Re: Attaching block devices to a VM in bhyve From: Jonathan Wong To: Allan Jude Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 22:16:21 -0000 Thanks. Do you think it would be a hard thing to add in to bhyve or is it related to something that's already being worked on? Thanks, Jon On Jan 13, 2015 1:47 PM, "Allan Jude" wrote: > On 2015-01-13 16:45, Jonathan Wong wrote: > > Hi. > > > > I was looking around and it seems is possible to attach zvols to a bhyve > > vm, at least according to the vmrc script. Is it possible to attach a > block > > device to a VM after it has been launched? > > > > Thanks, > > Jon > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > > > > You can attach a block device, just use the path to the device in the > place of a filename, same way you do with a zvol. > > However, it is not currently possible to attach a block device after > bhyve has started. > > -- > Allan Jude > > From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 13 22:22:11 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E7AA8BAC for ; Tue, 13 Jan 2015 22:22:11 +0000 (UTC) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id ABF9768E for ; Tue, 13 Jan 2015 22:22:11 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTP id 08CDC1277F; Wed, 14 Jan 2015 08:22:10 +1000 (EST) Received: from Peters-MacBook-Pro.local ([64.245.0.210]) by dommail.onthenet.com.au (MOS 4.4.4-GA) with ESMTP id CAX03441 (AUTH peterg@ptree32.com.au); Wed, 14 Jan 2015 08:22:09 +1000 Message-ID: <54B59A8E.8090206@freebsd.org> Date: Tue, 13 Jan 2015 14:22:06 -0800 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Zaphod Beeblebrox Subject: re: There's a whole lot of X34x0's out there that fail. Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 22:22:12 -0000 Hi, (moving this to the -virtualization list) > From what I've read around here, an entire swatch of CPUs have a > problem --- they have all the virtualization gunk save the ability to run > in16 bit mode in emulation. An example would be the X34x0 ... the > cornerstone processor for an entire generation of servers. I, for > instance, have four of them. While it is unfortunate that bhyve doesn't fully support Nehalems, we decided that the amount of work required to complete the support wasn't worth it, given that all follow-on models have 16-bit support. That being said, I'm certainly willing to provide help to anyone who wants to take on that work. > The "fail" in the subject line is about their inability to do two > things under bhyve: > > 1) launch linux through grub > > 2) have a VM with more than one processor > > Can we adopt what linux does to work around it? The various Linux VM solutions use an instruction emulator to run until the CPU is about to transition to 32-bit paged mode, and then cut over to using VT-x. Pulling this in to bhyve is a decidely non-trivial amount of work, especially since the majority of x86 emulator codebases are GPL'd. > Is it even necessary to boot linux with Grub? It's not Grub that is the issue, but the fact that the processor can't run in anything less than 32-bit paged mode. Grub boots Linux in either 16-bit real mode or 32-bit protected mode with paging disabled. However, here is a 64-bit entry point available for Linux/x64 (see "64-bit BOOT PROTOCOL" at https://www.kernel.org/doc/Documentation/x86/boot.txt). For anyone willing to have a go at this, my recommendation would be to modify grub-bhyve to boot Linux/x64 directly into 64-bit mode. This is already done for kFreeBSD/amd64 in grub (and for FreeBSD in bhyveload/libvmmapi) so that setup code could be reused. For MP support, the work would be to look at the Linux/FreeBSD MP trampolines and write an emulator that could either recognize those and skip them, or emulate only the instructions required to bring the vCPU into 32-bit protected mode, and then switch over to VT-x h/w emulation. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 13 23:23:22 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF61EE5A; Tue, 13 Jan 2015 23:23:22 +0000 (UTC) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 90023D0C; Tue, 13 Jan 2015 23:23:22 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTP id A2EE612751; Wed, 14 Jan 2015 09:23:19 +1000 (EST) Received: from Peters-MacBook-Pro.local ([64.245.0.210]) by dommail.onthenet.com.au (MOS 4.4.4-GA) with ESMTP id CAX07486 (AUTH peterg@ptree32.com.au); Wed, 14 Jan 2015 09:23:18 +1000 Message-ID: <54B5A8E4.7070906@freebsd.org> Date: Tue, 13 Jan 2015 15:23:16 -0800 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Jonathan Wong Subject: Re: Attaching block devices to a VM in bhyve References: <54B5925F.8010308@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-virtualization@freebsd.org, Allan Jude X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 23:23:22 -0000 Hi Jon, > Do you think it would be a hard thing to add in to bhyve or is it > related to something that's already being worked on? Quite difficult as it stands with bhyve: there's currently no way to dynamically modify a running VM's configuration. This limitation will be removed at some point but not in the short term. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Wed Jan 14 01:05:14 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07E65CDA; Wed, 14 Jan 2015 01:05:14 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B2162901; Wed, 14 Jan 2015 01:05:13 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id w8so3868374qac.13; Tue, 13 Jan 2015 17:05:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vZJeCe1Xw1g2p2kHpDZASgKrnNHhGLlROP+ZM3vFW1c=; b=eb4p5Mrd5vflxs9o/deRxYYe18KKhBpO/CDvyfdM5OkOTBwNJIC8rw35bquKsDXFmR vWi7kWMZBBsEIiJtyRbfR9Xr/SkUmsWX0MBunv+9oL709sbEvYo5BJ1u+Y3m+LXOvzwA pJw/aLZv1PRC4BadH69U+fysUIPONxhRRCjKReSdFamiAHkjaKrdeR1i60K7BZq8No8o TPCvmlgD63z9l6lMj84Q9s3SDXfu6TeWzF+QHkkypSxuZkX1K3na6iWCiz/w4N0EStvH Shp/WcKb/zTIPk7BNNHdnWye3PJTO2iNsDKc2TCZVsNmxbtanGAO778GtZaNR4EAM7hm Ibwg== MIME-Version: 1.0 X-Received: by 10.140.28.200 with SMTP id 66mr2461725qgz.16.1421197512774; Tue, 13 Jan 2015 17:05:12 -0800 (PST) Received: by 10.140.96.13 with HTTP; Tue, 13 Jan 2015 17:05:12 -0800 (PST) In-Reply-To: <54B5A8E4.7070906@freebsd.org> References: <54B5925F.8010308@freebsd.org> <54B5A8E4.7070906@freebsd.org> Date: Tue, 13 Jan 2015 17:05:12 -0800 Message-ID: Subject: Re: Attaching block devices to a VM in bhyve From: Jonathan Wong To: Peter Grehan Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-virtualization@freebsd.org, Allan Jude X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 01:05:14 -0000 Thanks. Last question. I also noticed that iSCSI targets are supported. I might have missed it, but is there a link to documentation somewhere? (Does one simply use the "default" portal for iSCSI? or how are portals specified.) Thanks, Jon On Tue, Jan 13, 2015 at 3:23 PM, Peter Grehan wrote: > Hi Jon, > > Do you think it would be a hard thing to add in to bhyve or is it >> related to something that's already being worked on? >> > > Quite difficult as it stands with bhyve: there's currently no way to > dynamically modify a running VM's configuration. This limitation will be > removed at some point but not in the short term. > > later, > > Peter. > > From owner-freebsd-virtualization@FreeBSD.ORG Wed Jan 14 01:10:22 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1D01BD5B for ; Wed, 14 Jan 2015 01:10:22 +0000 (UTC) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8BAB6921 for ; Wed, 14 Jan 2015 01:10:21 +0000 (UTC) Received: by mail-lb0-f170.google.com with SMTP id 10so5558939lbg.1 for ; Tue, 13 Jan 2015 17:10:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=m+iiM5MIjhhsMm7SRj/VqYbywg/ne+x8KjTfsAGLBS0=; b=LdNBtj6TsB4m1Evq0JUEBxZUrOEasHHRWkY0ld8679GBeJc1rGmAPqEKVE7FSjqSw3 jkWvvjrqHcCObwDWjySxGLA8jP0L6DyPPVsefzetDRPI9sJA/A+LY3OIs5ejyTcOhchP Y6q/kMOwVOhfgqSKqgvvHODvVOqYQmITU3UsVvYZj8HUUEGnN9tuy/FFeQZZXKJSPEPp FG6AagzmsFMMO/Qxah/kt61HOb1WJLQvpBrId0T+mTgnPhFI7xqhbs8zabeKiQtWdDzf 4JB9a2xz1Z110yJDpk54BezzMD2WbEh6Q4dHbod3KqSjgciULStoNZvxbZhBu/EEw8Vu ka1Q== MIME-Version: 1.0 X-Received: by 10.152.203.201 with SMTP id ks9mr1088235lac.57.1421197819610; Tue, 13 Jan 2015 17:10:19 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.129.3 with HTTP; Tue, 13 Jan 2015 17:10:19 -0800 (PST) In-Reply-To: References: <54B5925F.8010308@freebsd.org> <54B5A8E4.7070906@freebsd.org> Date: Tue, 13 Jan 2015 17:10:19 -0800 X-Google-Sender-Auth: DQ6uI4bXURdcdx9al21vafmxWNM Message-ID: Subject: Re: Attaching block devices to a VM in bhyve From: Craig Rodrigues To: Jonathan Wong Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 01:10:22 -0000 On Tue, Jan 13, 2015 at 5:05 PM, Jonathan Wong wrote: > > Thanks. Last question. I also noticed that iSCSI targets are supported. I > might have missed it, but is there a link to documentation somewhere iSCSI Initiator and Target Configuration: https://www.freebsd.org/doc/handbook/network-iscsi.html -- Craig From owner-freebsd-virtualization@FreeBSD.ORG Wed Jan 14 01:18:37 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8DABFE1A; Wed, 14 Jan 2015 01:18:37 +0000 (UTC) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 4D7D99E8; Wed, 14 Jan 2015 01:18:36 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTP id 9C33112811; Wed, 14 Jan 2015 11:18:34 +1000 (EST) Received: from Peters-MacBook-Pro.local ([64.245.0.210]) by dommail.onthenet.com.au (MOS 4.4.4-GA) with ESMTP id CAX14633 (AUTH peterg@ptree32.com.au); Wed, 14 Jan 2015 11:18:32 +1000 Message-ID: <54B5C3E6.3010204@freebsd.org> Date: Tue, 13 Jan 2015 17:18:30 -0800 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Jonathan Wong Subject: Re: Attaching block devices to a VM in bhyve References: <54B5925F.8010308@freebsd.org> <54B5A8E4.7070906@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-virtualization@freebsd.org, Allan Jude X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 01:18:37 -0000 Hi Jon, > Last question. I also noticed that iSCSI targets are supported. > I might have missed it, but is there a link to documentation somewhere? > (Does one simply use the "default" portal for iSCSI? or how are portals > specified.) Targets are indirectly supported in that you have to create a block device on the host using the FreeBSD iscsi initiator, and then point bhyve at that block device (see the handbook section, https://www.freebsd.org/doc/handbook/network-iscsi.html). later, Peter.