Date: Thu, 15 Dec 2022 16:10:53 -0800 From: Rob Ballantyne <robballantyne3@gmail.com> To: leif@ofwilsoncreek.com Cc: freebsd-cloud@freebsd.org Subject: Re: What is a VPC (google's specifically but it could be more general) really? Message-ID: <CAKLrb5eGPp2-h3bWEt-ZicUdxWtvEUE4fG4ZAW8=y%2B16miuVtw@mail.gmail.com> In-Reply-To: <CAK-wPOhK1gX5%2BV1Z9nNauRF-oC85Jc-vmi-bKq_-YTKVt10C8Q@mail.gmail.com> References: <CAKLrb5do6Evnn2WKKeAsUJrHWExCp5N=QF5wvTituoFyYmOc0A@mail.gmail.com> <CAK-wPOhK1gX5%2BV1Z9nNauRF-oC85Jc-vmi-bKq_-YTKVt10C8Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000038b1af05efe6ce04 Content-Type: text/plain; charset="UTF-8" Thank you Leif, I probably should have mentioned that I got the OpenVPN tunnel working as well. I was confused as to what was going on until I looked carefully at what Google had installed in the routing table and saw what ought to be a link local route (which would normally just be directed at a link#k entry in the Gateway Field) was actually directed to what I believe is the VPC router interface in the subnet (10.1.1.1 above). It's working now but I've got an uneasy feeling I haven't done it 'right.' If this were ordinary VLAN/Ethernet stuff, it would work like this too (I think) but it would be incurring an extra L3 hop through the router when it could have gone over the VLAN/Ethernet fabric direct. Thanks again! Rob On Thu, Dec 15, 2022 at 2:27 PM Leif Pedersen <leif@ofwilsoncreek.com> wrote: > Hi, > > I don't have a direct answer, but as a user I can confirm that OpenVPN in > layer 3 mode works for me. I simply haven't tried it in layer 2 mode with > GCE (because I've no need for layer 2 and it incurs the extra overhead of > broadcast packets). Layer 2 mode probably won't work anyway because the MTU > has to be reduced to 1460, unless you do that on all participating hosts. > Point is, if that's an option for you it might be worth exploring. > > As a side note, I configure the tun devices with the same IP address at > the vtnet device. That actually works perfectly, even though the two > endpoints are on wildly different networks, and avoids maintaining DNS > entries and routes for the point to point network. > > For example: > vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric > 0 mtu 1460 > inet 10.1.2.3 netmask 0xffffffff broadcast 10.1.2.3 > inet 130.x.x.x netmask 0xffffffff broadcast 130.x.x.x > tun5000: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 > inet 10.1.2.3 --> 10.16.0.1 netmask 0xffffffff > > (The internal IP on my home router's ethernet interface is 10.16.0.1/16, > overlapping with its tun interface.) > > -Leif > > > > > > On Thu, Dec 15, 2022 at 4:03 PM Rob Ballantyne <robballantyne3@gmail.com> > wrote: > >> Hello, >> >> I have a question about what the internal structure and forwarding is >> within Google's VPCs. >> >> I started into a project using OpenVPN to bind my home network to an >> isolated VPC in Google's Cloud when I discovered the routing didn't work >> quite the way I thought. I had assumed that VPCs would look like a private >> VLAN (Layer2) into which Google's infrastructure would inject L3 router >> interfaces and/or ip/ethernet filters. >> >> I set up a private VPC and two test FreeBSD boxes to test and see >> exactly how VPC configures routing. >> >> First, I just used a standard install of 13.1 and the routing table >> after everything is up and configured looks like: >> >> ---- >> Internet: >> Destination Gateway Flags Netif Expire >> default 10.1.1.1 UGS vtnet0 >> 10.1.1.1 link#1 UHS vtnet0 >> 10.1.1.20 link#1 UH lo0 >> 127.0.0.1 link#2 UH lo0 >> ---- >> >> This looked a little unusual to me so (there was no link local route >> for all the addresses in the VPC), I commented out the rc.conf entry >> 'google_network_daemon_enable=YES' and setup the vtnet0 interface up >> manually with: 'ifconfig_vtnet0="inet 10.1.1.20 netmask 255.255.255.0"' >> The resulting routing table: >> >> ---- >> Internet: >> Destination Gateway Flags Netif Expire >> 10.1.1.0/24 link#1 U vtnet0 >> 10.1.1.20 link#1 UHS lo0 >> 127.0.0.1 link#2 UH lo0 >> ---- >> >> This configuration wasn't able to communicate. The latter routing table >> looks more usual though, with a 10.1.1.0/24 route to the local link. >> >> So, it appears to me that VPCs are really configured to be a >> point-to-point (star really) network where the Google router interface >> (10.1.1.1 in this case) has to handle all forwarding between nodes of a >> network. >> >> I've searched around the web to try and confirm this but there is scant >> detail on how exactly forwarding works within a single VPC. >> >> My VPN project involved using a bastion VPN host that would have >> terminated the VPN/SSL tunnel and routed traffic between my home network >> and the isolated network behind the bastion. >> >> Before I make final decisions on configuration, I wanted to know if my >> understanding is correct and whether there is any documentation on this >> that I've somehow missed. >> >> FreeBSD is, of course, the host of choice for this operation! >> >> If anyone does know any details, any info would be greatly appreciated. >> >> Many Thanks, >> Rob Ballantyne >> > --00000000000038b1af05efe6ce04 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:monospac= e,monospace">Thank you Leif,</div><div class=3D"gmail_default" style=3D"fon= t-family:monospace,monospace"><br></div><div class=3D"gmail_default" style= =3D"font-family:monospace,monospace">=C2=A0 I probably should have mentione= d that I got the OpenVPN tunnel working as well.=C2=A0 I was confused as to= what was going on until I looked carefully at what Google had installed in= the routing table and saw what ought to be a link local route (which would= normally just be directed at a link#k entry in the Gateway Field) was actu= ally directed to what I believe is the VPC router interface in the subnet (= 10.1.1.1 above).</div><div class=3D"gmail_default" style=3D"font-family:mon= ospace,monospace"><br></div><div class=3D"gmail_default" style=3D"font-fami= ly:monospace,monospace">=C2=A0 It's working now but I've got an une= asy feeling I haven't done it 'right.'=C2=A0 If this were ordin= ary VLAN/Ethernet stuff, it would work like this too (I think) but it would= be incurring an extra L3 hop through the router when it could have gone ov= er the VLAN/Ethernet fabric direct.</div><div class=3D"gmail_default" style= =3D"font-family:monospace,monospace"><br></div><div class=3D"gmail_default"= style=3D"font-family:monospace,monospace">=C2=A0 Thanks again!</div><div c= lass=3D"gmail_default" style=3D"font-family:monospace,monospace">Rob</div><= /div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">O= n Thu, Dec 15, 2022 at 2:27 PM Leif Pedersen <<a href=3D"mailto:leif@ofw= ilsoncreek.com">leif@ofwilsoncreek.com</a>> wrote:<br></div><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px sol= id rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><d= iv>I don't have a direct answer, but as a user I can confirm that OpenV= PN in layer 3 mode works for me. I simply haven't tried it in layer 2 m= ode with GCE (because I've no need for layer 2 and it incurs the extra = overhead of broadcast packets). Layer 2 mode probably won't work anyway= because the MTU has to be reduced to 1460, unless you do that on all parti= cipating hosts. Point is, if that's an option for you it might be worth= exploring.</div><div><br></div><div>As a side note, I configure the tun de= vices with the same IP address at the vtnet device. That actually works per= fectly, even though the two endpoints are on wildly different networks, and= avoids maintaining DNS entries and routes for the point to point network.<= /div><div><br></div><div>For example:</div><div>vtnet0: flags=3D8943<UP,= BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1460<br></div>= <div>=C2=A0 =C2=A0 inet 10.1.2.3 netmask 0xffffffff broadcast 10.1.2.3<br>= =C2=A0 =C2=A0 inet 130.x.x.x netmask 0xffffffff broadcast 130.x.x.x<br>tun5= 000: flags=3D8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500= <br>=C2=A0 =C2=A0 inet 10.1.2.3 --> 10.16.0.1 netmask 0xffffffff<br></di= v><div><br></div><div>(The internal IP on my home router's ethernet int= erface is <a href=3D"http://10.16.0.1/16" target=3D"_blank">10.16.0.1/16</a= >, overlapping with its tun interface.)</div><div><br></div><div>-Leif</div= ><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div= class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Dec 15= , 2022 at 4:03 PM Rob Ballantyne <<a href=3D"mailto:robballantyne3@gmail= .com" target=3D"_blank">robballantyne3@gmail.com</a>> wrote:<br></div><b= lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le= ft:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div class= =3D"gmail_default" style=3D"font-family:monospace,monospace">Hello,</div><d= iv class=3D"gmail_default" style=3D"font-family:monospace,monospace"><br></= div><div class=3D"gmail_default" style=3D"font-family:monospace,monospace">= =C2=A0 I have a question about what the internal structure and forwarding i= s within Google's VPCs.</div><div class=3D"gmail_default" style=3D"font= -family:monospace,monospace"><br></div><div class=3D"gmail_default" style= =3D"font-family:monospace,monospace">=C2=A0 I started into a project using = OpenVPN to bind my home network to an isolated VPC in Google's Cloud wh= en I discovered the routing didn't work quite the way I thought.=C2=A0 = I had assumed that VPCs would look like a private VLAN (Layer2) into which = Google's infrastructure would inject L3 router interfaces and/or ip/eth= ernet filters.</div><div class=3D"gmail_default" style=3D"font-family:monos= pace,monospace"><br></div><div class=3D"gmail_default" style=3D"font-family= :monospace,monospace">=C2=A0 I set up a private VPC and two test FreeBSD bo= xes to test and see exactly how VPC configures routing.=C2=A0=C2=A0</div><d= iv class=3D"gmail_default" style=3D"font-family:monospace,monospace"><br></= div><div class=3D"gmail_default" style=3D"font-family:monospace,monospace">= =C2=A0 First, I just used a standard install of 13.1 and the routing table = after everything is up and configured looks like:</div><div class=3D"gmail_= default" style=3D"font-family:monospace,monospace"><br></div><div class=3D"= gmail_default" style=3D"font-family:monospace,monospace">----</div><div cla= ss=3D"gmail_default" style=3D"font-family:monospace,monospace">Internet:<br= >Destination =C2=A0 =C2=A0 =C2=A0 =C2=A0Gateway =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0Flags =C2=A0 =C2=A0 Netif Expire<br>default =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A010.1.1.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UGS =C2= =A0 =C2=A0 =C2=A0vtnet0<br>10.1.1.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 link= #1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UHS =C2=A0 =C2=A0 =C2=A0vtnet0= <br>10.1.1.20 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#1 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lo0<br>127.0.0.1= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lo0<br></div><div class=3D"= gmail_default" style=3D"font-family:monospace,monospace"><div class=3D"gmai= l_default">----</div><br></div><div class=3D"gmail_default" style=3D"font-f= amily:monospace,monospace">=C2=A0 This looked a little unusual to me so (th= ere was no link local route for all the addresses in the VPC), I commented = out the rc.conf entry 'google_network_daemon_enable=3DYES' and setu= p the vtnet0 interface up manually with: 'ifconfig_vtnet0=3D"inet = 10.1.1.20 netmask 255.255.255.0"'=C2=A0 The resulting routing tabl= e:</div><div class=3D"gmail_default" style=3D"font-family:monospace,monospa= ce"><br></div><div class=3D"gmail_default" style=3D"font-family:monospace,m= onospace">----</div><div class=3D"gmail_default" style=3D"font-family:monos= pace,monospace">Internet:<br>Destination =C2=A0 =C2=A0 =C2=A0 =C2=A0Gateway= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Flags =C2=A0 =C2=A0 Netif Expire<= br><a href=3D"http://10.1.1.0/24" target=3D"_blank">10.1.1.0/24</a> =C2=A0 = =C2=A0 =C2=A0 =C2=A0link#1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 U =C2= =A0 =C2=A0 =C2=A0 =C2=A0vtnet0<br>10.1.1.20 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0link#1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UHS =C2=A0 =C2=A0 =C2= =A0 =C2=A0 lo0<br>127.0.0.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#2 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lo= 0<br></div><div class=3D"gmail_default" style=3D"font-family:monospace,mono= space">----</div><div class=3D"gmail_default" style=3D"font-family:monospac= e,monospace"><br></div><div class=3D"gmail_default" style=3D"font-family:mo= nospace,monospace">=C2=A0 This configuration wasn't able to communicate= . The latter routing table looks more usual though, with a <a href=3D"http:= //10.1.1.0/24" target=3D"_blank">10.1.1.0/24</a> route to the local link.</= div><div class=3D"gmail_default" style=3D"font-family:monospace,monospace">= <br></div><div class=3D"gmail_default" style=3D"font-family:monospace,monos= pace">=C2=A0 So, it appears to me that VPCs=C2=A0are really configured to b= e a point-to-point (star really) network where the Google router interface = (10.1.1.1 in this case) has to handle all forwarding between nodes of a net= work.</div><div class=3D"gmail_default" style=3D"font-family:monospace,mono= space"><br></div><div class=3D"gmail_default" style=3D"font-family:monospac= e,monospace">=C2=A0 I've searched around the web to try and confirm thi= s but there is scant detail on how exactly forwarding works within a single= VPC.</div><div class=3D"gmail_default" style=3D"font-family:monospace,mono= space"><br></div><div class=3D"gmail_default" style=3D"font-family:monospac= e,monospace">=C2=A0 My VPN project involved using a bastion VPN host that w= ould have terminated the VPN/SSL tunnel and routed traffic between my home = network and the isolated network behind the bastion.</div><div class=3D"gma= il_default" style=3D"font-family:monospace,monospace"><br></div><div class= =3D"gmail_default" style=3D"font-family:monospace,monospace">=C2=A0 Before = I make final decisions on configuration, I wanted=C2=A0to know if my unders= tanding is correct and whether there is any documentation on this that I= 9;ve somehow missed.</div><div class=3D"gmail_default" style=3D"font-family= :monospace,monospace"><br></div><div class=3D"gmail_default" style=3D"font-= family:monospace,monospace">=C2=A0 FreeBSD is, of course, the host of choic= e for this operation!</div><div class=3D"gmail_default" style=3D"font-famil= y:monospace,monospace"><br></div><div class=3D"gmail_default" style=3D"font= -family:monospace,monospace">=C2=A0 If anyone does know any details, any in= fo would be greatly appreciated.</div><div class=3D"gmail_default" style=3D= "font-family:monospace,monospace"><br></div><div class=3D"gmail_default" st= yle=3D"font-family:monospace,monospace">Many Thanks,</div><div class=3D"gma= il_default" style=3D"font-family:monospace,monospace">Rob Ballantyne</div><= /div> </blockquote></div> </blockquote></div> --00000000000038b1af05efe6ce04--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKLrb5eGPp2-h3bWEt-ZicUdxWtvEUE4fG4ZAW8=y%2B16miuVtw>