Date: Sat, 18 May 2013 23:29:45 +0000 From: "Teske, Devin" <Devin.Teske@fisglobal.com> To: Joe <fbsd8@a1poweruser.com> Cc: "<freebsd-questions@freebsd.org>" <freebsd-questions@freebsd.org> Subject: Re: netgraph network setup for jail(8) vnet jails. Message-ID: <13CA24D6AB415D428143D44749F57D7201F5076A@ltcfiswmsgmb26> In-Reply-To: <5197A06A.9080401@a1poweruser.com> References: <5197A06A.9080401@a1poweruser.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Sorry for top-post, but just wanted to add a quick note:
The output of "ngctl dot" would be very helpful to others in debugging your=
setup.
--=20
Devin
On May 18, 2013, at 8:38 AM, Joe wrote:
> Hello list
>=20
> I cant get to the internet using this netgraph setup script.
> I sure would appreciate giving this console log a look over for
> errors. My netgraph knowledge level is not sufficient to see what is
> wrong. The goal is to run this script to setup and break down a netgraph
> network for a single vnet jail at a time. rl0 is the real nic interface
> device name of the nic facing the internet. This box is on my lan and
> the gateway box does NAT for all lan boxes. The host running this script =
can ping the internet ok.
>=20
> Thank you very much for your help.
>=20
>=20
>=20
>=20
>=20
> The host's kernel has modules with vimage & ipfw compiled in.
>=20
> From the host
> # /root >ifconfig
> rl0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> options=3D2008<VLAN_MTU,WOL_MAGIC>
> ether 00:0c:6e:09:8b:74
> inet 10.0.10.5 netmask 0xfffffff8 broadcast 10.0.10.7
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> plip0: flags=3D8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> ipfw0: flags=3D8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
>=20
> The jails config file
> # /root >cat /usr/local/etc/vnet/vdir4
> vdir4 {
> host.hostname =3D "vdir4";
> path =3D "/usr/jails/vdir4";
> mount.fstab =3D "/usr/local/etc/fstab/vdir4";
> vnet;
> persist;
> }
>=20
> The netgraph script
> # /root >cat /usr/local/bin/vnet.ng.test
> #!/bin/sh
> # snip comments for displaying here
> # This script is based on this /usr/share/examples/netgraph/virtual.lan
>=20
> # Give the name of ethernet interface.
> ETHER_INTF=3D"rl0"
>=20
> # List the names of virtual nodes and their IP addresses. Use ':'
> # character to separate node name from node IP address and netmask.
>=20
> #TARGET_TOPOLOGY=3D"c1|10.0.2.20/24 c2|10.0.2.21/24 c3|10.0.2.22/24"
> TARGET_TOPOLOGY=3D"vdir4|10.0.2.20/24"
>=20
> # MAC manufacturer prefix. This can be modified according to needs.
> MAC_PREFIX=3D"00:1d:92"
>=20
> # Temporary file is important for proper execution of script.
> TEMP_FILE=3D"/var/tmp/virtual.lan.tmp"
>=20
> virtual_lan_start() {
>=20
> # Load netgraph KLD's as necessary.
>=20
> for KLD in ng_ether ng_bridge ng_eiface; do
> if ! kldstat -v | grep -qw ${KLD}; then
> echo -n "Loading ${KLD}.ko... "
> kldload ${KLD} || exit 1
> echo "done"
> fi
> done
>=20
> # Reset all interfaces and jails. If temporary file can not be found
> # script assumes that there is no previous configuration.
>=20
> if [ ! -e ${TEMP_FILE} ]; then
> echo "No previous configuration(${TEMP_FILE}) found to clean-up."
> else
> echo -n "Cleaning previous configuration..."
> virtual_lan_stop
> echo "done"
> fi
>=20
> # Create temporary file for usage. This file includes generated
> # interface names and jail names. All bridges, interfaces and jails
> # are written to file while created. In clean-up process written
> # objects are cleaned (i.e. removed) from system.
>=20
> if [ -e ${TEMP_FILE} ]; then
> touch ${TEMP_FILE}
> fi
>=20
> echo -n "Verifying ethernet interface existence..."
> # Verify ethernet interface exist.
> if ! ngctl info ${ETHER_INTF}: >/dev/null 2>&1; then
> echo "Error: interface ${ETHER_INTF} does not exist"
> exit 1
> fi
>=20
> ifconfig ${ETHER_INTF} up || exit 1
> echo "done"
>=20
> # Get current number of bridge interfaces in the system. This number
> # is used to create a name for new bridge.
> BRIDGE_COUNT=3D`ngctl l | grep bridge | wc -l | sed -e "s/ //g"`
> BRIDGE_NAME=3D"bridge${BRIDGE_COUNT}"
>=20
> # Create new ng_bridge(4) node and attach it to the ethernet interface.
> # Connect ng_ether:lower hook to bridge:link0 when creating bridge and
> # connect ng_ether:upper hook to bridge:link1 after bridge name is set.
>=20
> echo "Creating bridge interface: ${BRIDGE_NAME}..."
> ngctl mkpeer ${ETHER_INTF}: bridge lower link0 || exit 1
> ngctl name ${ETHER_INTF}:lower ${BRIDGE_NAME} || exit 1
> ngctl connect ${ETHER_INTF}: ${BRIDGE_NAME}: upper link1 || exit 1
> echo "Bridge ${BRIDGE_NAME} is created and ${ETHER_INTF} is connected."
>=20
> # In the above code block two hooks are connected to bridge interface,
> # therefore LINKNUM is set to 2 indicating total number of connected
> # hooks on the bridge interface.
> LINKNUM=3D2
>=20
> # Write name of the bridge to temp file. Clean-up procedure will use
> # this name to shutdown bridge interface.
> echo "bridge ${BRIDGE_NAME}" > ${TEMP_FILE}
>=20
>=20
> # Attach vnet jail.
> for NODE in ${TARGET_TOPOLOGY}; do
>=20
> # Virtual nodes are defined in TARGET_TOPOLOGY variable. They
> # have the form of 'nodeName|IPaddr'. Below two lines split
> # node definition to get node name and node IP.
>=20
> NODE_NAME=3D`echo ${NODE} | awk -F"|" '{print $1}'`
> NODE_IP=3D`echo ${NODE} | awk -F"|" '{print $2}'`
>=20
> # Create virtual node (jail) with given name
> echo -n "Creating virtual node (jail) ${NODE_NAME}..."
>=20
> jail -f "/usr/local/etc/vnet/${NODE_NAME}" -c ${NODE_NAME}
> if [ $? -ne 0 ]; then
> echo "Error: /usr/sbin/jail failed to start jail ${NODE_NAME}."
> virtual_lan_stop
> exit 2
> fi
> echo "done"
>=20
> # Write name of the jail to temp file. Clean-up procedure will
> # use this name to remove jail.
>=20
> echo "node ${NODE_NAME}" >> ${TEMP_FILE}
>=20
> # Create a ng_eiface object for virtual node. ng_eiface
> # object has a hook that can be connected to one of bridge
> # links. After creating interface get its automatically
> # generated name for further usage.
>=20
> echo "Creating eiface interface for virtual node ${NODE_NAME}."
> ngctl mkpeer eiface ether ether
> EIFACE=3D`ngctl l | grep ngeth | tail -n 1| awk '{print $2}'`
> echo "Interface ${EIFACE} is created."
>=20
> # Write name of the interface to temp file. Clean-up procedure
> # will use this name to shutdown interface.
>=20
> echo "interface ${EIFACE}" >> ${TEMP_FILE}
>=20
> # Move virtual interface to virtual node. Note that Interface
> # name will not be changed at the end of this movement. Moved
> # interface can be seen at the output of ifconfig command in
> # jail: 'jexec jailname ifconfig'
>=20
> echo "Moving ${EIFACE} to ${NODE_NAME}"
> ifconfig ${EIFACE} vnet ${NODE_NAME}
>=20
> # Make lo0 interface localhost.
> jexec ${NODE_NAME} ifconfig lo0 localhost
>=20
> # Generate a random mac address for virtual interface. First
> # three octets can be changed by user. Last three octets are
> # generated randomly.
> M4=3D`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
> awk '{ print $1 % 256 }'`
> M5=3D`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
> awk '{ print $1 % 256 }'`
> M6=3D`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
> awk '{ print $1 % 256 }'`
>=20
> MAC=3D`printf ${MAC_PREFIX}:%02x:%02x:%02x ${M4} ${M5} ${M6}`
>=20
> # Set the link address (mac address) of virtual interface in
> # virtual node to randomly generated MAC.
> echo "Setting MAC address of ${EIFACE} to '${MAC}'"
> jexec ${NODE_NAME} ifconfig ${EIFACE} link $MAC
>=20
> # Either IPv4 or IPv6 can be used in this script. Ifconfig
> # IP setting syntax differs slightly for two IP versions.
> # For version 4 'inet' keyword is used whereas for version 6
> # 'inet6' is used. Below line tries to decide which IP version
> # is given and sets IPVER to 'inet' or 'inet6'.
>=20
> IPVER=3D`echo ${NODE_IP} | awk -F"." '{ split($4,last,"/"); \
> if( NF=3D=3D4 && $1>0 && $1<256 && $2<256 && $3<256 && \
> last[1]<256) print "inet"; else print "inet6"}'`
>=20
> # Set IP address of virtual interface in virtual node.
> echo "Setting IP address of ${EIFACE} to '${NODE_IP}'"
> jexec ${NODE_NAME} ifconfig ${EIFACE} ${IPVER} ${NODE_IP}
>=20
> # Connect virtual interface to bridge interface. Syntax is :
> # ngctl connect INTERFACE: BRIDGE: INTERFACE_HOOK EMPTY_LINK.
> # Interface has one hook named 'ether' and below line connects
> # ether hook to bridge's first unconnected link.
>=20
> echo -n "Connecting ${EIFACE}:ether to ${BRIDGE_NAME}:link${LINKNUM}..."
>=20
> ngctl connect ${EIFACE}: ${BRIDGE_NAME}: ether link${LINKNUM} \
> || exit 1
> echo "done"
>=20
> # Now, bridge has one more connected link thus link count is
> # incremented.
> LINKNUM=3D`expr ${LINKNUM} + 1`
> done
> echo "Virtual LAN established successfully!"
>=20
> }
>=20
>=20
> # Stop routine.
> virtual_lan_stop() {
>=20
> if [ ! -e ${TEMP_FILE} ]; then
> echo "Nothing to stop! ${TEMP_FILE}: temp file not found"
> else
> echo -n "Shutdown bridge interface.."
> OBJECTS=3D`cat ${TEMP_FILE} | grep bridge | awk '{print $2}'`
> for BRIDGE in ${OBJECTS}; do
> ngctl shutdown ${BRIDGE}: >/dev/null 2>&1
> done
> echo "done"
>=20
> echo -n "Shutdown all eiface interfaces..."
> OBJECTS=3D`cat ${TEMP_FILE} | grep interface | awk '{print $2}'`
> for INTERFACE in ${OBJECTS}; do
> ngctl shutdown ${INTERFACE}: >/dev/null 2>&1
> done
> echo "done"
>=20
> echo -n "Removing all jails..."
> OBJECTS=3D`cat ${TEMP_FILE} | grep node | awk '{print $2}'`
> for NODE in ${OBJECTS}; do
> jail -f "/usr/local/etc/vnet/${NODE}" -r ${NODE}
> done
> echo "done"
>=20
> echo "Removing tempfile ${TEMP_FILE}"
> rm ${TEMP_FILE}
> fi
> echo "Virtual LAN objects removed successfully!"
>=20
> }
>=20
>=20
> # Main entry point.
>=20
> case $# in
> 1)
> case $1 in
> start)
> echo -n "Creating default target topology:"
> echo " ${TARGET_TOPOLOGY}"
> virtual_lan_start
> ;;
> stop)
>=20
> if [ ! -e ${TEMP_FILE} ]; then
> echo -n "Noting to stop! ${TEMP_FILE}:"
> echo " temp file not found"
> else
> virtual_lan_stop
> fi
> ;;
> help)
> virtual_lan_usage
> exit 1
> ;;
> *)
> virtual_lan_usage
> exit 1
>=20
> esac
> ;;
> 2)
> case $1 in
> start)
> TARGET_TOPOLOGY=3D$2
> echo -n "Creating target topology:"
> echo "${TARGET_TOPOLOGY}"
> virtual_lan_start
> ;;
> *)
> virtual_lan_usage
> exit 1
> esac
> ;;
>=20
> *)
> virtual_lan_usage
> exit 1
> esac
>=20
>=20
> # /root >vnet.ng.test start
> Creating default target topology: vdir4|10.0.2.20/24
> Loading ng_ether.ko... done
> Loading ng_bridge.ko... done
> Loading ng_eiface.ko... done
> No previous configuration(/var/tmp/virtual.lan.tmp) found to clean-up.
> Verifying ethernet interface existence...done
> Creating bridge interface: bridge0...
> Bridge bridge0 is created and rl0 is connected.
> Creating virtual node (jail) vdir4...vdir4: created
> done
> Creating eiface interface for virtual node vdir4.
> Interface ngeth0 is created.
> Moving ngeth0 to vdir4
> Setting MAC address of ngeth0 to '00:1d:92:df:92:8e'
> Setting IP address of ngeth0 to '10.0.2.20/24'
> Connecting ngeth0:ether to bridge0:link2...done
> Virtual LAN established successfully!
>=20
>=20
> # /root >ngctl ls -l
> There are 5 total nodes:
> Name: rl0 Type: ether ID: 00000001 Num hooks: 2
> Local hook Peer name Peer type Peer ID Peer
> hook
> ---------- --------- --------- -------
> ---------
> upper bridge0 bridge 00000006 link1
>=20
> lower bridge0 bridge 00000006 link0
>=20
> Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0
> Name: bridge0 Type: bridge ID: 00000006 Num hooks: 3
> Local hook Peer name Peer type Peer ID Peer
> hook
> ---------- --------- --------- -------
> ---------
> link2 ngeth0 eiface 0000000a ether
>=20
> link1 rl0 ether 00000001 upper
>=20
> link0 rl0 ether 00000001 lower
>=20
> Name: ngeth0 Type: eiface ID: 0000000a Num hooks: 1
> Local hook Peer name Peer type Peer ID Peer
> hook
> ---------- --------- --------- -------
> ---------
> ether bridge0 bridge 00000006 link2
>=20
> Name: ngctl1513 Type: socket ID: 0000000d Num hooks: 0
>=20
>=20
> # /root >jexec vdir4 tcsh
> vdir4 / >ping -c 1 8.8.178.135
> PING 8.8.178.135 (8.8.178.135): 56 data bytes
> ping: sendto: No route to host
>=20
> --- 8.8.178.135 ping statistics ---
> 1 packets transmitted, 0 packets received, 100.0% packet loss
> vdir4 / >exit
> exit
>=20
>=20
> # /root >vnet.ng.test stop
> Shutdown bridge interface..done
> Shutdown all eiface interfaces...done
> Removing all jails...vdir4: removed
> done
> Removing tempfile /var/tmp/virtual.lan.tmp
> Virtual LAN objects removed successfully!
>=20
> # /root >jls
> JID IP Address Hostname Path
>=20
> # /root >ngctl ls -l
> There are 3 total nodes:
> Name: ngctl1540 Type: socket ID: 00000010 Num hooks: 0
> Name: rl0 Type: ether ID: 00000001 Num hooks: 0
> Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0
>=20
>=20
>=20
>=20
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.o=
rg"
_____________
The information contained in this message is proprietary and/or confidentia=
l. If you are not the intended recipient, please: (i) delete the message an=
d all copies; (ii) do not disclose, distribute or use the message in any ma=
nner; and (iii) notify the sender immediately. In addition, please be aware=
that any message addressed to our domain is subject to archiving and revie=
w by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?13CA24D6AB415D428143D44749F57D7201F5076A>