Date: Thu, 9 May 2002 17:28:13 -0500 (CDT) From: Nick Rogness <nick@rogness.net> To: Daniel Lang <dl@leo.org> Cc: freebsd-net@FreeBSD.ORG Subject: Re: Multiple NICs on the same subnet Message-ID: <Pine.BSF.4.21.0205091710500.21635-100000@cody.jharris.com> In-Reply-To: <20020509203930.GA32900@atrbg11.informatik.tu-muenchen.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 9 May 2002, Daniel Lang wrote:
> Hi,
>
> I want to use multiple NICs on the same subnet.
> Thats my setup:
>
> FreeBSD 4.5-STABLE (updated 2-3 weeks ago, no 4.6-PRERELEASE yet),
> 3 NICs inside: 2 x 3com 905C-TX, 1 x D-Link DGE-500SX (Level 1
> Gigabit).
>
> Now I want to assign a couple of IP addresses to those NICs. All IP
> addresses are on the same subnet, and they correspond to different
> kind of services available from this host, like FTP, CVSup, http,
> rsync, etc.
>
> Previously all services have been served by a single NIC, and I've
> used IP aliases to assign the service bound addresses to the NIC. Now
> I want to assign a certain set of services to different NICs (which
> all have separate uplinks at their switches and thus can achieve more
> bandwidth), like bind the FTP service to the lge interface, and maybe
> rsync and CVSup to one of the 3coms and the rest to the remaining
> 3com.
>
> This does not seem to be possible. If I want to assign an address to
> one of those NICs, I get an error from an ioctl like "file exists".
> Reading the archives, etc, it seems like I need to assign this address
> as an alias. This sort of worked. I could assign an address to one of
> the other NICs by choosing a netmask of /32 and stating alias on using
> ifconfig. But, our switch did not register any packets from one of the
> other NICs other than the default interface (one of the 3coms). It
> seems, like the address may be assigned to one of the NICs, but even
> if I use an application which explicitly binds to a certain source IP,
> the packet does not get sent by the interface, which has this IP
> address assigned, but by the default interface.
The problem you are having is not an alias problem but a routing
one. Packets come in to your alias on the proper interface but
when the reply packet gets sent it uses the default route which
goes out your default route.
In other words, packets that arrive inbound on an interface will
not necessarily leave that same interface on the outbound reply,
if it doesn't have a route to that network via that
interface. Instead, it leaves through the default gateway
interface (because of the default route).
The best way to handle this is with ipfw fwd. Basically you
forward packets trying to leave the default gateway with the
aliased address of a different interface out the right interface.
For example:
xl0 --> alias= 1.1.1.1/32 , (default gateway out this interface)
xl1 --> alias= 1.1.1.2/32
lge0 --> alias= 1.1.1.3/32
ipfw ruleset:
# FOrward packets properly
ipfw fwd $IP_OF_NEXT_HOP_xl1 ip from 1.1.1.2/32 to any out via xl0
ipfw fwd $IP_OF_NEXT_HOP_lge0 ip from 1.1.1.3/32 to any out via xl0
. . . [rest of firewall] . . .
You will need your kernel build with 'options IPFIREWALL_FORWARD'.
>
> This would not be that much of a problem so far. The problem really
> showed up, when it seemed like the Gigabit interface did not seem to
> work as expected. A couple of possible problems may be the cause,
> symptoms beeing "lge0: watchdog timeout" messages (which may be due to
> hardware/cabling problems), "sendto: no buffer space availble"
> messages (no idea where this comes from, any hints appreaciated,
> kern.ipc.nmbclusters and kern.maxusers etc, are bumped enough and did
> not max out (according to netstat -m)).
This is another problem altogether.
> So I thought, I run the production services on the reliable xl0 and
> xl1 interfaces and assign a test IP address to the lge0 interface, and
> if services from that interface run reliable, too, I switch the others
> over, benefitting from the increased bandwith.
>
> Well, thats how I would like it to work. But it doesn't. Is there any
> chance in FreeBSD to do it, and if, how?
>
> I've heard some rumours, it is not possible to use multiple NICs on
> the same subnet, since the IP stack would not know which interface to
> use to transmit packets, since it could not use its routing table (as
> the network is the same). But my argument would be, of course it
> should use the interface, which was configured with the address, the
> service bound its source port to. If any service did bind to
> IN_ADDR_ANY (0.0.0.0), the ip stack may choose the default interface
> or just any interface to transmit the packet from.
Nick Rogness <nick@rogness.net>
- Don't mind me...I'm just sniffing your packets
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0205091710500.21635-100000>