From owner-freebsd-net@FreeBSD.ORG Fri Dec 17 08:29:27 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36E841065675 for ; Fri, 17 Dec 2010 08:29:27 +0000 (UTC) (envelope-from patrick.bihan-faou@teambox.fr) Received: from smtp.teambox.fr (dedibox.teambox.fr [88.191.109.88]) by mx1.freebsd.org (Postfix) with ESMTP id DD47E8FC16 for ; Fri, 17 Dec 2010 08:29:26 +0000 (UTC) Received: from crest.teambox.fr (crest.mindstep.com [88.167.204.204]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: teambox) by smtp.teambox.fr (Postfix) with ESMTPSA id D0598A244CF for ; Fri, 17 Dec 2010 09:12:38 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by kawa.local.mindstep.fr (Postfix) with ESMTP id 70260FDBE69 for ; Fri, 17 Dec 2010 09:12:38 +0100 (CET) (envelope-from patrick.bihan-faou@teambox.fr) Received: from kawa.local.mindstep.fr ([127.0.0.1]) by localhost (kawa.local.mindstep.fr [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 4XIYluCRY22p for ; Fri, 17 Dec 2010 09:12:38 +0100 (CET) Received: from [127.0.0.1] (unknown [192.168.25.162]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by kawa.local.mindstep.fr (Postfix) with ESMTP id 39E38FDB831 for ; Fri, 17 Dec 2010 09:12:38 +0100 (CET) (envelope-from patrick.bihan-faou@teambox.fr) Message-ID: <4D0B1B76.1000207@teambox.fr> Date: Fri, 17 Dec 2010 09:12:38 +0100 From: Patrick Bihan-Faou Organization: TeamBox SARL User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <17835728.248313.1292567569116.JavaMail.root@sz0077a.emeryville.ca.mail.comcast.net> <4D0B0E03.2020707@freebsd.org> In-Reply-To: <4D0B0E03.2020707@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: Web Server supporting up to 4 WANs/Interfaces X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Dec 2010 08:29:27 -0000 Le 17/12/2010 08:15, Julian Elischer a écrit : > On 12/16/10 10:32 PM, Jayster wrote: >> I am in need of immediate help and while I rarely post on boards or >> lists (I promise I'll start sharing more of my own wisdom in the >> future), this issue has me stumped like no other. First of all, I >> need multiple gateways. Yes, I understand there is no such thing as >> multiple "default" gateways and I have a solid grasp of networking in >> general... . I also have a reasonable grasp of FreeBSD, though I am >> no where near the expert of many. I actually come from the world of >> programming and I've only used Linux and Windows in the past. >> >> I am currently working on a big project that involves many sites and >> a custom appliance with multiple applications designed in-house. In >> fact, I might be the ONLY person in snowy Michigan hiring right >> now... . After an exhaustive investigation, I chose FreeBSD over all >> other OSs. It was not a light choice, but I've been very pleased >> despite having zero knowledge at the start of it. The main reasons >> were flexibility, licensing and most important, I'm still shocked how >> open the community is to answering questions in posts, instead of >> putting people down for having less knowledge than a rude poster. I >> have found an answer to the most obscure questions through very quick >> google searches...until now. Kudos to those who take the >> time...FreeBSD far outnumbers other OS communities. Please help me >> avoid this issue being the big letdown, possibly forcing us to leave >> FreeBSD. I like the OS way too much already. >> >> Now the issue. Without too much detail, my device has 4 GigE ports on >> it. Each will be attached to a routed network. There is NO routing >> required between networks inside the box (not a router or firewall) >> and in fact, it CANNOT be allowed to happen because of security. >> Instead, each WAN port needs access to this box, but nothing beyond. >> The access consists of a Web Server, though several other Ports are >> required, such as SNMP Traps, Syslog, etc. Getting to the box is >> easy, routers do all the work. The issue is getting traffic back >> through the same interface it came in on and through the same router >> gateway. As we all know, only 1 gateway can be assigned in FreeBSD, >> unlike other flavors of Linux. Even the ones who don't offer single >> line gateway support can use IPTables to accomplish this task. But >> IPTables is not supported in FreeBSD. Not a bad thing as long as >> comparable solutions exist. >> >> Setting up static routes is not the solution. The problem with it is >> that multiple sites which will have this box will not have access to >> the next hop info from the gateway (the next hop gateway and subnet >> on the other side of the router). So I cannot use static routes. >> >> PFSense appears to support this (though not tested by me). I REALLY >> do not want to go that route. We have invested 3 months into adding >> many apps to the FreeBSD we have. PFSense is a custom FreeBSD kernel >> with many changes. Many message boards claim it breaks many Ports and >> changes other behaviors. Even if it didn't, we are under deadline and >> moving everything over to a new FreeBSD Version and then extensively >> testing everything repeatedly again would be a nightmare. I am >> interested in experiences with it if it becomes the last resort, though. >> >> I have tried both PF and IPFW. Different posts around the web claim >> Multiple Gateway solutions using both of them. I have tried each of >> the recommended setups, but had no luck. If you read the last >> responses to each of those posts, others also state they could not >> duplicate what is claimed, as well. PF looks the most promising. It >> has "if-bound", which is supposed to keep interface traffic on the >> same interface. That is a good first step. But pointing it to the >> gateway on that interface is still an issue. Please HELP!!! I haven't >> slept in days and I've been stuck for a week now!!! This is our last >> showstopper. >> > > if you are running on freeBSD 8 or newer you have two solutions open > to you (maybe 3). > > * Firstly, you can assign a completely different routing table to each > socket so that > packets from one socket only see things through the perspective of one > routing > table but packets from another socket behave according to the rules of > a completely > different routing table. alternatively you can assign a different > routing table to a > process and its descendants. > you can also use ipfw fwd to remap ports and addresses, in conjunction > with the > different routing tables. Routing tables are sometimes called FIBs > (Forwarding Information Bases) > > man setfib(1) and setfib(2) for more details. > Also see the setfib socket option in setsockopt(2). > > There are details that are still open for development (like IPV6 > support) but > it sounds like it will do what you want. > > * The second option is the new jail support. > > while setfib and friends can easily allow a single process to act > differently on a socket by socket basis > the new jail facilities allow you to take multiple interfaces and > assign them to different jails, > and each jail can be given a completely different routing table or in > fact a completely different SET of routing tables. > > man jail and man ifconfig (ifconfig vnet) > > * As a poor third contender you can do really funky things with the > ipfw 'fwd' command. > > julian > > (let me know offline a bit more about what you want and maybe I can be > a bit more specific about how to do it.) A third solution that does not involve jails or fib (which to me sounds like the better approach btw) is to use pf and "route-to" rules such as this one: pass out quick on ! bge1 route-to (bge1 ip.of.gateway.on.bge1) from bge1 to any and repeat for all four network interface. This should work properly for all TCP based protocols as the source IP on return packet from your server will be set to the IP of the interface the trafic came in from. The route-to mechanism will force the gateway and interface for the return packets based on that knowledge. I use this on FreeBSD 6 based machines where the new setfib functionality is not available and that works like a charm. I don't think any specific kernel option is needed for that. I tried the forward rules on ipfw, but they were not as easy to setup and not working reliably in my setup. Best regards, Patrick Bihan-Faou