Date: Thu, 06 Apr 2000 13:50:25 -0400 From: Javier Frias <javier@nyi.net> To: Joe Greco <jgreco@ns.sol.net> Cc: dev@inetu.net, isp@freebsd.org Subject: Re: flat network Message-ID: <38ECCE61.511B5A98@nyi.net> References: <200004080654.BAA20869@aurora.sol.net>
next in thread | previous in thread | raw e-mail | index | archive | help
From experience, the best solution is to implement vlans in your
network.
Joe Greco wrote:
>
> > I know this may be a bit more of a network
> > problem, but in my experience, freebsd people have
> > the best skills here to :)
> >
> > We have a server farm of about 200 servers.
> >
> > We have a single router which connects to our bay
> > switches (about 10 switches, all uplink into 1 100
> > mbps switch).
> >
> > The first 140+ servers were added with random ip
> > addresses assigned to random servers (a block of
> > 20 here, a block of 40 ip's there).
> >
> > Since then, we have started assigned logical
> > blocks (/28, /29, etc.) to servers and routing the
> > block directly to the server's main ip address (to
> > cut down on required arp entries in router).
> >
> > We have a problem where new servers, that don't
> > receive much traffic, tend to drop off the
> > network. After you ping them for about 30 seconds
> > plus they will return.
> >
> > If you constantly ping them, they will not fall
> > off the network (0% packet loss with over 64,000
> > packets sent during the night).
> >
> > I was wondering if anyone had experienced similiar
> > problems.
> >
> > I think either our router or switch is expiring
> > the arp entry and taking time to re-learn it (due
> > to the large size of our flat network). But how
> > does one actually tell if this is the problem.
> >
> > Any assistance would be greatly apprecaited.
>
> You have 200 servers, or 200 virtual hosts on N (N << 200) servers?
>
> Adding additional alias interfaces is generally not the real cool
> way to do web service, in any event. It is the first obvious mistake
> that many ISP's make... the advertising of crap on large flat networks
> via ARP. I've seen an ISP that did its dial-in pool as a /18 and used
> ARP so that folks with static IP addresses worked. I've seen places
> with /16's with a 0xffff0000 netmask - which caused the obvious problems
> with all sorts of networking devices, since the network had ~8,000 nodes
> or so on it.
>
> Use routing protocols. Break down and learn OSPF. If you have ten
> switches being aggregated into a 100mbps switch, dump the 100mbps
> switch and replace it with a router with a bunch of 100mbps ports.
> Take each junior switch, put it on a 0xffffffe0 network off of the
> router, and populate that with ten or twenty machines that are
> running your servers. Then you allocate a bunch of address space
> for virtual services, and you use OSPF to advertise each. You bind
> additional aliases to lo0 and advertise them as stubs or something
> like that, I've explained methods here before. Then you can even do
> clever things like redundant ethernets for instant, automatic
> failover. This sort of design should allow you to go up to a few
> hundred physical servers supporting thousands of virtual web sites,
> with no problem.
> --
> ... Joe
>
> -------------------------------------------------------------------------------
> Joe Greco - Systems Administrator jgreco@ns.sol.net
> Solaria Public Access UNIX - Milwaukee, WI 414/342-4847
>
> To Unsubscribe: send mail to majordomo@FreeBSD.org
> with "unsubscribe freebsd-isp" in the body of the message
--
MMM \|/ www __^__
(o o) @ @ (O-O) /(o o)\
-ooO-(_)-Ooo---oOO-(_)-OOo---oOO--(_)--OOo---oOO==(_)==OOo
Javier A. Frias <javier@nyi.net>
Sr. System Administrator
The New York Internet Company <http://www.nyi.net>;
20 Exchange Place 21st Floor
New York, N.Y. 10005
"Error #152 - Windows not found: (C)heer (P)arty (D)ance"
---------------------------------------------------------
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?38ECCE61.511B5A98>