Date: Sat, 8 Apr 2000 01:54:53 -0500 (CDT) From: Joe Greco <jgreco@ns.sol.net> To: dev@inetu.net Cc: isp@freebsd.org Subject: Re: flat network Message-ID: <200004080654.BAA20869@aurora.sol.net>
next in thread | raw e-mail | index | archive | help
> I know this may be a bit more of a network > problem, but in my experience, freebsd people have > the best skills here to :) > > We have a server farm of about 200 servers. > > We have a single router which connects to our bay > switches (about 10 switches, all uplink into 1 100 > mbps switch). > > The first 140+ servers were added with random ip > addresses assigned to random servers (a block of > 20 here, a block of 40 ip's there). > > Since then, we have started assigned logical > blocks (/28, /29, etc.) to servers and routing the > block directly to the server's main ip address (to > cut down on required arp entries in router). > > We have a problem where new servers, that don't > receive much traffic, tend to drop off the > network. After you ping them for about 30 seconds > plus they will return. > > If you constantly ping them, they will not fall > off the network (0% packet loss with over 64,000 > packets sent during the night). > > I was wondering if anyone had experienced similiar > problems. > > I think either our router or switch is expiring > the arp entry and taking time to re-learn it (due > to the large size of our flat network). But how > does one actually tell if this is the problem. > > Any assistance would be greatly apprecaited. You have 200 servers, or 200 virtual hosts on N (N << 200) servers? Adding additional alias interfaces is generally not the real cool way to do web service, in any event. It is the first obvious mistake that many ISP's make... the advertising of crap on large flat networks via ARP. I've seen an ISP that did its dial-in pool as a /18 and used ARP so that folks with static IP addresses worked. I've seen places with /16's with a 0xffff0000 netmask - which caused the obvious problems with all sorts of networking devices, since the network had ~8,000 nodes or so on it. Use routing protocols. Break down and learn OSPF. If you have ten switches being aggregated into a 100mbps switch, dump the 100mbps switch and replace it with a router with a bunch of 100mbps ports. Take each junior switch, put it on a 0xffffffe0 network off of the router, and populate that with ten or twenty machines that are running your servers. Then you allocate a bunch of address space for virtual services, and you use OSPF to advertise each. You bind additional aliases to lo0 and advertise them as stubs or something like that, I've explained methods here before. Then you can even do clever things like redundant ethernets for instant, automatic failover. This sort of design should allow you to go up to a few hundred physical servers supporting thousands of virtual web sites, with no problem. -- ... Joe ------------------------------------------------------------------------------- Joe Greco - Systems Administrator jgreco@ns.sol.net Solaria Public Access UNIX - Milwaukee, WI 414/342-4847 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200004080654.BAA20869>