From owner-freebsd-isp Thu Apr 6 10:50:20 2000 Delivered-To: freebsd-isp@freebsd.org Received: from cpq.nyi.net (cpq.nyi.net [204.248.157.72]) by hub.freebsd.org (Postfix) with ESMTP id E955B37BBD4 for ; Thu, 6 Apr 2000 10:50:03 -0700 (PDT) (envelope-from javier@nyi.net) Received: from nyi.net (root@cpq.nyi.net [204.248.157.72]) by cpq.nyi.net (8.9.3/8.9.3) with ESMTP id RAA02067; Thu, 6 Apr 2000 17:50:25 GMT (envelope-from javier@nyi.net) Message-ID: <38ECCE61.511B5A98@nyi.net> Date: Thu, 06 Apr 2000 13:50:25 -0400 From: Javier Frias Organization: NYI X-Mailer: Mozilla 4.72 [en] (X11; I; FreeBSD 4.0-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Joe Greco Cc: dev@inetu.net, isp@freebsd.org Subject: Re: flat network References: <200004080654.BAA20869@aurora.sol.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org From experience, the best solution is to implement vlans in your network. Joe Greco wrote: > > > I know this may be a bit more of a network > > problem, but in my experience, freebsd people have > > the best skills here to :) > > > > We have a server farm of about 200 servers. > > > > We have a single router which connects to our bay > > switches (about 10 switches, all uplink into 1 100 > > mbps switch). > > > > The first 140+ servers were added with random ip > > addresses assigned to random servers (a block of > > 20 here, a block of 40 ip's there). > > > > Since then, we have started assigned logical > > blocks (/28, /29, etc.) to servers and routing the > > block directly to the server's main ip address (to > > cut down on required arp entries in router). > > > > We have a problem where new servers, that don't > > receive much traffic, tend to drop off the > > network. After you ping them for about 30 seconds > > plus they will return. > > > > If you constantly ping them, they will not fall > > off the network (0% packet loss with over 64,000 > > packets sent during the night). > > > > I was wondering if anyone had experienced similiar > > problems. > > > > I think either our router or switch is expiring > > the arp entry and taking time to re-learn it (due > > to the large size of our flat network). But how > > does one actually tell if this is the problem. > > > > Any assistance would be greatly apprecaited. > > You have 200 servers, or 200 virtual hosts on N (N << 200) servers? > > Adding additional alias interfaces is generally not the real cool > way to do web service, in any event. It is the first obvious mistake > that many ISP's make... the advertising of crap on large flat networks > via ARP. I've seen an ISP that did its dial-in pool as a /18 and used > ARP so that folks with static IP addresses worked. I've seen places > with /16's with a 0xffff0000 netmask - which caused the obvious problems > with all sorts of networking devices, since the network had ~8,000 nodes > or so on it. > > Use routing protocols. Break down and learn OSPF. If you have ten > switches being aggregated into a 100mbps switch, dump the 100mbps > switch and replace it with a router with a bunch of 100mbps ports. > Take each junior switch, put it on a 0xffffffe0 network off of the > router, and populate that with ten or twenty machines that are > running your servers. Then you allocate a bunch of address space > for virtual services, and you use OSPF to advertise each. You bind > additional aliases to lo0 and advertise them as stubs or something > like that, I've explained methods here before. Then you can even do > clever things like redundant ethernets for instant, automatic > failover. This sort of design should allow you to go up to a few > hundred physical servers supporting thousands of virtual web sites, > with no problem. > -- > ... Joe > > ------------------------------------------------------------------------------- > Joe Greco - Systems Administrator jgreco@ns.sol.net > Solaria Public Access UNIX - Milwaukee, WI 414/342-4847 > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-isp" in the body of the message -- MMM \|/ www __^__ (o o) @ @ (O-O) /(o o)\ -ooO-(_)-Ooo---oOO-(_)-OOo---oOO--(_)--OOo---oOO==(_)==OOo Javier A. Frias Sr. System Administrator The New York Internet Company 20 Exchange Place 21st Floor New York, N.Y. 10005 "Error #152 - Windows not found: (C)heer (P)arty (D)ance" --------------------------------------------------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message