Date: Tue, 24 Sep 1996 12:06:11 -0400 (EDT) From: Brian Tao <taob@io.org> To: Joe Greco <jgreco@brasil.moneng.mei.com> Cc: freebsd-isp@FreeBSD.ORG Subject: Re: Thoughts on a news server cluster Message-ID: <Pine.NEB.3.92.960923122044.24621R-100000@zap.io.org> In-Reply-To: <199609231420.JAA15753@brasil.moneng.mei.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 23 Sep 1996, Joe Greco wrote:
>
> You get more concurrency if you use ccd too :-)
True enough, but I didn't feel ccd was stable enough when I first
built our news server (late last year).
> If nothing else, you can get a "SCSI-SCSI" translator (made by Mylex
> and others) where you just provide a fast/wide SCSI adaptor (2940,
> etc) and let the black box handle the RAID aspects.
Good news... Open Storage says they will have a 5x4GB CRD-5300
(might be a bit off on the model number) with 64MB cache available for
me in the next couple of days. The PPro systems are arriving this
afternoon, and I'm going to order a bunch of 2GB drives in a rackmount
chassis for next week. That will give me one system with a single F/W
drive, a ccd of 2GB drives, a Streamlogic hardware RAID and a CMD
hardware RAID for benchmark comparisons. The bits will be flying. ;-)
> Support is probably going to appear for several host adapter RAID
> solutions in the not too distant future, if I believe what people are
> telling me :-)
Anything happening with the effort to pool some money together to
pay a programmer to accelerate his port the DPT drivers? I *might* be
able to convince the company to toss in some money towards such an
effort.
> You do NOT want to NFS mount!!! I have done it. If you have the I/O
> bandwidth and CPU (but not the RAM) to spare on a machine, it may be a
> worthwhile option... but the tax on the host is high. And you take a
> major reliability hit if that host goes down.
I'm trying to do a simple sort of cost-benefit analysis. Two F/W
controllers and level 5 RAID with 25GB of usuable capacity costs in
the $25000 range. Per machine. For that kind of money, I'm
definitely willing to give NFS-mounted reader servers a try.
> It gives me 9 days retention on most stuff, 12 on alt, 2 on
> alt.binaries. It supports 150 users and _flies_, and even at 200 the
> performance is "acceptable" (but starts to degrade.. pretty much
> simultaneously RAM and I/O bandwidth start to dry up).
The only performance problem I'm seeing is long delays or timeouts
when attempting to open an NNRP session. Once I'm in, the server is
niec and fast. I haven't tried anything special with starting
in.nnrpd's out of inetd and running innd on a different port, etc. It
seems to be related to the number of incoming innxmit connections.
> PPro200? Heavy iron for news... what exactly are you doing with all
> that horsepower... :-)
They were roughly same price as Pentium 200's (a couple hundred
dollars difference). Maybe I'll start playing with on-the-fly
compression of news articles. ;-)
> That is one way to handle it, but I find that running XREPLIC off of
> the feeds system is perfectly acceptable... if I was going to have a
> separate "reader" fan-out machine I would probably STILL run it as an
> XREPLIC slave from the feeder machine... convenience.
I don't want to "lock" myself into using XREPLIC though. If the
main feeder blows up and I have to newfs the spool, it'll take extra
work to resync those article numbers. If I just treat the feeder and
the primary reader machine as entirely autonomous servers, something
that goes wrong with one is less likely to affect the other. Also,
isn't slave mode required for XREPLIC? If the feeder server is
unavailable, none of the reader machines will be able to post. I've
not played with XREPLIC before, so my understanding may be off.
> I don't know, I would think that I would start seeing some I/O
> contention with just one machine..
I don't think we're going to hit 1000 simultaneous readers at this
POP for a while yet. It will be a gradual curve up, so any
anticipated I/O bottlenecks can be headed off before they become a
problem. Do we have any kernel optimizations yet for PPro memory-
intensive operations?
> And I have not seen any basis for supporting that many readers on a
> single machine.. how big is your active file? What does "top"'s
> output look like on one of your readers? Enquiring minds want to know :-)
It's a pretty small active file, just under 9000 groups (407187
bytes). 'top' looks like this:
load averages: 0.36, 0.42, 0.41 11:52:58
109 processes: 1 running, 118 sleeping
Cpu states: 2.7% user, 1.5% nice, 14.2% system, 2.3% interrupt, 79.2% idle
Mem: 82M Active, 6152K Inact, 20M Wired, 19M Cache, 7785K Buf, 176K Free
Swap: 262M Total, 8336K Used, 254M Free, 3% Inuse
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
27230 news -6 0 24M 24M biowai 95:05 13.08% 11.02% innd.nodebug
27238 root 29 0 352K 808K RUN 0:00 3.15% 0.57% top
26658 news 2 4 316K 708K select 0:01 0.38% 0.38% in.nnrpd
25061 news 2 0 220K 352K sbwait 0:22 0.31% 0.31% innxmit
27200 news 2 4 292K 868K sbwait 0:00 0.23% 0.23% in.nnrpd
27235 news 2 0 292K 992K select 0:00 0.38% 0.19% in.nnrpd
27233 news -6 0 152K 484K piperd 0:00 0.20% 0.15% overchan
27150 news 2 4 288K 728K sbwait 0:00 0.08% 0.08% in.nnrpd
27190 news 2 4 284K 692K sbwait 0:00 0.08% 0.08% in.nnrpd
26803 news 2 4 292K 732K sbwait 0:00 0.04% 0.04% in.nnrpd
26480 news 2 0 448K 548K select 0:04 0.04% 0.04% innxmit
23024 news 2 0 220K 308K sbwait 0:31 0.04% 0.04% innxmit
[...]
Assuming 32MB for kernel and OS stuff, 32MB for innd, 150MB for 500
readers and no feeds, that still leaves ~40MB for disk cache and other
processes (like expires) on a 256MB machine.
--
Brian Tao (BT300, taob@io.org, taob@ican.net)
Senior Systems and Network Administrator, Internet Canada Corp.
"Though this be madness, yet there is method in't"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.92.960923122044.24621R-100000>