Date: Wed, 19 Sep 2001 16:23:25 -0400 (EDT) From: "Gregory (Grisha) Trubetskoy" <grisha@verio.net> To: Jesper Skriver <jesper@skriver.dk> Cc: <freebsd-hubs@freebsd.org> Subject: Re: 4.4 Availability Message-ID: <Pine.BSF.4.32.0109191610120.99936-100000@localhost> In-Reply-To: <20010919220258.F79240@skriver.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
ftp2 can't really do more than 100Mbps since it's on 100Mb link, but it
sure tried today to get as close to 100 as it could.
I'm seeing sporadic freezes, during which I get no response for up to 2
minutes. The machine stays up, but won't even respond to ICMP. This can be
seen on the graph below.
The thing about these things is they are so hard to troubleshoot because I
don't want to do a lot of experimenting because I want the server to stay
up, but I can't think of a way such a load can be simulated for test
purposes...
The spec is I think a 500MHz with 512MB RAM and an Adaptec 2100s RAID
controller, this is an Intel ISP 2150 (L440GX motherboard) - a pretty
solid box.
In the kernel I have "maxusers 256" - I wonder if this is to low. The ftpd
is stock ftpd, currently running as daemon, but usually I just have inetd.
[16:16:42 root@ftp2 /usr/home/grisha]# netstat -m
2916/6416/18432 mbufs in use (current/peak/max):
2428 mbufs allocated to data
488 mbufs allocated to packet headers
2243/4608/4608 mbuf clusters in use (current/peak/max)
10820 Kbytes allocated to network (78% of mb_map in use)
181910 requests for memory denied
288 requests for memory delayed
0 calls to protocol drain routines
[16:16:45 root@ftp2 /usr/home/grisha]# vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 md0 in sy cs us sy id
13 2 0 39768 16728 117 12 5 0 269 132 0 0 578 678 309 1 5 94
15 2 0 39768 21152 33 14 7 3 1217 2412 86 0 2231 21964 2935 60 40 0
10 2 0 38452 16744 131 24 9 0 1269 0 96 0 2124 22292 2724 65 35 0
17 2 0 40416 21056 52 6 5 0 1152 2125 102 0 2200 21973 2950 64 36 1
17 3 0 39852 16132 24 10 6 0 1319 0 109 0 2446 21412 3236 55 45 0
15 2 0 44392 21128 78 9 5 0 1478 2696 114 0 2281 22112 2961 57 43 0
[16:19:53 root@ftp2 /usr/home/grisha]# ps -ax | grep ftpd | wc
181 1697 19882
The top of top looks like this:
last pid: 64160; load averages: 2.35, 1.68, 1.70 up 63+19:01:47 16:22:16
217 processes: 14 running, 199 sleeping, 4 zombie
CPU states: 34.7% user, 0.0% nice, 49.5% system, 15.8% interrupt, 0.0% idle
Mem: 322M Active, 50M Inact, 107M Wired, 20M Cache, 61M Buf, 992K Free
Swap: 512M Total, 7268K Used, 505M Free, 1% Inuse
On Wed, 19 Sep 2001, Jesper Skriver wrote:
> On Wed, Sep 19, 2001 at 10:06:46AM -0400, Gregory (Grisha) Trubetskoy wrote:
> >
> >
> > This is fun to watch:
> >
> > ftp://ftp2.freebsd.org/etc/traffic-day.gif
> >
> > Traffic went from 5 to 35 megabits on ftp2 in the past couple of hours.
>
> ftp.FreeBSD.org has pushed between 200 and 260 Mbps most of the
> day.
>
> But there's a problem, when the number of connections increase to
> more than approx 900, the load skyrockets, often with a loadindex
> of 150 or more :-(
>
> This is using the stock ftpd from FreeBSD - it was worse using
> dgftp, havn't figured why yet.
>
> Currently ftpd is started from tcpserver, which allows to limit
> the number of concurrent connections, which helps to keep the load
> under control.
>
> A current snapshot:
>
> jesper@ftp% vmstat 1
> procs memory page disks faults cpu
> r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id
> 137 9 0 375124109428 337 0 0 0 1811 1625 0 224 5390 3322 54380 1 80 19
> 18415 0 379112 84108 839 0 0 0 6404 0 0 244 14738 6646 283041 1 99 0
> 23011 0 379764 73868 645 0 0 0 2829 0 1 117 6997 8067 108663 1 99 0
> 21413 0 379536102636 855 0 0 0 14339 21608 2 180 34044 13818 711264 0 100 0
> 19616 0 379948 70680 520 0 0 0 7760 0 0 177 18985 4797 399380 1 99 0
> 21618 0 378992102292 125 7 0 0 2694 10744 0 154 7207 1426 155766 0 100 0
> 19317 0 379460 88880 388 0 1 0 3269 0 1 177 7419 2344 139995 0 100 0
> 19119 0 376524105820 567 0 0 0 6599 10775 0 146 15907 3559 353796 0 100 0
> 17614 0 373716 74020 231 0 0 0 7512 0 0 147 18143 3417 430951 1 99 0
>
> jesper@ftp% ps aux | grep ^ftp | wc -l
> 840
>
> PS: It's a dual 800 MHz PIII with 2 GB of memory, storage is raid5
> using a adaptec 3210 controller, but it's not disk I/O that's
> the problem.
>
> /Jesper
>
> --
> Jesper Skriver, jesper(at)skriver(dot)dk - CCIE #5456
> Work: Network manager @ AS3292 (Tele Danmark DataNetworks)
> Private: FreeBSD committer @ AS2109 (A much smaller network ;-)
>
> One Unix to rule them all, One Resolver to find them,
> One IP to bring them all and in the zone to bind them.
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hubs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.32.0109191610120.99936-100000>