Date: Sat, 17 Jan 2004 14:24:51 -0800 From: Elliott Freis <efreis@opentable.com> To: "'freebsd-stable@freebsd.org'" <freebsd-stable@freebsd.org> Subject: Odd lag/hanging issue with production ftp server - Please help AS AP! Message-ID: <34BD9C97B714D511B01C00D0B73EC07303EEE65B@mail.opentable.com>
next in thread | raw e-mail | index | archive | help
I am forwarding this to this list per recommendation.
Note on recent test: Trying to do a transfer on the public LAN off the ftp
server, I am only able to get a bursty 20k/s! down from the ftp server.
This was with nfs completely unmounted, ftping from a local drive so its not
an NFS problem. It should get a full 2-3mb a sec at least. It used to. Up
is still full speed. The machine has a Intel pro NIC, direct to a Cisco
ArrowPoint. Both are set to 100 FD. ifconfig_fxp2="inet 66.151.XXX.XXX
netmask 255.255.255.224 media 100baseTX"
Any help GREATLY appreciated, I have exhausted all avenues I can think of,
including hardware swaps. Here is some diag:
last pid: 49903; load averages: 0.12, 0.20, 0.25
up 0+15:19:19 14:19:00
73 processes: 1 running, 72 sleeping
CPU states: 1.4% user, 0.0% nice, 17.1% system, 2.3% interrupt, 79.2%
idle
Mem: 24M Active, 1233M Inact, 178M Wired, 68M Cache, 163M Buf, 3600K Free
Swap: 1024M Total, 12K Used, 1024M Free
353/17120/262144 mbufs in use (current/peak/max):
271 mbufs allocated to data
82 mbufs allocated to packet headers
213/16806/65536 mbuf clusters in use (current/peak/max)
37892 Kbytes allocated to network (19% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs
Coll
fxp2 1500 <Link#3> 00:02:a5:13:fc:b5 30214691 0 22672636 0
589096
fxp2 1500 66.151.XXX.XXX ftp 30150589 - 22683851
- -
fxp2 1500 fe80:3::202 fe80:3::202:a5ff: 0 - 0 -
-
> -----Original Message-----
> From: Elliott Freis
> Sent: Friday, January 09, 2004 3:28 PM
> To: 'freebsd-questions@FreeBSD.ORG'
> Subject: Odd lag/hanging issue with production ftp server - Please
> help ASAP!
>
> I have been struggling with this one for over a month now. Here is a
> quick layout of my setup:
>
> Primary FTP server:
> Compaq DL380 1.5gb ram
> FreeBSD 4.5
> 3x36gb RAID 5 drives as local boot/storage
> ProFTPd
>
> Array server for FTP:
> AMD Athlon 2200 512mb ram
> FreeBSD 4.8
> 7x36gb Fiber channel drives, RAID 5 via Vinum.
>
> Both machines are connected via a cross-over cable, that has been tested
> good and swapped just in case. The primary storage for FTP is done on the
> Fiber drives via nfs from FTP to Array server. NFS options are -U -3.
>
> My problem is this. As more users connect and store files, the primary
> FTP machine becomes increasingly unresponsive. Currently, I max at about
> 350 concurrent FTP connections. The most basic test I have been doing is
> just holding down enter on an SSH session. As you hold enter down, you
> see it visually just hang for a second or more (up to about 5 seconds
> depending on the load). It is even worse if I spam "df -k" for example.
> For part of the time, it responds fine, though its randomly a second to
> multiple seconds. In other words, it is randomly responsive and not
> responsive every 5 seconds or so. During the "hanging" time, ftp sessions
> are also hung. So you see very bursty data transfers. Now thankfully, no
> ftp sessions drop, so we do get the data we need. But this is a terrible
> thing to be happening to a production server.
>
> One other thing of note, this happened to me about 4-5 months ago, but a
> reboot fixed it for some reason. So I concluded it was just a hiccup.
> But it has returned after a different reboot, and won't go away.
>
> Any help in troubleshooting this is very appreciated! Happy new year,
>
> -Elliott
>
> Example of "enter" latency (this is from a LAN connection):
> At shell prompt ">"
> >
> >
>
>
>
>
>
>
>
>
>
>
>
>
> >
> >
> >
> >
> >
>
> >
>
> >
> >
> >
>
> >
> >
> >
>
> >
> >
>
>
>
> >
> >
> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?34BD9C97B714D511B01C00D0B73EC07303EEE65B>