Date: Tue, 06 May 1997 00:38:14 -0600 From: Scott Halbert <thor@thuntek.net> To: Marc Slemko <marcs@znep.com> Cc: freebsd-isp@FreeBSD.ORG Subject: Re: Apache Socket Starvation Bug in 1.2 Message-ID: <3.0.32.19970506003705.01730f30@thuntek.net>
next in thread | raw e-mail | index | archive | help
At 11:25 PM 5/5/97 -0600, Marc Slemko wrote: >On Mon, 5 May 1997, Scott Halbert wrote: > >> Is anyone familiar with the problems with Apache versions 1.2 with socket >> starvation? The problem manifests itself as the server locking itself up >> and clients hanging with "waiting for reply". For me a KILL -HUP will get >> things going again. >> >> To quote the apacheweek article, "This problem only affects systems which >> do not define USE_FLOCK_SERIALIZED_ACCEPT or USE_FCNTL_SERIALIZED_ACCEPT >> in conf.h (this includes SunOS 4, FreeBSD, BSDI and some others)." >> >> Here are some URLS documenting the bug: >> >> http://www.apache.org/bugdb.cgi/full/467 >> http://www.apacheweek.com/issues/97-04-25 >> >> My problem is that the Apache workaround does not work for me. Turning >> on USE_FLOCK_SERIALIZED_ACCEPT does not help and USE_FCNTL_SERIALIZED_ACCEPT >> hangs the server immediately. > >What version of FreeBSD are you using? You may not have a /usr/tmp and >Apache is braindead and just uses /usr/tmp Just now I'm running FreeBSD 2.2-BETA_A. I do have a /usr/tmp but it is not set up like a normal /tmp (no group or world write). I'm not sure it access this directory as my 'User' user or as root. If root, it should have been able to write to it. >Try changing the: > > strncpy(lock_fname, "/usr/tmp/htlock.XXXXXX", sizeof(lock_fname)-1); > >lines (note there are two; one is compiled for >USE_FLOCK_SERIALIZED_ACCEPT, one for USE_FCNTL_SERIALIZED_ACCEPT) to >reference /var/tmp. This is a bad thing that I have been meaning to get >around to fix, but didn't get around to it. > >Grr. The reason why the FCNTL doesn't work is a bug caused by someone >being lazy. Fixed it somewhere else but guess not there... The patch >included at the end of this message should fix that one; should apply >cleanly against 1.2b10, but I haven't tried. I tried the above with 1.2b8, but am running vanilla 1.2b10. >> It sure seems to me that I've got something unique in my configuration. >> If this happened to all apache 1.2 servers under FreeBSD, I'd hear this >> group freaking out. I daresay that a very high majority of FreeBSD ISPs >> use Apache. Maybe some of you have seen it and were in the same boat? >> >> This problem is upsetting my customers and making me pull my hair out. >> >> Maybe this is not exactly the bug I'm chasing, though it sure looks like >> it. Maybe I've got my timeouts and idles messed up in my configuration >> file. > >Are you using multiple Listen directives? That is the only case where the >bug you refer to above shows up. Definately. Right now I have about 100 domains. The thing I noticed though is that when I put them all in one server, they system seems to lock up immediately. So, I divided my servers into about 6 servers with no more than 15 servers in year. This was a wild goose chase. >Are you sure you aren't running into processes per uid or file descriptor >limits? No, I checked that and definately not. I did notice that my shells were not passing through some quota limits to 'unlimited'. I fixed that, but still it wasn't a problem. >If your config file isn't overly complex, try it with as close to the >distributed config file as possible. I'll check it. My config files are generated by scripts. >How much traffic does your site see? How many virtual hosts does it have? >How often does the problem happen? Not huge amounts on these servers. 100 virtual servers, but as I said, I've got those running on 6 separate servers (they're all running on 1 machine). Seems like the problem happens a few times daily. It's hard to tell what actually is hanging up and which servers are hanging. It also seems to clear itself up. I can artificially clear it with kill -HUP. >When it happens, do a ps to see how many child processes there are (ie. >not the one running as root, but the ones that serve requests as whatever >your User directive is set to). Do a ktrace (eg. ktrace -p 1234) on a >child and see where it is hanging. If that fails, compile with -g and >attach gdb to a child process and see where it is hanging. If the child >is blocked waiting for a lock, then it could be the problem you mention >above. I'll look. >Rest assured that this problem does _not_ happen all the time. Many of >the developers (including me) use FreeBSD both for development and >production machines pushing high volumes of traffic. I figured. Otherwise, I'm sure I would have heard. I modified http_main.c on 1.2b10 changed /usr/tmp to /var/tmp Added USE_FLOCK_SERIALIZED_ACCEPT to FreeBSD profile in conf.h We'll see how that works. I guess I should certainly see files showing up in /var/tmp now. ---Scott Halbert Thunder Network Technologies, Inc.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3.0.32.19970506003705.01730f30>