Date: Tue, 10 Nov 1998 13:15:41 -0500 From: Steven Yang <syang@directhit.com> To: "'dg@root.com'" <dg@root.com>, Mike Smith <mike@smith.net.au> Cc: Steven Yang <syang@directhit.com>, "'Open Systems Networking'" <opsys@mail.webspan.net>, "'freebsd-hackers@freebsd.org'" <freebsd-hackers@FreeBSD.ORG> Subject: RE: FW: Can't get rid of my mbufs. Message-ID: <839A86AB6CE4D111A52200104B938D430B066B@MOE>
next in thread | raw e-mail | index | archive | help
Refresher: I started this thread two weeks ago and haven't had a time to reply until now. I'm using FreeBSD 2.2.5 (I previously stated 2.2.6, but I was wrong) with Apache 1.2.4 using FastCGI. A typical server response is about 22K of text, and under heavy load (20+ requests/second), my mbufs (as seen through netstat -m) keep increasing until the server reboots itself (when I have > ~10000 mbufs) It appears that all of the requests are getting valid replies (we check the returned web page for a string), even at loads around 100 requests/second. We do not do reverse-DNS. My original question and one of the replies is attached at the bottom of this email. The higher the load, the faster the mbufs increase. Under low load, the problem does not arise and new mbufs are not allocated. I was requested to give you guys the output of "netstat -n", as shown below. Big questions: do I have an mbuf leak? Is it possibly my fault? Could it be my version of FastCGI? Will upgrading my OS to 2.2.7 solve the problem? Will upgrading Apache solve the problem? > netstat -n Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp 0 0 10.4.18.101.6010 10.4.18.101.1031 ESTABLISHED tcp 0 0 10.4.18.101.1031 10.4.18.101.6010 ESTABLISHED tcp 0 20 10.4.18.101.22 10.4.18.198.2097 ESTABLISHED tcp 0 0 10.4.18.101.22 10.4.18.199.1529 ESTABLISHED tcp 0 0 10.4.18.101.22 10.4.18.210.4307 ESTABLISHED tcp 0 0 10.4.18.101.22 10.4.18.210.4168 ESTABLISHED udp 0 0 10.4.18.101.1022 10.4.18.5.2049 udp 0 0 10.4.18.101.1023 10.4.18.5.2049 Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr f48daf00 stream 0 0 f4926700 0 0 0 /tmp/OM_WS_84.21 f486d000 stream 0 0 f487a380 0 0 0 /tmp/OM_WS_83.21 f489d000 stream 0 0 f487eb00 0 0 0 /tmp/OM_WS_82.21 f4885200 stream 0 0 f47a0f80 0 0 0 /tmp/OM_WS_81.21 f4900100 stream 0 0 f4935e80 0 0 0 /tmp/OM_WS_80.21 f47b6d00 stream 0 0 f47ca980 0 0 0 /tmp/OM_WS_79.21 f48f8800 stream 0 0 f47b3e00 0 0 0 /tmp/OM_WS_78.21 f47b4500 stream 0 0 f482f380 0 0 0 /tmp/OM_WS_77.21 f4879900 stream 0 0 f47bea00 0 0 0 /tmp/OM_WS_76.21 f47ce900 stream 0 0 f4925a00 0 0 0 /tmp/OM_WS_75.21 f4791300 stream 0 0 f4934b80 0 0 0 /tmp/OM_WS_74.21 f48bf500 stream 0 0 f47a8b80 0 0 0 /tmp/OM_WS_73.21 f47b4900 stream 0 0 f4834700 0 0 0 /tmp/OM_WS_72.21 f478a200 stream 0 0 f4790180 0 0 0 /tmp/OM_WS_71.21 f4830000 dgram 0 0 0 f31a4594 0 f31f4e94 f48bf300 dgram 0 0 0 f31a4594 0 f31d9f14 f47c4000 dgram 0 0 0 f31a4594 0 f31c4b94 f4887100 dgram 0 0 0 f31a4594 0 f31d6e14 f47d0400 dgram 0 0 0 f31a4594 0 f31b5a14 f4786b00 dgram 0 0 0 f31a4594 0 0 f48d5d00 dgram 0 0 0 f31a4594 0 f31a4a14 f4742900 dgram 0 0 f4743680 0 f31c2694 0 /var/run/log > -----Original Message----- > From: David Greenman [SMTP:dg@root.com] > Sent: Wednesday, October 28, 1998 12:37 AM > To: Mike Smith > Cc: Steven Yang; 'Open Systems Networking'; > 'freebsd-hackers@freebsd.org' > Subject: Re: FW: Can't get rid of my mbufs. > > >> >If the machine is left idle for 2 hours, and presumably from this > we > >> >would expect that all open connections were closed > >> > >> That would be presuming too much. > > > >How about we ask the tester? > > I already did several messages back. I said "What does netstat -n > show?" > Note that I'm not asking for -m. I'm interested in the state of the > (apparantly still open) connections, not in the memory summaries. -n > simply > because doing reverse DNS for 6000+ connections might take a very long > time. > > -DG > > David Greenman > Co-founder/Principal Architect, The FreeBSD Project > > > -----Original Message----- > > From: Steven Yang > > Sent: Tuesday, October 20, 1998 11:04 AM > > To: 'freebsd-questions@freebsd.org' > > Subject: Can't get rid of my mbufs. > > > > Hi, we are running a test on FreeBSD 2.2.6, using Apache 1.2.4 with > > FastCGI. We are performing about 100 requests per second over a > > high-speed switch. On average, each request returns about 20,000 > > bytes, so we are transferring about 2MB/sec. The problem is, our > > machine ends up rebooting itself after a couple hours. > > Here are some sample outputs after pounding on our machine for over 1 > > hour: > > # netstat -m > > 4449 mbufs in use: > > 4437 mbufs allocated to data > > 1 mbufs allocated to packet headers > > 7 mbufs allocated to protocol control blocks > > 4 mbufs allocated to socket names and addresses > > 4263/4314 mbuf clusters in use > > 9184 Kbytes allocated to network (98% in use) > > 0 requests for memory denied > > 0 requests for memory delayed > > 0 calls to protocol drain routines > > > > #netstat -r > > Routing tables > > > > Internet: > > Destination Gateway Flags Refs Use > > Netif Expire > > default 10.4.18.1 UGSc 1 0 > > fxp0 > > 10.4.18/24 link#1 UC 0 0 > > 10.4.18.1 0:10:7b:a6:a6:e8 UHLW 2 0 > > fxp0 684 > > moe 0:10:4b:93:8d:43 UHLW 0 43 > > fxp0 1186 > > 10.4.18.198 0:10:4b:99:c5:e2 UHLW 1 9215795 > > fxp0 972 > > localhost localhost UH 0 0 > > lo0 > > > > Symptoms: I can sit around for a long time with no network activity, > > and the mbufs won't decrease. If I exceed 10,000 mbuf clusters, I'm > > in danger of hitting a server reboot. I can kill Apache, but that > > doesn't help. My memory usage increases linearly with increasing mbuf > > clusters. mbuf clusters only seem to increase under heavy load. > > Basically, once an mbuf is allocated, it's never deallocated, and we > > don't have memory leaks. > > How do I solve this problem? > > > > Thanks, > > Steven To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?839A86AB6CE4D111A52200104B938D430B066B>