From owner-freebsd-hackers@FreeBSD.ORG Wed May 14 13:46:37 2008
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
by hub.freebsd.org (Postfix) with ESMTP id 980B4106566C
for <freebsd-hackers@freebsd.org>; Wed, 14 May 2008 13:46:37 +0000 (UTC)
(envelope-from msaad@datapipe.com)
Received: from exchfe01.datapipe-corp.net (exchfe01.datapipe-corp.net
[64.106.130.69])
by mx1.freebsd.org (Postfix) with ESMTP id 65CA08FC13
for <freebsd-hackers@freebsd.org>; Wed, 14 May 2008 13:46:37 +0000 (UTC)
(envelope-from msaad@datapipe.com)
Received: from divide.lan (192.168.128.20) by exchfe01.datapipe-corp.net
(64.106.130.71) with Microsoft SMTP Server id 8.0.783.2;
Wed, 14 May 2008 09:46:36 -0400
Message-ID: <482AED3B.1020307@datapipe.com>
Date: Wed, 14 May 2008 09:46:35 -0400
From: Mark Saad <msaad@datapipe.com>
User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421)
MIME-Version: 1.0
To: Mikolaj Golub <to.my.trociny@gmail.com>, <freebsd-hackers@freebsd.org>
References: <482A2639.7000401@datapipe.com> <81zlqtfazy.fsf@zhuzha.ua1>
In-Reply-To: <81zlqtfazy.fsf@zhuzha.ua1>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
Cc:
Subject: Re: Socket leak
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
Reply-To: msaad@datapipe.com
List-Id: Technical Discussions relating to FreeBSD
<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 14 May 2008 13:46:37 -0000
Mikolaj
Thanks for the input, did you change any of the options for
TimeoutLinger or TimeoutIdle ?
The Proftpd I am running is build for 6.3-RELEASE here are the build
options
Compile-time Settings:
Version: 1.3.0a
Platform: FREEBSD6 (FREEBSD6_3)
Built With:
configure CPPFLAGS=-DHAVE_OPENSSL --localstatedir=/var/run
--disable-sendfile --disable-ipv6
--with-modules=mod_sql:mod_sql_mysql:mod_check_mysql:mod_check_digest
--prefix=/usr/local
--with-includes=/usr/local/include/mysql:/usr/include/openssl
--with-libraries=/usr/local/lib/mysql
This is not built from ports, one of the other S/A who works on this
setup built proftpd with MySQL support, not sure why
this is not from ports , from what I understand this mysql patch is not
in ports.
Mikolaj Golub wrote:
> On Tue, 13 May 2008 19:37:29 -0400 Mark Saad wrote:
>
> MS> I started logging the values of kern.ipc.numopensockets and I noticed
> MS> that something is leaking sockets. Here is a sample of the log
>
> MS> 2008-04-29--15:04.10 ____ kern.ipc.numopensockets: 1501
> MS> 2008-04-29--16:04.01 ____ kern.ipc.numopensockets: 1535
> MS> 2008-04-29--17:04.00 ____ kern.ipc.numopensockets: 1617
> MS> 2008-04-29--18:04.00 ____ kern.ipc.numopensockets: 1710
>
> MS> This continues until kern.ipc.maxsockets its reached or the box is
> MS> rebooted.
>
> MS> The other thing we looked at was the output from vmstat -z
> MS> The first thing was the high amount of malloc 128 bucket failures
>
> MS> 128 Bucket: 524, 0, 2489, 80, 8364, 23055239
>
> MS> I also logged the mbuf clusters, we never reached the max mbuf clusters
>
> MS> Its almost like there are stale sockets. Here is a snapshot of the server now
>
> MS> ewr# sockstat -4u |wc -l
> MS> 139
> MS> ewr# sysctl kern.ipc.numopensockets
> MS> kern.ipc.numopensockets: 13935
>
> MS> ewr# uptime
> MS> 7:30PM up 6 days, 26 mins, 3 users, load averages: 0.18, 0.25, 0.17
>
> We had the same problem on one of hosts running 6.2-RELEASE-p11. The situation
> was complicated by the fact that I didn't have root access to the host and
> there were problems with getting more debugging or running tcpdump.
>
> Eventually, it appeared that problem was caused by proftpd. One of our clients
> connected to ftp server every five minutes looking for new file to
> download. When there was the file everything was good. But when there wasn't,
> some strange things happened. In proftpd logs we had:
>
> FTP session opened.
> mod_delay/0.5: delaying for 28 usecs
> user fake authenticated by mod_auth_pam.c
> USER fake: Login successful.
> Preparing to chroot to directory '/var/ftp/fake'
> Environment successfully chroot()ed.
> mod_delay/0.5: delaying for 621 usecs
> Entering Passive Mode (XX,YY,ZZ,213,241,70).
> FTP session closed.
>
> i.e. the client connected to server, had login successful, created new DATA
> connection in passive mode and then exited. But although proftpd reported that
> connection closed and proftpd process exited we still had this orphaned
> connection in our system reported by netstat in ESTABLISHED state. sockstat
> did not display this connections. Some of these connections could be in
> CLOSE_WAIT mode instead of ESTABLISHED. Such connection was seen by netstat
> for several hours and then disappeared but I suspect that the socket buffer
> was not freed and numopensockets counter did not decrease.
>
> Unfortunately, I did not managed to persuade admin to increase DebugLevel in
> proftpd.conf and run tcpdump to investigate more what was going on. It turned
> out that we had proftpd built for FREEBSD5_4:
>
> Compile-time Settings:
> Version: 1.3.0
> Platform: FREEBSD5 (FREEBSD5_4)
> Built With:
> configure --localstatedir=/var/run --sysconfdir=/usr/local/etc --disable-sendfile --disable-ipv6 --with-modules=mod_ratio:mod_readme:mod_rewrite:mod_wrap:mod_ifsession --prefix=/usr/local i386-portbld-freebsd5.4
>
> Upgrade to more recent proftpd built for proper platform resolved the problem.
>
> So I would recommend to look for process that could cause this leak. In my
> case careful investigation of netstat output history and comparing with
> sockstat output helped to find guilty. May be it would help to restart daemons
> one by one and see if sockets are freed.
>
> You can surely increase kern.ipc.maxsockets as workaround until you identify
> what causes the problem.
>
> --
> Mikolaj Golub
> _______________________________________________
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
>
--
Mark Saad
msaad@datapipe.com