From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 8 23:50:15 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A4EE2C1E for ; Sat, 8 Dec 2012 23:50:15 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 138D38FC08 for ; Sat, 8 Dec 2012 23:50:14 +0000 (UTC) Received: (qmail 60999 invoked from network); 9 Dec 2012 01:19:48 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 9 Dec 2012 01:19:48 -0000 Message-ID: <50C3D22D.3060008@freebsd.org> Date: Sun, 09 Dec 2012 00:50:05 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Richard Sharpe Subject: Re: Possible obscure socket leak when system under load and listener is slow to accept Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Dec 2012 23:50:15 -0000 > Hi folks, > > Our QA group (at xxx) using Samba and smbtorture has been seeing a > lot of cases where accept returns ECONNABORTED because the system load > is high and Samba has a large listen backlog. > > Every now and then we get a crash in smbd or in winbindd and winbindd > complains of too many open files in the system. > > In looking at kern_accept, it seems to me that FreeBSD can leak a socket > when kern_accept calls soaccept on it but gets ECONNABORTED. This error > is the only error returned from tcp_usr_accept. > > It seems like the socket taken off so_comp is never freed in this case > and that there has been a call on soref on it as well, so that something > like the following is needed in the error path: > > ==== //some-path/freebsd/sys/kern/uipc_syscalls.c#1 > - /home/rsharpe/dev-src/packages/freebsd/sys/kern/uipc_syscalls.c ==== > @@ -433,6 +433,14 @@ > */ > if (name) > *namelen = 0; > + /* > + * We need to close the socket we unlinked > + * so we do not leak it. > + */ > + ACCEPT_LOCK(); > + SOCK_LOCK(so); > + soclose(so); > goto noconnection; > } > if (sa == NULL) { > > I think an soclose is needed at this point because soisconnected has > been called on the socket. > > Do you think this analysis is reasonable? > > We are using FreeBSD 8.0 but it seems the same is true for 9.0. However, > maybe I am wrong since I am not sure if the fdclose call would free the > socket, but a quick look suggested that it doesn't. The fdclose should properly tear down the file descriptor. The call graph is: fdclose() -> fdrop() -> _fdrop() -> fo_close()/soo_close() -> soclose() -> sorele() -> sofree() -> sodealloc(). A socket leak would not count against "kern.maxfiles" unless the file descriptor leaks as well. So it is unlikely that this is the problem. Samba may open a large number of files (real files and sockets) and you may run into the maxfiles limit. You can check the limit with "sysctl kern.maxfiles" and increase it at boot time in boot/loader.conf with "kern.maxfiles=100000" for example. -- Andre