From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 9 17:57:34 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A0B4219F for ; Sun, 9 Dec 2012 17:57:34 +0000 (UTC) (envelope-from rsharpe@richardsharpe.com) Received: from zmail.servaris.com (zmail.servaris.com [107.6.51.160]) by mx1.freebsd.org (Postfix) with ESMTP id 5E63A8FC0C for ; Sun, 9 Dec 2012 17:57:34 +0000 (UTC) Received: (qmail 47081 invoked by uid 89); 9 Dec 2012 17:57:32 -0000 Received: from unknown (HELO ?192.168.2.23?) (rsharpe@richardsharpe.com@108.225.16.199) by mail.richardsharpe.com with ESMTPA; 9 Dec 2012 17:57:32 -0000 Subject: Re: Possible obscure socket leak when system under load and listener is slow to accept From: Richard Sharpe To: Alfred Perlstein In-Reply-To: <50C4475D.9020300@mu.org> References: <50C3D22D.3060008@freebsd.org> <1355015131.6752.12.camel@localhost.localdomain> <50C4475D.9020300@mu.org> Content-Type: text/plain; charset="UTF-8" Date: Sun, 09 Dec 2012 09:57:30 -0800 Message-ID: <1355075850.6752.15.camel@localhost.localdomain> Mime-Version: 1.0 X-Mailer: Evolution 2.32.3 (2.32.3-1.fc14) Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Andre Oppermann X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Dec 2012 17:57:34 -0000 On Sun, 2012-12-09 at 00:10 -0800, Alfred Perlstein wrote: > On 12/8/12 5:05 PM, Richard Sharpe wrote: > > On Sun, 2012-12-09 at 00:50 +0100, Andre Oppermann wrote: > >>> Hi folks, > >>> > >>> Our QA group (at xxx) using Samba and smbtorture has been seeing a > >>> lot of cases where accept returns ECONNABORTED because the system load > >>> is high and Samba has a large listen backlog. > >>> > >>> Every now and then we get a crash in smbd or in winbindd and winbindd > >>> complains of too many open files in the system. > >>> > >>> In looking at kern_accept, it seems to me that FreeBSD can leak a socket > >>> when kern_accept calls soaccept on it but gets ECONNABORTED. This error > >>> is the only error returned from tcp_usr_accept. > >>> > >>> It seems like the socket taken off so_comp is never freed in this case > >>> and that there has been a call on soref on it as well, so that something > >>> like the following is needed in the error path: > >>> > >>> ==== //some-path/freebsd/sys/kern/uipc_syscalls.c#1 > >>> - /home/rsharpe/dev-src/packages/freebsd/sys/kern/uipc_syscalls.c ==== > >>> @@ -433,6 +433,14 @@ > >>> */ > >>> if (name) > >>> *namelen = 0; > >>> + /* > >>> + * We need to close the socket we unlinked > >>> + * so we do not leak it. > >>> + */ > >>> + ACCEPT_LOCK(); > >>> + SOCK_LOCK(so); > >>> + soclose(so); > >>> goto noconnection; > >>> } > >>> if (sa == NULL) { > >>> > >>> I think an soclose is needed at this point because soisconnected has > >>> been called on the socket. > >>> > >>> Do you think this analysis is reasonable? > >> > > >>> We are using FreeBSD 8.0 but it seems the same is true for 9.0. However, > >>> maybe I am wrong since I am not sure if the fdclose call would free the > >>> socket, but a quick look suggested that it doesn't. > >> The fdclose should properly tear down the file descriptor. The call > >> graph is: fdclose() -> fdrop() -> _fdrop() -> fo_close()/soo_close() -> > >> soclose() -> sorele() -> sofree() -> sodealloc(). > >> > >> A socket leak would not count against "kern.maxfiles" unless the file > >> descriptor leaks as well. So it is unlikely that this is the problem. > > OK, thanks for the feedback. I will keep looking. > > > >> Samba may open a large number of files (real files and sockets) and > >> you may run into the maxfiles limit. You can check the limit with > >> "sysctl kern.maxfiles" and increase it at boot time in boot/loader.conf > >> with "kern.maxfiles=100000" for example. > > Well, some of the smbds are dying, but it is possible that there is a > > file leak in Samba or our VFS that we are tripping as well. > > lsof and sockstat can be helpful. lsof may be able to help determine if > there's a leak because it MAY will find sockets not associated with a > process. > > Hope this helps. Thanks Alfred. After following through the call graph and confirming (with the code) that it was correct, I am now pretty convinced that I was wrong in assuming that it was a socket leak. However, lsof will be useful in allowing me to see how many FDs each smdb in this test is using. We have, I am told, kern.maxfiles set to 65536, which I think might be a little low for the test they are running.