From owner-freebsd-net@FreeBSD.ORG  Sat Mar 27 00:04:54 2004
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id A0C9616A4CE
	for <freebsd-net@freebsd.org>; Sat, 27 Mar 2004 00:04:54 -0800 (PST)
Received: from relay.pair.com (relay.pair.com [209.68.1.20])
	by mx1.FreeBSD.org (Postfix) with SMTP id 3B93443D1D
	for <freebsd-net@freebsd.org>; Sat, 27 Mar 2004 00:04:54 -0800 (PST)
	(envelope-from silby@silby.com)
Received: (qmail 27260 invoked from network); 27 Mar 2004 08:04:53 -0000
Received: from niwun.pair.com (HELO localhost) (209.68.2.70)
  by relay.pair.com with SMTP; 27 Mar 2004 08:04:53 -0000
X-pair-Authenticated: 209.68.2.70
Date: Sat, 27 Mar 2004 02:04:52 -0600 (CST)
From: Mike Silbersack <silby@silby.com>
To: Kevin Day <toasty@dragondata.com>
In-Reply-To: <23E302EC-7F7F-11D8-AA0F-000A95A8A1F2@dragondata.com>
Message-ID: <20040327020154.S2845@odysseus.silby.com>
References: <23E302EC-7F7F-11D8-AA0F-000A95A8A1F2@dragondata.com>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
cc: freebsd-net@freebsd.org
Subject: Re: sendfile returning ENOTCONN under heavy load
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 27 Mar 2004 08:04:54 -0000


On Fri, 26 Mar 2004, Kevin Day wrote:

> I'm using thttpd on a server that pushes 300-400mbps of static content,
> using sendfile(2).
>
> Once the load reaches a certain point (around 800-1000 clients
> downloading, anywhere from 150-250mbps), sendfile() will start randomly
> returning ENOTCONN, and the client is disconnected. I've raised
> kern.ipc.nsfbufs pretty high and that hasn't made any difference. Is
> there any easy way to tell exactly why the sockets are being closed? I
> can't seem to find any obvious signs of memory exhaustion or anything.
>
> Thanks,
>
> Kevin

The only place where I see sendfile returning ENOTCONN is:

        if ((so->so_state & SS_ISCONNECTED) == 0) {
                error = ENOTCONN;
                goto done;
        }

So presumably the connection was terminated at the TCP layer, not due to a
shortage in sfbufs.  Maybe you should set up your testbed that creates
this load, then set up one additional test computer.  Have that computer
tcpdump all of its traffic, hope that one of the dropped connections
happens to it, and see if you can find it in the dump.

Mike "Silby" Silbersack