From owner-freebsd-current@FreeBSD.ORG Tue Nov 6 11:53:17 2007 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC52F16A421; Tue, 6 Nov 2007 11:53:17 +0000 (UTC) (envelope-from ianf@clue.co.za) Received: from munchkin.clue.co.za (munchkin.clue.co.za [66.219.59.160]) by mx1.freebsd.org (Postfix) with ESMTP id 6BD0E13C4B7; Tue, 6 Nov 2007 11:53:17 +0000 (UTC) (envelope-from ianf@clue.co.za) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=20070313; d=clue.co.za; h=Received:Received:Received:To:cc:From:Subject:In-Reply-To:X-Attribution:Date:Message-Id; b=iTLmHDwANqZULL0Xw2BjZVWSXaNkA3nNFGF5oKtewLTSvFqkSpP8mMcksufB1UcybZsTXAbzJ6b9ZDejNEvkzbv0ccGRFb6qEUL9nEX2xlJVPvs4Y/QJc4rq5HNcCAKIw1qDI5kFCVZKC1YK+Xi8Wf7TZ345VceXG33o2zFg9FNT5FUrtQppbyMR8/Im8DcEn3iRhn6QloVnbQjmGZj52YX+mtBfp6QDtG4DeIS/4wZLb4qXEKnHpDEvf3r8rR2j; Received: from uucp by munchkin.clue.co.za with local-rmail (Exim 4.67) (envelope-from ) id 1IpMzW-0004mU-Gn; Tue, 06 Nov 2007 11:53:10 +0000 Received: from ianf.clue.co.za ([10.0.0.6] helo=clue.co.za) by urchin.clue.co.za with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.66) (envelope-from ) id 1IpMyT-0000ZZ-Cw; Tue, 06 Nov 2007 11:52:05 +0000 Received: from localhost ([127.0.0.1] helo=clue.co.za) by clue.co.za with esmtp (Exim 4.68 (FreeBSD)) (envelope-from ) id 1IpMyS-0001ck-F6; Tue, 06 Nov 2007 13:52:04 +0200 To: Andre Oppermann From: Ian FREISLICH In-Reply-To: Message from Andre Oppermann of "Mon, 05 Nov 2007 21:05:30 +0100." <472F778A.6080206@freebsd.org> X-Attribution: BOFH Date: Tue, 06 Nov 2007 13:52:04 +0200 Message-Id: Cc: current@freebsd.org Subject: Re: sendfile() not detecting closed connections. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 11:53:17 -0000 Andre Oppermann wrote: > Ian FREISLICH wrote: > > Andre Oppermann wrote: > > > >>Ian FREISLICH wrote: > >> > >>>Hi > >>> > >>>System is 8.0-CURRENT. I have the following piece of code: > >>> > >>> rename(path, data); > >>> stat(data, &sb); > >>> len = snprintf(buffer, MAXBUFLEN, "BYTES %lld\r\n", sb.st_size); > >>> write(connection, buffer, len); > >>> sleep(10); > >>> if ((sendfile(fd, connection, 0, sb.st_size, NULL, > >>> &sbytes, 0)) == -1 || sbytes != sb.st_size) { > >>> syslog(facility, "Problem writing data: %s, wrote %lld", > >>> strerror(errno), sbytes); > >>> respool(fd, path); > >>> unlink(data) > >>> close(fd); > >>> return(-1); > >>> } > >>> else > >>> syslog(facility, "Download successful %ld", sbytes); > >>> close(fd); > >>> unlink(data); > >>> > >>>If, during the sleep, I terminate the connection so that netstat > >>>reports: > >>> > >>>tcp4 0 0 127.0.0.1.666 127.0.0.1.58239 CLOSE_W A > > > > IT > > > >>>tcp4 0 0 127.0.0.1.58239 127.0.0.1.666 FIN_WAI T > > > > _2 > > > >>>sendfile() reports success for files less than about 64k in size, > >>>but I haven't been able to figure out where the threshold is. It > >>>erroneously reports that 41000 of the 64k were sent, but will say > >>>the whole file was transferred up to about 64k. The connection > >>>filedescriptor is blocking. > >>> > >>>Any ideas? > >> > >>sendfile() reports the bytes written into the send socket buffer. If > >>there is a connection error it doesn't (and never did) look at how > >>much data was still in the socket buffer. The sendfile(2) man page > >>says: "[sbytes] If non-NULL, the system will write the total number > >>of bytes sent on the socket to the variable pointed to by sbytes." > >>This could be changed to subtract the remaining data in the socket > >>buffer before reporting back. One has to be careful though about > >>other writes so that the number never goes negative. There may be > >>more data remaining in the socket buffer than from this write attempt > >>alone. > > > > > > The connection was closed about 6 seconds before I called sendfile(). > > Would sendfile() write to the socket buffer of a socket closed that > > long ago? > > The sendfile(2) syscall checks if the socket is still connected: > > if ((so->so_state & SS_ISCONNECTED) == 0) { > error = ENOTCONN; > goto out; > } > > Due to timing circumstances the connection may still be technically > connected while at the same time disconnecting when FIN exchange has > not completed yet. > > To prevent a sendfile(2) call on a disconnecting socket another test > has to be added: > > if (so->so_state & SS_ISDISCONNECTING) { > error = EPIPE; /* or ESHUTDOWN? */ > goto out; > } > > Please add this test after line 1841 in sys/kern/uipc_syscalls.c and > test again. The error code ESHUTDOWN seems more appropriate but is > so far not documented in the sendfile(2) man page. As it seems this > bug has been present since the introduction of the sendfile syscall. I'm not sure that this has changed much. It errors with errno == EPIPE, but still claims it's placed 40960 bytes in the socket buffer. A tight loop of netstat -anf inet never reveals a Send-Q greater than 0 bytes, although it's entirely likely it missed the event. But if I terminate the connection mid stream I can see a send queue sticking around for a bit: tcp4 0 43640 127.0.0.1.666 127.0.0.1.50916 CLOSE_WAIT Ian -- Ian Freislich