Date: Tue, 06 Nov 2007 13:52:04 +0200 From: Ian FREISLICH <ianf@clue.co.za> To: Andre Oppermann <andre@freebsd.org> Cc: current@freebsd.org Subject: Re: sendfile() not detecting closed connections. Message-ID: <E1IpMyS-0001ck-F6@clue.co.za> In-Reply-To: Message from Andre Oppermann <andre@freebsd.org> of "Mon, 05 Nov 2007 21:05:30 %2B0100." <472F778A.6080206@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Andre Oppermann wrote:
> Ian FREISLICH wrote:
> > Andre Oppermann wrote:
> >
> >>Ian FREISLICH wrote:
> >>
> >>>Hi
> >>>
> >>>System is 8.0-CURRENT. I have the following piece of code:
> >>>
> >>> rename(path, data);
> >>> stat(data, &sb);
> >>> len = snprintf(buffer, MAXBUFLEN, "BYTES %lld\r\n", sb.st_size);
> >>> write(connection, buffer, len);
> >>> sleep(10);
> >>> if ((sendfile(fd, connection, 0, sb.st_size, NULL,
> >>> &sbytes, 0)) == -1 || sbytes != sb.st_size) {
> >>> syslog(facility, "Problem writing data: %s, wrote %lld",
> >>> strerror(errno), sbytes);
> >>> respool(fd, path);
> >>> unlink(data)
> >>> close(fd);
> >>> return(-1);
> >>> }
> >>> else
> >>> syslog(facility, "Download successful %ld", sbytes);
> >>> close(fd);
> >>> unlink(data);
> >>>
> >>>If, during the sleep, I terminate the connection so that netstat
> >>>reports:
> >>>
> >>>tcp4 0 0 127.0.0.1.666 127.0.0.1.58239 CLOSE_W
A
> >
> > IT
> >
> >>>tcp4 0 0 127.0.0.1.58239 127.0.0.1.666 FIN_WAI
T
> >
> > _2
> >
> >>>sendfile() reports success for files less than about 64k in size,
> >>>but I haven't been able to figure out where the threshold is. It
> >>>erroneously reports that 41000 of the 64k were sent, but will say
> >>>the whole file was transferred up to about 64k. The connection
> >>>filedescriptor is blocking.
> >>>
> >>>Any ideas?
> >>
> >>sendfile() reports the bytes written into the send socket buffer. If
> >>there is a connection error it doesn't (and never did) look at how
> >>much data was still in the socket buffer. The sendfile(2) man page
> >>says: "[sbytes] If non-NULL, the system will write the total number
> >>of bytes sent on the socket to the variable pointed to by sbytes."
> >>This could be changed to subtract the remaining data in the socket
> >>buffer before reporting back. One has to be careful though about
> >>other writes so that the number never goes negative. There may be
> >>more data remaining in the socket buffer than from this write attempt
> >>alone.
> >
> >
> > The connection was closed about 6 seconds before I called sendfile().
> > Would sendfile() write to the socket buffer of a socket closed that
> > long ago?
>
> The sendfile(2) syscall checks if the socket is still connected:
>
> if ((so->so_state & SS_ISCONNECTED) == 0) {
> error = ENOTCONN;
> goto out;
> }
>
> Due to timing circumstances the connection may still be technically
> connected while at the same time disconnecting when FIN exchange has
> not completed yet.
>
> To prevent a sendfile(2) call on a disconnecting socket another test
> has to be added:
>
> if (so->so_state & SS_ISDISCONNECTING) {
> error = EPIPE; /* or ESHUTDOWN? */
> goto out;
> }
>
> Please add this test after line 1841 in sys/kern/uipc_syscalls.c and
> test again. The error code ESHUTDOWN seems more appropriate but is
> so far not documented in the sendfile(2) man page. As it seems this
> bug has been present since the introduction of the sendfile syscall.
I'm not sure that this has changed much. It errors with errno ==
EPIPE, but still claims it's placed 40960 bytes in the socket buffer.
A tight loop of netstat -anf inet never reveals a Send-Q greater
than 0 bytes, although it's entirely likely it missed the event.
But if I terminate the connection mid stream I can see a send queue
sticking around for a bit:
tcp4 0 43640 127.0.0.1.666 127.0.0.1.50916 CLOSE_WAIT
Ian
--
Ian Freislich
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E1IpMyS-0001ck-F6>