Date: Mon, 10 Dec 2001 14:12:30 -0800 (PST) From: Justin Erenkrantz <jerenkrantz@apache.org> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/32684: sendfile(2) with threads (libc_r) does not work in non-blocking mode Message-ID: <200112102212.fBAMCU147995@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 32684 >Category: kern >Synopsis: sendfile(2) with threads (libc_r) does not work in non-blocking mode >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Dec 10 14:20:01 PST 2001 >Closed-Date: >Last-Modified: >Originator: Justin Erenkrantz >Release: -CURRENT >Organization: Apache Group >Environment: FreeBSD <name-deleted> 5.0-CURRENT FreeBSD 5.0-CURRENT #6: Thu Dec 6 23:59:25 PST 2001 >Description: sendfile(2) with threads enabled is broken. You do not explicitly require threads - just compile -pthreads to enable the libc_r wrapper. There appears to be some disrecepencies between the libc_r/uthread/uthread_sendfile.c and the sendfile syscall. With -CURRENT, the headers are not sent correctly. We are sending ~80020 bytes in the headers. This is more than can be done in one non-blocking writev call. The libc_r wrapper does not handle this case correctly. Also, the libc_r wrapper does not properly adjust the length parameters to the sendfile syscall. The syscall requires headers+length, but the libc_r wrapper doesn't subtract the headers from the length (it handles the writev call for the headers itself). However, Alfred has a patch that fixes these problems (see fix below). But, we are now seeing that the file is sent twice. It looks like the sendfile syscall is returning -1/EAGAIN with 0 for sbytes even when it might be sending data. More examination is needed. This perhaps could be a problem in our test case or a still lingering problem in FreeBSD somewhere. >How-To-Repeat: We have a reproducable test case in our APR (Apache Portability Runtime) library. If time permits, I will try to de-APRize this test case and post an addendum with a URL to the testfile. 1) Apache 2.0 is based off of APR. You can grab the latest CVS copy of APR from: :pserver:anoncvs@cvs.apache.org:/home/cvspublic (anoncvs password is "anoncvs") cvs co apr 2) Run ./buildconf to generate all of the build stuff (we require libtool and autoconf...) 3) Run ./configure with --enable-threads to override our disabling of FreeBSD thread support. (You may want to add --disable-shared.) 4) make all (duh) 5) If you want some extra help from the sendfile test program, go to: http://www.apache.org/~jerenkrantz/freebsd/sendfile.patch and apply it. It's a slightly modified version of the one in the repository to give more output. It also allows for remote connection of the sendfile test suite. I'm not too comfortable committing it just yet, so the web site will have to do. =) 6) make sendfile in test/ dir. 7) Okay, our test program needs a client and server. So: ./sendfile server ./sendfile client {blocking|nonblocking|timeout} If you apply the patch, this version also allows for: ./sendfile client nonblocking 192.168.0.2 to have the client connect to a remote server - that may or may not be of help. It is for me, so it's there. =) Otherwise, the server and client must be on the same machine. FWIW, I run the server like so: while true; do ./sendfile server; done (Timeout is essentially equivalent to non-blocking but places an upper limit on how long we will wait. I wouldn't worry about timeout that much - if non-blocking works, it should work as well.) 8) We send the data with the following segments: - 15 byte header (1234567890ABCD) - 5 byte header (EFGH) - 80,000 byte header (only ^) - A 200,000 byte file containing only 0s. (this is the file.) - 19 byte trailer (IJKLMNOPQRSTUVWXYZ) - 10 byte trailer (!@#$%&*()\n) - 90,000 byte trailer (only @) 9) APR's actual sendfile() call is located in network_io/unix/sendrecv.c. Look for __FreeBSD__. =) 10) It works with blocking mode in all cases. Non-blocking mode only work with threads disabled (--disable-threads). 11) With the patch, we definitely made some progress as we are getting past the headers and are sending the file. However, I will attempt to track this down time permitting, but since you know FreeBSD much better than I do, I think you guys have better luck. >Fix: A partial fix is here: http://people.freebsd.org/~alfred/uthread_sf.diff >Release-Note: >Audit-Trail: >Unformatted: To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200112102212.fBAMCU147995>