From owner-freebsd-net Thu Nov 7 20:30:46 2002 Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C8DD537B401 for ; Thu, 7 Nov 2002 20:30:43 -0800 (PST) Received: from lawyers.ir.bbn.com (d53-203-209.clv.wideopenwest.com [64.53.209.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id AE64743E75 for ; Thu, 7 Nov 2002 20:30:42 -0800 (PST) (envelope-from mallman@lawyers.ir.bbn.com) Received: from lawyers (mallman@localhost) by lawyers.ir.bbn.com (8.9.3/8.9.3) with ESMTP id XAA29327; Thu, 7 Nov 2002 23:28:33 -0500 Message-Id: <200211080428.XAA29327@lawyers.ir.bbn.com> To: freebsd-net@FreeBSD.ORG Cc: Fran.Lawas-Grodek@grc.nasa.gov, Cindy.Tran@grc.nasa.gov Cc: "Joseph Ishac" , bdimond@grc.nasa.gov From: Mark Allman Reply-To: mallman@grc.nasa.gov Subject: Re: Problem in High Speed and Long Delay with FreeBSD In-Reply-To: <20021101112001.E3259@grc.nasa.gov> Organization: BBN Technologies/NASA GRC Song-of-the-Day: Born in the USA Date: Thu, 07 Nov 2002 23:28:33 -0500 Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Folks- Lots of interesting thoughts on this thread already. But, we have not yet figured it out. So, a further data point... I have been playing this evening on my machine at home -- a way old p5 running freebsd 4.7. I am seeing the same problem as we see at GRC on the freebsd 4.1 boxes. As I see it we can have one of a few possible problems: * For some reason the application is not able to fill the socket buffer. * For some reason TCP is not grabbing everything from the socket buffer. * TCP's congestion control is getting hosed somehow into thinking we need to slow down. So, my idea goes something like this: Say I have a connection with a X byte socket buffer. I start a timer, open a TCP connection, write Y bytes (Y < X), close the connection and stop the timer. The time should be very small in this case, because I am really timing how long it take the app to send data to the OS. (I am not enabling SO_LINGER). So, I ran some tests, as follows (sockbuf == X, dataen == Y): sockbuf datalen snd_time rcv_time --------------------------------------------- 16384 15000 0.000 0.617 150000 140000 0.003 4.021 500000 495000 0.015 14.083 1000000 995000 0.042 28.577 1500000 1490000 0.079 47.986 1600000 1590000 0.088 44.055 1800000 1790000 0.108 50.810 1900000 1890000 0.117 55.010 2000000 1990000 1.011 57.666 2100000 2090000 3.845 60.233 3000000 2990000 39.594 122.308 So, except for the last 3 lines all the tests easily filled up the socket buffers and exited. But, in the third line from the bottom we started to see something else. We "filled" the buffers until we could fill no more and had to wait for some ACKs to allow the buffer to drain before dumping everything in it. But, the socket buffer is bigger than the data stream so this should not be an issue, right? The situation only gets worse as we increase the transfer sizes (and socket buffer sizes). The break point seems to be somewhere near 2MB. For instance, when sending almost 3MB the sender though the transfer took ~40 seconds, while the receiver accurately shows ~122 seconds. That shows that the sender had to wait for a third of the data to drain from the network to get all its data into the socket buffers. But, that should not have been. One more piece of information... The socket descriptor is placed in non-blocking mode. I write in 8KB chunks. So, I do a select(), and write accordingly. However, even though I cannot write into the socket buffer at times, select() never fails to say that I can write the descriptor *and* I never see a short write(). But, clearly I am blocking or I'd shoot everything into the socket buffer. Does any of this make any sense to anyone? Any ideas on what might be wrong here? Any suggestions on places to start looking? Thanks! allman -- Mark Allman -- NASA GRC/BBN -- http://roland.grc.nasa.gov/~mallman/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message