From owner-freebsd-net@FreeBSD.ORG Tue Nov 21 21:23:15 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 85F1D16A407; Tue, 21 Nov 2006 21:23:15 +0000 (UTC) (envelope-from pr@isprime.com) Received: from radio.isprime.com (radio.isprime.com [66.230.159.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id EE05E43DD5; Tue, 21 Nov 2006 21:20:01 +0000 (GMT) (envelope-from pr@isprime.com) Received: from [64.111.209.130] (unknown [64.111.209.130]) by radio.isprime.com (Postfix) with ESMTP id 4207A1B68F7; Tue, 21 Nov 2006 16:20:18 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <2815E433-4AB2-491D-B62A-D685F6B85F35@isprime.com> Content-Transfer-Encoding: 7bit From: Phil Rosenthal Date: Tue, 21 Nov 2006 16:20:17 -0500 To: andre@freebsd.org X-Mailer: Apple Mail (2.752.2) Cc: freebsd-net@freebsd.org Subject: Re: Automatic TCP send socker buffer sizing X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2006 21:23:15 -0000 Hello, I've tested this out and had a few comments. 1) I've seen in production that some sockets get large very quickly during periods of high latency (eg: when sending to a user downloading from a cablemodem and their headend gets temporarily saturated and has large buffers, which raises the RTT under saturation, which increases the bandwidth delay product), but then as there isn't any code to shrink the buffers. This would probably need to be in the timers to notice the case of the sender temporarily stopping sending - eg in a keepalive http socket (a separate, but related issue). 2) In our code, and I believe in other code, we watch the tcp send buffer drain before closing the sockets, which in our implementation works by checking with kqueue how many bytes we can send to that socket. If the buffer size keeps changing, we would now have to call getsockopt() periodically to get the current size, which would obviously have a polling-type performance problem. Is it possible to make a kqueue hook to let us know when the socket buffer size changes? 3) There is a potential for a DoS here as you could have a large number of clients simulate a very large bandwidth delay product (eg: ipfw delay 100,000 ms) and start up a lot of downloads. This could cause your total socket buffers to rapidly grow much larger than normal and eat up all the available kernel memory. Perhaps having another tunable for a soft limit on total socket size past which we get more conservative on buffer sizing, and reduce the size of the largest buffers in exchange for making the smallest buffers larger. 4) What happens if you call setsockopt() to set the size? Perhaps we can have a new setsockopt() flag to disable automatic sizing on a socket if we have a specific size we want a buffer to be and don't want it dynamically sized. After some refining, this does have some very good potential to be very useful for us. -P