From owner-freebsd-net@FreeBSD.ORG Wed Feb 15 19:43:23 2006 Return-Path: X-Original-To: net@freebsd.org Delivered-To: freebsd-net@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A920316A420 for ; Wed, 15 Feb 2006 19:43:23 +0000 (GMT) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id D5A7443D62 for ; Wed, 15 Feb 2006 19:43:16 +0000 (GMT) (envelope-from andre@freebsd.org) Received: (qmail 81119 invoked from network); 15 Feb 2006 19:39:29 -0000 Received: from c00l3r.networx.ch (HELO freebsd.org) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 15 Feb 2006 19:39:29 -0000 Message-ID: <43F3846E.846ED620@freebsd.org> Date: Wed, 15 Feb 2006 20:43:42 +0100 From: Andre Oppermann X-Mailer: Mozilla 4.8 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Ivan Voras References: <43F0CE40.5040800@fer.hr> <43F1A2D1.7040402@packetfront.com> <43F24E7E.4060503@fer.hr> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Ragnar Lonn , net@freebsd.org Subject: Re: TCP Performance advice needed [long!] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2006 19:43:23 -0000 Ivan Voras wrote: > > Ragnar Lonn wrote: > > I don't know what tcp.inflight does but I know that this type of > > application > > protocol, that expects a reply before sending the next piece of data, will > > always be completely dependent upon roundtrip times for its throughput - > > the roundtrip time for the exchange "transmit-reply" will limit the > > possible throughput you can get so if you want higher performance, either > > I understand this bottleneck, and know (at least in theory :) ) how it > could be solved, but my problems are not directly related to that: > > - For small (but consistent in size) packet sizes, I get randomly > varying round-trip times, and much lower packets-per-second ratio then > with big packets (consistent in size) with the exact same lock-step > protocol. Packet generation and processing are not CPU intensive. This is because of the Nagle algo which tries to accumulate some more data in packets and to piggyback ACKs to replies. Enable TCP_NODELAY on the socket and it should go away. Disabling delayed ACKs may help in your case too. > - When using big packets (actually, when switching back and forth from > small packets to big packets), the PPS performance starts low and climbs > to "normal" levels, and I'd like to avoid this. This is a local network > with 0 errors. > > (if you like it, replace the word "packets" with "messages" in the above > explanation :) ) > > I think the above problems are not directly related to the protocol > (which could be better, I agree, but it won't happen at least until I > understand what is happening with this version) but on fine-tuning of > the network or socket options. This is 'normal' TCP socket behaviour which is obviously not good with your application. -- Andre