From owner-freebsd-net@FreeBSD.ORG  Sun Sep 21 23:14:28 2003
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 9F89316A4B3
	for <freebsd-net@freebsd.org>; Sun, 21 Sep 2003 23:14:28 -0700 (PDT)
Received: from frenchfries.net
	(lsanca1-ar14-4-60-145-009.lsanca1.dsl-verizon.net [4.60.145.9])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 9AC9D43FF9
	for <freebsd-net@freebsd.org>; Sun, 21 Sep 2003 23:14:25 -0700 (PDT)
	(envelope-from pherman@frenchfries.net)
Received: by frenchfries.net (Postfix, from userid 1000)
	id BAF3D4B89; Sun, 21 Sep 2003 23:14:23 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
	by frenchfries.net (Postfix) with ESMTP id 787A24A65
	for <freebsd-net@freebsd.org>; Sun, 21 Sep 2003 23:14:23 -0700 (PDT)
Date: Sun, 21 Sep 2003 23:14:23 -0700 (PDT)
From: Paul Herman <pherman@frenchfries.net>
X-X-Sender: pherman@mammoth.eat.frenchfries.net
To: freebsd-net@freebsd.org
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Message-Id: <20030922061423.BAF3D4B89@frenchfries.net>
Subject: Question on TCP segment sizes
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 22 Sep 2003 06:14:28 -0000

Hi -net,

While pondering sending large files across a jumbo frame network
using FreeBSD, I decided to first see how well the loopback
interface does.  Using ttcp on the same machine:

  ttcp -s -r &
  ttcp -s -t 127.0.0.1

I noticed that although the MSS is 16k, I don't ever see a full 16k
segment.  In fact, 16k packets in the form of one 2k packet and one
14k packet are sent, as the following tcpdump output shows:

.706018 127.0.0.1.1026 > 127.0.0.1.5001: S 3254883299:3254883299(0) win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 112512 0> (DF)
.706108 127.0.0.1.5001 > 127.0.0.1.1026: S 2952251081:2952251081(0) ack 3254883300 win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 112512 112512> (DF)
.706140 127.0.0.1.1026 > 127.0.0.1.5001: . ack 1 win 57344 <nop,nop,timestamp 112512 112512> (DF)
.707454 127.0.0.1.1026 > 127.0.0.1.5001: P 1:8193(8192) ack 1 win 57344 <nop,nop,timestamp 112512 112512> (DF)
.708308 127.0.0.1.1026 > 127.0.0.1.5001: P 8193:22529(14336) ack 1 win 57344 <nop,nop,timestamp 112512 112512> (DF)
.708346 127.0.0.1.5001 > 127.0.0.1.1026: . ack 22529 win 43008 <nop,nop,timestamp 112512 112512> (DF)
.708375 127.0.0.1.1026 > 127.0.0.1.5001: P 22529:24577(2048) ack 1 win 57344 <nop,nop,timestamp 112512 112512> (DF)
.708508 127.0.0.1.1026 > 127.0.0.1.5001: P 24577:38913(14336) ack 1 win 57344 <nop,nop,timestamp 112512 112512> (DF)
.708530 127.0.0.1.5001 > 127.0.0.1.1026: . ack 38913 win 43008 <nop,nop,timestamp 112512 112512> (DF)
.708549 127.0.0.1.1026 > 127.0.0.1.5001: P 38913:40961(2048) ack 1 win 57344 <nop,nop,timestamp 112512 112512> (DF)
.708617 127.0.0.1.1026 > 127.0.0.1.5001: P 40961:55297(14336) ack 1 win 57344 <nop,nop,timestamp 112512 112512> (DF)
.708638 127.0.0.1.5001 > 127.0.0.1.1026: . ack 55297 win 43008 <nop,nop,timestamp 112512 112512> (DF)
[...repeats...]

The same happens phenomenon with FTP so I don't think there's any
voodoo going on with ttcp.  Interestingly enough, raising the MTU
on lo0 above 16k changes nothing.

Also, IPv6 (FTP) shows similar behavior:

.449842 ::1.1036 > ::1.49153: P 92161:93185(1024) ack 1 win 57344 <nop,nop,timestamp 292774 292774>
.449992 ::1.1036 > ::1.49153: P 93185:107521(14336) ack 1 win 57344 <nop,nop,timestamp 292774 292774>
.450018 ::1.49153 > ::1.1036: . ack 107521 win 43008 <nop,nop,timestamp 292774 292774> [flowlabel 0x645a7]
.450099 ::1.1036 > ::1.49153: P 107521:108545(1024) ack 1 win 57344 <nop,nop,timestamp 292774 292774>
.450250 ::1.1036 > ::1.49153: P 108545:122881(14336) ack 1 win 57344 <nop,nop,timestamp 292774 292774>
.450275 ::1.49153 > ::1.1036: . ack 122881 win 43008 <nop,nop,timestamp 292774 292774> [flowlabel 0x645a7]
.450354 ::1.1036 > ::1.49153: P 122881:123905(1024) ack 1 win 57344 <nop,nop,timestamp 292774 292774>
.450504 ::1.1036 > ::1.49153: P 123905:138241(14336) ack 1 win 57344 <nop,nop,timestamp 292774 292774>

So the question is, why don't we get two full sized 16344 (or at
least two 14336) sized segments?  It seems it would be more
efficient that way, no?

4.9-PRERELEASE from yesterday, all sysctls are system defaults...

-Paul.