From owner-freebsd-hackers  Tue Apr 10 11: 5:18 2001
Delivered-To: freebsd-hackers@freebsd.org
Received: by hub.freebsd.org (Postfix, from userid 618)
	id E229D37B422; Tue, 10 Apr 2001 11:05:14 -0700 (PDT)
Subject: sendfile() vs. gigabit ethernet
To: hackers@freebsd.org
Date: Tue, 10 Apr 2001 11:05:14 -0700 (PDT)
X-Mailer: ELM [version 2.4ME+ PL54 (25)]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Message-Id: <20010410180514.E229D37B422@hub.freebsd.org>
From: wpaul@FreeBSD.ORG (Bill Paul)
Sender: owner-freebsd-hackers@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.ORG

People are constantly asking me about the performance of gigabit ethernet
cards with FreeBSD. I'm constantly wishing they would all go to hell --
directly to hell, without passing "GO" or collecting $200 -- but their
persistence has led me to ask a few questions of my own. One thing everyone
notices right away is that some of FreeBSD's defaults aren't terribly
optimal for gigabit ethernet, the main one being socket buffer sizes.
At least as far as TCP is concerned, you can do one of two things:

- Change your application to call setsockopt() and increase the SO_SNDBUF
  and SO_RCVBUF values.
- Use sysctl to change the net.inet.tcp.sendspace and net.inet.tcp.recvspace
  values.

The former option affectly only a given application, while the latter
affects the entire system. Most people will bump the sendspace and
recvspace variables since it's easier than modifying their application.
Changing the default from 16K to 64K usually has a drastic effect on
performance with something like ttcp, which is what I typically use
for testing. You can also use the -B flag to ttcp to have it increase
the SNDBUF/RCVBUF values just for itself and achieve the same effect.

But then what usually happens is people will attempt to perform a file
transfer test using, of all things. FTP, and suddenly they notice that
performance doesn't really improve all that much. I could never understand
just why this was so, until I realized that /usr/libexec/ftpd in FreeBSD
now uses sendfile(). The ttcp utility on the other hand uses write(2)
(as does netperf for that matter).

So my question is: has anyone done any tests with sendfile() over a
high speed network link and tried to vary the sendspace/recvspace
defaults to improve performance? If so, does it make any difference?
My feeling is that it should, but from what I can tell, sendfile()
only passes data to sosend() in page-sized chunks, and this is somehow
negating the effect of increasing the socket buffer size. Am I right,
or am I just missing something obvious, as usual?

-Bill

=============================================================================
-Bill Paul            (925) 691-2800 | Systems Programmer, Master of Unix-Fu
                  wpaul@osd.bsdi.com | BSDi Open Source Solutions
=============================================================================
"I like zees guys. Zey are fonny guys. Just keel one of zem." -- The 3 Amigos
=============================================================================

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message