Date: Thu, 31 Oct 2002 15:02:17 -0800 From: Luigi Rizzo <rizzo@icir.org> To: Fran.Lawas-Grodek@grc.nasa.gov, Cindy.Tran@grc.nasa.gov Cc: freebsd-net@FreeBSD.ORG, Mark.Allman@grc.nasa.gov Subject: Re: Problem in High Speed and Long Delay with FreeBSD Message-ID: <20021031150217.A35590@carp.icir.org> In-Reply-To: <20021031135601.B23802@grc.nasa.gov>; from Fran.Lawas-Grodek@grc.nasa.gov on Thu, Oct 31, 2002 at 01:56:01PM -0500 References: <20021031135601.B23802@grc.nasa.gov>
next in thread | previous in thread | raw e-mail | index | archive | help
you might want to have a look at the sysctl variable
kern.ipc.sockbuf_waste_factor too.
Remember that memory is charged to socket buffers depending on how
many clusters are allocated, even if they are not fully used.
E.g. in your example you are probably doing 1KB writes each of
which consumes a 2KB cluster plus a 256byte mbuf, so no
matter what you will never manage to reach more than (roughly)
a window larger than kern.ipc.maxsockbuf/2.
The max raw amounf of memory allocated in a socket buffer is
typically
min( tcp_{send|recv}buf * kern.ipc.sockbuf_waste_factor,
kern.ipc.maxsockbuf)
and probably you are hitting the roof on the second one.
cheers
luigi
On Thu, Oct 31, 2002 at 01:56:01PM -0500, Fran Lawas-Grodek wrote:
> Hello,
>
> Hopefully someone might have some advice on our problem.
>
> We are setting up a testbed consisting of FreeBSD 4.1 on the sender
> and receiver machines. (This older version of FreeBSD is necessary
> due to subsequent TCP development patches that are to be tested.)
> The problem that we are seeing is that on a 100Mbps link with a
> 250millisecond delay, our overall throughput does not exceed 32Mbps.
> We expect to see at least 60Mbps with the 250ms delay. Without a
> delay, throughput is acceptable at 87Mbps. Tcptrace output shows no
> retransmissions and no out-of-order packets under the delay.
>
> With the delay, we are setting up a 3MByte sender and receiver socket
> buffer size, but through the time sequence plot, we only see about 1MB
> used of the buffer on the sender side. Additional tests were run with an
> 80ms delay and requesting a 1MB buffer, and yet it appears only are
> allowed to use up to 500KB of the window. Raising the requested
> socket buffer size shows no affect -- we keep running into some
> invisible threshold that is much smaller than the requested socket
> buffer size. A hacked version of the ttcp application prints out that
> what we request in the -b buffer size via setsockopt, is what we get
> (via getsockopt). Unfortunately, the xplots show that our transfers
> reach a lower threshold (1MB instead of 3MB, 500KB instead of 1MB).
>
> Testing commands used:
> receiver> ttcp -b 312500 -l 1024 -r -s
> sender> ttcp -b 312500 -l 1024 -n 100000 -s -t receiverhost
>
> The -b above is set to the value of the Bandwidth Delay Product
> for a 100Mbps link and 250ms delay.
>
> Per the Pittsburgh Supercomputing site, we have already increased the
> maxsockbuf, nmbclusters, memory, tcp_sendspace, tcp_recvspace sysctl
> values, but are still resulting in this low throughput.
>
> The network cards that we are using are 3Com95-TX 100BaseT cards. The
> machines with the FreeBSD installations are Pentium II 400Mhz with
> Xeon chips, 400 Mb RAM each.
>
> Using two other non-FreeBSD machines, we have verified that it is not
> the intermediary network routers or delay emulator equipment, as the
> non-FreeBSD machines give the expected throughput under delayed and
> no delay conditions. (60+Mbps under 250ms delay)
>
> Has anyone else any experiences in this type of testing?
> Perhaps there might be something wrong with our network card driver?
> Any other suggested network cards to try?
> Does anyone know of a limit that needs to be tweaked in the TCP
> stack or FreeBSD operating system that would allow us to actually
> use more than this invisible socket buffer limit? One thought is
> that there is some sort of calculated limit that won't allow us to
> send more packets than our requested socket buffer will allow, but
> not having any the kernel expertise, we are not sure where to look.
>
>
> Thanks in advance for any suggestions,
>
> Fran Lawas-Grodek
> Cindy Tran
> NASA Glenn Research Center
> Cleveland OH USA
>
> (ps: We have already consulted with Mark Allman at our lab, and he
> is just as stumped. The feeling is that the cause of the problem
> is buried somewhere in the kernel, not due to the network cards.)
>
> --
> ________________________________________________________________
>
> Frances J. Lawas-Grodek |
> NASA Glenn Research Center | phone: (216) 433-5052
> 21000 Brookpark Rd, MS 142-4 | fax : (216) 433-8000
> Cleveland, Ohio 44135 | email: fran@grc.nasa.gov
> ________________________________________________________________
>
>
>
> To Unsubscribe: send mail to majordomo@FreeBSD.org
> with "unsubscribe freebsd-net" in the body of the message
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021031150217.A35590>