Date: Mon, 13 Feb 2006 19:21:52 +0100 From: Ivan Voras <ivoras@fer.hr> To: net@freebsd.org Subject: TCP Performance advice needed [long!] Message-ID: <43F0CE40.5040800@fer.hr>
next in thread | raw e-mail | index | archive | help
I'm writing a simple client-server application and I think I have a
performance problem. The basic communication between the server and the
client is like this: the server sends messages at most MAX_SIZE in
length, one by one, and waits until the client acknowleges each message
before sending the next one. Messages of maximum length (MAX_SIZE) make
about 75% or more of total messages. Client responses are small and of
fixed size (<10 bytes). I have TCP_NODELAY turned on on both. All sends
are done with a blocking send() call of required size. Client's response
processing time doesn't depend notably on the message size.
I've done some simple benchmarks and I'm puzzled about the results. In
the benchmarks, the MAX_SIZE is periodically increased from 256 to 65536
in power-of-2 steps. In the benchmark setup the server and client are
connected directly via 100Mbps connection without transmission problems
(FTP pushes ~12MB/s in both directions).
Problem: It seems that the packets-per-second rate sent by the server
increases from 1000 at MAX_SIZE=256 to 2500 at MAX_SIZE=4096, then
sharply drops to 400 at MAX_SIZE=8192 and above. Remembering a
discussion on Samba performance, I tried disabling tcp.inflight.enable,
and the results for large packet sizes became very good, about 4000pps,
until MAX_SIZE=32768, where it drops to 400 again. What does
tcp.inflight do, and can it be done without changing sysctls?
Another problem: the starting rates, e.g. ~1000pps at MAX_SIZE=256, seem
too low to me - this pushes only 256kB/s. This is a problem because in
real usage such small sizes could be common. Looking at netstat, packet
rates seem to vary randomly. Here's netstat log for session with
MAX_SIZE=512:
1 0 66 1 0 178 0
378 0 27956 378 0 132192 0
1405 0 103962 1405 0 488770 0
1193 0 88274 1193 0 414994 0
1093 0 80874 1093 0 380194 0
1010 0 74732 1011 0 351658 0
930 0 68812 930 0 323732 0
797 0 58970 796 0 276576 0
95 0 7022 95 0 32890 0
input (Total) output
packets errs bytes packets errs bytes colls
201 0 14866 201 0 69778 0
495 0 36622 495 0 172202 0
739 0 54678 739 0 257002 0
1291 0 95526 1292 0 449728 0
1134 0 83908 1133 0 394094 0
1026 0 75916 1026 0 356616 0
987 0 73030 987 0 343306 0
873 0 64594 873 0 303634 0
855 0 63262 855 0 297370 0
360 0 26632 360 0 125372 0
1002 0 74140 1002 0 348264 0
1166 0 86276 1166 0 405860 0
1056 0 78136 1056 0 367056 0
937 0 69330 937 0 325906 0
753 0 55714 753 0 261874 0
111 0 8206 111 0 38458 0
143 0 10566 144 0 49692 0
1 0 66 1 0 178 0
1 0 66 1 0 178 0
Here's for MAX_SIZE=8192:
input (Total) output
packets errs bytes packets errs bytes colls
1082 0 74292 1438 0 1580040 0
1287 0 88366 1701 0 1885062 0
1403 0 96342 1867 0 2044074 0
1763 0 121046 2346 0 2578684 0
1532 0 105208 2026 0 2236048 0
1866 0 128124 2463 0 2726554 0
2401 0 164874 3192 0 3508384 0
2797 0 192058 3719 0 4095824 0
2791 0 191646 3714 0 4070142 0
2773 0 190410 3692 0 4052512 0
2583 0 177358 3438 0 3780296 0
2947 0 202358 3927 0 4307032 0
2963 0 203462 3943 0 4324596 0
2946 0 202292 3926 0 4307032 0
2965 0 203594 3946 0 4341638 0
2989 0 205242 3977 0 4359552 0
2928 0 201056 3898 0 4289006 0
2995 0 205654 3983 0 4368192 0
2965 0 203594 3945 0 4333018 0
2983 0 204830 3973 0 4359598 0
2978 0 204484 3968 0 4359380 0
input (Total) output
packets errs bytes packets errs bytes colls
2986 0 205028 3976 0 4359910 0
2979 0 204566 3969 0 4359578 0
2123 0 145790 2827 0 3089432 0
It seems that the transmission for MAX_SIZE=8192 has slow start but I
don't know how to disable it?
Help on any of these points, or suggestions where to investigate further
will be appreciated!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43F0CE40.5040800>