Date: Tue, 14 Feb 2006 10:28:49 +0100 From: Ragnar Lonn <raglon@packetfront.com> To: Ivan Voras <ivoras@fer.hr> Cc: net@freebsd.org Subject: Re: TCP Performance advice needed [long!] Message-ID: <43F1A2D1.7040402@packetfront.com> In-Reply-To: <43F0CE40.5040800@fer.hr> References: <43F0CE40.5040800@fer.hr>
next in thread | previous in thread | raw e-mail | index | archive | help
I don't know what tcp.inflight does but I know that this type of application protocol, that expects a reply before sending the next piece of data, will always be completely dependent upon roundtrip times for its throughput - the roundtrip time for the exchange "transmit-reply" will limit the possible throughput you can get so if you want higher performance, either change the protocol so it works more like TCP with a "window" of data that is being sent. I.e. send several packets at once without waiting for replies in between. Or...use several concurrent TCP streams to speed things up (like NNTP servers do, for instance). Maybe this isn't what you wanted to know, but I thought I'd write it anyhow :) /Ragnar Ivan Voras wrote: > I'm writing a simple client-server application and I think I have a > performance problem. The basic communication between the server and > the client is like this: the server sends messages at most MAX_SIZE in > length, one by one, and waits until the client acknowleges each > message before sending the next one. Messages of maximum length > (MAX_SIZE) make about 75% or more of total messages. Client responses > are small and of fixed size (<10 bytes). I have TCP_NODELAY turned on > on both. All sends are done with a blocking send() call of required > size. Client's response processing time doesn't depend notably on the > message size. > > I've done some simple benchmarks and I'm puzzled about the results. In > the benchmarks, the MAX_SIZE is periodically increased from 256 to > 65536 in power-of-2 steps. In the benchmark setup the server and > client are connected directly via 100Mbps connection without > transmission problems (FTP pushes ~12MB/s in both directions). > > Problem: It seems that the packets-per-second rate sent by the server > increases from 1000 at MAX_SIZE=256 to 2500 at MAX_SIZE=4096, then > sharply drops to 400 at MAX_SIZE=8192 and above. Remembering a > discussion on Samba performance, I tried disabling > tcp.inflight.enable, and the results for large packet sizes became > very good, about 4000pps, until MAX_SIZE=32768, where it drops to 400 > again. What does tcp.inflight do, and can it be done without changing > sysctls? > > Another problem: the starting rates, e.g. ~1000pps at MAX_SIZE=256, > seem too low to me - this pushes only 256kB/s. This is a problem > because in real usage such small sizes could be common. Looking at > netstat, packet rates seem to vary randomly. Here's netstat log for > session with MAX_SIZE=512: > > > 1 0 66 1 0 178 0 > 378 0 27956 378 0 132192 0 > 1405 0 103962 1405 0 488770 0 > 1193 0 88274 1193 0 414994 0 > 1093 0 80874 1093 0 380194 0 > 1010 0 74732 1011 0 351658 0 > 930 0 68812 930 0 323732 0 > 797 0 58970 796 0 276576 0 > 95 0 7022 95 0 32890 0 > input (Total) output > packets errs bytes packets errs bytes colls > 201 0 14866 201 0 69778 0 > 495 0 36622 495 0 172202 0 > 739 0 54678 739 0 257002 0 > 1291 0 95526 1292 0 449728 0 > 1134 0 83908 1133 0 394094 0 > 1026 0 75916 1026 0 356616 0 > 987 0 73030 987 0 343306 0 > 873 0 64594 873 0 303634 0 > 855 0 63262 855 0 297370 0 > 360 0 26632 360 0 125372 0 > 1002 0 74140 1002 0 348264 0 > 1166 0 86276 1166 0 405860 0 > 1056 0 78136 1056 0 367056 0 > 937 0 69330 937 0 325906 0 > 753 0 55714 753 0 261874 0 > 111 0 8206 111 0 38458 0 > 143 0 10566 144 0 49692 0 > 1 0 66 1 0 178 0 > 1 0 66 1 0 178 0 > > Here's for MAX_SIZE=8192: > > input (Total) output > packets errs bytes packets errs bytes colls > 1082 0 74292 1438 0 1580040 0 > 1287 0 88366 1701 0 1885062 0 > 1403 0 96342 1867 0 2044074 0 > 1763 0 121046 2346 0 2578684 0 > 1532 0 105208 2026 0 2236048 0 > 1866 0 128124 2463 0 2726554 0 > 2401 0 164874 3192 0 3508384 0 > 2797 0 192058 3719 0 4095824 0 > 2791 0 191646 3714 0 4070142 0 > 2773 0 190410 3692 0 4052512 0 > 2583 0 177358 3438 0 3780296 0 > 2947 0 202358 3927 0 4307032 0 > 2963 0 203462 3943 0 4324596 0 > 2946 0 202292 3926 0 4307032 0 > 2965 0 203594 3946 0 4341638 0 > 2989 0 205242 3977 0 4359552 0 > 2928 0 201056 3898 0 4289006 0 > 2995 0 205654 3983 0 4368192 0 > 2965 0 203594 3945 0 4333018 0 > 2983 0 204830 3973 0 4359598 0 > 2978 0 204484 3968 0 4359380 0 > input (Total) output > packets errs bytes packets errs bytes colls > 2986 0 205028 3976 0 4359910 0 > 2979 0 204566 3969 0 4359578 0 > 2123 0 145790 2827 0 3089432 0 > > It seems that the transmission for MAX_SIZE=8192 has slow start but I > don't know how to disable it? > > Help on any of these points, or suggestions where to investigate > further will be appreciated! > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43F1A2D1.7040402>