Date: Thu, 27 Apr 2023 14:30:33 +0530 From: biggy pardeshi <biggy.pardeshi@gmail.com> To: freebsd-transport@freebsd.org Cc: rscheff@freebsd.org Subject: FreeBSD CUBIC - Extremely low performance for short RTT setups experiencing congestion Message-ID: <CAD%2Bhhtso5GkhVWi0YwqA%2BFaJtqWiC=sNRecdTzWp=9KRKCNnNQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--00000000000087cdd005fa4d97b9 Content-Type: text/plain; charset="UTF-8" Hi team, I am a networking datapath engineer working at VMware. I primarily work on VMware's ESX's TCP/IP stack. We have been playing around with FreeBSD's CUBIC Congestion Control Algorithm (CCA) for a while. In most of our performance tests, CUBIC is performing fine. However, we saw that CUBIC results in performance degradation of around 100-150% in setups where the following conditions are met 1. sender and receiver are back-to-back connected (short RTT) 2. back-to-back connection/link experiences congestion We saw that the congestion window (cwnd) is not increasing fast enough in the case of CUBIC, after experiencing a congestion event. On the same setup, NewReno is performing very well and provides a very fast increase in the cwnd after a congestion event. https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-cubic-06 - CUBIC Internet-Draft (I-D) As per the I-D, it is mentioned that for short RTT (or even low BDP) networks standard TCP CCAs perform better than CUBIC. Hence, in such cases, the TCP-friendly window size (equation 4) will always come out to be greater than the cubic window size (equation 1). Hence, we will focus our discussion only on the TCP-friendly window size being calculated during the congestion avoidance phase. Theoretically, CUBIC should perform at least as better as standard TCP for short RTT setups. However, in reality, this is not being observed. We find that the granularity of the system's ticks value is causing CUBIC to not grow the cwnd faster. I will explain this issue with an example. ------- Consider that we have a short RTT setup where the RTT is 0.05ms. Let us assume that the system's tick frequency is 1000 HZ, i.e. the ticks value will increase by 1, once every 1ms (FreeBSD probably has a 1ms tick timer). Now let us consider a period of 1s. During this 1s period, the sender will receive around 1s/0.05ms = 1000ms/0.05ms = 20000 ACKs. Consequently, we will be calling the cc_ack_received() callback for each of these ACKs. For NewReno, the cwnd will be increased for each of those ACKs, till the TCP flow becomes limited by the receiver's window. However, in CUBIC, even though the cubic_ack_received() callback is invoked for each of those 20000 ACKs, the cwnd will not be increased for each of those ACKs. This is because of the "ticks" value used to calculate the time elapsed from the last congestion event. In FreeBSD for a period of 1ms, the ticks value will be the same. In 1ms, we will receive 1ms/0.05ms = 20 ACKs. For all of these 20 ACKs, ticks value will be the same, the time elapsed from the last congestion will be the same, and finally, the TCP-friendly window estimate will be the same. Hence, cwnd will not increase for the entire 1ms duration. If a system has some other timer period (for eg. 10ms), the cwnd will stay the same for that entire period. Of these 20000 ACKs, NewReno will try to increase the cwnd value for almost every ACK. However, CUBIC will increase it only for 1000 ACKs. That too distributed over a period of 1s. I hope the issue is now clear. ---- We wanted to discuss the solution to this issue. Currently, we have thought of falling back to using the newreno way of doing congestion avoidance when we are dealing with short RTT connections. This means that if the mean RTT value maintained by CUBIC private data is less than or equal to 1, we will use NewReno's way of doing congestion avoidance to get a TCP-friendly window estimate. In other cases (non-short RTT), we will use equation 4 of I-D to get the TCP-friendly estimate. This will resolve the issue we are seeing. However, the only concern we have in this approach is that this will make CUBIC RTT-dependent for short RTT networks. As per the I-D I see the following Another notable feature of CUBIC is that its window increase rate is mostly independent of RTT, and follows a (cubic) function of the elapsed time from the beginning of congestion avoidance. So, we are not sure if logically this solution is right or not. Also, we are not sure of any other implications this change might cause in CUBIC. ---- Adding Richard to the thread directly, as I have been following his work on CUBIC for some time. Thanks, Bhaskar Pardeshi (bpardeshi@vmware.com) VMware, Inc. --00000000000087cdd005fa4d97b9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi team,<div><br></div><div>I am a networking datapath eng= ineer working at VMware. I primarily work on VMware's ESX's TCP/IP = stack. We have been playing around with FreeBSD's CUBIC Congestion Cont= rol Algorithm (CCA) for a while. In most of our performance tests, CUBIC is= performing fine. However, we saw that CUBIC results in performance degrada= tion of around 100-150% in setups where the following conditions are met</d= iv><div><ol><li>sender and receiver are back-to-back connected (short RTT)<= /li><li>back-to-back connection/link experiences congestion</li></ol><div>W= e saw that the congestion window (cwnd) is not increasing fast enough in th= e case of CUBIC, after experiencing a congestion event. On the same setup, = NewReno is performing=C2=A0very well and provides a very fast increase in t= he cwnd after a congestion event.</div></div><div><br></div><div><a href=3D= "https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-cubic-06">https://da= tatracker.ietf.org/doc/html/draft-ietf-tcpm-cubic-06</a> - CUBIC Internet-D= raft (I-D)<br></div><div>As per the I-D, it is mentioned that for short RTT= (or even low BDP) networks standard TCP CCAs perform=C2=A0better than CUBI= C. Hence, in such cases, the TCP-friendly window size (equation 4) will alw= ays come out to be greater than the cubic window size (equation 1). Hence, = we will focus our discussion only on the TCP-friendly window size being cal= culated during the congestion avoidance phase.</div><div><br></div><div>The= oretically,=C2=A0CUBIC should perform at least as better as standard TCP fo= r short RTT setups. However, in reality, this is not being observed. We fin= d that the granularity of the system's ticks value is causing CUBIC to = not grow the cwnd faster. I will explain this issue with an example.</div><= div><br></div><div>-------</div><div><br></div><div>Consider that we have a= short RTT setup where the RTT is 0.05ms. Let us assume that the system'= ;s tick frequency is 1000 HZ, i.e. the ticks value will increase by 1, once= every 1ms (FreeBSD probably has a 1ms tick timer).</div><div><br></div><di= v>Now let us consider a period of 1s. During this 1s period, the sender wil= l receive around 1s/0.05ms =3D 1000ms/0.05ms =3D 20000 ACKs. Consequently, = we will be calling the cc_ack_received() callback for each of these ACKs. F= or NewReno, the cwnd will be increased for each of those ACKs, till the TCP= flow becomes limited by the receiver's window. However, in CUBIC, even= though the cubic_ack_received() callback is invoked for each of those 2000= 0 ACKs, the cwnd will not be increased for each of those ACKs. This is beca= use of the "ticks" value used to calculate the time elapsed from = the last congestion event. In FreeBSD for a period of 1ms, the ticks value = will be the same. In 1ms, we will receive 1ms/0.05ms =3D 20 ACKs. For all o= f these 20 ACKs, ticks value will be the same, the time elapsed from the la= st congestion will be the same, and finally, the TCP-friendly window estima= te will be the same. Hence, cwnd will not increase for the entire 1ms durat= ion. If a system has some other timer period (for eg. 10ms), the cwnd will = stay the same for that entire period.</div><div><br></div><div>Of these 200= 00 ACKs,=C2=A0 NewReno will try to increase the cwnd value for almost every= ACK. However, CUBIC will increase it only for 1000 ACKs. That too distribu= ted over a period of 1s. I hope the issue is now clear.</div><div><br></div= ><div>----</div><div>We wanted to discuss the solution to this issue. Curre= ntly, we have thought of falling back to using the newreno way of doing con= gestion avoidance when we are dealing with short RTT connections. This mean= s that if the mean RTT value maintained by CUBIC private data is less than = or equal to 1, we will use NewReno's way of doing congestion avoidance = to get a TCP-friendly window estimate. In other cases (non-short RTT), we w= ill use equation 4 of I-D to get the TCP-friendly estimate.</div><div><br><= /div><div>This will resolve the issue we are seeing. However, the only conc= ern we have in this approach is that this will make CUBIC RTT-dependent for= short RTT networks. As per the I-D I see the following</div><div><pre clas= s=3D"gmail-c-mrkdwn__pre" style=3D"box-sizing:inherit;margin-top:4px;margin= -bottom:4px;padding:8px;font-size:12px;font-variant-ligatures:none;line-hei= ght:1.50001;white-space:pre-wrap;word-break:normal;border-radius:4px;overfl= ow-y:hidden;color:rgb(209,210,211);font-family:Monaco,Menlo,Consolas,"= Courier New",monospace"><span style=3D"background-color:rgb(255,0,0)">= Another notable feature of CUBIC is that its window increase rate is mostly independent of RTT, and follows a (cubic) function of the elapsed time from the beginning of congestion avoidance.</span></pre></d= iv><div><div>So, we are not sure if logically this solution is right or not= . Also, we are not sure of any other implications this change might cause i= n CUBIC.</div><div><br></div><div>----</div><div>Adding Richard to the thre= ad directly, as I have been following his work on CUBIC for some time.</div= ><div><br></div><div>Thanks,</div><div>Bhaskar Pardeshi (<a href=3D"mailto:= bpardeshi@vmware.com">bpardeshi@vmware.com</a>)</div></div><div>VMware, Inc= .</div></div> --00000000000087cdd005fa4d97b9--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD%2Bhhtso5GkhVWi0YwqA%2BFaJtqWiC=sNRecdTzWp=9KRKCNnNQ>