From nobody Thu Apr 27 09:00:33 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q6V825HNBz46blK for ; Thu, 27 Apr 2023 09:00:50 +0000 (UTC) (envelope-from biggy.pardeshi@gmail.com) Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q6V7z2D7Gz3BpH; Thu, 27 Apr 2023 09:00:47 +0000 (UTC) (envelope-from biggy.pardeshi@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=oX6op5ax; spf=pass (mx1.freebsd.org: domain of biggy.pardeshi@gmail.com designates 2607:f8b0:4864:20::52c as permitted sender) smtp.mailfrom=biggy.pardeshi@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-51b33c72686so6079892a12.1; Thu, 27 Apr 2023 02:00:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682586045; x=1685178045; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=RVlgNp5utacLhGTLgoCcFQh+xIdZW+b1qTYPrf5wYEw=; b=oX6op5ax48sBtCBR4h9rMk7UjsXzHNKK1BFeaSw0bcWcjBGUOO1dmDcCClTMMBZFmG f3VjWrGxx20bdVl+wZM4p0TtFzR72hCDKyD2tsYWKPQAW+5hPnV3dmD0sz1+zA8ZB40Y sRKOoWOl0AR968DrLhYqxCrDehc/3bsbNhfjo0mub5JtT5ePLj6nGg7J4OMmiPgAREZJ Ln25OKK93N/yn/A/wINXf2wR9xJ6A/FKlc70IatLQpZTBRd4kvpy1bXVfB9BmrRHGCUv mEQO7s/eWRkmEj+EWoH9iW0cAtsIZl+2MAA7jsW+pDKqt+hDDQ4m6bOELAPwrobq8BU2 EuwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682586045; x=1685178045; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=RVlgNp5utacLhGTLgoCcFQh+xIdZW+b1qTYPrf5wYEw=; b=Qhp1Oh/f+xP6ZVjL1Ni3lg9DY/ANbaYTPkKnEf3oOF3W+n9EpmQ9Djy5yL3otGhteC z4HfWNYn37/HZL2HRauoA9ytPYbq8U55BvI6A9ooObqAGNN+uS2kwmW3JwUr+j7ghP3W rw6OTRNVJwXOyNX5k7d5vcIluIsvqUXRx4fLrTtyfY4YvXMK3ZPIeN/x4z8QxDhjzjIv 3+6ue+FGkZ3d1zlMABuTGm202h6Cdo/AsRHjYjr+5/rSDO/Id1MIxLUIOT+9UbWHTGGq 7uMBAEGX6WOPOX9T/rx8IQ5J32wZcIr1YJGInPxYqM5j2Mf8oQwZHejT8S0b5xg86dWT uorQ== X-Gm-Message-State: AC+VfDznQlMDBKxVf1g2G3rkyXtDyhZIjF9aIoJjXUbiWDZR1bCV8sRV y72eDapmHcDwk0bUfeXssllfAIcnwsGHm4p+rS8+0Ke5Eag= X-Google-Smtp-Source: ACHHUZ6uBpBcHAxbJ92hQBVdurvQmBmycuN6NXvq6of2Oor3yXy4VxM6xyG9/SNti/gP6HrjIlVSAsXM031+yTFULcM= X-Received: by 2002:a17:90b:3a8d:b0:23f:2661:f94c with SMTP id om13-20020a17090b3a8d00b0023f2661f94cmr1087572pjb.47.1682586045237; Thu, 27 Apr 2023 02:00:45 -0700 (PDT) List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 From: biggy pardeshi Date: Thu, 27 Apr 2023 14:30:33 +0530 Message-ID: Subject: FreeBSD CUBIC - Extremely low performance for short RTT setups experiencing congestion To: freebsd-transport@freebsd.org Cc: rscheff@freebsd.org Content-Type: multipart/alternative; boundary="00000000000087cdd005fa4d97b9" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::52c:server fail]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52c:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-transport@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; TAGGED_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Q6V7z2D7Gz3BpH X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --00000000000087cdd005fa4d97b9 Content-Type: text/plain; charset="UTF-8" Hi team, I am a networking datapath engineer working at VMware. I primarily work on VMware's ESX's TCP/IP stack. We have been playing around with FreeBSD's CUBIC Congestion Control Algorithm (CCA) for a while. In most of our performance tests, CUBIC is performing fine. However, we saw that CUBIC results in performance degradation of around 100-150% in setups where the following conditions are met 1. sender and receiver are back-to-back connected (short RTT) 2. back-to-back connection/link experiences congestion We saw that the congestion window (cwnd) is not increasing fast enough in the case of CUBIC, after experiencing a congestion event. On the same setup, NewReno is performing very well and provides a very fast increase in the cwnd after a congestion event. https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-cubic-06 - CUBIC Internet-Draft (I-D) As per the I-D, it is mentioned that for short RTT (or even low BDP) networks standard TCP CCAs perform better than CUBIC. Hence, in such cases, the TCP-friendly window size (equation 4) will always come out to be greater than the cubic window size (equation 1). Hence, we will focus our discussion only on the TCP-friendly window size being calculated during the congestion avoidance phase. Theoretically, CUBIC should perform at least as better as standard TCP for short RTT setups. However, in reality, this is not being observed. We find that the granularity of the system's ticks value is causing CUBIC to not grow the cwnd faster. I will explain this issue with an example. ------- Consider that we have a short RTT setup where the RTT is 0.05ms. Let us assume that the system's tick frequency is 1000 HZ, i.e. the ticks value will increase by 1, once every 1ms (FreeBSD probably has a 1ms tick timer). Now let us consider a period of 1s. During this 1s period, the sender will receive around 1s/0.05ms = 1000ms/0.05ms = 20000 ACKs. Consequently, we will be calling the cc_ack_received() callback for each of these ACKs. For NewReno, the cwnd will be increased for each of those ACKs, till the TCP flow becomes limited by the receiver's window. However, in CUBIC, even though the cubic_ack_received() callback is invoked for each of those 20000 ACKs, the cwnd will not be increased for each of those ACKs. This is because of the "ticks" value used to calculate the time elapsed from the last congestion event. In FreeBSD for a period of 1ms, the ticks value will be the same. In 1ms, we will receive 1ms/0.05ms = 20 ACKs. For all of these 20 ACKs, ticks value will be the same, the time elapsed from the last congestion will be the same, and finally, the TCP-friendly window estimate will be the same. Hence, cwnd will not increase for the entire 1ms duration. If a system has some other timer period (for eg. 10ms), the cwnd will stay the same for that entire period. Of these 20000 ACKs, NewReno will try to increase the cwnd value for almost every ACK. However, CUBIC will increase it only for 1000 ACKs. That too distributed over a period of 1s. I hope the issue is now clear. ---- We wanted to discuss the solution to this issue. Currently, we have thought of falling back to using the newreno way of doing congestion avoidance when we are dealing with short RTT connections. This means that if the mean RTT value maintained by CUBIC private data is less than or equal to 1, we will use NewReno's way of doing congestion avoidance to get a TCP-friendly window estimate. In other cases (non-short RTT), we will use equation 4 of I-D to get the TCP-friendly estimate. This will resolve the issue we are seeing. However, the only concern we have in this approach is that this will make CUBIC RTT-dependent for short RTT networks. As per the I-D I see the following Another notable feature of CUBIC is that its window increase rate is mostly independent of RTT, and follows a (cubic) function of the elapsed time from the beginning of congestion avoidance. So, we are not sure if logically this solution is right or not. Also, we are not sure of any other implications this change might cause in CUBIC. ---- Adding Richard to the thread directly, as I have been following his work on CUBIC for some time. Thanks, Bhaskar Pardeshi (bpardeshi@vmware.com) VMware, Inc. --00000000000087cdd005fa4d97b9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi team,

I am a networking datapath eng= ineer working at VMware. I primarily work on VMware's ESX's TCP/IP = stack. We have been playing around with FreeBSD's CUBIC Congestion Cont= rol Algorithm (CCA) for a while. In most of our performance tests, CUBIC is= performing fine. However, we saw that CUBIC results in performance degrada= tion of around 100-150% in setups where the following conditions are met
  1. sender and receiver are back-to-back connected (short RTT)<= /li>
  2. back-to-back connection/link experiences congestion
W= e saw that the congestion window (cwnd) is not increasing fast enough in th= e case of CUBIC, after experiencing a congestion event. On the same setup, = NewReno is performing=C2=A0very well and provides a very fast increase in t= he cwnd after a congestion event.

As per the I-D, it is mentioned that for short RTT= (or even low BDP) networks standard TCP CCAs perform=C2=A0better than CUBI= C. Hence, in such cases, the TCP-friendly window size (equation 4) will alw= ays come out to be greater than the cubic window size (equation 1). Hence, = we will focus our discussion only on the TCP-friendly window size being cal= culated during the congestion avoidance phase.

The= oretically,=C2=A0CUBIC should perform at least as better as standard TCP fo= r short RTT setups. However, in reality, this is not being observed. We fin= d that the granularity of the system's ticks value is causing CUBIC to = not grow the cwnd faster. I will explain this issue with an example.
<= div>
-------

Consider that we have a= short RTT setup where the RTT is 0.05ms. Let us assume that the system'= ;s tick frequency is 1000 HZ, i.e. the ticks value will increase by 1, once= every 1ms (FreeBSD probably has a 1ms tick timer).

Now let us consider a period of 1s. During this 1s period, the sender wil= l receive around 1s/0.05ms =3D 1000ms/0.05ms =3D 20000 ACKs. Consequently, = we will be calling the cc_ack_received() callback for each of these ACKs. F= or NewReno, the cwnd will be increased for each of those ACKs, till the TCP= flow becomes limited by the receiver's window. However, in CUBIC, even= though the cubic_ack_received() callback is invoked for each of those 2000= 0 ACKs, the cwnd will not be increased for each of those ACKs. This is beca= use of the "ticks" value used to calculate the time elapsed from = the last congestion event. In FreeBSD for a period of 1ms, the ticks value = will be the same. In 1ms, we will receive 1ms/0.05ms =3D 20 ACKs. For all o= f these 20 ACKs, ticks value will be the same, the time elapsed from the la= st congestion will be the same, and finally, the TCP-friendly window estima= te will be the same. Hence, cwnd will not increase for the entire 1ms durat= ion. If a system has some other timer period (for eg. 10ms), the cwnd will = stay the same for that entire period.

Of these 200= 00 ACKs,=C2=A0 NewReno will try to increase the cwnd value for almost every= ACK. However, CUBIC will increase it only for 1000 ACKs. That too distribu= ted over a period of 1s. I hope the issue is now clear.

----
We wanted to discuss the solution to this issue. Curre= ntly, we have thought of falling back to using the newreno way of doing con= gestion avoidance when we are dealing with short RTT connections. This mean= s that if the mean RTT value maintained by CUBIC private data is less than = or equal to 1, we will use NewReno's way of doing congestion avoidance = to get a TCP-friendly window estimate. In other cases (non-short RTT), we w= ill use equation 4 of I-D to get the TCP-friendly estimate.

<= /div>
This will resolve the issue we are seeing. However, the only conc= ern we have in this approach is that this will make CUBIC RTT-dependent for= short RTT networks. As per the I-D I see the following
=
   Another notable feature of CUBIC is that its window increase rate is
   mostly independent of RTT, and follows a (cubic) function of the
   elapsed time from the beginning of congestion avoidance.
So, we are not sure if logically this solution is right or not= . Also, we are not sure of any other implications this change might cause i= n CUBIC.

----
Adding Richard to the thre= ad directly, as I have been following his work on CUBIC for some time.

Thanks,
Bhaskar Pardeshi (bpardeshi@vmware.com)
VMware, Inc= .
--00000000000087cdd005fa4d97b9-- From nobody Fri Apr 28 08:57:04 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q761H4m8yz48CJ1 for ; Fri, 28 Apr 2023 08:57:07 +0000 (UTC) (envelope-from rscheff@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q761H461Pz4RH6; Fri, 28 Apr 2023 08:57:07 +0000 (UTC) (envelope-from rscheff@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682672227; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=nso5Br0SKlS/lIIqAhxzKCv77XFZ0K0ZeGAqTJ0gI1U=; b=N1iTZsZ4XbRLZB0siPL2qk2iiA7S2A48sYrynmgqHBxud/eqMfSOaoeZOq19nL+z8Fx43B ma2uLdL5tFgThyY7QDuV7VI7FygQejloilJXN/MkE+sBh7mi2ntv+r72mH0PmbrQq2M4w9 wGaqF9hSWVoeyL2KWHQRgc1ymk3rpd3aJROhZ1ruIbRsoM6fca+EU0UO10U5vLHDdE50Fy cKy3CDJaxMmGjojKRerO6xGSlkTbfe6QV/Xr6NuGsUDwnMYuu9En38SqSWM02zeTfQULzb Fpb5L79IX4NcOBjAOwFjRqCPxNG1tYksB4PKX5tSrRnauVyOh4gbllojFVc7xA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682672227; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=nso5Br0SKlS/lIIqAhxzKCv77XFZ0K0ZeGAqTJ0gI1U=; b=DHunhFWtae2A74GbdduBVbGYbn8DjiXj90ERNIzY0VZaqRWM5JQ4xphTFqU9N7UtJF/dht hu+Dg8BNskbaqr4JEykQE/WfVSZjfnRsGVm6lCTqA6mKfemkRhw+Eoneya3sLLrMb+pd6+ qyDGfydZaoHp1Ue83EAkkxn6PPsUUJZ/a0ZnOFMJH6r2jHZb16QQcysDbiOZjmmvWj8zUS UF5IT0RpQP/AA0BGwGsdNFYq3x6+XL5xv7UmK4eanVxvg9m4h3aZS+ZOpAj/r4J9oDo04m XOaDqTeTOpKN5u8cWtULDWCvRcfKZ/O6dNIjTnAdc1OGSpSKAp/xio7pLH/aRQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682672227; a=rsa-sha256; cv=none; b=cxnmvZdwBGAQLtDGHZLqw4ZM4XT1xXhpEABTU56jGJLxjR8djJ/W3A/vRLbmGOX8xKHg5l 8llQRJO1EOmHcHzOjnDZnGOoP67H0mg0uQ4B8UXCNtl65I4YeWilRWlvnr7kXmA04ln5wI gI3waMaDv05UdYTPjzRv28aFM5Xp9wMaOrmv0Pg/6erF5YChELzWkTuD4xCRRex8PDxRTf ALUnwojuqTMjw4fpdzY+ZGb+t/HxXa/6+M73eSxFuIItMJiAYJ0IrwB7nYFU6vcF7ehrR0 m1aUKyLxKtWOcGr48RuEPR7jtTpx9Y3kXLE7DPAxPxuACDAY6ZKZwXhZ5CVMyg== Received: from [192.168.233.109] (unknown [185.236.167.136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: rscheff/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Q761G74Tdz1STX; Fri, 28 Apr 2023 08:57:06 +0000 (UTC) (envelope-from rscheff@freebsd.org) Message-ID: Date: Fri, 28 Apr 2023 10:57:04 +0200 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 To: biggy pardeshi , freebsd-transport@freebsd.org, cc@freebsd.org From: "Scheffenegger, Richard" Autocrypt: addr=rscheff@freebsd.org; keydata= xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iziPF0FzrN K1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9yZz7CmgQTFgoAQhYh BDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGABQsJCAcCAyICAQYVCgkICwIE FgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/nvht8kExJ31M+3qpjOqdVypMp+/Ojvh5Z lsk96QEA5HCBkteJcrohwRA7llZvLH3m25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdV AQUBAQdA1Dim8ZWpXRS5i9hb3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2 S7eZrINEWrPNtHEXvliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bA EPqOH+JCIND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8= Subject: Re: FreeBSD CUBIC - Extremely low performance for short RTT setups experiencing congestion Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------pGvOwGkAI8mGFBEmNpOOCLMo" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------pGvOwGkAI8mGFBEmNpOOCLMo Content-Type: multipart/mixed; boundary="------------odwtMt06NqkOZWgnb0llcX7c"; protected-headers="v1" From: "Scheffenegger, Richard" To: biggy pardeshi , freebsd-transport@freebsd.org, cc@freebsd.org Message-ID: Subject: Re: FreeBSD CUBIC - Extremely low performance for short RTT setups experiencing congestion --------------odwtMt06NqkOZWgnb0llcX7c Content-Type: multipart/mixed; boundary="------------QQBuQJ1QRY9fGRyIZlMQV2Qh" --------------QQBuQJ1QRY9fGRyIZlMQV2Qh Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 SGkgQmhhc2thciwNCg0KUGVyc29uYWxseSwgSSdtIG5vdCBhIGJpZyBmYW4gb2YgaGF2aW5n IGluaXRpYWwtUlRUIGRlcGVuZGVudCB0b2dnbGluZyANCm9mIHRoZSBiZWhhdmlvciBvZiBh IFRDUCBzZXNzaW9uLg0KDQpUd28gb3RoZXIgYXBwcm9hY2hlcyBjb21lIHRvIG1pbmQ6DQoN CmEpIFRoZSBzZXBhcmF0ZSBUQ1AgUkFDSyBzdGFjayBhbHJlYWR5IHVzZXMgVENQX0hQVFMg KGhpZ2ggcHJlY2ljaW9uIA0KdGltZXIgc2NoZWR1bGUpIHdoaWNoIHlpZWxkcyBhIGdyYW51 bGFyaXR5IG9mIDEgdXNlYy4gSG93ZXZlciwgSSBiZWxpZXZlIA0KdGhpcyBkb2VzIG5vdCBh Y3R1YWxseSBpbXByb3ZlIHRoZSBncmFudWxhcml0eSBvZiB0aGUgUlRUIG1lYXN1cmVtZW50 cyANCihhbmQgYnkgZXh0ZW50aW9uLCB5aWVsZCBtb3JlIHByZWNpc2lvbiBpbiB0aGUgQ3Vi aWMgY2FsY3VsYXRpb24pLiANCkhvd2V2ZXIsIGl0IG1heSBiZSBhbiBhcHByb2FjaCB0aGF0 IHdoZW4gVENQX0hQVFMgaXMgY29tcGlsZWQgaW4sIHRvIA0KYWxzbyBpbmNyZWFzZSB0aGUg UlRUIG1lYXN1cmVtZW50IGdyYW51bGFyaXR5IHRvIHVzZWMuDQoNCmIpIER1cmluZyBjb25n ZXN0aW9uIGF2b2lkYW5jZSAoQ0EpLCBDdWJpYyBhbHJlYWR5IGhhcyBhIGxpbmVhciBjd25k IA0KZ3Jvd3RoIGZ1bmN0aW9uLCB3aGljaCBpcyBtb2RlbGxlZCB0byBiZSBlcXVhbCB0byBO ZXdSZW5vIENBIGN3bmQgZ3Jvd3RoIA0KLSB3aGVuIHRoZSBjdWJlIGZ1bmN0aW9uIGlzIHZl cnkgY2xvc2UgdG8gaXRzIGluZmxlY3Rpb24gcG9pbnQsIGFuZCB0aHVzIA0KbmVhcmx5IGZs YXQuIEl0IHNob3VsZCBiZSByZWxhdGl2ZWx5IHN0cmFpZ2h0IGZvcndhcmQsIHRvIGFsc28g dXNlIHRoZSANCmFjdHVhbCBOZXdSZW5vIENBIGdyb3d0aCBmdW5jdGlvbiAoMSAvIGNud2Qp IGluIHRoYXQgY29uZGl0aW9uYWwgYnJhbmNoLg0KDQoNCkJ1dCB0aGUgb3B0aW1hbCBmaXgg aXMgdG8gaGVlZCB0aGUgODMxMmJpcyBkb2N1bWVudC4NCmh0dHBzOi8vZGF0YXRyYWNrZXIu aWV0Zi5vcmcvZG9jL2RyYWZ0LWlldGYtdGNwbS1yZmM4MzEyYmlzLw0KDQpUaGUgY3ViaWMg YWxnb3JpdGhtIGluIEZyZWVCU0QgcHJlZGF0ZXMgUkZDODMxMiwgYW5kIDgzMTJiaXMgaXMg aW4gdGhlIA0KZmluYWwgc3RhdGVzIG9mIGdldHRpbmcgcHVibGlzaGVkLg0KDQpXaGlsZSB0 aGUgb3JpZ2luYWwgUkZDIGluZGVlZCBzcGVjaWZpZWQgYSB0aW1lLWJhc2VkIGNhbGN1bGF0 aW9uIGluIHRoZSANCmNvbmNhdmUgcmVnaW9uLCB0aGUgLWJpcyBkb2N1bWVudCB1c2VzIHRo ZSBhY3R1YWwgYWNrZWQgc2VnbWVudHMgKGl0IGlzIA0Kd3JpdHRlbiB3aXRoIGEgc2VnbWVu dC1vcmllbnRlZCBzdGFjayBpbiBtaW5kLCBhY2tlZCBieXRlcyB3aGVuIGN3bmQgaXMgDQpp biBieXRlcyBpcyBlcXVhbGx5IHZhbGlkLg0KDQpVcGRhdGluZyB0aGUgRnJlZUJTRCBDdWJp YyBtb2R1bGUgdXNpbmcgODMxMmJpcyB3b3VsZCBiZSB0aGUgYmVzdCBwYXRoIA0KZm9yd2Fy ZC4NCg0KQmVzdCByZWdhcmRzLA0KICAgUmljaGFyZA0KDQotLS0tLS0tLS0tDQoNCkhpIHRl YW0sDQoNCkkgYW0gYSBuZXR3b3JraW5nIGRhdGFwYXRoIGVuZ2luZWVyIHdvcmtpbmcgYXQg Vk13YXJlLiBJIHByaW1hcmlseSB3b3JrIA0Kb24gVk13YXJlJ3MgRVNYJ3MgVENQL0lQIHN0 YWNrLiBXZSBoYXZlIGJlZW4gcGxheWluZyBhcm91bmQgd2l0aCANCkZyZWVCU0QncyBDVUJJ QyBDb25nZXN0aW9uIENvbnRyb2wgQWxnb3JpdGhtIChDQ0EpIGZvciBhIHdoaWxlLiBJbiBt b3N0IA0Kb2Ygb3VyIHBlcmZvcm1hbmNlIHRlc3RzLCBDVUJJQyBpcyBwZXJmb3JtaW5nIGZp bmUuIEhvd2V2ZXIsIHdlIHNhdyB0aGF0IA0KQ1VCSUMgcmVzdWx0cyBpbiBwZXJmb3JtYW5j ZSBkZWdyYWRhdGlvbiBvZiBhcm91bmQgMTAwLTE1MCUgaW4gc2V0dXBzIA0Kd2hlcmUgdGhl IGZvbGxvd2luZyBjb25kaXRpb25zIGFyZSBtZXQNCjEuCXNlbmRlciBhbmQgcmVjZWl2ZXIg YXJlIGJhY2stdG8tYmFjayBjb25uZWN0ZWQgKHNob3J0IFJUVCkNCjIuCWJhY2stdG8tYmFj ayBjb25uZWN0aW9uL2xpbmsgZXhwZXJpZW5jZXMgY29uZ2VzdGlvbg0KV2Ugc2F3IHRoYXQg dGhlIGNvbmdlc3Rpb24gd2luZG93IChjd25kKSBpcyBub3QgaW5jcmVhc2luZyBmYXN0IGVu b3VnaCANCmluIHRoZSBjYXNlIG9mIENVQklDLCBhZnRlciBleHBlcmllbmNpbmcgYSBjb25n ZXN0aW9uIGV2ZW50LiBPbiB0aGUgc2FtZSANCnNldHVwLCBOZXdSZW5vIGlzIHBlcmZvcm1p bmcgdmVyeSB3ZWxsIGFuZCBwcm92aWRlcyBhIHZlcnkgZmFzdCBpbmNyZWFzZSANCmluIHRo ZSBjd25kIGFmdGVyIGEgY29uZ2VzdGlvbiBldmVudC4NCg0KaHR0cHM6Ly9kYXRhdHJhY2tl ci5pZXRmLm9yZy9kb2MvaHRtbC9kcmFmdC1pZXRmLXRjcG0tY3ViaWMtMDYgLSBDVUJJQyAN CkludGVybmV0LURyYWZ0IChJLUQpDQpBcyBwZXIgdGhlIEktRCwgaXQgaXMgbWVudGlvbmVk IHRoYXQgZm9yIHNob3J0IFJUVCAob3IgZXZlbiBsb3cgQkRQKSANCm5ldHdvcmtzIHN0YW5k YXJkIFRDUCBDQ0FzIHBlcmZvcm0gYmV0dGVyIHRoYW4gQ1VCSUMuIEhlbmNlLCBpbiBzdWNo IA0KY2FzZXMsIHRoZSBUQ1AtZnJpZW5kbHkgd2luZG93IHNpemUgKGVxdWF0aW9uIDQpIHdp bGwgYWx3YXlzIGNvbWUgb3V0IHRvIA0KYmUgZ3JlYXRlciB0aGFuIHRoZSBjdWJpYyB3aW5k b3cgc2l6ZSAoZXF1YXRpb24gMSkuIEhlbmNlLCB3ZSB3aWxsIGZvY3VzIA0Kb3VyIGRpc2N1 c3Npb24gb25seSBvbiB0aGUgVENQLWZyaWVuZGx5IHdpbmRvdyBzaXplIGJlaW5nIGNhbGN1 bGF0ZWQgDQpkdXJpbmcgdGhlIGNvbmdlc3Rpb24gYXZvaWRhbmNlIHBoYXNlLg0KDQpUaGVv cmV0aWNhbGx5LCBDVUJJQyBzaG91bGQgcGVyZm9ybSBhdCBsZWFzdCBhcyBiZXR0ZXIgYXMg c3RhbmRhcmQgVENQIA0KZm9yIHNob3J0IFJUVCBzZXR1cHMuIEhvd2V2ZXIsIGluIHJlYWxp dHksIHRoaXMgaXMgbm90IGJlaW5nIG9ic2VydmVkLiANCldlIGZpbmQgdGhhdCB0aGUgZ3Jh bnVsYXJpdHkgb2YgdGhlIHN5c3RlbSdzIHRpY2tzIHZhbHVlIGlzIGNhdXNpbmcgDQpDVUJJ QyB0byBub3QgZ3JvdyB0aGUgY3duZCBmYXN0ZXIuIEkgd2lsbCBleHBsYWluIHRoaXMgaXNz dWUgd2l0aCBhbiANCmV4YW1wbGUuDQoNCi0tLS0tLS0NCg0KQ29uc2lkZXIgdGhhdCB3ZSBo YXZlIGEgc2hvcnQgUlRUIHNldHVwIHdoZXJlIHRoZSBSVFQgaXMgMC4wNW1zLiBMZXQgdXMg DQphc3N1bWUgdGhhdCB0aGUgc3lzdGVtJ3MgdGljayBmcmVxdWVuY3kgaXMgMTAwMCBIWiwg aS5lLiB0aGUgdGlja3MgdmFsdWUgDQp3aWxsIGluY3JlYXNlIGJ5IDEsIG9uY2UgZXZlcnkg MW1zIChGcmVlQlNEIHByb2JhYmx5IGhhcyBhIDFtcyB0aWNrIHRpbWVyKS4NCg0KTm93IGxl dCB1cyBjb25zaWRlciBhIHBlcmlvZCBvZiAxcy4gRHVyaW5nIHRoaXMgMXMgcGVyaW9kLCB0 aGUgc2VuZGVyIA0Kd2lsbCByZWNlaXZlIGFyb3VuZCAxcy8wLjA1bXMgPSAxMDAwbXMvMC4w NW1zID0gMjAwMDAgQUNLcy4gDQpDb25zZXF1ZW50bHksIHdlIHdpbGwgYmUgY2FsbGluZyB0 aGUgY2NfYWNrX3JlY2VpdmVkKCkgY2FsbGJhY2sgZm9yIGVhY2ggDQpvZiB0aGVzZSBBQ0tz LiBGb3IgTmV3UmVubywgdGhlIGN3bmQgd2lsbCBiZSBpbmNyZWFzZWQgZm9yIGVhY2ggb2Yg dGhvc2UgDQpBQ0tzLCB0aWxsIHRoZSBUQ1AgZmxvdyBiZWNvbWVzIGxpbWl0ZWQgYnkgdGhl IHJlY2VpdmVyJ3Mgd2luZG93LiANCkhvd2V2ZXIsIGluIENVQklDLCBldmVuIHRob3VnaCB0 aGUgY3ViaWNfYWNrX3JlY2VpdmVkKCkgY2FsbGJhY2sgaXMgDQppbnZva2VkIGZvciBlYWNo IG9mIHRob3NlIDIwMDAwIEFDS3MsIHRoZSBjd25kIHdpbGwgbm90IGJlIGluY3JlYXNlZCBm b3IgDQplYWNoIG9mIHRob3NlIEFDS3MuIFRoaXMgaXMgYmVjYXVzZSBvZiB0aGUgInRpY2tz IiB2YWx1ZSB1c2VkIHRvIA0KY2FsY3VsYXRlIHRoZSB0aW1lIGVsYXBzZWQgZnJvbSB0aGUg bGFzdCBjb25nZXN0aW9uIGV2ZW50LiBJbiBGcmVlQlNEIA0KZm9yIGEgcGVyaW9kIG9mIDFt cywgdGhlIHRpY2tzIHZhbHVlIHdpbGwgYmUgdGhlIHNhbWUuIEluIDFtcywgd2Ugd2lsbCAN CnJlY2VpdmUgMW1zLzAuMDVtcyA9IDIwIEFDS3MuIEZvciBhbGwgb2YgdGhlc2UgMjAgQUNL cywgdGlja3MgdmFsdWUgd2lsbCANCmJlIHRoZSBzYW1lLCB0aGUgdGltZSBlbGFwc2VkIGZy b20gdGhlIGxhc3QgY29uZ2VzdGlvbiB3aWxsIGJlIHRoZSBzYW1lLCANCmFuZCBmaW5hbGx5 LCB0aGUgVENQLWZyaWVuZGx5IHdpbmRvdyBlc3RpbWF0ZSB3aWxsIGJlIHRoZSBzYW1lLiBI ZW5jZSwgDQpjd25kIHdpbGwgbm90IGluY3JlYXNlIGZvciB0aGUgZW50aXJlIDFtcyBkdXJh dGlvbi4gSWYgYSBzeXN0ZW0gaGFzIHNvbWUgDQpvdGhlciB0aW1lciBwZXJpb2QgKGZvciBl Zy4gMTBtcyksIHRoZSBjd25kIHdpbGwgc3RheSB0aGUgc2FtZSBmb3IgdGhhdCANCmVudGly ZSBwZXJpb2QuDQoNCk9mIHRoZXNlIDIwMDAwIEFDS3MsICBOZXdSZW5vIHdpbGwgdHJ5IHRv IGluY3JlYXNlIHRoZSBjd25kIHZhbHVlIGZvciANCmFsbW9zdCBldmVyeSBBQ0suIEhvd2V2 ZXIsIENVQklDIHdpbGwgaW5jcmVhc2UgaXQgb25seSBmb3IgMTAwMCBBQ0tzLiANClRoYXQg dG9vIGRpc3RyaWJ1dGVkIG92ZXIgYSBwZXJpb2Qgb2YgMXMuIEkgaG9wZSB0aGUgaXNzdWUg aXMgbm93IGNsZWFyLg0KDQotLS0tDQpXZSB3YW50ZWQgdG8gZGlzY3VzcyB0aGUgc29sdXRp b24gdG8gdGhpcyBpc3N1ZS4gQ3VycmVudGx5LCB3ZSBoYXZlIA0KdGhvdWdodCBvZiBmYWxs aW5nIGJhY2sgdG8gdXNpbmcgdGhlIG5ld3Jlbm8gd2F5IG9mIGRvaW5nIGNvbmdlc3Rpb24g DQphdm9pZGFuY2Ugd2hlbiB3ZSBhcmUgZGVhbGluZyB3aXRoIHNob3J0IFJUVCBjb25uZWN0 aW9ucy4gVGhpcyBtZWFucyANCnRoYXQgaWYgdGhlIG1lYW4gUlRUIHZhbHVlIG1haW50YWlu ZWQgYnkgQ1VCSUMgcHJpdmF0ZSBkYXRhIGlzIGxlc3MgdGhhbiANCm9yIGVxdWFsIHRvIDEs IHdlIHdpbGwgdXNlIE5ld1Jlbm8ncyB3YXkgb2YgZG9pbmcgY29uZ2VzdGlvbiBhdm9pZGFu Y2UgDQp0byBnZXQgYSBUQ1AtZnJpZW5kbHkgd2luZG93IGVzdGltYXRlLiBJbiBvdGhlciBj YXNlcyAobm9uLXNob3J0IFJUVCksIA0Kd2Ugd2lsbCB1c2UgZXF1YXRpb24gNCBvZiBJLUQg dG8gZ2V0IHRoZSBUQ1AtZnJpZW5kbHkgZXN0aW1hdGUuDQoNClRoaXMgd2lsbCByZXNvbHZl IHRoZSBpc3N1ZSB3ZSBhcmUgc2VlaW5nLiBIb3dldmVyLCB0aGUgb25seSBjb25jZXJuIHdl IA0KaGF2ZSBpbiB0aGlzIGFwcHJvYWNoIGlzIHRoYXQgdGhpcyB3aWxsIG1ha2UgQ1VCSUMg UlRULWRlcGVuZGVudCBmb3IgDQpzaG9ydCBSVFQgbmV0d29ya3MuIEFzIHBlciB0aGUgSS1E IEkgc2VlIHRoZSBmb2xsb3dpbmcNCiAgICBBbm90aGVyIG5vdGFibGUgZmVhdHVyZSBvZiBD VUJJQyBpcyB0aGF0IGl0cyB3aW5kb3cgaW5jcmVhc2UgcmF0ZSBpcw0KICAgIG1vc3RseSBp bmRlcGVuZGVudCBvZiBSVFQsIGFuZCBmb2xsb3dzIGEgKGN1YmljKSBmdW5jdGlvbiBvZiB0 aGUNCiAgICBlbGFwc2VkIHRpbWUgZnJvbSB0aGUgYmVnaW5uaW5nIG9mIGNvbmdlc3Rpb24g YXZvaWRhbmNlLg0KU28sIHdlIGFyZSBub3Qgc3VyZSBpZiBsb2dpY2FsbHkgdGhpcyBzb2x1 dGlvbiBpcyByaWdodCBvciBub3QuIEFsc28sIHdlIA0KYXJlIG5vdCBzdXJlIG9mIGFueSBv dGhlciBpbXBsaWNhdGlvbnMgdGhpcyBjaGFuZ2UgbWlnaHQgY2F1c2UgaW4gQ1VCSUMuDQoN Ci0tLS0NCkFkZGluZyBSaWNoYXJkIHRvIHRoZSB0aHJlYWQgZGlyZWN0bHksIGFzIEkgaGF2 ZSBiZWVuIGZvbGxvd2luZyBoaXMgd29yayANCm9uIENVQklDIGZvciBzb21lIHRpbWUuDQoN ClRoYW5rcywNCkJoYXNrYXIgUGFyZGVzaGkgKGJwYXJkZXNoaUB2bXdhcmUuY29tKQ0KVk13 YXJlLCBJbmMuDQo= --------------QQBuQJ1QRY9fGRyIZlMQV2Qh Content-Type: application/pgp-keys; name="OpenPGP_0x17BE5899E0B1439B.asc" Content-Disposition: attachment; filename="OpenPGP_0x17BE5899E0B1439B.asc" Content-Description: OpenPGP public key Content-Transfer-Encoding: quoted-printable -----BEGIN PGP PUBLIC KEY BLOCK----- xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iz iPF0FzrNK1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9y Zz7CmgQTFgoAQhYhBDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGA BQsJCAcCAyICAQYVCgkICwIEFgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/n vht8kExJ31M+3qpjOqdVypMp+/Ojvh5Zlsk96QEA5HCBkteJcrohwRA7llZvLH3m 25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdVAQUBAQdA1Dim8ZWpXRS5i9hb 3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2S7eZrINEWrPNtHEX vliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bAEPqOH+JC IND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8=3D =3DBwxS -----END PGP PUBLIC KEY BLOCK----- --------------QQBuQJ1QRY9fGRyIZlMQV2Qh-- --------------odwtMt06NqkOZWgnb0llcX7c-- --------------pGvOwGkAI8mGFBEmNpOOCLMo Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQQ2S7eZrINEWrPNtHEXvliZ4LFDmwUCZEuKYAUDAAAAAAAKCRAXvliZ4LFDm1mS AQC1rcJ9WJDRej3oAZyuBy3JE2CJLmta+ipeFJY38ogwbwD+PHKrTGlCxqbbwGbTXLiKIr4ZevNB PK1q3i7XJLk9WAk= =GAqk -----END PGP SIGNATURE----- --------------pGvOwGkAI8mGFBEmNpOOCLMo--