Date: Fri, 11 Nov 2022 13:14:56 +0800 From: Zhenlei Huang <zlei.huang@gmail.com> To: "Scheffenegger, Richard" <Richard.Scheffenegger@netapp.com> Cc: Hans Petter Selasky <hps@selasky.org>, Michael Tuexen <michael.tuexen@lurchi.franken.de>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Too aggressive TCP ACKs Message-ID: <63BD8E79-29FA-47E6-BBF7-B755BA060FC7@gmail.com> In-Reply-To: <PH0PR06MB76397B14C0709EFE3E2AB48386019@PH0PR06MB7639.namprd06.prod.outlook.com> References: <75D35F36-7759-4168-ADBA-C2414F5B53BC@gmail.com> <712641B3-5196-40CC-9B64-04637F16F649@lurchi.franken.de> <62A0DD30-B3ED-48BE-9C01-146487599092@gmail.com> <0FED34A9-D093-442A-83B7-08C06D11F8B5@lurchi.franken.de> <330A9146-F7CC-4CAB-9003-2F90B872AC3E@gmail.com> <1ed66217-5463-fd4d-7e7a-58d9981bc44c@selasky.org> <5A501643-1E81-4A8C-8DDC-094371DC03D7@gmail.com> <7EDD65B7-5FCD-42E1-A9E8-AA5139B0A81E@gmail.com> <PH0PR06MB76397B14C0709EFE3E2AB48386019@PH0PR06MB7639.namprd06.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_31C73F77-BFEF-4968-AFBB-394C2135F9F6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 10, 2022, at 8:01 PM, Scheffenegger, Richard = <Richard.Scheffenegger@netapp.com> wrote: >=20 > This is the current draft in this space: > =20 > https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-rate-request/ = <https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-rate-request/>; > =20 > and it has been adopted as WG document at this weeks IETF, from what I = can tell. Thanks for that information ! > =20 > So it has traction =E2=80=93 if you want to give your feedback, please = subscribe to the tcpm mailing list, and discuss your use case and how/if = the approach aligns with this there. Subscribed. > =20 > Richard > =20 > =20 > =20 > From: owner-freebsd-net@freebsd.org = <mailto:owner-freebsd-net@freebsd.org> <owner-freebsd-net@freebsd.org = <mailto:owner-freebsd-net@freebsd.org>> On Behalf Of Zhenlei Huang > Sent: Donnerstag, 10. November 2022 09:07 > To: Hans Petter Selasky <hps@selasky.org <mailto:hps@selasky.org>> > Cc: Michael Tuexen <michael.tuexen@lurchi.franken.de = <mailto:michael.tuexen@lurchi.franken.de>>; freebsd-net@freebsd.org = <mailto:freebsd-net@freebsd.org> > Subject: Re: Too aggressive TCP ACKs > =20 > NetApp Security WARNING: This is an external email. Do not click links = or open attachments unless you recognize the sender and know the content = is safe.=20 >=20 >=20 >=20 > On Nov 9, 2022, at 11:18 AM, Zhenlei Huang <zlei.huang@gmail.com = <mailto:zlei.huang@gmail.com>> wrote: > =20 > =20 > On Oct 22, 2022, at 6:14 PM, Hans Petter Selasky <hps@selasky.org = <mailto:hps@selasky.org>> wrote: > =20 > Hi, >=20 > Some thoughts about this topic. > =20 > Sorry for late response. >=20 >=20 >=20 > Delaying ACKs means loss of performance when using Gigabit TCP = connections in data centers. There it is important to ACK the data as = quick as possible, to avoid running out of TCP window space. Thinking = about TCP connections at 30 GBit/s and above! > =20 > In data centers, the bandwidth is much more and the latency is = extremely low (compared to WAN), sub-milliseconds . > The TCP window space is bandwidth multiply RTT. For a 30 GBit/s = network it is about 750KiB . I think that is trivial for a > datacenter server. >=20 >=20 > 4.2.3.2 in RFC 1122 states: > > in a stream of full-sized segments there SHOULD be an ACK for at = least every second segment=20 > Even if the ACK every tenth segment, the impact of delayed ACKs on TCP = window is not significant ( at most > ten segments not ACKed in TCP send window ). > =20 > Anyway, for datacenter usage the bandwidth is symmetric and the = reverse path ( TX path of receiver ) is sufficient. > Servers can even ACK every segment (no delaying ACK). > =20 >=20 > I think the implementation should be exactly like it is. >=20 > There is a software LRO in FreeBSD to coalesce the ACKs before they = hit the network stack, so there are no real problems there. > =20 > I'm OK with the current implementation. > =20 > I think upper layers (or application) have (business) information to = indicate whether delaying ACKs should be employed. > After googling I found there's a draft [1]. > =20 > [1] Sender Control of Delayed Acknowledgments in TCP: = https://www.ietf.org/archive/id/draft-gomez-tcpm-delack-suppr-reqs-01.xml = <https://www.ietf.org/archive/id/draft-gomez-tcpm-delack-suppr-reqs-01.xml= > > =20 > Found the html / pdf / txt version of the draft RFC. > https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-pull/ = <https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-pull/>; >=20 >=20 >=20 >=20 >=20 > --HPS >=20 >=20 > =20 > Best regards, > Zhenlei --Apple-Mail=_31C73F77-BFEF-4968-AFBB-394C2135F9F6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" = class=3D""><div><blockquote type=3D"cite" class=3D""><div class=3D"">On = Nov 10, 2022, at 8:01 PM, Scheffenegger, Richard <<a = href=3D"mailto:Richard.Scheffenegger@netapp.com" = class=3D"">Richard.Scheffenegger@netapp.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><div style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D"">This is the current draft in this space:<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""> </o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D""><a = href=3D"https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-rate-request= /" style=3D"color: blue; text-decoration: underline;" = class=3D"">https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-rate-requ= est/</a><o:p class=3D""></o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D""><o:p class=3D""> </o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">and it has been = adopted as WG document at this weeks IETF, from what I can = tell.</span></div></div></div></blockquote><div><br = class=3D""></div><div>Thanks for that information !</div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><div style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""></o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D""><o:p class=3D""> </o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">So it has = traction =E2=80=93 if you want to give your feedback, please subscribe = to the tcpm mailing list, and discuss your use case and how/if the = approach aligns with this = there.</span></div></div></div></blockquote><div><br = class=3D""></div><div>Subscribed.</div><div><br = class=3D""></div><blockquote type=3D"cite" class=3D""><div class=3D""><div= class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><div style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""></o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D""><o:p class=3D""> </o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">Richard<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""> </o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D""><o:p class=3D""> </o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span class=3D""><o:p = class=3D""> </o:p></span></div><div class=3D""><div = style=3D"border-style: solid none none; border-top-width: 1pt; = border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;" = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""><b class=3D""><span lang=3D"EN-US" = class=3D"">From:</span></b><span lang=3D"EN-US" class=3D""><span = class=3D"Apple-converted-space"> </span><a = href=3D"mailto:owner-freebsd-net@freebsd.org" style=3D"color: blue; = text-decoration: underline;" = class=3D"">owner-freebsd-net@freebsd.org</a><span = class=3D"Apple-converted-space"> </span><<a = href=3D"mailto:owner-freebsd-net@freebsd.org" style=3D"color: blue; = text-decoration: underline;" = class=3D"">owner-freebsd-net@freebsd.org</a>><span = class=3D"Apple-converted-space"> </span><b class=3D"">On Behalf = Of<span class=3D"Apple-converted-space"> </span></b>Zhenlei = Huang<br class=3D""><b class=3D"">Sent:</b><span = class=3D"Apple-converted-space"> </span>Donnerstag, 10. November = 2022 09:07<br class=3D""><b class=3D"">To:</b><span = class=3D"Apple-converted-space"> </span>Hans Petter Selasky <<a = href=3D"mailto:hps@selasky.org" style=3D"color: blue; text-decoration: = underline;" class=3D"">hps@selasky.org</a>><br class=3D""><b = class=3D"">Cc:</b><span = class=3D"Apple-converted-space"> </span>Michael Tuexen <<a = href=3D"mailto:michael.tuexen@lurchi.franken.de" style=3D"color: blue; = text-decoration: underline;" = class=3D"">michael.tuexen@lurchi.franken.de</a>>;<span = class=3D"Apple-converted-space"> </span><a = href=3D"mailto:freebsd-net@freebsd.org" style=3D"color: blue; = text-decoration: underline;" class=3D"">freebsd-net@freebsd.org</a><br = class=3D""><b class=3D"">Subject:</b><span = class=3D"Apple-converted-space"> </span>Re: Too aggressive TCP = ACKs<o:p class=3D""></o:p></span></div></div></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div><table class=3D"MsoNormalTable" border=3D"1" = cellspacing=3D"0" cellpadding=3D"0" align=3D"left" width=3D"200" = style=3D"width: 150pt; background-color: rgb(0, 103, 197); = background-position: initial initial; background-repeat: initial = initial;"><tbody class=3D""><tr class=3D""><td nowrap=3D"" = style=3D"padding: 6pt;" class=3D""><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><b class=3D""><span = style=3D"font-size: 10pt; font-family: Arial, sans-serif; color: = yellow;" class=3D"">NetApp Security WARNING</span></b><span = style=3D"font-size: 10pt; font-family: Arial, sans-serif; color: = yellow;" class=3D"">: This is an external email. Do not click links or = open attachments unless you recognize the sender and know the content is = safe.<span class=3D"Apple-converted-space"> </span></span><o:p = class=3D""></o:p></div></td></tr></tbody></table><p class=3D"MsoNormal" = style=3D"margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, = sans-serif;"><br class=3D""><br class=3D""><o:p class=3D""></o:p></p><div = class=3D""><div class=3D""><blockquote style=3D"margin-top: 5pt; = margin-bottom: 5pt;" class=3D""><div class=3D""><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">On = Nov 9, 2022, at 11:18 AM, Zhenlei Huang <<a = href=3D"mailto:zlei.huang@gmail.com" style=3D"color: blue; = text-decoration: underline;" class=3D"">zlei.huang@gmail.com</a>> = wrote:<o:p class=3D""></o:p></div></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div><div class=3D""><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><o:p class=3D""> </o:p></div><div = class=3D""><blockquote style=3D"margin-top: 5pt; margin-bottom: 5pt;" = class=3D""><div class=3D""><div style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D"">On Oct 22, 2022, at 6:14 = PM, Hans Petter Selasky <<a href=3D"mailto:hps@selasky.org" = style=3D"color: blue; text-decoration: underline;" = class=3D"">hps@selasky.org</a>> wrote:<o:p = class=3D""></o:p></div></div><div style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div><div class=3D""><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D"">Hi,<br class=3D""><br class=3D"">Some thoughts = about this topic.<o:p class=3D""></o:p></div></div></div></blockquote><div= class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><div class=3D""><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">Sorry = for late response.<o:p class=3D""></o:p></div></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><br = class=3D""><br class=3D""><o:p class=3D""></o:p></div><blockquote = style=3D"margin-top: 5pt; margin-bottom: 5pt;" class=3D""><div = class=3D""><div class=3D""><div style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><br class=3D"">Delaying = ACKs means loss of performance when using Gigabit TCP connections in = data centers. There it is important to ACK the data as quick as = possible, to avoid running out of TCP window space. Thinking about TCP = connections at 30 GBit/s and above!<o:p = class=3D""></o:p></div></div></div></blockquote><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><o:p class=3D""> </o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">In data centers, the bandwidth is much = more and the latency is extremely low (compared to = WAN), sub-milliseconds .<o:p class=3D""></o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">The TCP window space is bandwidth = multiply RTT. For a 30 GBit/s network it is about 750KiB . I think that = is trivial for a<o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D"">datacenter server.<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><br = class=3D""><br class=3D""><o:p class=3D""></o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">4.2.3.2 in RFC 1122 states:<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">> in a stream of full-sized segments there = SHOULD be an ACK for at least every second segment <o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">Even if = the ACK every tenth segment, the impact of delayed ACKs on TCP = window is not significant ( at most<o:p class=3D""></o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""> ten segments not ACKed in = TCP send window ).<o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><o:p class=3D""> </o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">Anyway, for datacenter usage = the bandwidth is symmetric and the reverse path ( TX path of = receiver ) is sufficient.<o:p class=3D""></o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">Servers can even ACK every segment = (no delaying ACK).<o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><blockquote style=3D"margin-top: 5pt; = margin-bottom: 5pt;" class=3D""><div class=3D""><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><br class=3D"">I think the implementation should = be exactly like it is.<br class=3D""><br class=3D"">There is a software = LRO in FreeBSD to coalesce the ACKs before they hit the network stack, = so there are no real problems there.<o:p = class=3D""></o:p></div></div></div></blockquote><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><o:p class=3D""> </o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">I'm OK with the current = implementation.<o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><o:p class=3D""> </o:p></div></div><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">I think upper layers (or application) = have (business) information to indicate whether delaying ACKs should be = employed.<o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D"">After googling I found there's a draft [1].<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><div class=3D""><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">[1] Sender Control of Delayed Acknowledgments in = TCP:<span class=3D"Apple-converted-space"> </span><a = href=3D"https://www.ietf.org/archive/id/draft-gomez-tcpm-delack-suppr-reqs= -01.xml" style=3D"color: blue; text-decoration: underline;" = class=3D"">https://www.ietf.org/archive/id/draft-gomez-tcpm-delack-suppr-r= eqs-01.xml</a><o:p = class=3D""></o:p></div></div></div></div></div></blockquote><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><div class=3D""><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">Found = the html / pdf / txt version of the draft RFC.<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><a = href=3D"https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-pull/" = style=3D"color: blue; text-decoration: underline;" = class=3D"">https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-pull/</a>= <o:p class=3D""></o:p></div></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><br class=3D""><br = class=3D""><o:p class=3D""></o:p></div><blockquote style=3D"margin-top: = 5pt; margin-bottom: 5pt;" class=3D""><div class=3D""><div class=3D""><div = class=3D""><div style=3D"margin: 0cm; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""><br class=3D""><br class=3D""><o:p = class=3D""></o:p></div><blockquote style=3D"margin-top: 5pt; = margin-bottom: 5pt;" class=3D""><div class=3D""><div class=3D""><p = class=3D"MsoNormal" style=3D"margin: 0cm 0cm 12pt; font-size: 11pt; = font-family: Calibri, sans-serif;"><br class=3D"">--HPS<br class=3D""><br = class=3D""><o:p class=3D""></o:p></p></div></div></blockquote></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><o:p class=3D""> </o:p></div><div = class=3D""><div class=3D""><div style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D"">Best regards,<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">Zhenlei</div></div></div></div></div></blockquote></div></div><= /div></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_31C73F77-BFEF-4968-AFBB-394C2135F9F6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?63BD8E79-29FA-47E6-BBF7-B755BA060FC7>