Date: Sun, 17 Mar 2024 11:40:54 -0400 From: "Drew Gallatin" <gallatin@freebsd.org> To: "Nuno Teixeira" <eduardo@freebsd.org>, tuexen@freebsd.org Cc: garyj@gmx.de, current@freebsd.org, net@freebsd.org, "Randall Stewart" <rrs@freebsd.org> Subject: Re: Request for Testing: TCP RACK Message-ID: <fe9097f3-532d-4569-880b-a8a672ad8384@app.fastmail.com> In-Reply-To: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> References: <42C327BD-6CE4-43AA-A1AE-3BEC08D623DB@freebsd.org> <CAFDf7U%2BU3QpgsG-hY0u3tC_X9WKSRzuAM7Nfhzu-%2BqPBbs1nng@mail.gmail.com> <486915F0-456B-4B09-A8BC-93BBA79C4CA1@freebsd.org> <20240313080624.6c73908c@ernst.home> <508E3B47-8E1B-469F-97B1-2171A3098888@freebsd.org> <86a5n1i0xg.fsf@ltc.des.dev> <AE656D0C-FA5B-4194-8B57-4140CD57ABF1@freebsd.org> <CAFDf7UJRitdHz5QebNH9A4mNo0pKCBRfZCfmFgjNVSdCr_JXrw@mail.gmail.com> <78D1FF09-71A3-4486-B934-D8332F54B237@freebsd.org> <20240316104053.20bef8c2@ernst.home> <CAFDf7UKT=FYsaXRacwsnjDkYUjT4_1QnHynWAc67yxQS8XA0Yw@mail.gmail.com> <20240316115128.33d11f7b@ernst.home> <CAFDf7U%2BBgyw2e1hJySt-4GTX3=qn6LX7B2Ebzcmvr%2BD6MJqfpg@mail.gmail.com> <7367F29A-D52B-4828-B79A-AA2667E81E7D@freebsd.org> <CAFDf7UKRLpBt%2B11GmWO5p0b-L9ZMve6MyCxRUuVm32EBZY0MWA@mail.gmail.com> <BB57B7CC-1506-4B4C-8A04-26F67DE30EDA@freebsd.org> <CAFDf7ULtN9owoH-ns2OfR6ZhypNGxuNzkQbb2P9zR8ceFgaj5A@mail.gmail.com> <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <CAFDf7U%2BAjfeY%2Bqjq%2B-R71w5i1pRoxQdOmqJ9w4s1U13AA8-duA@mail.gmail.com> <C5D50314-4B0C-42F6-AA67-B5A32A4BA335@freebsd.org> <CAFDf7UKL6vtKo1Mn9Vw_5OD9Xubuw%2BdgS83WKwsiTUaXHs8D6Q@mail.gmail.com> <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--e58bff945182484499f23ee3bb676b2a
Content-Type: multipart/alternative;
boundary=43607613842b4113a49b4bd429aef6fd
--43607613842b4113a49b4bd429aef6fd
Content-Type: text/plain
Resending with the patch as an attachment.
Drew
On Sun, Mar 17, 2024, at 11:39 AM, Drew Gallatin wrote:
> I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct?
>
> If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like:
>
> diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c
> index e9a16cd0b36e..54b540c97123 100644
> --- a/sys/kern/subr_trap.c
> +++ b/sys/kern/subr_trap.c
> @@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame)
> * Software Timer Support for Network Processing"
> * by Mohit Aron and Peter Druschel.
> */
> - tcp_hpts_softclock();
> + /*tcp_hpts_softclock();*/
> /*
> * Let the scheduler adjust our priority etc.
> */
>
>
> If that fixes it, I suspect we should either make this hook optional for casual users of tcp_hpts(), or add some kind of "last called" timestamp to prevent it being called over and over and over on workloads which are syscall heavy.
>
> Note that for non-casual users of hpts (like Netflix, with hundreds of thousands of TCP connections managed by hpts), this call is a huge win, so I think we'd prefer that it remain in some form.
>
> Drew
>
--43607613842b4113a49b4bd429aef6fd
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html><html><head><title></title><style type=3D"text/css">
p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div>Resending =
with the patch as an attachment.<br></div><div><br></div><div>Drew<br></=
div><div><br></div><div>On Sun, Mar 17, 2024, at 11:39 AM, Drew Gallatin=
wrote:<br></div><blockquote type=3D"cite" id=3D"qt" style=3D""><div>I d=
on't have the full context, but it seems like the complaint is a perform=
ance regression in bonnie++ and perhaps other things when tcp_hpts is lo=
aded, even when it is not used. Is that correct?<br></div><div><br=
></div><div>If so, I suspect its because we drive the tcp_hpts_softclock=
() routine from userret(), in order to avoid tons of timer interrupts an=
d context switches. To test this theory, you could apply a p=
atch like:<br></div><div><br></div><div>diff --git a/sys/kern/subr_trap.=
c b/sys/kern/subr_trap.c<br></div><div>index e9a16cd0b36e..54b540c97123 =
100644<br></div><div>--- a/sys/kern/subr_trap.c<br></div><div>+++ b/sys/=
kern/subr_trap.c<br></div><div>@@ -138,7 +138,7 @@ userret(struct thread=
*td, struct trapframe *frame)<br></div><div> &nb=
sp; * Software Timer Support for Network Processing"<b=
r></div><div> * by Mohit=
Aron and Peter Druschel.<br></div><div> &n=
bsp; */<br></div><div>- =
tcp_hpts_softclock();<br></div><div>+  =
; /*tcp_hpts_softclock();*/<br></div><div> =
/*<br></div><div> =
* Let the scheduler adjust our priority etc.<br></div><div> =
*/<br></div><div><br></div><d=
iv><br></div><div>If that fixes it, I suspect we should either make this=
hook optional for casual users of tcp_hpts(), or add some kind of "last=
called" timestamp to prevent it being called over and over and over on =
workloads which are syscall heavy.<br></div><div><br></div><div>Note tha=
t for non-casual users of hpts (like Netflix, with hundreds of thousands=
of TCP connections managed by hpts), this call is a huge win, so I thin=
k we'd prefer that it remain in some form.<br></div><div><br></div><div>=
Drew<br></div><div><br></div></blockquote></body></html>
--43607613842b4113a49b4bd429aef6fd--
--e58bff945182484499f23ee3bb676b2a
Content-Disposition: attachment; filename="nerf_tcp_hpts_softclock.diff"
Content-Type: application/octet-stream; name="nerf_tcp_hpts_softclock.diff"
Content-Transfer-Encoding: BASE64
ZGlmZiAtLWdpdCBhL3N5cy9rZXJuL3N1YnJfdHJhcC5jIGIvc3lzL2tlcm4vc3Vicl90cmFw
LmMKaW5kZXggZTlhMTZjZDBiMzZlLi41NGI1NDBjOTcxMjMgMTAwNjQ0Ci0tLSBhL3N5cy9r
ZXJuL3N1YnJfdHJhcC5jCisrKyBiL3N5cy9rZXJuL3N1YnJfdHJhcC5jCkBAIC0xMzgsNyAr
MTM4LDcgQEAgdXNlcnJldChzdHJ1Y3QgdGhyZWFkICp0ZCwgc3RydWN0IHRyYXBmcmFtZSAq
ZnJhbWUpCiAJICogU29mdHdhcmUgVGltZXIgU3VwcG9ydCBmb3IgTmV0d29yayBQcm9jZXNz
aW5nIgogCSAqIGJ5IE1vaGl0IEFyb24gYW5kIFBldGVyIERydXNjaGVsLgogCSAqLwotCXRj
cF9ocHRzX3NvZnRjbG9jaygpOworCS8qdGNwX2hwdHNfc29mdGNsb2NrKCk7Ki8KIAkvKgog
CSAqIExldCB0aGUgc2NoZWR1bGVyIGFkanVzdCBvdXIgcHJpb3JpdHkgZXRjLgogCSAqLwo=
--e58bff945182484499f23ee3bb676b2a--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fe9097f3-532d-4569-880b-a8a672ad8384>