Date: Sun, 17 Mar 2024 11:39:06 -0400 From: "Drew Gallatin" <gallatin@freebsd.org> To: "Nuno Teixeira" <eduardo@freebsd.org>, tuexen@freebsd.org Cc: garyj@gmx.de, current@freebsd.org, net@freebsd.org, "Randall Stewart" <rrs@freebsd.org> Subject: Re: Request for Testing: TCP RACK Message-ID: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> In-Reply-To: <CAFDf7UKL6vtKo1Mn9Vw_5OD9Xubuw%2BdgS83WKwsiTUaXHs8D6Q@mail.gmail.com> References: <42C327BD-6CE4-43AA-A1AE-3BEC08D623DB@freebsd.org> <CAFDf7U%2BU3QpgsG-hY0u3tC_X9WKSRzuAM7Nfhzu-%2BqPBbs1nng@mail.gmail.com> <486915F0-456B-4B09-A8BC-93BBA79C4CA1@freebsd.org> <20240313080624.6c73908c@ernst.home> <508E3B47-8E1B-469F-97B1-2171A3098888@freebsd.org> <86a5n1i0xg.fsf@ltc.des.dev> <AE656D0C-FA5B-4194-8B57-4140CD57ABF1@freebsd.org> <CAFDf7UJRitdHz5QebNH9A4mNo0pKCBRfZCfmFgjNVSdCr_JXrw@mail.gmail.com> <78D1FF09-71A3-4486-B934-D8332F54B237@freebsd.org> <20240316104053.20bef8c2@ernst.home> <CAFDf7UKT=FYsaXRacwsnjDkYUjT4_1QnHynWAc67yxQS8XA0Yw@mail.gmail.com> <20240316115128.33d11f7b@ernst.home> <CAFDf7U%2BBgyw2e1hJySt-4GTX3=qn6LX7B2Ebzcmvr%2BD6MJqfpg@mail.gmail.com> <7367F29A-D52B-4828-B79A-AA2667E81E7D@freebsd.org> <CAFDf7UKRLpBt%2B11GmWO5p0b-L9ZMve6MyCxRUuVm32EBZY0MWA@mail.gmail.com> <BB57B7CC-1506-4B4C-8A04-26F67DE30EDA@freebsd.org> <CAFDf7ULtN9owoH-ns2OfR6ZhypNGxuNzkQbb2P9zR8ceFgaj5A@mail.gmail.com> <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <CAFDf7U%2BAjfeY%2Bqjq%2B-R71w5i1pRoxQdOmqJ9w4s1U13AA8-duA@mail.gmail.com> <C5D50314-4B0C-42F6-AA67-B5A32A4BA335@freebsd.org> <CAFDf7UKL6vtKo1Mn9Vw_5OD9Xubuw%2BdgS83WKwsiTUaXHs8D6Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--49acd8c4f2f346cdb093c5dcbb4bfe48
Content-Type: text/plain
I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct?
If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like:
diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c
index e9a16cd0b36e..54b540c97123 100644
--- a/sys/kern/subr_trap.c
+++ b/sys/kern/subr_trap.c
@@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame)
* Software Timer Support for Network Processing"
* by Mohit Aron and Peter Druschel.
*/
- tcp_hpts_softclock();
+ /*tcp_hpts_softclock();*/
/*
* Let the scheduler adjust our priority etc.
*/
If that fixes it, I suspect we should either make this hook optional for casual users of tcp_hpts(), or add some kind of "last called" timestamp to prevent it being called over and over and over on workloads which are syscall heavy.
Note that for non-casual users of hpts (like Netflix, with hundreds of thousands of TCP connections managed by hpts), this call is a huge win, so I think we'd prefer that it remain in some form.
Drew
--49acd8c4f2f346cdb093c5dcbb4bfe48
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}</style></head><body><div>I don't have th=
e full context, but it seems like the complaint is a performance regress=
ion in bonnie++ and perhaps other things when tcp_hpts is loaded, even w=
hen it is not used. Is that correct?<br></div><div><br></div><div>=
If so, I suspect its because we drive the tcp_hpts_softclock() routine f=
rom userret(), in order to avoid tons of timer interrupts and context sw=
itches. To test this theory, you could apply a patch like:<b=
r></div><div><br></div><div>diff --git a/sys/kern/subr_trap.c b/sys/kern=
/subr_trap.c<br></div><div>index e9a16cd0b36e..54b540c97123 100644<br></=
div><div>--- a/sys/kern/subr_trap.c<br></div><div>+++ b/sys/kern/subr_tr=
ap.c<br></div><div>@@ -138,7 +138,7 @@ userret(struct thread *td, struct=
trapframe *frame)<br></div><div> &nb=
sp; * Software Timer Support for Network Processing"<br></div><div=
> * by Mohit Aron and Pe=
ter Druschel.<br></div><div> &n=
bsp; */<br></div><div>- tcp_hpts_sof=
tclock();<br></div><div>+ /*tcp_hpts=
_softclock();*/<br></div><div> =
/*<br></div><div> * Let=
the scheduler adjust our priority etc.<br></div><div> =
*/<br></div><div><br></div><div><br></div=
><div>If that fixes it, I suspect we should either make this hook option=
al for casual users of tcp_hpts(), or add some kind of "last called" tim=
estamp to prevent it being called over and over and over on workloads wh=
ich are syscall heavy.<br></div><div><br></div><div>Note that for non-ca=
sual users of hpts (like Netflix, with hundreds of thousands of TCP conn=
ections managed by hpts), this call is a huge win, so I think we'd prefe=
r that it remain in some form.<br></div><div><br></div><div>Drew<br></di=
v><div><br></div></body></html>
--49acd8c4f2f346cdb093c5dcbb4bfe48--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6e795e9c-8de4-4e02-9a96-8fabfaa4e66f>