From nobody Mon Mar 18 19:41:47 2024 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tz4xM0w25z5DrFT; Mon, 18 Mar 2024 19:41:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tz4xL64Xyz47lj; Mon, 18 Mar 2024 19:41:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 42IJflGB000230; Mon, 18 Mar 2024 21:41:50 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 42IJflGB000230 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 42IJflUr000229; Mon, 18 Mar 2024 21:41:47 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 18 Mar 2024 21:41:47 +0200 From: Konstantin Belousov To: Drew Gallatin Cc: Mike Karels , tuexen , Nuno Teixeira , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Subject: Re: Request for Testing: TCP RACK Message-ID: References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4Tz4xL64Xyz47lj On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > I got the idea from > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > The gist is that the TCP pacing stuff needs to run frequently, and > rather than run it out of a clock interrupt, its more efficient to run > it out of a system call context at just the point where we return to > userspace and the cache is trashed anyway. The current implementation > is fine for our workload, but probably not idea for a generic system. > Especially one where something is banging on system calls. > > Ast's could be the right tool for this, but I'm super unfamiliar with > them, and I can't find any docs on them. > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > what's happening here? This call would need some AST number added, and then it registers the ast to run on next return to userspace, for the current thread. Is it enough? > > Drew > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > > > >> > > > >> Hello all! > > > >> > > > >> It works just fine! > > > >> System performance is OK. > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > >> > > > >> --- > > > >> net.inet.tcp.functions_available: > > > >> Stack D Alias PCB count > > > >> freebsd freebsd 0 > > > >> rack * rack 38 > > > >> --- > > > >> > > > >> It would be so nice that we can have a sysctl tunnable for this patch > > > >> so we could do more tests without recompiling kernel. > > > > Thanks for testing! > > > > > > > > @gallatin: can you come up with a patch that is acceptable for Netflix > > > > and allows to mitigate the performance regression. > > > > > > Ideally, tcphpts could enable this automatically when it starts to be > > > used (enough?), but a sysctl could select auto/on/off. > > There is already a well-known mechanism to request execution of the > > specific function on return to userspace, namely AST. The difference > > with the current hack is that the execution is requested for one callback > > in the context of the specific thread. > > > > Still, it might be worth a try to use it; what is the reason to hit a thread > > that does not do networking, with TCP processing? > > > > > > > > Mike > > > > > > > Best regards > > > > Michael > > > >> > > > >> Thanks all! > > > >> Really happy here :) > > > >> > > > >> Cheers, > > > >> > > > >> Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) 20:26): > > > >>> > > > >>> Hello, > > > >>> > > > >>>> I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct? > > > >>>> > > > >>>> If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like: > > > >>> > > > >>> It's affecting overall system performance, bonnie was just a way to > > > >>> get some numbers to compare. > > > >>> > > > >>> Tomorrow I will test patch. > > > >>> > > > >>> Thanks! > > > >>> > > > >>> -- > > > >>> Nuno Teixeira > > > >>> FreeBSD Committer (ports) > > > >> > > > >> > > > >> > > > >> -- > > > >> Nuno Teixeira > > > >> FreeBSD Committer (ports) > > > > >