From owner-dev-commits-src-all@freebsd.org Wed Sep 15 19:20:53 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 530BF66CC09; Wed, 15 Sep 2021 19:20:53 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4H8qnK11XSz3C6r; Wed, 15 Sep 2021 19:20:53 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pl1-x62f.google.com with SMTP id d18so2230384pll.11; Wed, 15 Sep 2021 12:20:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=NRaYvUmtdIz9Cg5c1vWT0Sj1Hh9plnsyXaItoGi+kXA=; b=dtduJffXb9IBko8BuCejD3aQm8xV+9WMXECYtFpruHQxLpHIVv/6Y1GhxOeBwzPx/m Tvn3otepGUl2/8jYuT4x+dVrDBjISBmdfkye8+kZsnBnh5G+zkP03407JdVfvG2jUhT/ lV03BX1Qw1JbhqhnkovRPOu4omdI57jqGHj23CFByqzeeSpRf0TGG9/I1z/dS6NDHAvo KqldvsCLwnUvm6zxfPK6HL9M08YXfYhAFJW9Lki82KSqwuExMqIPk3MiG+o3xqezUHin r2voh+CsSD3GVRheTMHFVXfYziamqwECiwVogI6yc6uXsOaZVXDRUWQSJ4iTdETxv6TJ MdUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:subject:to:cc:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=NRaYvUmtdIz9Cg5c1vWT0Sj1Hh9plnsyXaItoGi+kXA=; b=7BUDmKoTnUUUt40PGV7nSxvzpuIl/OJLQ7mv1sVWwM0L5CuQmbILXFNKJEaqe5Emej 6/Qdco3C9loIG3KYR8+O/0bC36I1WQ/DZfi/4QDSwwNumRq1vXoqtBSYYg+7felDtBg6 D2U67r9Q9L3c/gAljOUUyqlyW4yZzU7ocgfrP0zPZk4Vt6uLZrMVzIuP1Q9bCv7q/hMs 9W8KptflV7KVrmxBsiZj3j2pGLu5ZtuIqDPQlCFDnkrWMybMqUK38XjbOEKwePw2Dw5Y rwtCLxmcN4mUtyfFdk67HCNAwAgplnX+vqeaAPztMIf6MHX+WEAkxN1Kg9NutzYhENu3 GaJw== X-Gm-Message-State: AOAM533d2IZKt5c9TUAfXz07BDJawKuaQKytXkYgrC1MddNSiNOmPtkO UK4xPbexdI/VA+XWQuBVyDNL2vRQGyM= X-Google-Smtp-Source: ABdhPJyfqlvrimhuON4TVP7VY6iIZCiHwtLTkTVpXh7A3UChMLE+E/2+v667daKF8bgpL0yetURyLQ== X-Received: by 2002:a17:90b:80a:: with SMTP id bk10mr1415805pjb.127.1631733651792; Wed, 15 Sep 2021 12:20:51 -0700 (PDT) Received: from [10.192.166.0] (stargate.chelsio.com. [12.32.117.8]) by smtp.googlemail.com with ESMTPSA id c23sm645447pgb.74.2021.09.15.12.20.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 15 Sep 2021 12:20:51 -0700 (PDT) Sender: Navdeep Parhar Subject: Re: git: 1ecbc1d8e9d3 - main - cxgbe tom: Don't queue AIO requests on listen sockets. To: Alan Somers , John Baldwin Cc: src-committers , dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org References: <202109142046.18EKkfEN043871@gitrepo.freebsd.org> <0cec1c04-0b42-6297-37c5-43d1dcb0f8d5@FreeBSD.org> From: Navdeep Parhar Message-ID: Date: Wed, 15 Sep 2021 12:20:45 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4H8qnK11XSz3C6r X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Sep 2021 19:20:53 -0000 On 9/15/21 11:05 AM, Alan Somers wrote: > On Wed, Sep 15, 2021 at 11:32 AM John Baldwin > wrote: >=20 > On 9/15/21 8:47 AM, Alan Somers wrote: > > On Wed, Sep 15, 2021 at 9:21 AM John Baldwin > wrote: > > > >> On 9/14/21 1:53 PM, Alan Somers wrote: > >>> On Tue, Sep 14, 2021 at 2:46 PM John Baldwin > wrote: > >>> > >>>> The branch main has been updated by jhb: > >>>> > >>>> URL: > >>>> > >> > https://cgit.FreeBSD.org/src/commit/?id=3D1ecbc1d8e9d3fbcd8e68fc68f= 0a32944a12ddb1e > > >>>> > >>>> commit 1ecbc1d8e9d3fbcd8e68fc68f0a32944a12ddb1e > >>>> Author:=C2=A0 =C2=A0 =C2=A0John Baldwin > >>>> AuthorDate: 2021-09-14 20:46:14 +0000 > >>>> Commit:=C2=A0 =C2=A0 =C2=A0John Baldwin > >>>> CommitDate: 2021-09-14 20:46:14 +0000 > >>>> > >>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0cxgbe tom: Don't queue AIO requests= on listen sockets. > >>>> > >>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0This is similar to the fixes in 141= fe2dceeae.=C2=A0 One > difference is > >> that > >>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0TOE sockets do not change states (l= isten vs non-listen) once > >> created, > >>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0so no lock is needed for SOLISTENIN= G(). > >>>> > >>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0Sponsored by:=C2=A0 =C2=A0Chelsio C= ommunications > >>>> > >>> > >>> I've always wondered: what's the point to using AIO with > sockets?=C2=A0 Can't > >>> everything socket-related be done better with non-blocking > read/write and > >>> kqueue? > >> > >> Zero-copy operation with TOE is why TOE uses AIO.=C2=A0 Zero-co= py of user > >> buffers > >> can't really work with the non-AIO APIs because the user buffer= > is free to > >> be reused immediately after write(2) (and on the read side you > don't know > >> the buffer in advance to allow the NIC to write directly into > the use > >> buffer). > >> > >> In theory we could support zero-copy using mb_ext_pgs for > aio_write() for > >> the non-TOE case similar to what sendfile() does. > >> > >> -- > >> John Baldwin > >> > > > > Interesting.=C2=A0 Do you know of any common applications that i= nclude > this > > optimization?=C2=A0 I've been working on the AIO ecosystem for R= ust.=20 > It would > > be good to ensure that this use case works, especially if > zero-copy ever > > works for non-TOE. >=20 > I do not, and I rely on patches I merged upstream to netperf (-a an= d > -A flags) > to test it.=C2=A0 I believe there might be some proprietary bits in= some > FreeBSD > downstreams that might make use of this. >=20 > --=20 > John Baldwin >=20 >=20 > Do you mean these -a and -A flags, or am I looking in the wrong place? > =C2=A0 =C2=A0 =C2=A0=C2=A0 -a sizespec > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Alter the send and re= ceive buffer alignments on the local > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 system.=C2=A0 This de= faults to 8 bytes. >=20 > =C2=A0 =C2=A0 =C2=A0 =C2=A0-A sizespec > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 As -a, but for the re= mote system. The aio options are available for the TCP tests but you might have to=20 build netperf with a non-default option (that's what I had to do a long=20 time back and I haven't checked if it's still needed). # netperf -t TCP_STREAM -- -h =2E.. TCP/UDP BSD Sockets Test Options: -a Use aio_write(2) -A Use aio_read(2) =2E.. # pkg options netperf netperf - EXAMPLES: off netperf - EXS: on netperf - HISTOGRAM: on netperf - INFO: off netperf - OMNI: off netperf - SCTP: on netperf - SOCKETS: on Regards, Navdeep