From owner-svn-src-stable@freebsd.org Mon Oct 10 18:34:09 2016 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 365A8C0CAAC; Mon, 10 Oct 2016 18:34:09 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-qt0-f177.google.com (mail-qt0-f177.google.com [209.85.216.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DE45CEB9; Mon, 10 Oct 2016 18:34:08 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-qt0-f177.google.com with SMTP id q7so63503723qtq.1; Mon, 10 Oct 2016 11:34:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=ca1aqN8BPnAqQEme5Q0GzkPFrbVT/qf31+HfWHtiVwo=; b=Lx3PK3oBUr8H8w+DMUVwb2OivLUC5M4Mjid2PHrLXYNQ+GRNQgnmFDmqiOKdk+nKS/ Llsz7DMjI+hYTCdZRv7ywlKGAt7eIFRngFHfLoBBl0zFswcrBD5YrE9fdv+8iO4PfrzO +tHnzm+h/b8TPZFsx6nmKEPpwEabE5hbmQSiPeeDaIZ6vTuDTExpjH7LuCA4ynD3873F WvUTNvObbtvoAcdyVw6axMHsDuzElA0EqhYu+MObUmkI2gLkSaDc83np+vdw283oqqaf ouygKscJFoyWk0kc48KCn8otXme+1d4rcWQeTlJd9KDlC2Y/jwOyGNApDegWszLC/msy ABvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=ca1aqN8BPnAqQEme5Q0GzkPFrbVT/qf31+HfWHtiVwo=; b=Hem3lgaWh8+rTMLFZIK42pqzChairOwgOx37S7QmbiRnZurF0M9UsTtGhuDkzHXRqF Ilg8OWZzHhz+uOi2XckDf2FnCKDQVRQetyo4dXX5+cM03H8AdZNjaww2dfuUWyZU7hZd GEU08TDkM80e0hMMA9NC9xUJX0gv17G7lB8mnDl2enmmtBeJUwzP++Sc+c4fD1yqeRHL qTuFw4wdeGm8O9Mob2B+sscLj4KXZ2kl8pNFSvlG1YUGnFf9YEb73e+z7DS5l3LHGTvv X4GIoNHEcNGm79xN7V2HyzewMDif75P9Yvu/1eKZ8u3u5pag7eCtH9/bbt7XP2NfiG8N xf1Q== X-Gm-Message-State: AA6/9RmP3ilLNkxIECfuQ79tEbg7ecJM2RNqe2MrmcRq7AEPeyCwbdixLYSGDfonqNAg1Q== X-Received: by 10.200.42.130 with SMTP id b2mr34048357qta.102.1476124382114; Mon, 10 Oct 2016 11:33:02 -0700 (PDT) Received: from [10.192.166.0] (stargate.chelsio.com. [12.32.117.8]) by smtp.googlemail.com with ESMTPSA id j2sm1126049qtb.46.2016.10.10.11.33.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Oct 2016 11:33:01 -0700 (PDT) Subject: Re: svn commit: r306661 - in stable/11/sys/dev/cxgbe: . tom To: Slawa Olhovchenkov , John Baldwin References: <201610032315.u93NFiHE057529@repo.freebsd.org> <20161010110901.GH6177@zxy.spb.ru> <1660024.uzJn2AtV1k@ralph.baldwin.cx> <20161010182821.GZ54003@zxy.spb.ru> Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org From: Navdeep Parhar Message-ID: Date: Mon, 10 Oct 2016 11:32:59 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161010182821.GZ54003@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2016 18:34:09 -0000 On 10/10/2016 11:28, Slawa Olhovchenkov wrote: > On Mon, Oct 10, 2016 at 10:46:27AM -0700, John Baldwin wrote: > >> On Monday, October 10, 2016 02:09:01 PM Slawa Olhovchenkov wrote: >>> On Mon, Oct 03, 2016 at 11:15:44PM +0000, John Baldwin wrote: >>> >>>> Author: jhb >>>> Date: Mon Oct 3 23:15:44 2016 >>>> New Revision: 306661 >>>> URL: https://svnweb.freebsd.org/changeset/base/306661 >>>> >>>> Log: >>>> MFC 303405: Add support for zero-copy aio_write() on TOE sockets. >>>> >>>> AIO write requests for a TOE socket on a Chelsio T4+ adapter can now >>>> DMA directly from the user-supplied buffer. This is implemented by >>>> wiring the pages backing the user-supplied buffer and queueing special >>>> mbufs backed by raw VM pages to the socket buffer. The TOE code >>>> recognizes these special mbufs and builds a sglist from the VM page >>>> array associated with the mbuf when queueing a work request to the TOE. >>>> >>>> Because these mbufs do not have an associated virtual address, m_data >>>> is not valid. Thus, the AIO handler does not invoke sosend() directly >>>> for these mbufs but instead inlines portions of sosend_generic() and >>>> tcp_usr_send(). >>>> >>>> An aiotx_buffer structure is used to describe the user buffer (e.g. >>>> it holds the array of VM pages and a reference to the AIO job). The >>>> special mbufs reference this structure via m_ext. Note that a single >>>> job might be split across multiple mbufs (e.g. if it is larger than >>>> the socket buffer size). The 'ext_arg2' member of each mbuf gives an >>>> offset relative to the backing aiotx_buffer. The AIO job associated >>>> with an aiotx_buffer structure is completed when the last reference to >>>> the structure is released. >>>> >>>> Zero-copy aio_write()'s for connections associated with a given >>>> adapter can be enabled/disabled at runtime via the >>>> 'dev.t[45]nex.N.toe.tx_zcopy' sysctl. >>>> >>>> Sponsored by: Chelsio Communications >>> >>> Do you have any public available application patches for support this? >>> May be nginx? >> >> Applications need to use aio_read(), ideally with at least 2 buffers (so >> queue two reads, then when a read completes, consume the data and do the >> next read). I'm not sure nginx will find this but so useful as web servers >> tend to send a lot more data than they receive. The only software I have >> patched explicitly for this is netperf. > > Hm, this is like only aio_read() on sokets give performance boost, not > aio_write()? > Both. tx_zcopy is the knob for aio_write and ddp is the knob for aio_read. ddp will probably get renamed to rx_zcopy (while still working with the 'ddp' name for backward compat).