From owner-freebsd-drivers@freebsd.org Mon Dec 17 11:24:47 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A45D133B369; Mon, 17 Dec 2018 11:24:47 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E7F58FFD3; Mon, 17 Dec 2018 11:24:44 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wm1-x330.google.com with SMTP id n190so12000299wmd.0; Mon, 17 Dec 2018 03:24:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=m3FCx/ilHbCbH1PQeEV/LpvDplBkw2pi8TS/G2qT+n8=; b=luoc9Ie6ouvWM2LnX6AsQmojsvBzpoowC7wSoFIgsXx7tN61tkPQsCGbiCfjdVquCb 5clW1guLol7+oqPdQjjKXJUUBoQlcnP0VVepxUBiKCsZ3lULqMoeMcKArb0Qyi4x/bV6 Kj0kABSvEM6wOcWnr8IrXC8YFyjOnOq+ybIe03/eDfA+kZunk65Kpd+UH/LI7N3k7krd BRrVeSa4aACKCBaIuy7l2XmLHkxVgRS9Hxz2SjMO4C+RQpE2vVwdG5xZl1UMROxWkoML tuojdb4lISMS/BU69YAv4LTGpbQsJuzOHIm5WHimv0aR1OgFceVrO6wxEBz2ztkRiwrt EefA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=m3FCx/ilHbCbH1PQeEV/LpvDplBkw2pi8TS/G2qT+n8=; b=nOF0Yfzv3qF5KrFGD0U2HKjFidJc/VIGu+txc7p+oSD8LM/faYU5mVJ7hIdzUI5jW4 Be4oQsvUpid/NDNRd+m4Ru/oZRmv4AP8A+ku5Waegdq/NuDQkpaj/Cily9v3XierAWyn +KkHFrLd7zJX8wHed2PUcj8uw2XAf44+C2x8lnpE/8FhSnkI77qNMQhW2nVcB5HzQTAN TCZbJStpWALgyEdl4XxmBaqHtZknu0nhYD5WXS09DmTVaQPqT69dfc0DIu3cCyNVDi86 49uVn5LKaRoOJHJTZcKx+ruhmG3go5BiyF4Wq/Wxgoz1tDZi8GzGyOZ6LliFAiANnsEt 3E4w== X-Gm-Message-State: AA+aEWZu3YIaMUALycghSKvuy4VeGV/gMwWS40x34zLi3L+dIU6NkSYF BLJxj4AurJ9KknkCxnAF0tj9sJU4RY+CF3NGkXKBuQ== X-Google-Smtp-Source: AFSGD/UnHmeJxIxkobBaC8usa0EzLrOVxsbdysvZeG/1ed7oDu+f/Ck32xdBBcmZA6om5OcexrlRfVHzuh4HamQ5/kI= X-Received: by 2002:a1c:8d12:: with SMTP id p18mr12264927wmd.31.1545045882795; Mon, 17 Dec 2018 03:24:42 -0800 (PST) MIME-Version: 1.0 References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> In-Reply-To: From: Rajesh Kumar Date: Mon, 17 Dec 2018 16:54:30 +0530 Message-ID: Subject: Re: How to use the DMA Engine in FreeBSD? To: cem@freebsd.org Cc: asomers@freebsd.org, freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org X-Rspamd-Queue-Id: 6E7F58FFD3 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=luoc9Ie6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2a00:1450:4864:20::330 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-5.41 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[0.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.53)[ip: (-9.65), ipnet: 2a00:1450::/32(-1.55), asn: 15169(-1.35), country: US(-0.08)]; NEURAL_HAM_SHORT(-0.88)[-0.876,0]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 11:24:47 -0000 Hi, Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to a peripheral device being made as the owner of the bus by the DMA engine and do data transfer. Please correct me if I am wrong. But, will this bus_dma API uses the DMA engine in hardware (does it support PCI busmastering)? As I understand, DMA engine will expose DMA channels, which can be used by the client drivers to queue their DMA requests (without CPU intervention) and completions are given through the callback functions. If this is considered the real DMA operation, I don't see options to request DMA channels using bus_dma API's, and the callbacks in bus_dma API is meant to receive just the DMA map details (doesn't indicate a DMA completion). ioat(4) driver seems to be doing what I described above, but that seems to be Intel specific. Can it be used with any DMA controllers? So, I am wondering what needs to be done in FreeBSD to do the actual DMA involving DMA engine in a more generic way? Thanks, Rajesh. On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer wrote: > On Fri, Dec 14, 2018 at 8:17 AM Alan Somers wrote: > > > For some Intel based server hardware, there is the "ioat" driver, which > > > allows for user code to schedule DMA operations. See ioat(4) for > > > details, including a pointer to the test program. > > This isn't true (or at least, only for testing). It's only usable by > the kernel at the moment. > > > * In what context are callbacks called? Are they called from a signal > > handler, or in a separate thread, or something else? > > Directly from the ithread, mostly. Callers can't take sleep locks or > do very much besides set a flag and wakeup, or kick off a callout or > taskqueue. > > > * Why isn't ioat.h installed? > > Why would it be? It's a kernel interface. > > > * Are "interrupts" synonymous with callbacks? > > I don't understand the question. Interrupts are synonymous with > interrupts. Like, MSI-X. > > > * Do you have a rough idea for about the minimum buffer size that > > makes sense to use with ioat? > > What makes sense will depend on your use case. It's not necessarily > faster than CPU memcpy, but it may be worth some slowdown to offload > latency-insensitive bulk copies. (It may make a lot of sense for > asynchronous page zero, if someone wants to bring that back.) It's > worth benchmarking your specific model. > > We use it for >= 8kB copies on Broadwell, although that's largely an > artifact of our use case (write sizes are exactly 8 bytes, 512 bytes, > or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with > some small number of lanes on Broadwell, although I may be mistaken; > and it may not communicate with RAM over the PCIe bus anyway, given > the tight integration with the CPU. > > Best, > Conrad > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" > From owner-freebsd-drivers@freebsd.org Tue Dec 18 08:29:20 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D8941345D3B; Tue, 18 Dec 2018 08:29:20 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DDCF861FC; Tue, 18 Dec 2018 08:29:19 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wr1-x429.google.com with SMTP id v13so14912579wrw.5; Tue, 18 Dec 2018 00:29:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4sLUyBLDZ7Hfb8aUbN7j7izvHmcH+tWtdcpArhN0RwI=; b=J6y4HftqT16G/gMtfWvxn5qiRrRVJi/B9Nm5b49pwdXnULbfPPc7vNZCM6ds3jk5nX pJ/Zuea0+xHWJfhr/5gDjTOUq2rC68FhdVWKvylL3rB6DrVpQKwvtgdv7H82f94eHGG/ YqsngcjSMH+5Lb+BGdOwQ6byq5fyQtgd3NjtWFnlLsIhCL1sannqArTC8bMlftNPS08r c2fRY8VBRpG2bGNvN7i8MRhkLqsxeg9SWl+Uyb1GvujRIXIGtMvNoMzsd4ODBKMtnOx5 3eWjDseWbgDFeIr3F5oNnr4z0vNQvkxcxyR/lS02BnWw5tSpY8/GHnalm/gI/c5YEOg9 p6Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4sLUyBLDZ7Hfb8aUbN7j7izvHmcH+tWtdcpArhN0RwI=; b=Cr9bfngLa36Pmry6gtwu4UzBij77W5/lNteiFDN3MBMZ/Qu2YPFoST8ePVPC/gx6FT y2zx0oB/LQu0Ks3jI2VTsay8fr3OxZJncNGM9I4CUb6JeBCp74d3Kj599mp3vCqa4vVd qW3fBx81OfC5+E6qqROHg/cXldSAwYoPfxV60QqokXcdh5HLU89tbGswgqQgVD8c36Ae U3y+wmJVYN6XK7vyrq7riNg6SrEY1LKxEUNBOEL4m52KXKYScTo2iloOxBHdd/qUAixE NDVhZ2mccLUkcysyoXQi1vJooJtEKhxagCh4xVKMP+WKFtyzO2V3rtj/uk2GVm0Ia1fJ /dtw== X-Gm-Message-State: AA+aEWY3CMoqQ59wlFzQEll4EAzNSSHfDRIY1BJ/9AlNtaS6z0h3UDy4 JDMH5D1FLQK8pvDgCoTOtxT3aY4taArgyuPErhY/HA== X-Google-Smtp-Source: AFSGD/WeHNTvrAjIcXg4xGDDudzWSaMkcqKdauf18Bsit/LWL3lox1AIREoSWP6wKrzuXDTJpH0g1sqVcRZv/dMr8XA= X-Received: by 2002:adf:94e4:: with SMTP id 91mr14518743wrr.322.1545121757998; Tue, 18 Dec 2018 00:29:17 -0800 (PST) MIME-Version: 1.0 References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> In-Reply-To: From: Rajesh Kumar Date: Tue, 18 Dec 2018 13:59:06 +0530 Message-ID: Subject: Re: How to use the DMA Engine in FreeBSD? To: cem@freebsd.org Cc: asomers@freebsd.org, freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org X-Rspamd-Queue-Id: 7DDCF861FC X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=J6y4Hftq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2a00:1450:4864:20::429 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-5.38 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[9.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-0.90)[-0.902,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.47)[ip: (-9.29), ipnet: 2a00:1450::/32(-1.58), asn: 15169(-1.38), country: US(-0.08)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Dec 2018 08:29:20 -0000 Any thoughts (or) suggestions here? On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar wrote: > Hi, > > Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to a > peripheral device being made as the owner of the bus by the DMA engine and > do data transfer. Please correct me if I am wrong. But, will this bus_dma > API uses the DMA engine in hardware (does it support PCI busmastering)? > > As I understand, DMA engine will expose DMA channels, which can be used by > the client drivers to queue their DMA requests (without CPU intervention) > and completions are given through the callback functions. If this is > considered the real DMA operation, I don't see options to request DMA > channels using bus_dma API's, and the callbacks in bus_dma API is meant to > receive just the DMA map details (doesn't indicate a DMA completion). > ioat(4) driver seems to be doing what I described above, but that seems to > be Intel specific. Can it be used with any DMA controllers? > > So, I am wondering what needs to be done in FreeBSD to do the actual DMA > involving DMA engine in a more generic way? > > Thanks, > Rajesh. > > > On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer wrote: > >> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers wrote: >> > > For some Intel based server hardware, there is the "ioat" driver, >> which >> > > allows for user code to schedule DMA operations. See ioat(4) for >> > > details, including a pointer to the test program. >> >> This isn't true (or at least, only for testing). It's only usable by >> the kernel at the moment. >> >> > * In what context are callbacks called? Are they called from a signal >> > handler, or in a separate thread, or something else? >> >> Directly from the ithread, mostly. Callers can't take sleep locks or >> do very much besides set a flag and wakeup, or kick off a callout or >> taskqueue. >> >> > * Why isn't ioat.h installed? >> >> Why would it be? It's a kernel interface. >> >> > * Are "interrupts" synonymous with callbacks? >> >> I don't understand the question. Interrupts are synonymous with >> interrupts. Like, MSI-X. >> >> > * Do you have a rough idea for about the minimum buffer size that >> > makes sense to use with ioat? >> >> What makes sense will depend on your use case. It's not necessarily >> faster than CPU memcpy, but it may be worth some slowdown to offload >> latency-insensitive bulk copies. (It may make a lot of sense for >> asynchronous page zero, if someone wants to bring that back.) It's >> worth benchmarking your specific model. >> >> We use it for >= 8kB copies on Broadwell, although that's largely an >> artifact of our use case (write sizes are exactly 8 bytes, 512 bytes, >> or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with >> some small number of lanes on Broadwell, although I may be mistaken; >> and it may not communicate with RAM over the PCIe bus anyway, given >> the tight integration with the CPU. >> >> Best, >> Conrad >> _______________________________________________ >> freebsd-drivers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >> To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org >> " >> > From owner-freebsd-drivers@freebsd.org Tue Dec 18 12:36:19 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CE131350559 for ; Tue, 18 Dec 2018 12:36:19 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by mx1.freebsd.org (Postfix) with ESMTP id 8F8036B08B for ; Tue, 18 Dec 2018 12:36:18 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from ppp121-45-40-54.bras2.adl4.internode.on.net (HELO midget.dons.net.au) ([121.45.40.54]) by ipmail06.adl6.internode.on.net with ESMTP; 18 Dec 2018 23:01:10 +1030 Received: from midget.dons.net.au (localhost [127.0.0.1]) by midget.dons.net.au (8.15.2/8.15.2) with ESMTPS id wBICUvp6040531 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 18 Dec 2018 23:01:04 +1030 (ACDT) (envelope-from darius@dons.net.au) Received: (from mailnull@localhost) by midget.dons.net.au (8.15.2/8.15.2/Submit) id wBICETA2025219 for ; Tue, 18 Dec 2018 22:44:29 +1030 (ACDT) (envelope-from darius@dons.net.au) X-Authentication-Warning: midget.dons.net.au: mailnull set sender to using -f Received: from [192.168.0.127] ([179.62.129.213]) by ppp121-45-40-54.bras2.adl4.internode.on.net (envelope-sender ) (MIMEDefang) with ESMTP id wBICE2dg025069; Tue, 18 Dec 2018 22:44:29 +1030 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: How to use the DMA Engine in FreeBSD? From: "O'Connor, Daniel" In-Reply-To: Date: Tue, 18 Dec 2018 09:13:55 -0300 Cc: cem@freebsd.org, FreeBSD Hackers , freebsd-drivers@freebsd.org, asomers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2DDD8ADE-4E34-4239-88EA-473B30FF8087@dons.net.au> References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> To: Rajesh Kumar X-Mailer: Apple Mail (2.3445.101.1) X-Spam-Score: 1.5 (*) No, score=1.5 required=5.0 tests=HELO_MISC_IP, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Scanned-By: MIMEDefang 2.83 on 10.0.2.1 X-Rspamd-Queue-Id: 8F8036B08B X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [4.14 / 15.00]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; HAS_XAW(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; MX_GOOD(-0.01)[cached: midget.dons.net.au]; NEURAL_HAM_SHORT(-0.40)[-0.401,0]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; RCVD_IN_DNSWL_LOW(-0.10)[145.137.101.150.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:4739, ipnet:150.101.0.0/16, country:AU]; SUBJECT_ENDS_QUESTION(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[54.40.45.121.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; IP_SCORE(0.28)[ipnet: 150.101.0.0/16(0.96), asn: 4739(0.46), country: AU(-0.03)]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-drivers@freebsd.org]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.87)[0.874,0]; DMARC_NA(0.00)[dons.net.au]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(1.00)[0.996,0]; R_SPF_NA(0.00)[] X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Dec 2018 12:36:19 -0000 > On 18 Dec 2018, at 05:29, Rajesh Kumar wrote: > Any thoughts (or) suggestions here? You don't need to "assign DMA channels" and a PCIe device doesn't = "master the bus" because it's all point to point. "DMA channels" are a legacy of ISA cards, although the terminology may = be reused for things like ioat. For PCI[e] devices which support bus mastering (ie virtually all of = them) the device itself issues the transfer. How that is actually done = depends on the device - usually you program registers with a = scatter/gather (SG) list, or with the physical address of an SG list in = system memory and then the device will do the job. The bus_dma* routines in the kernel are a tool box to let a driver tell = the kernel what kind of DMA a card is going to do and to help with = things like mapping it as necessary, cache coherency etc etc. Systems with ioat are pretty rare I believe, the vast majority of DMA in = a modern PC is done by PCI cards (or things which look like PCI cards to = the OS). > On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar = wrote: >=20 >> Hi, >>=20 >> Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers = to a >> peripheral device being made as the owner of the bus by the DMA = engine and >> do data transfer. Please correct me if I am wrong. But, will this = bus_dma >> API uses the DMA engine in hardware (does it support PCI = busmastering)? >>=20 >> As I understand, DMA engine will expose DMA channels, which can be = used by >> the client drivers to queue their DMA requests (without CPU = intervention) >> and completions are given through the callback functions. If this is >> considered the real DMA operation, I don't see options to request DMA >> channels using bus_dma API's, and the callbacks in bus_dma API is = meant to >> receive just the DMA map details (doesn't indicate a DMA completion). >> ioat(4) driver seems to be doing what I described above, but that = seems to >> be Intel specific. Can it be used with any DMA controllers? >>=20 >> So, I am wondering what needs to be done in FreeBSD to do the actual = DMA >> involving DMA engine in a more generic way? >>=20 >> Thanks, >> Rajesh. >>=20 >>=20 >> On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer wrote: >>=20 >>> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers = wrote: >>>>> For some Intel based server hardware, there is the "ioat" driver, >>> which >>>>> allows for user code to schedule DMA operations. See ioat(4) for >>>>> details, including a pointer to the test program. >>>=20 >>> This isn't true (or at least, only for testing). It's only usable = by >>> the kernel at the moment. >>>=20 >>>> * In what context are callbacks called? Are they called from a = signal >>>> handler, or in a separate thread, or something else? >>>=20 >>> Directly from the ithread, mostly. Callers can't take sleep locks = or >>> do very much besides set a flag and wakeup, or kick off a callout or >>> taskqueue. >>>=20 >>>> * Why isn't ioat.h installed? >>>=20 >>> Why would it be? It's a kernel interface. >>>=20 >>>> * Are "interrupts" synonymous with callbacks? >>>=20 >>> I don't understand the question. Interrupts are synonymous with >>> interrupts. Like, MSI-X. >>>=20 >>>> * Do you have a rough idea for about the minimum buffer size that >>>> makes sense to use with ioat? >>>=20 >>> What makes sense will depend on your use case. It's not necessarily >>> faster than CPU memcpy, but it may be worth some slowdown to offload >>> latency-insensitive bulk copies. (It may make a lot of sense for >>> asynchronous page zero, if someone wants to bring that back.) It's >>> worth benchmarking your specific model. >>>=20 >>> We use it for >=3D 8kB copies on Broadwell, although that's largely = an >>> artifact of our use case (write sizes are exactly 8 bytes, 512 = bytes, >>> or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with >>> some small number of lanes on Broadwell, although I may be mistaken; >>> and it may not communicate with RAM over the PCIe bus anyway, given >>> the tight integration with the CPU. >>>=20 >>> Best, >>> Conrad >>> _______________________________________________ >>> freebsd-drivers@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >>> To unsubscribe, send any mail to = "freebsd-drivers-unsubscribe@freebsd.org >>> " >>>=20 >>=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum From owner-freebsd-drivers@freebsd.org Wed Dec 19 09:45:59 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D44731352274; Wed, 19 Dec 2018 09:45:58 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C57A672606; Wed, 19 Dec 2018 09:45:57 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wm1-x32c.google.com with SMTP id f188so4960938wmf.5; Wed, 19 Dec 2018 01:45:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=GwomkPl3xq3IpnQyA1dZXsw2iVM32YY4qcVxeAgP4QY=; b=NLyyhedeAtPLKUPMqyX/Rjtv2sKXPvVafrPLcZzBXeJZlKGIiuo9Czx9/T4TE4AC/y oxUEqPrJk/YAYOfbRQ5YKTO3cVe0/9MBu8ALbcT4v/flH+k0/feEU3L5aM2cMf+LlmQC gjjr6jN29GavC/H47Vk/1vz9oQjTwx5m90S5jVrodQoKU0jTQ0nBoCLlaf6xxyGAO1++ X3o2OnUCNtkTMTNJCKciYjlRlLbi8VM8C2XKRvDQhNdpexHMpmsL194YIIg1azk2k3co LJYzkjSx+ifbF7WMVA0fRxB0tcW2Z0rAchlV4MrhvNN3OJMmgWQYzsvRnMfSDyITQUXf c9lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=GwomkPl3xq3IpnQyA1dZXsw2iVM32YY4qcVxeAgP4QY=; b=b8bMv43NdW6bWs/ZoZmK5I0u5StzaLX8K8z7wEL73Wt4vWqRaD4Eit01YAygDmc1EY KR2vxy53oWveXyM6q8sI2ct8Nny+jYQzuu+eBZfO0GqydhnnvHEGou3xvQOG6f0jgx3V Z+l4aw39aatWM3HUzknHtePbI/wtXAgJtmQIjrfbtfc1/ik4Wg4oLoRfOaEgeivQl5Dt shI0JwsGVZUEOgraWjopl93wmyn7SfWFyoR5YYcMIOOOsJS6M1Ksbe3YGxMZWz3Jg+P3 4leK8Mzj88P0Hbv5AHdOubrT1pp44j+FEtTPlhV9yVgDQvS6pYz/oG4OrODp3JSuzzAL fFzw== X-Gm-Message-State: AA+aEWamtzKN9lBatBtadEL8+U9KFA309sc5FmlLOQJNJ6Lq/z2ogVwI U3p3UJdCckyOs5/FkHaSTfK7qjDHbGzGzCADZPLuui0q X-Google-Smtp-Source: AFSGD/U9uzBuGDpZtmm8lDeKOrf8OgHFUd1O0bNiAnrxl8dTEezii4kJ5TfH5R4bafxSfrbZX4JrQH1moTvIUPrHNW8= X-Received: by 2002:a1c:8d12:: with SMTP id p18mr6969993wmd.31.1545212756495; Wed, 19 Dec 2018 01:45:56 -0800 (PST) MIME-Version: 1.0 References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> <2DDD8ADE-4E34-4239-88EA-473B30FF8087@dons.net.au> In-Reply-To: <2DDD8ADE-4E34-4239-88EA-473B30FF8087@dons.net.au> From: Rajesh Kumar Date: Wed, 19 Dec 2018 15:15:44 +0530 Message-ID: Subject: Re: How to use the DMA Engine in FreeBSD? To: "O'Connor, Daniel" Cc: cem@freebsd.org, FreeBSD Hackers , freebsd-drivers@freebsd.org, asomers@freebsd.org X-Rspamd-Queue-Id: C57A672606 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=NLyyhede; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2a00:1450:4864:20::32c as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-5.41 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[c.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-0.89)[-0.885,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.51)[ip: (-9.51), ipnet: 2a00:1450::/32(-1.58), asn: 15169(-1.38), country: US(-0.08)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 09:45:59 -0000 Thanks for your explanation Daniel. I will see how to initiate the DMA transfer in my case after the bus_dma API's has done its part. On Tue, Dec 18, 2018 at 6:01 PM O'Connor, Daniel wrote: > > > > On 18 Dec 2018, at 05:29, Rajesh Kumar wrote: > > Any thoughts (or) suggestions here? > > You don't need to "assign DMA channels" and a PCIe device doesn't "master > the bus" because it's all point to point. > > "DMA channels" are a legacy of ISA cards, although the terminology may be > reused for things like ioat. > > For PCI[e] devices which support bus mastering (ie virtually all of them) > the device itself issues the transfer. How that is actually done depends on > the device - usually you program registers with a scatter/gather (SG) list, > or with the physical address of an SG list in system memory and then the > device will do the job. > > The bus_dma* routines in the kernel are a tool box to let a driver tell > the kernel what kind of DMA a card is going to do and to help with things > like mapping it as necessary, cache coherency etc etc. > > Systems with ioat are pretty rare I believe, the vast majority of DMA in a > modern PC is done by PCI cards (or things which look like PCI cards to the > OS). > > > On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar wrote: > > > >> Hi, > >> > >> Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to > a > >> peripheral device being made as the owner of the bus by the DMA engine > and > >> do data transfer. Please correct me if I am wrong. But, will this > bus_dma > >> API uses the DMA engine in hardware (does it support PCI busmastering)? > >> > >> As I understand, DMA engine will expose DMA channels, which can be used > by > >> the client drivers to queue their DMA requests (without CPU > intervention) > >> and completions are given through the callback functions. If this is > >> considered the real DMA operation, I don't see options to request DMA > >> channels using bus_dma API's, and the callbacks in bus_dma API is meant > to > >> receive just the DMA map details (doesn't indicate a DMA completion). > >> ioat(4) driver seems to be doing what I described above, but that seems > to > >> be Intel specific. Can it be used with any DMA controllers? > >> > >> So, I am wondering what needs to be done in FreeBSD to do the actual DMA > >> involving DMA engine in a more generic way? > >> > >> Thanks, > >> Rajesh. > >> > >> > >> On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer wrote: > >> > >>> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers > wrote: > >>>>> For some Intel based server hardware, there is the "ioat" driver, > >>> which > >>>>> allows for user code to schedule DMA operations. See ioat(4) for > >>>>> details, including a pointer to the test program. > >>> > >>> This isn't true (or at least, only for testing). It's only usable by > >>> the kernel at the moment. > >>> > >>>> * In what context are callbacks called? Are they called from a signal > >>>> handler, or in a separate thread, or something else? > >>> > >>> Directly from the ithread, mostly. Callers can't take sleep locks or > >>> do very much besides set a flag and wakeup, or kick off a callout or > >>> taskqueue. > >>> > >>>> * Why isn't ioat.h installed? > >>> > >>> Why would it be? It's a kernel interface. > >>> > >>>> * Are "interrupts" synonymous with callbacks? > >>> > >>> I don't understand the question. Interrupts are synonymous with > >>> interrupts. Like, MSI-X. > >>> > >>>> * Do you have a rough idea for about the minimum buffer size that > >>>> makes sense to use with ioat? > >>> > >>> What makes sense will depend on your use case. It's not necessarily > >>> faster than CPU memcpy, but it may be worth some slowdown to offload > >>> latency-insensitive bulk copies. (It may make a lot of sense for > >>> asynchronous page zero, if someone wants to bring that back.) It's > >>> worth benchmarking your specific model. > >>> > >>> We use it for >= 8kB copies on Broadwell, although that's largely an > >>> artifact of our use case (write sizes are exactly 8 bytes, 512 bytes, > >>> or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with > >>> some small number of lanes on Broadwell, although I may be mistaken; > >>> and it may not communicate with RAM over the PCIe bus anyway, given > >>> the tight integration with the CPU. > >>> > >>> Best, > >>> Conrad > >>> _______________________________________________ > >>> freebsd-drivers@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > >>> To unsubscribe, send any mail to " > freebsd-drivers-unsubscribe@freebsd.org > >>> " > >>> > >> > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > > From owner-freebsd-drivers@freebsd.org Sat Dec 22 09:19:30 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BF99134C85D for ; Sat, 22 Dec 2018 09:19:30 +0000 (UTC) (envelope-from info-a3614-122728-123034-d084ed0c=80750009=8@worldclassministries.com) Received: from master.p2.class.worldclassministries.com (master.p2.class.worldclassministries.com [51.158.71.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8ACB180788 for ; Sat, 22 Dec 2018 09:19:29 +0000 (UTC) (envelope-from info-a3614-122728-123034-d084ed0c=80750009=8@worldclassministries.com) Subject: Weihnachtsangebot 2018! 90%-95% Rabatt!! From: "COOP" To: freebsd-drivers@freebsd.org Sender: info@worldclassministries.com Reply-To: info@worldclassministries.com Date: 22 Dec 2018 09:19:27 -0000 X-CampaignID: s4:122728-9b93180b77a2d4c2 Message-ID: X-Mailer-Info: 8.E2M2EDN.xIjM3IDO.mJXZlJ2ck1CZylmdlJ3cAZmclVmYzRmLvJ3Z.gDM3UDMwATO.xIzMwMDN MIME-Version: 1.0 X-Rspamd-Queue-Id: 8ACB180788 X-Spamd-Bar: / X-Spamd-Result: default: False [0.21 / 15.00]; HAS_REPLYTO(0.00)[info@worldclassministries.com]; R_SPF_ALLOW(-0.20)[+ip4:51.158.71.217]; TO_DN_NONE(0.00)[]; REPLYTO_ADDR_EQ_FROM(0.00)[]; DKIM_TRACE(0.00)[worldclassministries.com:+]; DMARC_POLICY_ALLOW(-0.50)[worldclassministries.com,quarantine]; MX_GOOD(-0.01)[cached: master.p2.class.worldclassministries.com]; SUBJECT_ENDS_EXCLAIM(0.00)[]; FORGED_SENDER(0.30)[info@worldclassministries.com,info-a3614-122728-123034-d084ed0c=80750009=8@worldclassministries.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+,1:+]; IP_SCORE(-0.04)[asn: 12876(-0.15), country: FR(-0.02)]; ASN(0.00)[asn:12876, ipnet:51.158.0.0/15, country:FR]; FROM_NEQ_ENVFROM(0.00)[info@worldclassministries.com,info-a3614-122728-123034-d084ed0c=80750009=8@worldclassministries.com]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.67)[-0.666,0]; R_DKIM_ALLOW(-0.20)[worldclassministries.com:s=default]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HTML_SHORT_LINK_IMG_1(2.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_SPAM_SHORT(0.57)[0.573,0]; NEURAL_HAM_LONG(-0.94)[-0.943,0]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Dec 2018 09:19:30 -0000 Willkommen bei Coop! -------------------- Herzliche Gl=C3=BCckw=C3=BCnsche! ----------------------- Wir haben 250 Konsumenten aus Switzerland, um an einer kurzen Befragung von Coop teilzunehmen. Alle Teilnehmer werden (1) optionale Belohnung erhalten. Sie haben die Wahl zwischen Geschenkgutscheinen, Elektronikartikeln, Hautpflegeprodukten und vielem mehr. Ihr einmaliger Bonuscode lautet: CP42A facebook twitter ( http://chef.worldclassministries.com/ga/click/2-80750009= -3614-67444-123034-68915-633cde3df1-42c0a7535e ) Aufgrund der hohen Nachfrage ist lediglich ein begrenztes Angebot verf=C3=BCgbar. Klick hier ( http://chef.worldclassministries.com/ga/click/2-80750009-3614-= 67444-123034-68915-633cde3df1-42c0a7535e ) Abmelden ( http://chef.worldclassministries.com/ga/unsubscribe/2-80750009-3= 614-67444-123034-c600b1e6992ad46-42c0a7535e )=