From owner-freebsd-drivers@freebsd.org Tue Dec 18 12:36:19 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CE131350559 for ; Tue, 18 Dec 2018 12:36:19 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by mx1.freebsd.org (Postfix) with ESMTP id 8F8036B08B for ; Tue, 18 Dec 2018 12:36:18 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from ppp121-45-40-54.bras2.adl4.internode.on.net (HELO midget.dons.net.au) ([121.45.40.54]) by ipmail06.adl6.internode.on.net with ESMTP; 18 Dec 2018 23:01:10 +1030 Received: from midget.dons.net.au (localhost [127.0.0.1]) by midget.dons.net.au (8.15.2/8.15.2) with ESMTPS id wBICUvp6040531 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 18 Dec 2018 23:01:04 +1030 (ACDT) (envelope-from darius@dons.net.au) Received: (from mailnull@localhost) by midget.dons.net.au (8.15.2/8.15.2/Submit) id wBICETA2025219 for ; Tue, 18 Dec 2018 22:44:29 +1030 (ACDT) (envelope-from darius@dons.net.au) X-Authentication-Warning: midget.dons.net.au: mailnull set sender to using -f Received: from [192.168.0.127] ([179.62.129.213]) by ppp121-45-40-54.bras2.adl4.internode.on.net (envelope-sender ) (MIMEDefang) with ESMTP id wBICE2dg025069; Tue, 18 Dec 2018 22:44:29 +1030 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: How to use the DMA Engine in FreeBSD? From: "O'Connor, Daniel" In-Reply-To: Date: Tue, 18 Dec 2018 09:13:55 -0300 Cc: cem@freebsd.org, FreeBSD Hackers , freebsd-drivers@freebsd.org, asomers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2DDD8ADE-4E34-4239-88EA-473B30FF8087@dons.net.au> References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> To: Rajesh Kumar X-Mailer: Apple Mail (2.3445.101.1) X-Spam-Score: 1.5 (*) No, score=1.5 required=5.0 tests=HELO_MISC_IP, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Scanned-By: MIMEDefang 2.83 on 10.0.2.1 X-Rspamd-Queue-Id: 8F8036B08B X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [4.14 / 15.00]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; HAS_XAW(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; MX_GOOD(-0.01)[cached: midget.dons.net.au]; NEURAL_HAM_SHORT(-0.40)[-0.401,0]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; RCVD_IN_DNSWL_LOW(-0.10)[145.137.101.150.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:4739, ipnet:150.101.0.0/16, country:AU]; SUBJECT_ENDS_QUESTION(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[54.40.45.121.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; IP_SCORE(0.28)[ipnet: 150.101.0.0/16(0.96), asn: 4739(0.46), country: AU(-0.03)]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-drivers@freebsd.org]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.87)[0.874,0]; DMARC_NA(0.00)[dons.net.au]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(1.00)[0.996,0]; R_SPF_NA(0.00)[] X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Dec 2018 12:36:19 -0000 > On 18 Dec 2018, at 05:29, Rajesh Kumar wrote: > Any thoughts (or) suggestions here? You don't need to "assign DMA channels" and a PCIe device doesn't = "master the bus" because it's all point to point. "DMA channels" are a legacy of ISA cards, although the terminology may = be reused for things like ioat. For PCI[e] devices which support bus mastering (ie virtually all of = them) the device itself issues the transfer. How that is actually done = depends on the device - usually you program registers with a = scatter/gather (SG) list, or with the physical address of an SG list in = system memory and then the device will do the job. The bus_dma* routines in the kernel are a tool box to let a driver tell = the kernel what kind of DMA a card is going to do and to help with = things like mapping it as necessary, cache coherency etc etc. Systems with ioat are pretty rare I believe, the vast majority of DMA in = a modern PC is done by PCI cards (or things which look like PCI cards to = the OS). > On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar = wrote: >=20 >> Hi, >>=20 >> Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers = to a >> peripheral device being made as the owner of the bus by the DMA = engine and >> do data transfer. Please correct me if I am wrong. But, will this = bus_dma >> API uses the DMA engine in hardware (does it support PCI = busmastering)? >>=20 >> As I understand, DMA engine will expose DMA channels, which can be = used by >> the client drivers to queue their DMA requests (without CPU = intervention) >> and completions are given through the callback functions. If this is >> considered the real DMA operation, I don't see options to request DMA >> channels using bus_dma API's, and the callbacks in bus_dma API is = meant to >> receive just the DMA map details (doesn't indicate a DMA completion). >> ioat(4) driver seems to be doing what I described above, but that = seems to >> be Intel specific. Can it be used with any DMA controllers? >>=20 >> So, I am wondering what needs to be done in FreeBSD to do the actual = DMA >> involving DMA engine in a more generic way? >>=20 >> Thanks, >> Rajesh. >>=20 >>=20 >> On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer wrote: >>=20 >>> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers = wrote: >>>>> For some Intel based server hardware, there is the "ioat" driver, >>> which >>>>> allows for user code to schedule DMA operations. See ioat(4) for >>>>> details, including a pointer to the test program. >>>=20 >>> This isn't true (or at least, only for testing). It's only usable = by >>> the kernel at the moment. >>>=20 >>>> * In what context are callbacks called? Are they called from a = signal >>>> handler, or in a separate thread, or something else? >>>=20 >>> Directly from the ithread, mostly. Callers can't take sleep locks = or >>> do very much besides set a flag and wakeup, or kick off a callout or >>> taskqueue. >>>=20 >>>> * Why isn't ioat.h installed? >>>=20 >>> Why would it be? It's a kernel interface. >>>=20 >>>> * Are "interrupts" synonymous with callbacks? >>>=20 >>> I don't understand the question. Interrupts are synonymous with >>> interrupts. Like, MSI-X. >>>=20 >>>> * Do you have a rough idea for about the minimum buffer size that >>>> makes sense to use with ioat? >>>=20 >>> What makes sense will depend on your use case. It's not necessarily >>> faster than CPU memcpy, but it may be worth some slowdown to offload >>> latency-insensitive bulk copies. (It may make a lot of sense for >>> asynchronous page zero, if someone wants to bring that back.) It's >>> worth benchmarking your specific model. >>>=20 >>> We use it for >=3D 8kB copies on Broadwell, although that's largely = an >>> artifact of our use case (write sizes are exactly 8 bytes, 512 = bytes, >>> or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with >>> some small number of lanes on Broadwell, although I may be mistaken; >>> and it may not communicate with RAM over the PCIe bus anyway, given >>> the tight integration with the CPU. >>>=20 >>> Best, >>> Conrad >>> _______________________________________________ >>> freebsd-drivers@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >>> To unsubscribe, send any mail to = "freebsd-drivers-unsubscribe@freebsd.org >>> " >>>=20 >>=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum