From owner-freebsd-drivers@freebsd.org Tue Dec 18 08:29:20 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D8941345D3B; Tue, 18 Dec 2018 08:29:20 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DDCF861FC; Tue, 18 Dec 2018 08:29:19 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wr1-x429.google.com with SMTP id v13so14912579wrw.5; Tue, 18 Dec 2018 00:29:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4sLUyBLDZ7Hfb8aUbN7j7izvHmcH+tWtdcpArhN0RwI=; b=J6y4HftqT16G/gMtfWvxn5qiRrRVJi/B9Nm5b49pwdXnULbfPPc7vNZCM6ds3jk5nX pJ/Zuea0+xHWJfhr/5gDjTOUq2rC68FhdVWKvylL3rB6DrVpQKwvtgdv7H82f94eHGG/ YqsngcjSMH+5Lb+BGdOwQ6byq5fyQtgd3NjtWFnlLsIhCL1sannqArTC8bMlftNPS08r c2fRY8VBRpG2bGNvN7i8MRhkLqsxeg9SWl+Uyb1GvujRIXIGtMvNoMzsd4ODBKMtnOx5 3eWjDseWbgDFeIr3F5oNnr4z0vNQvkxcxyR/lS02BnWw5tSpY8/GHnalm/gI/c5YEOg9 p6Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4sLUyBLDZ7Hfb8aUbN7j7izvHmcH+tWtdcpArhN0RwI=; b=Cr9bfngLa36Pmry6gtwu4UzBij77W5/lNteiFDN3MBMZ/Qu2YPFoST8ePVPC/gx6FT y2zx0oB/LQu0Ks3jI2VTsay8fr3OxZJncNGM9I4CUb6JeBCp74d3Kj599mp3vCqa4vVd qW3fBx81OfC5+E6qqROHg/cXldSAwYoPfxV60QqokXcdh5HLU89tbGswgqQgVD8c36Ae U3y+wmJVYN6XK7vyrq7riNg6SrEY1LKxEUNBOEL4m52KXKYScTo2iloOxBHdd/qUAixE NDVhZ2mccLUkcysyoXQi1vJooJtEKhxagCh4xVKMP+WKFtyzO2V3rtj/uk2GVm0Ia1fJ /dtw== X-Gm-Message-State: AA+aEWY3CMoqQ59wlFzQEll4EAzNSSHfDRIY1BJ/9AlNtaS6z0h3UDy4 JDMH5D1FLQK8pvDgCoTOtxT3aY4taArgyuPErhY/HA== X-Google-Smtp-Source: AFSGD/WeHNTvrAjIcXg4xGDDudzWSaMkcqKdauf18Bsit/LWL3lox1AIREoSWP6wKrzuXDTJpH0g1sqVcRZv/dMr8XA= X-Received: by 2002:adf:94e4:: with SMTP id 91mr14518743wrr.322.1545121757998; Tue, 18 Dec 2018 00:29:17 -0800 (PST) MIME-Version: 1.0 References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> In-Reply-To: From: Rajesh Kumar Date: Tue, 18 Dec 2018 13:59:06 +0530 Message-ID: Subject: Re: How to use the DMA Engine in FreeBSD? To: cem@freebsd.org Cc: asomers@freebsd.org, freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org X-Rspamd-Queue-Id: 7DDCF861FC X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=J6y4Hftq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2a00:1450:4864:20::429 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-5.38 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[9.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-0.90)[-0.902,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.47)[ip: (-9.29), ipnet: 2a00:1450::/32(-1.58), asn: 15169(-1.38), country: US(-0.08)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Dec 2018 08:29:20 -0000 Any thoughts (or) suggestions here? On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar wrote: > Hi, > > Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to a > peripheral device being made as the owner of the bus by the DMA engine and > do data transfer. Please correct me if I am wrong. But, will this bus_dma > API uses the DMA engine in hardware (does it support PCI busmastering)? > > As I understand, DMA engine will expose DMA channels, which can be used by > the client drivers to queue their DMA requests (without CPU intervention) > and completions are given through the callback functions. If this is > considered the real DMA operation, I don't see options to request DMA > channels using bus_dma API's, and the callbacks in bus_dma API is meant to > receive just the DMA map details (doesn't indicate a DMA completion). > ioat(4) driver seems to be doing what I described above, but that seems to > be Intel specific. Can it be used with any DMA controllers? > > So, I am wondering what needs to be done in FreeBSD to do the actual DMA > involving DMA engine in a more generic way? > > Thanks, > Rajesh. > > > On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer wrote: > >> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers wrote: >> > > For some Intel based server hardware, there is the "ioat" driver, >> which >> > > allows for user code to schedule DMA operations. See ioat(4) for >> > > details, including a pointer to the test program. >> >> This isn't true (or at least, only for testing). It's only usable by >> the kernel at the moment. >> >> > * In what context are callbacks called? Are they called from a signal >> > handler, or in a separate thread, or something else? >> >> Directly from the ithread, mostly. Callers can't take sleep locks or >> do very much besides set a flag and wakeup, or kick off a callout or >> taskqueue. >> >> > * Why isn't ioat.h installed? >> >> Why would it be? It's a kernel interface. >> >> > * Are "interrupts" synonymous with callbacks? >> >> I don't understand the question. Interrupts are synonymous with >> interrupts. Like, MSI-X. >> >> > * Do you have a rough idea for about the minimum buffer size that >> > makes sense to use with ioat? >> >> What makes sense will depend on your use case. It's not necessarily >> faster than CPU memcpy, but it may be worth some slowdown to offload >> latency-insensitive bulk copies. (It may make a lot of sense for >> asynchronous page zero, if someone wants to bring that back.) It's >> worth benchmarking your specific model. >> >> We use it for >= 8kB copies on Broadwell, although that's largely an >> artifact of our use case (write sizes are exactly 8 bytes, 512 bytes, >> or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with >> some small number of lanes on Broadwell, although I may be mistaken; >> and it may not communicate with RAM over the PCIe bus anyway, given >> the tight integration with the CPU. >> >> Best, >> Conrad >> _______________________________________________ >> freebsd-drivers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >> To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org >> " >> >