From owner-freebsd-hackers@freebsd.org Mon Dec 17 11:24:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A45D133B369; Mon, 17 Dec 2018 11:24:47 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E7F58FFD3; Mon, 17 Dec 2018 11:24:44 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wm1-x330.google.com with SMTP id n190so12000299wmd.0; Mon, 17 Dec 2018 03:24:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=m3FCx/ilHbCbH1PQeEV/LpvDplBkw2pi8TS/G2qT+n8=; b=luoc9Ie6ouvWM2LnX6AsQmojsvBzpoowC7wSoFIgsXx7tN61tkPQsCGbiCfjdVquCb 5clW1guLol7+oqPdQjjKXJUUBoQlcnP0VVepxUBiKCsZ3lULqMoeMcKArb0Qyi4x/bV6 Kj0kABSvEM6wOcWnr8IrXC8YFyjOnOq+ybIe03/eDfA+kZunk65Kpd+UH/LI7N3k7krd BRrVeSa4aACKCBaIuy7l2XmLHkxVgRS9Hxz2SjMO4C+RQpE2vVwdG5xZl1UMROxWkoML tuojdb4lISMS/BU69YAv4LTGpbQsJuzOHIm5WHimv0aR1OgFceVrO6wxEBz2ztkRiwrt EefA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=m3FCx/ilHbCbH1PQeEV/LpvDplBkw2pi8TS/G2qT+n8=; b=nOF0Yfzv3qF5KrFGD0U2HKjFidJc/VIGu+txc7p+oSD8LM/faYU5mVJ7hIdzUI5jW4 Be4oQsvUpid/NDNRd+m4Ru/oZRmv4AP8A+ku5Waegdq/NuDQkpaj/Cily9v3XierAWyn +KkHFrLd7zJX8wHed2PUcj8uw2XAf44+C2x8lnpE/8FhSnkI77qNMQhW2nVcB5HzQTAN TCZbJStpWALgyEdl4XxmBaqHtZknu0nhYD5WXS09DmTVaQPqT69dfc0DIu3cCyNVDi86 49uVn5LKaRoOJHJTZcKx+ruhmG3go5BiyF4Wq/Wxgoz1tDZi8GzGyOZ6LliFAiANnsEt 3E4w== X-Gm-Message-State: AA+aEWZu3YIaMUALycghSKvuy4VeGV/gMwWS40x34zLi3L+dIU6NkSYF BLJxj4AurJ9KknkCxnAF0tj9sJU4RY+CF3NGkXKBuQ== X-Google-Smtp-Source: AFSGD/UnHmeJxIxkobBaC8usa0EzLrOVxsbdysvZeG/1ed7oDu+f/Ck32xdBBcmZA6om5OcexrlRfVHzuh4HamQ5/kI= X-Received: by 2002:a1c:8d12:: with SMTP id p18mr12264927wmd.31.1545045882795; Mon, 17 Dec 2018 03:24:42 -0800 (PST) MIME-Version: 1.0 References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> In-Reply-To: From: Rajesh Kumar Date: Mon, 17 Dec 2018 16:54:30 +0530 Message-ID: Subject: Re: How to use the DMA Engine in FreeBSD? To: cem@freebsd.org Cc: asomers@freebsd.org, freebsd-hackers@freebsd.org, freebsd-drivers@freebsd.org X-Rspamd-Queue-Id: 6E7F58FFD3 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=luoc9Ie6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2a00:1450:4864:20::330 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-5.41 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[0.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.53)[ip: (-9.65), ipnet: 2a00:1450::/32(-1.55), asn: 15169(-1.35), country: US(-0.08)]; NEURAL_HAM_SHORT(-0.88)[-0.876,0]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 11:24:47 -0000 Hi, Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to a peripheral device being made as the owner of the bus by the DMA engine and do data transfer. Please correct me if I am wrong. But, will this bus_dma API uses the DMA engine in hardware (does it support PCI busmastering)? As I understand, DMA engine will expose DMA channels, which can be used by the client drivers to queue their DMA requests (without CPU intervention) and completions are given through the callback functions. If this is considered the real DMA operation, I don't see options to request DMA channels using bus_dma API's, and the callbacks in bus_dma API is meant to receive just the DMA map details (doesn't indicate a DMA completion). ioat(4) driver seems to be doing what I described above, but that seems to be Intel specific. Can it be used with any DMA controllers? So, I am wondering what needs to be done in FreeBSD to do the actual DMA involving DMA engine in a more generic way? Thanks, Rajesh. On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer wrote: > On Fri, Dec 14, 2018 at 8:17 AM Alan Somers wrote: > > > For some Intel based server hardware, there is the "ioat" driver, which > > > allows for user code to schedule DMA operations. See ioat(4) for > > > details, including a pointer to the test program. > > This isn't true (or at least, only for testing). It's only usable by > the kernel at the moment. > > > * In what context are callbacks called? Are they called from a signal > > handler, or in a separate thread, or something else? > > Directly from the ithread, mostly. Callers can't take sleep locks or > do very much besides set a flag and wakeup, or kick off a callout or > taskqueue. > > > * Why isn't ioat.h installed? > > Why would it be? It's a kernel interface. > > > * Are "interrupts" synonymous with callbacks? > > I don't understand the question. Interrupts are synonymous with > interrupts. Like, MSI-X. > > > * Do you have a rough idea for about the minimum buffer size that > > makes sense to use with ioat? > > What makes sense will depend on your use case. It's not necessarily > faster than CPU memcpy, but it may be worth some slowdown to offload > latency-insensitive bulk copies. (It may make a lot of sense for > asynchronous page zero, if someone wants to bring that back.) It's > worth benchmarking your specific model. > > We use it for >= 8kB copies on Broadwell, although that's largely an > artifact of our use case (write sizes are exactly 8 bytes, 512 bytes, > or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with > some small number of lanes on Broadwell, although I may be mistaken; > and it may not communicate with RAM over the PCIe bus anyway, given > the tight integration with the CPU. > > Best, > Conrad > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" >