From owner-freebsd-drivers@freebsd.org Sat Dec 15 00:59:56 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9382A13278DB; Sat, 15 Dec 2018 00:59:56 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it1-f179.google.com (mail-it1-f179.google.com [209.85.166.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9366B73870; Sat, 15 Dec 2018 00:59:55 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it1-f179.google.com with SMTP id c9so12298907itj.1; Fri, 14 Dec 2018 16:59:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=EU5UMKETp3I2V730UX34nZFrsQLTwDvGGqEmFD4No18=; b=nFyfrZSodB8TGGdP4cdVuI44Wax0iYG3DUe611GXw2Y3l5b/1Q72SMrM6vRU4o4TaZ cZLwlM+/Djb7WVYL44fRBC4mgQPByFON1jN+RIoHmVRdCyKdDqluMvZyjcsLUdWtJo6f tuYBdC5M4FPhl1ZEfeKTSlepUxU7d8S1AtKQm5qZd5J6TIDFCv3eOnObqu/6OVgn8481 6eoDFAzkDM1ufuOzwIQxTusYFwl2BHPod/HURYsx2IC1KgTR/lpPwaYF+jdEXAlwVr9a k06ribbhJ2tzp0x+R40kw2vX825zM6cAlXRApmU2yK2IuE9srSnoRiOWEW86TKj3rGNa UWKw== X-Gm-Message-State: AA+aEWbOOW/aqyXmoX+l7XgmKzY1rpK+Q4e0wyB916blsRCRHRpUyPSU 7aDyxzkMg0Chmo4M6jP9jSEaFfnS X-Google-Smtp-Source: AFSGD/XUX3GrM+fBKa+bw8roDrvBqNMlKlvrkvbC79tNWb9nI696Zz6ZOOvL5qEmnSirJqgx2V/auA== X-Received: by 2002:a24:4ac3:: with SMTP id k186mr4514863itb.76.1544835588913; Fri, 14 Dec 2018 16:59:48 -0800 (PST) Received: from mail-io1-f51.google.com (mail-io1-f51.google.com. [209.85.166.51]) by smtp.gmail.com with ESMTPSA id 196sm3419524itu.37.2018.12.14.16.59.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 14 Dec 2018 16:59:48 -0800 (PST) Received: by mail-io1-f51.google.com with SMTP id o13so2406987iom.4; Fri, 14 Dec 2018 16:59:48 -0800 (PST) X-Received: by 2002:a6b:6111:: with SMTP id v17mr4620029iob.107.1544835588458; Fri, 14 Dec 2018 16:59:48 -0800 (PST) MIME-Version: 1.0 References: <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> In-Reply-To: Reply-To: cem@freebsd.org From: Conrad Meyer Date: Fri, 14 Dec 2018 16:59:37 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: How to use the DMA Engine in FreeBSD? To: Alan Somers Cc: "freebsd-hackers@freebsd.org" , freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 9366B73870 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of csecem@gmail.com designates 209.85.166.179 as permitted sender) smtp.mailfrom=csecem@gmail.com X-Spamd-Result: default: False [-4.83 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[cem@freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.95)[-0.948,0]; FORGED_SENDER(0.30)[cem@freebsd.org,csecem@gmail.com]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_NEQ_ENVFROM(0.00)[cem@freebsd.org,csecem@gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[179.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.87)[ip: (-9.42), ipnet: 209.85.128.0/17(-3.53), asn: 15169(-1.33), country: US(-0.09)]; RWL_MAILSPIKE_POSSIBLE(0.00)[179.166.85.209.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2018 00:59:56 -0000 On Fri, Dec 14, 2018 at 8:17 AM Alan Somers wrote: > > For some Intel based server hardware, there is the "ioat" driver, which > > allows for user code to schedule DMA operations. See ioat(4) for > > details, including a pointer to the test program. This isn't true (or at least, only for testing). It's only usable by the kernel at the moment. > * In what context are callbacks called? Are they called from a signal > handler, or in a separate thread, or something else? Directly from the ithread, mostly. Callers can't take sleep locks or do very much besides set a flag and wakeup, or kick off a callout or taskqueue. > * Why isn't ioat.h installed? Why would it be? It's a kernel interface. > * Are "interrupts" synonymous with callbacks? I don't understand the question. Interrupts are synonymous with interrupts. Like, MSI-X. > * Do you have a rough idea for about the minimum buffer size that > makes sense to use with ioat? What makes sense will depend on your use case. It's not necessarily faster than CPU memcpy, but it may be worth some slowdown to offload latency-insensitive bulk copies. (It may make a lot of sense for asynchronous page zero, if someone wants to bring that back.) It's worth benchmarking your specific model. We use it for >= 8kB copies on Broadwell, although that's largely an artifact of our use case (write sizes are exactly 8 bytes, 512 bytes, or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with some small number of lanes on Broadwell, although I may be mistaken; and it may not communicate with RAM over the PCIe bus anyway, given the tight integration with the CPU. Best, Conrad