From owner-freebsd-arch@FreeBSD.ORG Mon Apr 27 14:46:53 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70B46DAB for ; Mon, 27 Apr 2015 14:46:53 +0000 (UTC) Received: from mail-ig0-x234.google.com (mail-ig0-x234.google.com [IPv6:2607:f8b0:4001:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 373A61E03 for ; Mon, 27 Apr 2015 14:46:53 +0000 (UTC) Received: by igbyr2 with SMTP id yr2so64564113igb.0 for ; Mon, 27 Apr 2015 07:46:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ESLr29EpsUU5sWCCSPZoZhz8DpYA3n/0OyEY5KHIggo=; b=wIJUQ7mzZz5UUmc7py9SYgixKDwo0LPZbLpUS80cShxbdfc2Rg7j9jAe+DKmq5yLmq REAoIQLJPrIc3oFbHrxixXdqTCUshpLWm9nTJRIf6HLexv54RzSoT3/RXq9FCvb1f4wy EWNAm3bA8aXd50DncDhF1GGHIng/pGsNmelGJtRnKD1O6e33orLBGXGLhnz5FP/VTCkb FpodAZORovbcvL3S/JYFpA3j780y65XhdPDfKQ2kWvEuJEKtf4nnHb4groXMPc6yfzEG RG8QB7CqQ9N5taSIEf+PncySXI2NTVyHb09LMP3Ke8f8fHNcc10Oi8YYTr98jSlg0q4R x7RQ== MIME-Version: 1.0 X-Received: by 10.50.6.4 with SMTP id w4mr13928585igw.36.1430146012552; Mon, 27 Apr 2015 07:46:52 -0700 (PDT) Received: by 10.36.106.70 with HTTP; Mon, 27 Apr 2015 07:46:52 -0700 (PDT) In-Reply-To: References: <20150425094152.GE2390@kib.kiev.ua> Date: Mon, 27 Apr 2015 09:46:52 -0500 Message-ID: Subject: Re: bus_dmamap_sync() for bounced client buffers from user address space From: Jason Harmening To: Svatopluk Kraus Cc: Konstantin Belousov , FreeBSD Arch Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Apr 2015 14:46:53 -0000 > > Using of vslock() is proposed method in bus_dma man page. IMO, the >> function looks complex and can be a big time eater. However, are you >> saying that vslock() does not work for that? Then for what reason >> does that function exist? >> > There's been some misunderstanding here, I think. If you use vslock (or vm_map_wire, which vslock wraps), then the UVAs should be safe from teardown and you should be able to use bcopy if you are in the correct context. See the post elsewhere in this thread where I dig through the sys_munmap path and find vm_map_delete waiting on system-wired PTEs. > > >> >> >> > The only safe method to work with the userspace regions is to >> > vm_fault_quick_hold() them to get hold on the pages, and then either >> > pass pages array down, or remap them in the KVA with pmap_qenter(). >> > >> >> >> So, even vm_fault_quick_hold() does not keep valid user mapping? >> > vm_fault_quick_hold_pages() doesn't do any bookkeeping on UVAs, only the underlying physical pages. That means it is possible for the UVA region to be munmap'ed if vm_fault_quick_hold_pages() has been used. So if you use vm_fault_quick_hold_pages() instead of vslock(), you can't use bus_dmamap_load_uio(UIO_USERSPACE) because that assumes valid UVA mappings. You must instead deal only with the underlying vm_page_t's, which means using _bus_dmamap_load_ma(). Here's my take on it: vslock(), as you mention, is very complex. It not only keeps the physical pages from being swapped out, but it also removes them from page queues (see https://lists.freebsd.org/pipermail/freebsd-current/2015-March/054890.html) and does a lot of bookkeeping on the UVA mappings for those pages. Part of that involves simulating a pagefault, which as kib mentions can lead to a lot of UVA fragmentation. vm_fault_quick_hold_pages() is much cheaper and seems mostly intended for short-term DMA operations. So, you might use vslock() + bus_dmamap_load_uio() for long-duration DMA transfers, like continuous streaming to a circular buffer that could last minutes or longer. Then, the extra cost of the vslock will be amortized over the long time of the transfer, and UVA fragmentation will be less of a concern since you presumably will have a limited number of vslock() calls over the lifetime of the process. Also, you will probably be keeping the DMA map for a long duration anyway, so it should be OK to wait and call bus_dmamap_sync() in the process context. Since vslock() removed the pages from the page queues, there will also be less work for pagedaemon to do during the long transfer. OTOH, vm_fault_quick_hold_pages() + _bus_dmamap_load_ma() seems much better to do for frequent short transfers to widely-varying buffers, such as block I/O. The extra pagedaemon work is inconsequential here, and since the DMA operations are frequent and you may have many in-flight at once, the reduced setup cost and fragmentation are much more important.