From owner-freebsd-current@freebsd.org  Wed Apr 27 16:17:00 2016
Return-Path: <owner-freebsd-current@freebsd.org>
Delivered-To: freebsd-current@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id C221EB1EBA5
 for <freebsd-current@mailman.ysv.freebsd.org>;
 Wed, 27 Apr 2016 16:17:00 +0000 (UTC) (envelope-from jhb@freebsd.org)
Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1])
 (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id A47B51FC4;
 Wed, 27 Apr 2016 16:17:00 +0000 (UTC) (envelope-from jhb@freebsd.org)
Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net
 [73.231.226.104])
 by bigwig.baldwin.cx (Postfix) with ESMTPSA id BC3C3B93C;
 Wed, 27 Apr 2016 12:16:59 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: freebsd-current@freebsd.org
Cc: Shawn Webb <shawn.webb@hardenedbsd.org>, 'Luigi Rizzo' <luigi@freebsd.org>
Subject: Re: Netmap LOR
Date: Wed, 27 Apr 2016 08:34:32 -0700
Message-ID: <2137327.T2i4J3cQgy@ralph.baldwin.cx>
User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; )
In-Reply-To: <20160425232700.GB99221@mutt-hardenedbsd>
References: <20160425232700.GB99221@mutt-hardenedbsd>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
 (bigwig.baldwin.cx); Wed, 27 Apr 2016 12:16:59 -0400 (EDT)
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Apr 2016 16:17:00 -0000

On Monday, April 25, 2016 07:27:00 PM Shawn Webb wrote:
> Here's the backtrace:
> 
> lock order reversal: (sleepable after non-sleepable)
>  1st 0xfffff8002a914840 vm object (vm object) @ /usr/src/sys/vm/vm_fault.c:361
>  2nd 0xffffffff822bf9c8 (&nm_mem)->nm_mtx ((&nm_mem)->nm_mtx) @ /usr/src/sys/dev/netmap/netmap_mem2.c:490
> stack backtrace:
> #0 0xffffffff80bb3790 at witness_debugger+0x70
> #1 0xffffffff80bb3684 at witness_checkorder+0xe54
> #2 0xffffffff80b5f132 at _sx_xlock+0x72
> #3 0xffffffff8075c43d at netmap_mem2_ofstophys+0x2d
> #4 0xffffffff80759a2b at netmap_dev_pager_fault+0x3b
> #5 0xffffffff80e8b991 at dev_pager_getpages+0x61
> #6 0xffffffff80eb440a at vm_pager_get_pages+0x4a
> #7 0xffffffff80e98210 at vm_fault_hold+0x780
> #8 0xffffffff80e97a48 at vm_fault+0x78
> #9 0xffffffff81028745 at trap_pfault+0x115
> #10 0xffffffff81027dd2 at trap+0x342
> #11 0xffffffff81008981 at calltrap+0x8
> 
> Thanks,

Yes, it's not safe to acquire an sx lock in a d_mmap callback.

OTOH, if the buffers are static (and they should be since d_mmap mappings
are cached forever once established), then netmap could create an sglist
of the physaddrs of the backing pages and create an OBJT_SG VM object
backed by that sglist.  It could then return that object via a d_mmap_single
callback (which can grab sx locks).

-- 
John Baldwin