From owner-freebsd-scsi@freebsd.org  Sun Aug  2 13:39:16 2015
Return-Path: <owner-freebsd-scsi@freebsd.org>
Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A1F19B1737
 for <freebsd-scsi@mailman.ysv.freebsd.org>;
 Sun,  2 Aug 2015 13:39:16 +0000 (UTC)
 (envelope-from kostikbel@gmail.com)
Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1])
 (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 0EAD71B31
 for <freebsd-scsi@freebsd.org>; Sun,  2 Aug 2015 13:39:15 +0000 (UTC)
 (envelope-from kostikbel@gmail.com)
Received: from tom.home (kostik@localhost [127.0.0.1])
 by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t72DdAgi079195
 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO);
 Sun, 2 Aug 2015 16:39:10 +0300 (EEST)
 (envelope-from kostikbel@gmail.com)
DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t72DdAgi079195
Received: (from kostik@localhost)
 by tom.home (8.15.2/8.15.2/Submit) id t72DdAS6079194;
 Sun, 2 Aug 2015 16:39:10 +0300 (EEST)
 (envelope-from kostikbel@gmail.com)
X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com
 using -f
Date: Sun, 2 Aug 2015 16:39:10 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Max Gurtovoy <maxg@mellanox.com>
Cc: freebsd-scsi@freebsd.org
Subject: Re: PIM_UNMAPPED question
Message-ID: <20150802133910.GS2072@kib.kiev.ua>
References: <VI1PR05MB1440A7BED8C6434EE01360CFDE880@VI1PR05MB1440.eurprd05.prod.outlook.com>
 <55BE0B55.8070807@mellanox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <55BE0B55.8070807@mellanox.com>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00,
 DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no
 autolearn_force=no version=3.4.1
X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home
X-BeenThere: freebsd-scsi@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: SCSI subsystem <freebsd-scsi.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-scsi>,
 <mailto:freebsd-scsi-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-scsi/>
List-Post: <mailto:freebsd-scsi@freebsd.org>
List-Help: <mailto:freebsd-scsi-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-scsi>,
 <mailto:freebsd-scsi-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 02 Aug 2015 13:39:16 -0000

On Sun, Aug 02, 2015 at 03:21:41PM +0300, Max Gurtovoy wrote:
> However, I noticed that the bio_ma vector consists of physically 
> contiguous addresses. I expected that user-space
> allocations would not be physically contiguous. Is it possible that I'm 
> still not seeing the user IO buffers?

Large enough userspace allocations, on a machine with enough free or
reusable memory, most likely trigger reservations to allow the mappings
to be promoted to superpages.  This means that on amd64 aligned
2M chunks are often contiguous.

You could verify the guess by looking at the procstat(1) -v output
for your process. In particular, look for the S flag for the i/o memory
mapping.

There may be other reasons why you see physically contiguous bio_ma's,
but the explanation above is the most straightforward.