From owner-freebsd-scsi@freebsd.org Thu Dec 10 15:02:15 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F10619D6796; Thu, 10 Dec 2015 15:02:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7376818F3; Thu, 10 Dec 2015 15:02:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBAF2AlF026079 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 10 Dec 2015 17:02:10 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBAF2AlF026079 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBAF2Awr026078; Thu, 10 Dec 2015 17:02:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 10 Dec 2015 17:02:10 +0200 From: Konstantin Belousov To: Max Gurtovoy Cc: freebsd-scsi@freebsd.org, Sagi Grimberg , Oren Duer , Hans Petter Selasky , freebsd-geom@freebsd.org Subject: Re: splitting iovecs to bios Message-ID: <20151210150210.GY82577@kib.kiev.ua> References: <56696E03.8050202@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56696E03.8050202@mellanox.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 15:02:16 -0000 On Thu, Dec 10, 2015 at 02:20:19PM +0200, Max Gurtovoy wrote: > Hi all, > > I'm developing an iSER (iSCSI extensions for RDMA) driver for FreeBSD > 11-Current and I encounter some weired behaviour while testing IOs with > iovcnt > 1. > I wrote a small test program that creates an iovec struct (let's say in > size of 4) and each iov_base starts from page begining and 4k len: > 0 iov_base 0xe25000 iov_len 4096 > 1 iov_base 0xe26000 iov_len 4096 > 2 iov_base 0xe27000 iov_len 4096 > 3 iov_base 0xe28000 iov_len 4096 > > I use readv/writev to send this iovec data. > I saw that my driver get 4 different bio requests even thought this > vector is aligned. This is surprising becasue in other OS I get only 1 bio. > I have noticed the the physio function in sys/kern/kern_physio.c is > praparing one bio for each iovec entry. > > is there a reason for this kind of implementation ? is there an option > to send this array using 1 bio (some flag) ? > We can improve performance if we send it in 1 bio instead of 4. > My driver supports BIO_UNMAPPED. There might be indeed a reason, it could be that some drivers expect blocking to be done by the userspace. The drivers could have some restrictions on transfer sizes and atomicity of transfer, which would be broken by the unconditional merge. I cannot give you an example of such driver, known block-aware drivers like sa(4) only require the bio size to be multiple of the basic block size. OTOH, I see no issue with adding a SI_PHYSIOMERGE flag and doing the merges for the driver in physio(), when unmapped request has consequtive iov elements ending and starting at the page boundary.