From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 09:42:06 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03396A79 for ; Fri, 30 Jan 2015 09:42:06 +0000 (UTC) Received: from mail-wg0-f50.google.com (exprod7og128.obsmtp.com [64.18.2.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 77FF5E0B for ; Fri, 30 Jan 2015 09:42:05 +0000 (UTC) Received: from mail-wg0-f50.google.com ([74.125.82.50]) (using TLSv1) by exprod7ob128.postini.com ([64.18.6.12]) with SMTP ID DSNKVMtR5VGrpfhom74BQVlrbAxcs/2Fawrw@postini.com; Fri, 30 Jan 2015 01:42:05 PST Received: by mail-wg0-f50.google.com with SMTP id b13so25749937wgh.9 for ; Fri, 30 Jan 2015 01:41:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=uwhquk7V8zzNmzamNtIoVKcLPlYr2NYhWDdwNuZ8cPU=; b=hPeG6PRGgWUW3j7T/Hhv6yIYzpLP2NYLdPgDwe3aY/rC0by9lztkhg+v3nyJZPURnS V73tiwBBMQEIZsv9qGc+yfxioyCKWy0qh+8mf8K3COEYXVH5+ELr8pt/e0hugyCUVMaX pW6dumrjgh+e8mPjeDP/UJYEfHg4KyNaURP0ljAuy2/YN7crwcQvqHOcMWM/ctwSxoUg 23yN2W+jywuqLfR4BxHldcBbs2I0gw35jxWGPhtc+5Ldy19qUOQ81MLeUmvgGig5ZCE7 yEmdFwJkURORlIUiFcLudCHlb8WLSwwN7gvS9LOUUNHKDKY9P8vYg0vnsikV1BYmE/2l l+Jg== X-Gm-Message-State: ALoCoQkgkHIXMIAKMqjXTUMVEsOUN9FSdItGHtngtyYSN02sOwlnlq69UH0ByYTr2N91KWJlApVrOtDEwZtW5GtuNTbgpw0reOD/1GfvQx6JjlK9jT/tvqrhxuEFxWWWVoWrddL8RQiGY6pWIopUkBeKG2rLaQ+j2A== X-Received: by 10.194.200.1 with SMTP id jo1mr10505303wjc.64.1422610917611; Fri, 30 Jan 2015 01:41:57 -0800 (PST) X-Received: by 10.194.200.1 with SMTP id jo1mr10505262wjc.64.1422610917368; Fri, 30 Jan 2015 01:41:57 -0800 (PST) From: Sibananda Sahu References: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> In-Reply-To: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQJ90Y9Ud/YDW7817ZAJGKXgu7SlNwGkTx3Um3AbLDA= Date: Fri, 30 Jan 2015 15:11:56 +0530 Message-ID: <923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com> Subject: RE: How to send 1MB I/O size in a single I/O request without split To: Scott Long Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 09:42:06 -0000 Hey Scoot, Thanks very much for your reply. >> - How can I get more sge count in an I/O request? >> >If you want to test multiple segments, I suggest leaving you system runnin= g >for a long time with multiple processes freeing and allocating memory so >that the system becomes >fragmented. You can also modify the bus_dma_tag >in your driver to specify a maximum segment size of 4k instead of >(presumably in your case) something larger. That will force >busdma to >stop merging adjacent segments. As you have said: "The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segment= s reduces processing overhead in the driver and the hardware." If I will modify the bus_dma_tag in our driver to have a max segment of 4K then what is the performance impact? Thanks, Sibananda Sahu -----Original Message----- From: Scott Long [mailto:scott4long@yahoo.com] Sent: Friday, January 30, 2015 1:47 AM To: Sibananda Sahu Cc: freebsd-scsi@freebsd.org Subject: Re: How to send 1MB I/O size in a single I/O request without split > On Jan 29, 2015, at 11:56 AM, Sibananda Sahu > wrote: > > Hi All, > > > > Recently we have added large I/O size of 1MB in our LSI controller and > for that we have implemented the same in driver. > > But I have observed that the large I/O that an application is able to > send is 128KB in one I/O request. > > > > I used the following command to send 1MB I/O: > > # ddpt if=3D/dev/da0 of=3D/dev/zero count=3D1 bs=3D1M > > > > But I have observed that the number of scatter gather elements per I/O > request always comes 1 and the I/O length comes as 128KB(max). > > [=E2=80=A6] > > > So my primary questions are: > > - How can I send a large I/O size of 1MB in a single I/O request > without any split? You answered this question already, you must redefine MAXPHYS. This can be done via a kernel compile option, and many users and companies already know how to do this. I plan to start a discussion on increasing the default size. > > - Why I am getting always 1 scatter gather element? > You are likely getting an allocation that is physically contiguous. This i= s especially true since you are using =E2=80=98dd=E2=80=99 from userland, and= the memory allocator in userland tries to useon 2MB superpages for allocations. The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segments reduces processing overhead in the driver and the hardware. > - How can I get more sge count in an I/O request? > If you want to test multiple segments, I suggest leaving you system running for a long time with multiple processes freeing and allocating memory so that the system becomes fragmented. You can also modify the bus_dma_tag in your driver to specify a maximum segment size of 4k instead of (presumably in your case) something larger. That will force busdma to stop merging adjacent segments. Scott