From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 10:18:37 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 563167B5 for ; Fri, 30 Jan 2015 10:18:37 +0000 (UTC) Received: from mail-we0-f174.google.com (exprod7og126.obsmtp.com [64.18.2.206]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF6FE1DF for ; Fri, 30 Jan 2015 10:18:35 +0000 (UTC) Received: from mail-we0-f174.google.com ([74.125.82.174]) (using TLSv1) by exprod7ob126.postini.com ([64.18.6.12]) with SMTP ID DSNKVMtae1E5JOshFg+Iepok+f6GPt9dRk4Q@postini.com; Fri, 30 Jan 2015 02:18:37 PST Received: by mail-we0-f174.google.com with SMTP id w55so20869039wes.5 for ; Fri, 30 Jan 2015 02:18:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=JDbkutA32hn0/cgY9QOdwMb5lzF0llOxaz5vY9vKloc=; b=kFXXAOSBEC843g7QZ+ONev/YsY+y/4tlW5IA9D0qvjGDbZo6aEoIXSFns5NwTEJTTR EKuJN+iOD6p/LXCGhxyU2XIkHpy89B/M4+kU2PweVRnIl7Yz0a5fLj+S4fVUlsH0EKrs w8oNJfGs/BnkqnFuEdhEKiWszaqp02F7AK+fBHcfikNz6hsHqt7g1DktqYw8MXJr6B9N RiAw27vw5zPl2wGbSnhNgveW21b0H20fL3q7F9rtK4EC/b1yV0Vh7RK1Aw3NGOM1Cz0s ZCrH6IpGiY0YMEGKpouXjH68qH4IFktah1FlfA4hCduXN5T9FHt2B2s+0PYurjQdGFAz xs9g== X-Gm-Message-State: ALoCoQk8w5I+alb1BzVQYCY4oTselsakWVNOBKsBruar02Yq16wIYWzvYhRPoWDweG9hXYVPCTWDBPMplLdnMVAzBPtwYdIjg00yxjEO10VNvSoslYuMVU6NDkrJRWMOhnOHUQqmpgE24Q1eKBnmEuxqQi0aNuIVSQ== X-Received: by 10.194.143.12 with SMTP id sa12mr10443337wjb.101.1422613114834; Fri, 30 Jan 2015 02:18:34 -0800 (PST) X-Received: by 10.194.143.12 with SMTP id sa12mr10443306wjb.101.1422613114627; Fri, 30 Jan 2015 02:18:34 -0800 (PST) From: Sibananda Sahu References: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> 923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com In-Reply-To: 923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQJ90Y9Ud/YDW7817ZAJGKXgu7SlNwGkTx3Um3AbLDCAAATKwA== Date: Fri, 30 Jan 2015 15:48:33 +0530 Message-ID: <37dd5147bdbf0ea417d2fdfb31565358@mail.gmail.com> Subject: RE: How to send 1MB I/O size in a single I/O request without split To: Scott Long Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 10:18:37 -0000 Hi Scott, One more thing to ask. I think even if I will modify the bus_dma_tag in our driver to have a max segment of 4K, the max I/O length per I/O request that will come is 128KB because of the MAXPHYS limitation. I have tested this by restricting the MAXPHYS to 128KB again and when I requested an I/O with bs=3D1M the I/O was split with various length I/O requests. Does this implies that tuning the MAXPHYS is the ultimate solution to get larger I/O size?? Thanks, Sibananda -----Original Message----- From: Sibananda Sahu [mailto:sibananda.sahu@avagotech.com] Sent: Friday, January 30, 2015 3:12 PM To: 'Scott Long' Cc: 'freebsd-scsi@freebsd.org' Subject: RE: How to send 1MB I/O size in a single I/O request without split Hey Scoot, Thanks very much for your reply. >> - How can I get more sge count in an I/O request? >> >If you want to test multiple segments, I suggest leaving you system runnin= g >for a long time with multiple processes freeing and allocating memory so >that the system becomes >fragmented. You can also modify the bus_dma_tag >in your driver to specify a maximum segment size of 4k instead of >(presumably in your case) something larger. That will force >busdma to >stop merging adjacent segments. As you have said: "The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segment= s reduces processing overhead in the driver and the hardware." If I will modify the bus_dma_tag in our driver to have a max segment of 4K then what is the performance impact? Thanks, Sibananda Sahu -----Original Message----- From: Scott Long [mailto:scott4long@yahoo.com] Sent: Friday, January 30, 2015 1:47 AM To: Sibananda Sahu Cc: freebsd-scsi@freebsd.org Subject: Re: How to send 1MB I/O size in a single I/O request without split > On Jan 29, 2015, at 11:56 AM, Sibananda Sahu > wrote: > > Hi All, > > > > Recently we have added large I/O size of 1MB in our LSI controller and > for that we have implemented the same in driver. > > But I have observed that the large I/O that an application is able to > send is 128KB in one I/O request. > > > > I used the following command to send 1MB I/O: > > # ddpt if=3D/dev/da0 of=3D/dev/zero count=3D1 bs=3D1M > > > > But I have observed that the number of scatter gather elements per I/O > request always comes 1 and the I/O length comes as 128KB(max). > > [=E2=80=A6] > > > So my primary questions are: > > - How can I send a large I/O size of 1MB in a single I/O request > without any split? You answered this question already, you must redefine MAXPHYS. This can be done via a kernel compile option, and many users and companies already know how to do this. I plan to start a discussion on increasing the default size. > > - Why I am getting always 1 scatter gather element? > You are likely getting an allocation that is physically contiguous. This i= s especially true since you are using =E2=80=98dd=E2=80=99 from userland, and= the memory allocator in userland tries to useon 2MB superpages for allocations. The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segments reduces processing overhead in the driver and the hardware. > - How can I get more sge count in an I/O request? > If you want to test multiple segments, I suggest leaving you system running for a long time with multiple processes freeing and allocating memory so that the system becomes fragmented. You can also modify the bus_dma_tag in your driver to specify a maximum segment size of 4k instead of (presumably in your case) something larger. That will force busdma to stop merging adjacent segments. Scott