From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 29 18:56:44 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F194A336 for ; Thu, 29 Jan 2015 18:56:43 +0000 (UTC) Received: from mail-wi0-f170.google.com (exprod7og120.obsmtp.com [64.18.2.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 24EF47C9 for ; Thu, 29 Jan 2015 18:56:42 +0000 (UTC) Received: from mail-wi0-f170.google.com ([209.85.212.170]) (using TLSv1) by exprod7ob120.postini.com ([64.18.6.12]) with SMTP ID DSNKVMqCY1801xSI/hEjSj5mCcP7dGHqSqEM@postini.com; Thu, 29 Jan 2015 10:56:43 PST Received: by mail-wi0-f170.google.com with SMTP id bs8so13989814wib.1 for ; Thu, 29 Jan 2015 10:56:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:mime-version:thread-index:date:message-id :subject:to:content-type; bh=4hZiQziQI1ABO59Noq2XGrtNhEpU/BSC9pL7SxSyMwk=; b=ERbcsyBo1V0cU1VA5kJFf9fjvoImXGZ+kITFjFYIRuidToCkDSG2biXeBr1sbwjUbN 4jkPiyO2nvi6DihCw/gltMc9kq4DjnMK/Bx9Majqd52gTcmduD0wJx7RDX34/oGvpgMn M+T6AbweUE+KvsqAiMRJbum8mD/F4+4dBpnU2hbMZ1DPHEriSXOkT/ISUqgZdk6HV1C+ sOfZDXzRlz7Fp/E0exC/4eqya1uwCAdmWUvE1JlgZEjgV5a31UhG+AkCDh6WVOfkbfn5 PLyciq8oRFfqqx2URs3/WZjxJ3A2EpH3Msey3CJnG085SPmTR7lwsTR6sV+zk10eWHbI 4E3A== X-Gm-Message-State: ALoCoQmOLVxWRSATJ3ii7adzPjEBwnSAuJtUp070uzS7wkCcHaGxCin/ABFcUWt2t1WH+8vljW3I0jAlz1b6lKyTFLSdshYjt877UGQjFk0xcw9qea7k0yWU9srTttDi5g7JQgV7dEPStF3rNzA83b3R4RALBk6uUg== X-Received: by 10.180.98.3 with SMTP id ee3mr4183287wib.12.1422557795371; Thu, 29 Jan 2015 10:56:35 -0800 (PST) X-Received: by 10.180.98.3 with SMTP id ee3mr4183262wib.12.1422557795153; Thu, 29 Jan 2015 10:56:35 -0800 (PST) From: Sibananda Sahu MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AdA79UEB0PauHoWcTx2d/YXUH38PpQ== Date: Fri, 30 Jan 2015 00:26:33 +0530 Message-ID: Subject: How to send 1MB I/O size in a single I/O request without split To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2015 18:56:44 -0000 Hi All, Recently we have added large I/O size of 1MB in our LSI controller and for that we have implemented the same in driver. But I have observed that the large I/O that an application is able to send is 128KB in one I/O request. I used the following command to send 1MB I/O: # ddpt if=/dev/da0 of=/dev/zero count=1 bs=1M But I have observed that the number of scatter gather elements per I/O request always comes 1 and the I/O length comes as 128KB(max). How can I get an I/O request from an application that will send 256 scatter gather elements of size 4K each(which makes a single 1MB I/O request). I have seen this kind of request in LINUX, but in FreeBSD I have received only 1 sge count and 128KB max I/O length. mrsas(4) driver reports to CAM layer that it supports 1MB I/O size. ccb->cpi.maxio = sc->max_num_sge * MRSAS_PAGE_SIZE; [Final value that is reported is (256 * 4096)] After a little investigation I found the following in sys/cam/scsi/scsi_da.c if (cpi.maxio == 0) softc->disk->d_maxsize = DFLTPHYS; /* traditional default */ else if (cpi.maxio > MAXPHYS) softc->disk->d_maxsize = MAXPHYS; /* for safety */ else softc->disk->d_maxsize = cpi.maxio; So even if the controller supports max I/O size greater than 128KB, it is restricted to 128KB only. I have changed the value of MAXPHYS from (128 * 1024) to (1024 * 1024) just to see if I can get 1MB I/O request to driver and simultaneously whether the driver can process 1M I/O size or not. I observed that after this modification I am able to get 1MB I/O request from the application. Number of sge is still 1 but the I/O length is now 1MB. Further information: Mrsas(4) driver supports UNMAPPED I/O. I have exercised the same thing with UNMAPPED I/O disabled but the result was same. So my primary questions are: - How can I send a large I/O size of 1MB in a single I/O request without any split? - Why I am getting always 1 scatter gather element? - How can I get more sge count in an I/O request? Thanks, Sibananda Sahu Device driver developer @ AVAGO Technologies. From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 29 20:17:15 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CE75F786 for ; Thu, 29 Jan 2015 20:17:15 +0000 (UTC) Received: from nm19-vm1.bullet.mail.gq1.yahoo.com (nm19-vm1.bullet.mail.gq1.yahoo.com [98.136.217.24]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97E48260 for ; Thu, 29 Jan 2015 20:17:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1422562628; bh=mx/1ih0Fd0gSjZl6GFX0YNzVTIni1gp6/p2gI1dNYHo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=LKWHArii1XA+YA2ct2eeBhXy5vJgIzeTVCEfYD/wzFgQw04d3xkirBA5Gqgn2rkvH7TkDCGP9+EQiC30tAp5J7sdGzi9TgllaL8gyDVZshADYJDhIfYUgYtMPSbs1PW4vEkYFSefHot+bbiRnMXyTgBqqgg44hYSnmb2ydsXXvVh6MSZMHeX6Y03SaLrcB0Q5BCe7sZjNiOmY3Gla3qXuwog1NM5IoN2fttPr1WvJh3YiEbANY5JxsbqG+xH4VYW9BEGQSUqkgiG+enNudr4mXrFu2PIHOvHguJUyXEKDVL2ejlgfub9IzjBGvLMutZPhbjhRvb3NvYI72r0SgFyYA== Received: from [98.137.12.55] by nm19.bullet.mail.gq1.yahoo.com with NNFMP; 29 Jan 2015 20:17:08 -0000 Received: from [208.71.42.202] by tm15.bullet.mail.gq1.yahoo.com with NNFMP; 29 Jan 2015 20:17:08 -0000 Received: from [127.0.0.1] by smtp213.mail.gq1.yahoo.com with NNFMP; 29 Jan 2015 20:17:08 -0000 X-Yahoo-Newman-Id: 286312.33753.bm@smtp213.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: ZxXRApgVM1lwAkM8fHn0tNk.UJOvi748AfHXORNdIi0tAtR p5W_q2fxlzOBj1au42bCffu5p4sxGG03qfQje3YFzp.oJSh8uuoJM.2Ce90P _2H1pdJrKO_Toy9HuFyWWuJTdq2v3NRZ9pgBN03gYoxRwZSs5wejJDoVQmSn AzAOcuF_mErjkhj2.X7dJc3rZTlanLHy8sCRcfLX1tvK.JC6u2WGX9h8wiWQ gIw8siO3LJQTQoQZ6AF3EiDAKzSMt66wrPKAs6t9VMpOC2dIZ4hDSb3yzZI7 jXeAczomIn97yW3nn8M4As6PYB62ef0gTEhK_NSEacwnrAhaBKTH3qNT4GRM 8OsUPS3CJLRcvXJCszs.ziC2AgzDTUWuCmV6V8QUqIAYcod5E_DWzSEIjLL0 9uwee3M1zYXXjI15jc2xy_oktRqTvQsn.xZDz5BC6CG06PRr.o3R6RBBzqlS 1e5.BVutpQ.7stzCPcFoiOv9zGOQrMIQrtKVGFO2VRpVqlw8AuTR4Ta08ljq KlUBbnAle38a1tiKdEmE8hERR9I2w56gkZebnYxDeJQ-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: How to send 1MB I/O size in a single I/O request without split From: Scott Long In-Reply-To: Date: Thu, 29 Jan 2015 13:17:06 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> References: To: Sibananda Sahu X-Mailer: Apple Mail (2.2070.6) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2015 20:17:16 -0000 > On Jan 29, 2015, at 11:56 AM, Sibananda Sahu = wrote: >=20 > Hi All, >=20 >=20 >=20 > Recently we have added large I/O size of 1MB in our LSI controller and = for > that we have implemented the same in driver. >=20 > But I have observed that the large I/O that an application is able to = send > is 128KB in one I/O request. >=20 >=20 >=20 > I used the following command to send 1MB I/O: >=20 > # ddpt if=3D/dev/da0 of=3D/dev/zero count=3D1 bs=3D1M >=20 >=20 >=20 > But I have observed that the number of scatter gather elements per I/O > request always comes 1 and the I/O length comes as 128KB(max). >=20 > [=E2=80=A6] >=20 >=20 > So my primary questions are: >=20 > - How can I send a large I/O size of 1MB in a single I/O = request > without any split? You answered this question already, you must redefine MAXPHYS. This can = be done via a kernel compile option, and many users and companies = already know how to do this. I plan to start a discussion on increasing = the default size. >=20 > - Why I am getting always 1 scatter gather element? >=20 You are likely getting an allocation that is physically contiguous. = This is especially true since you are using =E2=80=98dd=E2=80=99 from = userland, and the memory allocator in userland tries to useon 2MB = superpages for allocations. The busdma API will see that the allocation = is contiguous and attempt to merge the contiguous segments. This is = usually desirable since few segments reduces processing overhead in the = driver and the hardware. > - How can I get more sge count in an I/O request? >=20 If you want to test multiple segments, I suggest leaving you system = running for a long time with multiple processes freeing and allocating = memory so that the system becomes fragmented. You can also modify the = bus_dma_tag in your driver to specify a maximum segment size of 4k = instead of (presumably in your case) something larger. That will force = busdma to stop merging adjacent segments. Scott From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 05:55:18 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 386CC95C for ; Fri, 30 Jan 2015 05:55:18 +0000 (UTC) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09851619 for ; Fri, 30 Jan 2015 05:55:18 +0000 (UTC) Received: by mail-pa0-f52.google.com with SMTP id kx10so48165826pab.11 for ; Thu, 29 Jan 2015 21:55:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=7YSG4WD0GL1d3u4FihTk+bdql8QKqJ9DdqUZ00SM11Y=; b=sZydAJpjwzyA8bL+wkTKGJVK9Hmz2PFLEw5GeZpAwVoR/Yd4TZsLxZVd4QuOz1c16u hOcszMHzeMRlOcj8HV5IhV8bzwroMjzF+1iYE4QHw4hmBLG0zgOcDRnEbRNWWJrfoRFe PhnJ/Vi2eT9fMLRjT9EnjRmfmXlUu2V1QYbqykaX9lKpDwxwONarpiIQKdKYkGaV9aXl Ew7SYmBL10fCyc9d8+/496tH+t0rOEYt29NYRsybsGeWpGfPLvZ/efLwrXQOvfw/7HEa wMzsKRQAl8Ped7nCHOoHr98bwyfM62f1J5W0jvXYGZBZthZinJlfUCbN+V6lNIT/UsOd EWwg== MIME-Version: 1.0 X-Received: by 10.70.38.239 with SMTP id j15mr6330055pdk.115.1422597317648; Thu, 29 Jan 2015 21:55:17 -0800 (PST) Received: by 10.70.49.34 with HTTP; Thu, 29 Jan 2015 21:55:17 -0800 (PST) Date: Fri, 30 Jan 2015 13:55:17 +0800 Message-ID: Subject: bugfix: SA driver will unwind when close device even with SA_MODE_NOREWIND From: Peter Xu To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 05:55:18 -0000 Hi, In BSD system, when tape devices are discovered, CAM will create specific /dev/saX.[0-3] automatically. Here when we does not the tape to auto rewind when closing the device, we should access /dev/saX.1 (which means we are using SA_MODE_NOREWIND mode). Found one bug that tape will auto rewind even accessing /dev/saX.1. diff --git a/sys/cam/scsi/scsi_sa.c b/sys/cam/scsi/scsi_sa.c index 83b8345..55ee293 100644 --- a/sys/cam/scsi/scsi_sa.c +++ b/sys/cam/scsi/scsi_sa.c @@ -186,8 +186,8 @@ typedef enum { "\007NO_MODESEL" \ "\010NO_CPAGE" -#define SAMODE(z) (dev2unit(z) & 0x3) -#define SADENSITY(z) ((dev2unit(z) >> 2) & 0x3) +#define SAMODE(z) ((dev2unit(z) >> 2) & 0x3) +#define SADENSITY(z) (dev2unit(z) & 0x3) #define SA_IS_CTRL(z) (dev2unit(z) & (1 << 4)) #define SA_NOT_CTLDEV 0 Thanks. Peter From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 09:42:06 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03396A79 for ; Fri, 30 Jan 2015 09:42:06 +0000 (UTC) Received: from mail-wg0-f50.google.com (exprod7og128.obsmtp.com [64.18.2.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 77FF5E0B for ; Fri, 30 Jan 2015 09:42:05 +0000 (UTC) Received: from mail-wg0-f50.google.com ([74.125.82.50]) (using TLSv1) by exprod7ob128.postini.com ([64.18.6.12]) with SMTP ID DSNKVMtR5VGrpfhom74BQVlrbAxcs/2Fawrw@postini.com; Fri, 30 Jan 2015 01:42:05 PST Received: by mail-wg0-f50.google.com with SMTP id b13so25749937wgh.9 for ; Fri, 30 Jan 2015 01:41:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=uwhquk7V8zzNmzamNtIoVKcLPlYr2NYhWDdwNuZ8cPU=; b=hPeG6PRGgWUW3j7T/Hhv6yIYzpLP2NYLdPgDwe3aY/rC0by9lztkhg+v3nyJZPURnS V73tiwBBMQEIZsv9qGc+yfxioyCKWy0qh+8mf8K3COEYXVH5+ELr8pt/e0hugyCUVMaX pW6dumrjgh+e8mPjeDP/UJYEfHg4KyNaURP0ljAuy2/YN7crwcQvqHOcMWM/ctwSxoUg 23yN2W+jywuqLfR4BxHldcBbs2I0gw35jxWGPhtc+5Ldy19qUOQ81MLeUmvgGig5ZCE7 yEmdFwJkURORlIUiFcLudCHlb8WLSwwN7gvS9LOUUNHKDKY9P8vYg0vnsikV1BYmE/2l l+Jg== X-Gm-Message-State: ALoCoQkgkHIXMIAKMqjXTUMVEsOUN9FSdItGHtngtyYSN02sOwlnlq69UH0ByYTr2N91KWJlApVrOtDEwZtW5GtuNTbgpw0reOD/1GfvQx6JjlK9jT/tvqrhxuEFxWWWVoWrddL8RQiGY6pWIopUkBeKG2rLaQ+j2A== X-Received: by 10.194.200.1 with SMTP id jo1mr10505303wjc.64.1422610917611; Fri, 30 Jan 2015 01:41:57 -0800 (PST) X-Received: by 10.194.200.1 with SMTP id jo1mr10505262wjc.64.1422610917368; Fri, 30 Jan 2015 01:41:57 -0800 (PST) From: Sibananda Sahu References: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> In-Reply-To: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQJ90Y9Ud/YDW7817ZAJGKXgu7SlNwGkTx3Um3AbLDA= Date: Fri, 30 Jan 2015 15:11:56 +0530 Message-ID: <923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com> Subject: RE: How to send 1MB I/O size in a single I/O request without split To: Scott Long Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 09:42:06 -0000 Hey Scoot, Thanks very much for your reply. >> - How can I get more sge count in an I/O request? >> >If you want to test multiple segments, I suggest leaving you system runnin= g >for a long time with multiple processes freeing and allocating memory so >that the system becomes >fragmented. You can also modify the bus_dma_tag >in your driver to specify a maximum segment size of 4k instead of >(presumably in your case) something larger. That will force >busdma to >stop merging adjacent segments. As you have said: "The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segment= s reduces processing overhead in the driver and the hardware." If I will modify the bus_dma_tag in our driver to have a max segment of 4K then what is the performance impact? Thanks, Sibananda Sahu -----Original Message----- From: Scott Long [mailto:scott4long@yahoo.com] Sent: Friday, January 30, 2015 1:47 AM To: Sibananda Sahu Cc: freebsd-scsi@freebsd.org Subject: Re: How to send 1MB I/O size in a single I/O request without split > On Jan 29, 2015, at 11:56 AM, Sibananda Sahu > wrote: > > Hi All, > > > > Recently we have added large I/O size of 1MB in our LSI controller and > for that we have implemented the same in driver. > > But I have observed that the large I/O that an application is able to > send is 128KB in one I/O request. > > > > I used the following command to send 1MB I/O: > > # ddpt if=3D/dev/da0 of=3D/dev/zero count=3D1 bs=3D1M > > > > But I have observed that the number of scatter gather elements per I/O > request always comes 1 and the I/O length comes as 128KB(max). > > [=E2=80=A6] > > > So my primary questions are: > > - How can I send a large I/O size of 1MB in a single I/O request > without any split? You answered this question already, you must redefine MAXPHYS. This can be done via a kernel compile option, and many users and companies already know how to do this. I plan to start a discussion on increasing the default size. > > - Why I am getting always 1 scatter gather element? > You are likely getting an allocation that is physically contiguous. This i= s especially true since you are using =E2=80=98dd=E2=80=99 from userland, and= the memory allocator in userland tries to useon 2MB superpages for allocations. The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segments reduces processing overhead in the driver and the hardware. > - How can I get more sge count in an I/O request? > If you want to test multiple segments, I suggest leaving you system running for a long time with multiple processes freeing and allocating memory so that the system becomes fragmented. You can also modify the bus_dma_tag in your driver to specify a maximum segment size of 4k instead of (presumably in your case) something larger. That will force busdma to stop merging adjacent segments. Scott From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 10:18:37 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 563167B5 for ; Fri, 30 Jan 2015 10:18:37 +0000 (UTC) Received: from mail-we0-f174.google.com (exprod7og126.obsmtp.com [64.18.2.206]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF6FE1DF for ; Fri, 30 Jan 2015 10:18:35 +0000 (UTC) Received: from mail-we0-f174.google.com ([74.125.82.174]) (using TLSv1) by exprod7ob126.postini.com ([64.18.6.12]) with SMTP ID DSNKVMtae1E5JOshFg+Iepok+f6GPt9dRk4Q@postini.com; Fri, 30 Jan 2015 02:18:37 PST Received: by mail-we0-f174.google.com with SMTP id w55so20869039wes.5 for ; Fri, 30 Jan 2015 02:18:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=JDbkutA32hn0/cgY9QOdwMb5lzF0llOxaz5vY9vKloc=; b=kFXXAOSBEC843g7QZ+ONev/YsY+y/4tlW5IA9D0qvjGDbZo6aEoIXSFns5NwTEJTTR EKuJN+iOD6p/LXCGhxyU2XIkHpy89B/M4+kU2PweVRnIl7Yz0a5fLj+S4fVUlsH0EKrs w8oNJfGs/BnkqnFuEdhEKiWszaqp02F7AK+fBHcfikNz6hsHqt7g1DktqYw8MXJr6B9N RiAw27vw5zPl2wGbSnhNgveW21b0H20fL3q7F9rtK4EC/b1yV0Vh7RK1Aw3NGOM1Cz0s ZCrH6IpGiY0YMEGKpouXjH68qH4IFktah1FlfA4hCduXN5T9FHt2B2s+0PYurjQdGFAz xs9g== X-Gm-Message-State: ALoCoQk8w5I+alb1BzVQYCY4oTselsakWVNOBKsBruar02Yq16wIYWzvYhRPoWDweG9hXYVPCTWDBPMplLdnMVAzBPtwYdIjg00yxjEO10VNvSoslYuMVU6NDkrJRWMOhnOHUQqmpgE24Q1eKBnmEuxqQi0aNuIVSQ== X-Received: by 10.194.143.12 with SMTP id sa12mr10443337wjb.101.1422613114834; Fri, 30 Jan 2015 02:18:34 -0800 (PST) X-Received: by 10.194.143.12 with SMTP id sa12mr10443306wjb.101.1422613114627; Fri, 30 Jan 2015 02:18:34 -0800 (PST) From: Sibananda Sahu References: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> 923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com In-Reply-To: 923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQJ90Y9Ud/YDW7817ZAJGKXgu7SlNwGkTx3Um3AbLDCAAATKwA== Date: Fri, 30 Jan 2015 15:48:33 +0530 Message-ID: <37dd5147bdbf0ea417d2fdfb31565358@mail.gmail.com> Subject: RE: How to send 1MB I/O size in a single I/O request without split To: Scott Long Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 10:18:37 -0000 Hi Scott, One more thing to ask. I think even if I will modify the bus_dma_tag in our driver to have a max segment of 4K, the max I/O length per I/O request that will come is 128KB because of the MAXPHYS limitation. I have tested this by restricting the MAXPHYS to 128KB again and when I requested an I/O with bs=3D1M the I/O was split with various length I/O requests. Does this implies that tuning the MAXPHYS is the ultimate solution to get larger I/O size?? Thanks, Sibananda -----Original Message----- From: Sibananda Sahu [mailto:sibananda.sahu@avagotech.com] Sent: Friday, January 30, 2015 3:12 PM To: 'Scott Long' Cc: 'freebsd-scsi@freebsd.org' Subject: RE: How to send 1MB I/O size in a single I/O request without split Hey Scoot, Thanks very much for your reply. >> - How can I get more sge count in an I/O request? >> >If you want to test multiple segments, I suggest leaving you system runnin= g >for a long time with multiple processes freeing and allocating memory so >that the system becomes >fragmented. You can also modify the bus_dma_tag >in your driver to specify a maximum segment size of 4k instead of >(presumably in your case) something larger. That will force >busdma to >stop merging adjacent segments. As you have said: "The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segment= s reduces processing overhead in the driver and the hardware." If I will modify the bus_dma_tag in our driver to have a max segment of 4K then what is the performance impact? Thanks, Sibananda Sahu -----Original Message----- From: Scott Long [mailto:scott4long@yahoo.com] Sent: Friday, January 30, 2015 1:47 AM To: Sibananda Sahu Cc: freebsd-scsi@freebsd.org Subject: Re: How to send 1MB I/O size in a single I/O request without split > On Jan 29, 2015, at 11:56 AM, Sibananda Sahu > wrote: > > Hi All, > > > > Recently we have added large I/O size of 1MB in our LSI controller and > for that we have implemented the same in driver. > > But I have observed that the large I/O that an application is able to > send is 128KB in one I/O request. > > > > I used the following command to send 1MB I/O: > > # ddpt if=3D/dev/da0 of=3D/dev/zero count=3D1 bs=3D1M > > > > But I have observed that the number of scatter gather elements per I/O > request always comes 1 and the I/O length comes as 128KB(max). > > [=E2=80=A6] > > > So my primary questions are: > > - How can I send a large I/O size of 1MB in a single I/O request > without any split? You answered this question already, you must redefine MAXPHYS. This can be done via a kernel compile option, and many users and companies already know how to do this. I plan to start a discussion on increasing the default size. > > - Why I am getting always 1 scatter gather element? > You are likely getting an allocation that is physically contiguous. This i= s especially true since you are using =E2=80=98dd=E2=80=99 from userland, and= the memory allocator in userland tries to useon 2MB superpages for allocations. The busdma API will see that the allocation is contiguous and attempt to merge the contiguous segments. This is usually desirable since few segments reduces processing overhead in the driver and the hardware. > - How can I get more sge count in an I/O request? > If you want to test multiple segments, I suggest leaving you system running for a long time with multiple processes freeing and allocating memory so that the system becomes fragmented. You can also modify the bus_dma_tag in your driver to specify a maximum segment size of 4k instead of (presumably in your case) something larger. That will force busdma to stop merging adjacent segments. Scott From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 16:30:16 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07C2B95F for ; Fri, 30 Jan 2015 16:30:16 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id C9049155 for ; Fri, 30 Jan 2015 16:30:15 +0000 (UTC) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t0UGPD44012311 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 30 Jan 2015 16:25:14 GMT Date: Fri, 30 Jan 2015 16:25:13 +0000 From: Karl Pielorz To: freebsd-scsi@freebsd.org Subject: iSCSI target discovery... Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 16:30:16 -0000 Hi, I'm using iSCSI under FreeBSD 10.1-R (ctladm/iscsid et'al). Is there a command I can run that will dump a list of available targets on a remote host / portal? 'iscsictl -d' appears to do this - but also says 'After discovery is done, sessions will be added for each discovered target...' I just want something that will dump a list of available targets on a remote host? Sorry if it's obvious and I've missed it :( Thanks, -Karl From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 17:38:12 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 66E5B7D7 for ; Fri, 30 Jan 2015 17:38:12 +0000 (UTC) Received: from nm5-vm6.bullet.mail.gq1.yahoo.com (nm5-vm6.bullet.mail.gq1.yahoo.com [98.136.218.181]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2E7A2DE1 for ; Fri, 30 Jan 2015 17:38:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1422639485; bh=gZN+8ucR1i4g2/dsFmDTqkeuCHoOqABfeN+PI0e20zA=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=i8S90nCjRohkX5/spUSGDiFEs2KCGAsxP5pkNVY458A1zzh8/ewQIGEeH/KDVwiELiE9/es397LnF3Yps1Q0oO/n0W/ycU7AeM/Y35na53vuiWX7LCksXhc7cGxpmcfaRwn7w3GAgeGfuNn64MUIqk2I1Mfu/Y7jlfiv1MvCTcdiApmptwOCLsg1vZ66ov5BhJKZiz7RRfI3NTRl56pMNxwrOhQYjHhZBjX1rymrgsqZ28EsBolGx24aQBbwvM1eNiScjHgvF+SSpIjYp3nxuSJdeY6Dx0klMu8U1Al1eyhmbEMwIzRDHM7XwLYGiXAbbq/HoYxVwvVg3RSTHkAOyw== Received: from [98.137.12.59] by nm5.bullet.mail.gq1.yahoo.com with NNFMP; 30 Jan 2015 17:38:05 -0000 Received: from [208.71.42.193] by tm4.bullet.mail.gq1.yahoo.com with NNFMP; 30 Jan 2015 17:38:05 -0000 Received: from [127.0.0.1] by smtp204.mail.gq1.yahoo.com with NNFMP; 30 Jan 2015 17:38:05 -0000 X-Yahoo-Newman-Id: 233747.20230.bm@smtp204.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: Y.I3qTQVM1kedMIZI6pTM5iFvXANAN27AEQcHItcDGnrgmb bP6FKu93HP1GrdDciR2kaGcX3GStopvk.VMCrGSYZzxqIBEG4b9DjEynhCP8 JS6blZc9AzSdC7sNISbqCuVG.ZBMVFEZSuqorIkSmIPi7N1FteZAoTGiP.lh bzFkR0gphz4aLKcLhbOySRoV3uXMFB8s6ZVqTbyLQz37iMDDjOZDO.bijjwE jrY.69_IKo.vtRbG6FGqP1SFs6Sm4fw7n7vVayNTZOtNs.pLQJdq1VtEnNiq IcvLx3hijru.tDG.KSyFAL9OoNuPPfY4N50sgDOasMQFz8tS6VxK2M.Z4qQK uPlAwJANagYt_n.pgbTCyzOFY2uXoTmLz0yJXq4Ku4eRGdWNJD0CDHdxKSgN _K05JxwLAdG8GGpWa9NTbe_Op4vdxIqikueXw_EPbs2K9RqbJN7RfVaFw0dN awtbMWgGtC9J2vLzQ6F_W1MnEserf7a1u71V4rb29hZo4bpnnnsLCsC2YtZs r0WsHQ8SevZ6nea3tY4eALGTsyPf_3Ka_S49NAMrqng-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: How to send 1MB I/O size in a single I/O request without split From: Scott Long In-Reply-To: <37dd5147bdbf0ea417d2fdfb31565358@mail.gmail.com> Date: Fri, 30 Jan 2015 10:38:03 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> <923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com> <37dd5147bdbf0ea417d2fdfb31565358@mail.gmail.com> To: Sibananda Sahu X-Mailer: Apple Mail (2.2070.6) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 17:38:12 -0000 > On Jan 30, 2015, at 3:18 AM, Sibananda Sahu = wrote: >=20 > Hi Scott, >=20 > One more thing to ask. >=20 > I think even if I will modify the bus_dma_tag in our driver to have a = max > segment of 4K, the max I/O length per I/O request that will come is = 128KB > because of the MAXPHYS limitation. > I have tested this by restricting the MAXPHYS to 128KB again and when = I > requested an I/O with bs=3D1M the I/O was split with various length = I/O > requests. >=20 > Does this implies that tuning the MAXPHYS is the ultimate solution to = get > larger I/O size?? Yes. You will need to rebuild your kernel with the larger size. Some = day we might default to a larger size, or make it dynamic. Scott From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 17:42:16 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 61C3C95A for ; Fri, 30 Jan 2015 17:42:16 +0000 (UTC) Received: from nm21-vm4.bullet.mail.gq1.yahoo.com (nm21-vm4.bullet.mail.gq1.yahoo.com [98.136.217.51]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 29DC1EAB for ; Fri, 30 Jan 2015 17:42:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1422639735; bh=E1A7btyKeHDLVrN2HUVpvD21PvL4QD1xVMoC7nPtEnI=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=J4EbAANKxIzn8ed0FtY5zzMl6D5cTsXtrLcqUy3Ual9+801ZyZXZHEcO0Bpg4MaK5JjPVtXhMu6F7cx2ArqtO/sBf294WXF6F+v2mFjKe+6Ly6gLWJ9HAXJt5lP1X3T7+BCZaf3k4KgVK1hGdsbi8IvJh7tCzOk6rvPUoTQfoGp825NCHeQCL1JErvIxlXVrZ726BKV4DoNvdIXy4TJjmuK5+Ol+lfeUIJlel8GN+orbl9cF9hr7imNzuPNGf7GzIH12DnxbpK4FO+9rfLJuxRTdVFpep+jaT+2XwXbWK+7X27mp8PHWwE8U1D9kM6vL39IIWrio/+09ZHGswSzICA== Received: from [98.137.12.188] by nm21.bullet.mail.gq1.yahoo.com with NNFMP; 30 Jan 2015 17:42:15 -0000 Received: from [208.71.42.193] by tm9.bullet.mail.gq1.yahoo.com with NNFMP; 30 Jan 2015 17:42:15 -0000 Received: from [127.0.0.1] by smtp204.mail.gq1.yahoo.com with NNFMP; 30 Jan 2015 17:42:15 -0000 X-Yahoo-Newman-Id: 101408.20218.bm@smtp204.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: bbeqlXcVM1mWt43134G3T0cR4O_ekzDvR_MVYx3LoDdrJWw uXUvv3rgZ6R8CBGrYCsjWHAMfJGOs8J978Hg2G7moOn5auoBoHgdzOd.ywEX Naxn0q4torsLLrczTf3mloig8KxtaUchSK8zh995REezbdoH88P4Dq9sSaXN d_Neg1GmrxTcFtkpULFQdw_fcjGcYTT5yP0_zoeaBHRbboQfHfUXVGRBM9vg Z78k6KIPNd.sD3P4zXpvSEPJA3Y6QIE878IypoAW24dxNhYCAcwQBBgghsig .6FZrxs3bxmYISONeVBo4HemXaWUwvrW_427xweh6bU15Gvy_mfEliYOZIcb NwGrRemBt9MsTccD9CuBOuPSEEysDwPWf6mdKG0ZGhetFyZ_djdUwO_KGofz KMj1ioAUyR91JnWSe11_XjAvv9N1cwI_2YHSuSbgUXqcdj5noaUpKxhvS0ef g52ohGp2WyiRgXWE0sS5Ms4dVJL8nHGw_OeyQhQvLdDfJsJ1uGYBT.TD6P2W VXbkMjhLjSZVrsLdA6kMJBsXxS5L27LVSK7n2D3NiZtw- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: How to send 1MB I/O size in a single I/O request without split From: Scott Long In-Reply-To: <923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com> Date: Fri, 30 Jan 2015 10:42:11 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <840B5221-A5E5-4FBC-BA2A-F44EF4036C59@yahoo.com> References: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> <923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com> To: Sibananda Sahu X-Mailer: Apple Mail (2.2070.6) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 17:42:16 -0000 > On Jan 30, 2015, at 2:41 AM, Sibananda Sahu = wrote: >=20 > Hey Scoot, >=20 > Thanks very much for your reply. >=20 >>> - How can I get more sge count in an I/O request? >>>=20 >=20 >> If you want to test multiple segments, I suggest leaving you system = running >> for a long time with multiple processes freeing and allocating memory = so >> that the system becomes >fragmented. You can also modify the = bus_dma_tag >> in your driver to specify a maximum segment size of 4k instead of >> (presumably in your case) something larger. That will force >busdma = to >> stop merging adjacent segments. >=20 > As you have said: >=20 > "The busdma API will see that the allocation is contiguous and attempt = to > merge the contiguous segments. This is usually desirable since few = segments > reduces processing overhead in the driver and the hardware." >=20 > If I will modify the bus_dma_tag in our driver to have a max segment = of 4K > then what is the performance impact? >=20 Most drivers have a for() loop that iterates through the list of = segments from busdma and stores them into SGE elements for the hardware to use. = Obviously, fewer segments means fewer trips through this loop. It also means a = smaller SGE list, which means less command data transferred over the PCI bus. = Though I cannot speak for the architecture details of your hardware, often = times the same optimizations apply to the hardware internally. With modern storage = controllers that are doing millions of i/o=E2=80=99s per second, these small = improvements can add up. Scott From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 30 18:20:11 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6C5C182D for ; Fri, 30 Jan 2015 18:20:11 +0000 (UTC) Received: from mail-we0-f169.google.com (exprod7og119.obsmtp.com [64.18.2.16]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E44B4355 for ; Fri, 30 Jan 2015 18:20:10 +0000 (UTC) Received: from mail-we0-f169.google.com ([74.125.82.169]) (using TLSv1) by exprod7ob119.postini.com ([64.18.6.12]) with SMTP ID DSNKVMvLUx49o3WJZPhOHKYMsHmX7gektDdW@postini.com; Fri, 30 Jan 2015 10:20:11 PST Received: by mail-we0-f169.google.com with SMTP id u56so28670659wes.0 for ; Fri, 30 Jan 2015 10:20:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=DWZ7l06K5A2HurH3wcKjDAQSp8LJnmF3tWclMFjgPnk=; b=iE2tWAZrlia33ALll9NJ7tmaygymeCRwhQR51mYrOeKH07ONv6mVABAST69CAEf9Hl q2rntITbZNjhb4jow7DnoQU3xZQHplsXNC7pZZ4xDtPSD+/qppt8Y5aDGwzrDHv4XQBh Dy9utraTq7l5IRAJioQpn413NSqm55X9ezFhTSTBoiGBbQzc0S8jOA36Cc9zQ5VWkAHm wZiG0uxJPcUGHK3481HeHBZUrLuCVEm+HA9Ymnvx443AdSXjC9Kv60sZIdrCiJe+1hF9 wRBm2BHzXpuzXtD/YKhh9trkt/hFc6vdewHUsgzM8Jt6QQ2LgALpQCLk9ZB2WX3UcNEZ H4qw== X-Gm-Message-State: ALoCoQm4UpnJdBHefvhePTmbK9fvA5Xl+ofIGDjyEdxE2yYZDc22YlqATOMPwtsZUQZ8TulbKcpiUQ0z4zniSQa2d+y/cdanhizjtUu3Km05alKcBMdTq31Qpl2E7xmn8T11Yw0lY6sAgnmHCtwfrcWmmNtRGFe/9Q== X-Received: by 10.194.143.12 with SMTP id sa12mr14809999wjb.101.1422642002813; Fri, 30 Jan 2015 10:20:02 -0800 (PST) X-Received: by 10.194.143.12 with SMTP id sa12mr14809964wjb.101.1422642002630; Fri, 30 Jan 2015 10:20:02 -0800 (PST) From: Sibananda Sahu References: <8B56B74C-7EBC-4D1B-89AB-46DA8ED05DD5@yahoo.com> <923e4c2e65d29f2f39e0aa2f6d4ab38a@mail.gmail.com> <840B5221-A5E5-4FBC-BA2A-F44EF4036C59@yahoo.com> In-Reply-To: <840B5221-A5E5-4FBC-BA2A-F44EF4036C59@yahoo.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQJ90Y9Ud/YDW7817ZAJGKXgu7SlNwGkTx3UAb2EBL4BaZa2BptXc+jA Date: Fri, 30 Jan 2015 23:50:02 +0530 Message-ID: <7a375283987fdbae4d4b1db58021c9e4@mail.gmail.com> Subject: RE: How to send 1MB I/O size in a single I/O request without split To: Scott Long Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jan 2015 18:20:11 -0000 Thanks Scott for your great explanations. I really appreciate your time that helped me much in understanding the OS behavior. Thanks, Sibananda -----Original Message----- From: Scott Long [mailto:scott4long@yahoo.com] Sent: Friday, January 30, 2015 11:12 PM To: Sibananda Sahu Cc: freebsd-scsi@freebsd.org Subject: Re: How to send 1MB I/O size in a single I/O request without split > On Jan 30, 2015, at 2:41 AM, Sibananda Sahu > wrote: > > Hey Scoot, > > Thanks very much for your reply. > >>> - How can I get more sge count in an I/O request? >>> > >> If you want to test multiple segments, I suggest leaving you system >> running for a long time with multiple processes freeing and >> allocating memory so that the system becomes >fragmented. You can >> also modify the bus_dma_tag in your driver to specify a maximum >> segment size of 4k instead of (presumably in your case) something >> larger. That will force >busdma to stop merging adjacent segments. > > As you have said: > > "The busdma API will see that the allocation is contiguous and attempt > to merge the contiguous segments. This is usually desirable since few > segments reduces processing overhead in the driver and the hardware." > > If I will modify the bus_dma_tag in our driver to have a max segment > of 4K then what is the performance impact? > Most drivers have a for() loop that iterates through the list of segments from busdma and stores them into SGE elements for the hardware to use. Obviously, fewer segments means fewer trips through this loop. It also means a smaller SGE list, which means less command data transferred over th= e PCI bus. Though I cannot speak for the architecture details of your hardware, often times the same optimizations apply to the hardware internally. With modern storage controllers that are doing millions of i/o= =E2=80=99s per second, these small improvements can add up. Scott