From owner-freebsd-scsi@FreeBSD.ORG Mon Feb 11 16:12:20 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7BD17286; Mon, 11 Feb 2013 16:12:20 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog127.obsmtp.com (na3sys009aog127.obsmtp.com [74.125.149.107]) by mx1.freebsd.org (Postfix) with ESMTP id B61A0BF1; Mon, 11 Feb 2013 16:12:19 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob127.postini.com ([74.125.148.12]) with SMTP ID DSNKURkYYgRp8GqAW0NesCfc6pJa8DW0Ipkw@postini.com; Mon, 11 Feb 2013 08:12:20 PST Received: from PALHUB01.lsi.com (128.94.213.114) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.264.0; Mon, 11 Feb 2013 11:12:29 -0500 Received: from inbexch02.lsi.com (135.36.98.40) by PALHUB01.lsi.com (128.94.213.114) with Microsoft SMTP Server (TLS) id 8.3.264.0; Mon, 11 Feb 2013 11:12:17 -0500 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch02.lsi.com ([135.36.98.40]) with mapi; Mon, 11 Feb 2013 21:42:14 +0530 From: "Desai, Kashyap" To: "Kenneth D. Merry" Date: Mon, 11 Feb 2013 21:42:11 +0530 Subject: RE: Max Queue depth of HBA limited to 256 ? Thread-Topic: Max Queue depth of HBA limited to 256 ? Thread-Index: Ac4D5iOPnb8e42+1TjmX//UNm9rDLAEjBB1Q Message-ID: References: <20130121170529.GA64188@nargothrond.kdm.org> <20130205211642.GA75343@nargothrond.kdm.org> In-Reply-To: <20130205211642.GA75343@nargothrond.kdm.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-scsi@freebsd.org" , "McConnell, Stephen" , "jhb@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Feb 2013 16:12:20 -0000 Ken: I am still not able to pump more than 256 command. (This time I simulated s= imilar to your setup. I tried driver) For example: dev.mps.0.%desc: LSI SAS2008 dev.mps.0.%driver: mps dev.mps.0.%location: slot=3D0 function=3D0 dev.mps.0.%pnpinfo: vendor=3D0x1000 device=3D0x0072 subvendor=3D0x1000 subd= evice=3D0x0072 class=3D0x010700 dev.mps.0.%parent: pci2 dev.mps.0.debug_level: 0 dev.mps.0.disable_msix: 0 dev.mps.0.disable_msi: 0 dev.mps.0.firmware_version: 14.00.00.00 dev.mps.0.driver_version: 14.00.00.01-fbsd dev.mps.0.io_cmds_active: 256 dev.mps.0.io_cmds_highwater: 256 dev.mps.0.chain_free: 2048 dev.mps.0.chain_free_lowwater: 2047 dev.mps.0.max_chains: 2048 dev.mps.0.chain_alloc_fail: 0 io_cmds_highwater =3D 256 Here is the system information: at scbus0 target 10 lun 0 (pass3,da1) at scbus0 target 11 lun 0 (pass4,da2) at scbus0 target 12 lun 0 (pass5,da3) at scbus0 target 13 lun 0 (pass2,da0) at scbus0 target 14 lun 0 (pass6,= ses0) -- Expander Running 128 dd on each target using small shell scripts: --------------------------------------------------------- #!/bin/sh max=3D128 for i in `seq 1 $max` do dd if=3D/dev/da0 of=3D/dev/null bs=3D1m & dd if=3D/dev/da1 of=3D/dev/null bs=3D1m & dd if=3D/dev/da2 of=3D/dev/null bs=3D1m & dd if=3D/dev/da3 of=3D/dev/null bs=3D1m & echo "$i" done ---------------------------------------------------------------------------= ------------------ The individual drives logs:=20 dhcp-135-24-192-146# camcontrol tags da3 -v (pass5:mps0:0:12:0): dev_openings 248 (pass5:mps0:0:12:0): dev_active 7 (pass5:mps0:0:12:0): devq_openings 248 (pass5:mps0:0:12:0): devq_queued 0 (pass5:mps0:0:12:0): held 0 (pass5:mps0:0:12:0): mintags 2 (pass5:mps0:0:12:0): maxtags 255 dhcp-135-24-192-146# camcontrol tags da2 -v (pass4:mps0:0:11:0): dev_openings 155 (pass4:mps0:0:11:0): dev_active 100 (pass4:mps0:0:11:0): devq_openings 155 (pass4:mps0:0:11:0): devq_queued 0 (pass4:mps0:0:11:0): held 0 (pass4:mps0:0:11:0): mintags 2 (pass4:mps0:0:11:0): maxtags 255 dhcp-135-24-192-146# camcontrol tags da1 -v (pass3:mps0:0:10:0): dev_openings 145 (pass3:mps0:0:10:0): dev_active 110 (pass3:mps0:0:10:0): devq_openings 145 (pass3:mps0:0:10:0): devq_queued 0 (pass3:mps0:0:10:0): held 0 (pass3:mps0:0:10:0): mintags 2 (pass3:mps0:0:10:0): maxtags 255 dhcp-135-24-192-146# camcontrol tags da0 -v (pass2:mps0:0:13:0): dev_openings 233 (pass2:mps0:0:13:0): dev_active 22 (pass2:mps0:0:13:0): devq_openings 233 (pass2:mps0:0:13:0): devq_queued 0 (pass2:mps0:0:13:0): held 0 (pass2:mps0:0:13:0): mintags 2 (pass2:mps0:0:13:0): maxtags 255 Is there any change in latest Upstream kernel ? Mine is not very much lates= t FreeBSD OS..! I will try with latest upstream freebsd OS.=20 ` Kashyap > -----Original Message----- > From: Kenneth D. Merry [mailto:ken@freebsd.org] > Sent: Wednesday, February 06, 2013 2:47 AM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; jhb@freebsd.org; McConnell, Stephen > Subject: Re: Max Queue depth of HBA limited to 256 ? >=20 >=20 > I'm able to get more than 255 commands outstanding to the controller in > my configuration. >=20 > For example: >=20 > dev.mps.0.%desc: LSI SAS2116 > dev.mps.0.%driver: mps > dev.mps.0.%location: slot=3D6 function=3D0 handle=3D\_SB_.PCI0.S30_ > dev.mps.0.%pnpinfo: vendor=3D0x1000 device=3D0x0064 subvendor=3D0x1000 > subdevice=3D0x30c0 class=3D0x010700 > dev.mps.0.%parent: pci0 > dev.mps.0.debug_level: 4 > dev.mps.0.disable_msix: 0 > dev.mps.0.disable_msi: 0 > dev.mps.0.firmware_version: 13.00.01.00 > dev.mps.0.driver_version: 14.00.00.01-fbsd > dev.mps.0.io_cmds_active: 442 > dev.mps.0.io_cmds_highwater: 464 > dev.mps.0.chain_free: 354 > dev.mps.0.chain_free_lowwater: 181 > dev.mps.0.max_chains: 2048 > dev.mps.0.chain_alloc_fail: 0 >=20 > io_cmds_highwater is 464. Can you get more than 255 commands > outstanding if you use more than 1 target? >=20 > This is with 272 'dd' processes doing 1MB reads to 16 2TB and 3TB SAS > drives behind 2 3Gb Maxim expanders: >=20 > at scbus2 target 144 lun 0 > (pass4,sg4,da0) > at scbus2 target 145 lun 0 > (pass5,sg5,da1) > at scbus2 target 146 lun 0 > (pass6,sg6,da2) > at scbus2 target 147 lun 0 > (pass7,sg7,da3) > at scbus2 target 148 lun 0 > (pass8,sg8,da4) > at scbus2 target 149 lun 0 > (pass9,sg9,da5) > at scbus2 target 150 lun 0 > (pass10,sg10,da6) > at scbus2 target 151 lun 0 > (pass11,sg11,da7) > at scbus2 target 152 lun 0 > (pass12,sg12,da8) > at scbus2 target 153 lun 0 > (pass13,sg13,da9) > at scbus2 target 154 lun 0 > (pass14,sg14,da10) > at scbus2 target 155 lun 0 > (pass15,sg15,da11) > at scbus2 target 156 lun 0 > (pass16,sg16,da12) > at scbus2 target 157 lun 0 > (pass17,sg17,da13) > at scbus2 target 158 lun 0 > (pass18,sg18,da14) > at scbus2 target 159 lun 0 > (pass19,sg19,da15) >=20 > i.e. 17 iterations of this: >=20 > ((i=3D0)); while [ $i -le 15 ]; do dd if=3D/dev/da$i of=3D/dev/null bs=3D= 1m & > ((i++)); done >=20 > The individual drives see varying numbers of tags, but nowhere near the > maximum: >=20 > [root@storage-domain ~]# camcontrol tags da15 -v > (pass19:mps0:0:159:0): dev_openings 230 > (pass19:mps0:0:159:0): dev_active 25 > (pass19:mps0:0:159:0): devq_openings 230 > (pass19:mps0:0:159:0): devq_queued 0 > (pass19:mps0:0:159:0): held 0 > (pass19:mps0:0:159:0): mintags 2 > (pass19:mps0:0:159:0): maxtags 255 >=20 > What kind of drive is the target? >=20 > Ken >=20 > On Wed, Jan 23, 2013 at 00:44:31 +0530, Desai, Kashyap wrote: > > LSI h/w needs more outstanding command in FW to get better Perf counts > compare to other OS. > > > > Please suggest if whatever I have been observed is limitation from > FreeBSD or we can tune it in Driver ? > > My goals is to pump ~1000 outstanding IOs to the HBA. I see that it > never goes beyond 255. > > > > Thanks, > > Kashyap > > > > > -----Original Message----- > > > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > > > scsi@freebsd.org] On Behalf Of Desai, Kashyap > > > Sent: Monday, January 21, 2013 11:18 PM > > > To: Kenneth D. Merry > > > Cc: freebsd-scsi@freebsd.org; jhb@freebsd.org; McConnell, Stephen > > > Subject: RE: Max Queue depth of HBA limited to 256 ? > > > > > > > > > > > > > -----Original Message----- > > > > From: Kenneth D. Merry [mailto:ken@freebsd.org] > > > > Sent: Monday, January 21, 2013 10:35 PM > > > > To: Desai, Kashyap > > > > Cc: freebsd-scsi@freebsd.org; McConnell, Stephen; Saxena, Sumit; > > > > jhb@freebsd.org > > > > Subject: Re: Max Queue depth of HBA limited to 256 ? > > > > > > > > On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote: > > > > > Hi, > > > > > > > > > > I was trying to check few things on LSI controller, where we > > > > > have more > > > > than 256 queue depth support. > > > > > I added default maxtags in scsi/scsi_xpt.c as below. (Because I > > > > > don't > > > > want mattags to restrict any outstanding commands the LSI HBA. > > > > > > > > > > { > > > > > /* Default tagged queuing parameters for all devices */ > > > > > { > > > > > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED, > > > > > /*vendor*/"*", /*product*/"*", /*revision*/"*" > > > > > }, > > > > > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <--- > Default > > > > maxtags were 256. I increase it to 10234 > > > > > }, > > > > > > > > > > > > > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding > > > > commands in Firmware. But due to some reason, I am not able to > > > > pump more than 256 outstanding commands to the HBA. > > > > > > > > > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have > > > > sysctl parameter in Driver to display outstanding "FW commands". > > > > Max value for FW outstanding only goes up to 256. > > > > > > > > > > Also from some other mail thread Subject "mfi driver > > > > > performance", I > > > > found that folks talk about tuning queue depth _but_ nobody > > > > discussed to increase it beyond 256. Is there any limitation in > FreeBSD ? > > > > > > > > > > > > > As Jim pointed out, one thing to check is the values passed into > > > > cam_sim_alloc(). In the case of the mps(4) driver, the > > > > calculation is in mps_attach(): > > > > > > > > sc->num_reqs =3D MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit); > > > > > > > > What is reported for the RequestCredit on this particular adapter? > > > > > > > > The other question is, what does 'camcontrol tags daX -v' show > > > > when you are running the test? > > > > > > Below is output of camcontrol tags da1 -v. > > > > > > dhcp-135-24-192-127# camcontrol tags da13 -v > > > (pass13:mrsas0:0:13:0): dev_openings 1024 > > > (pass13:mrsas0:0:13:0): dev_active 0 > > > (pass13:mrsas0:0:13:0): devq_openings 1024 > > > (pass13:mrsas0:0:13:0): devq_queued 0 > > > (pass13:mrsas0:0:13:0): held 0 > > > (pass13:mrsas0:0:13:0): mintags 2 > > > (pass13:mrsas0:0:13:0): maxtags 1024 > > > dhcp-135-24-192-127# camcontrol tags da1 -v > > > (pass1:mrsas0:0:1:0): dev_openings 1024 > > > (pass1:mrsas0:0:1:0): dev_active 0 > > > (pass1:mrsas0:0:1:0): devq_openings 1024 > > > (pass1:mrsas0:0:1:0): devq_queued 0 > > > (pass1:mrsas0:0:1:0): held 0 > > > (pass1:mrsas0:0:1:0): mintags 2 > > > (pass1:mrsas0:0:1:0): maxtags 1024 > > > > > > Value 1024 is hard coded for my testing. In MegaRaid controller and > > > SAS- HBA Driver read max commands value from FW. > > > Similar to "RequestCredit".. Different FW has different value, but > > > they are every time above 255. > > > > > > > > > When I run IOs dev_active stays in range of 0-255 only. See below > > > output when I run IOs on /dev/da1 and /dev/da13. I expect total > > > dev_openings should go beyond 255, which is not happening. > > > > > > > > > dhcp-135-24-192-127# camcontrol tags da1 -v > > > (pass1:mrsas0:0:1:0): dev_openings 832 > > > (pass1:mrsas0:0:1:0): dev_active 192 > > > (pass1:mrsas0:0:1:0): devq_openings 832 > > > (pass1:mrsas0:0:1:0): devq_queued 0 > > > (pass1:mrsas0:0:1:0): held 0 > > > (pass1:mrsas0:0:1:0): mintags 2 > > > (pass1:mrsas0:0:1:0): maxtags 1024 > > > dhcp-135-24-192-127# camcontrol tags da13 -v > > > (pass13:mrsas0:0:13:0): dev_openings 881 > > > (pass13:mrsas0:0:13:0): dev_active 143 > > > (pass13:mrsas0:0:13:0): devq_openings 881 > > > (pass13:mrsas0:0:13:0): devq_queued 0 > > > (pass13:mrsas0:0:13:0): held 0 > > > (pass13:mrsas0:0:13:0): mintags 2 > > > (pass13:mrsas0:0:13:0): maxtags 1024 > > > > > > > > > > > > > > > Jim: > > > Below is my API call. I have hard code value "queue_depth" =3D 1024 > > > > > > sc->sim_0 =3D cam_sim_alloc(mrsas_action, mrsas_poll, "mrsas", sc= , > > > device_get_unit(sc->mrsas_dev), &sc->sim_lock, queue_depth, > > > queue_depth, devq); > > > > > > ~ Kashyap > > > > > > > > > > > Ken > > > > -- > > > > Kenneth Merry > > > > ken@FreeBSD.ORG > > > _______________________________________________ > > > freebsd-scsi@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > > To unsubscribe, send any mail to "freebsd-scsi- > unsubscribe@freebsd.org" >=20 > -- > Kenneth Merry > ken@FreeBSD.ORG