From owner-freebsd-scsi@FreeBSD.ORG Sun Sep 16 22:15:50 2012 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.ORG Received: by hub.freebsd.org (Postfix, from userid 821) id A6BAF1065670; Sun, 16 Sep 2012 22:15:50 +0000 (UTC) Date: Sun, 16 Sep 2012 22:15:50 +0000 From: John To: FreeBSD iSCSI Message-ID: <20120916221550.GA63055@FreeBSD.org> References: <20120915022437.GA90210@FreeBSD.org> <20120915023329.GA55292@nargothrond.kdm.org> <20120915031305.GA97685@FreeBSD.org> <20120915032826.GA63349@nargothrond.kdm.org> <20120915040907.GA5458@FreeBSD.org> <20120915043938.GA71754@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120915043938.GA71754@nargothrond.kdm.org> User-Agent: Mutt/1.4.2.1i Cc: "Kenneth D. Merry" Subject: Re: How to force a reset of a device (disk) in an enclosre slot X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Sep 2012 22:15:50 -0000 ----- Kenneth D. Merry's Original Message ----- > On Sat, Sep 15, 2012 at 04:09:07 +0000, John wrote: > > ----- Kenneth D. Merry's Original Message ----- > > > On Sat, Sep 15, 2012 at 03:13:05 +0000, John wrote: > > > > ----- Kenneth D. Merry's Original Message ----- > > > > > On Sat, Sep 15, 2012 at 02:24:37 +0000, John wrote: > > > > > > Hi Folks, > > > > > > > > > > > > I've been poking around and can't seem to find a way to reset and > > > > > > hopefully acquire access to a disk device in an enclosure. For instance: > > > > > > > > > > > > FreeBSD 9.1-PRERELEASE > > > > > > > > > > > > # camcontrol smpphylist ses4 > > > > > > 37 PHYs: > > > > > > PHY Attached SAS Address > > > > > > 0 0x5000039368233602 (pass105,da98) > > > > > > 1 0x5000039368238e3e (pass106,da99) > > > > > > 2 0x500003936823bca2 (pass107,da100) > > > > > > 3 0x500003936819507e (pass108,da101) > > > > > > 4 0x5000039368197d5a (pass109,da102) > > > > > > 5 0x5000039368197c6e (pass110,da103) > > > > > > 6 0x500003936818770e (pass111,da104) > > > > > > 7 0x5000039368238eba (pass112,da105) > > > > > > 8 0x5000039368232f42 (pass113,da106) > > > > > > 9 0x0000000000000000 > > > > > > 10 0x500003936813c31e > > > > > > 11 0x5000039368233892 (pass114,da107) > > > > > > 12 0x500003936813c2ca (pass115,da108) > > > > > > ... > > > > > > > > > > > > Note, bay/slot 10 has a listed device address. If I were to pull the > > > > > > drive and re-insert it, it would show up (as da390 in this case). > > > > > > The above is after a fresh reboot. Note da106 to da107 skipping > > > > > > slot 10 (slot 9 is empty). > > > > > > > > > > > > The smp utils provide a similar view: > > > > > > > > > > > > # smp_discover /dev/ses4 > > > > > > phy 0:D:attached:[5000039368233602:00 t(SSP)] 6 Gbps > > > > > > phy 1:D:attached:[5000039368238e3e:00 t(SSP)] 6 Gbps > > > > > > phy 2:D:attached:[500003936823bca2:00 t(SSP)] 6 Gbps > > > > > > phy 3:D:attached:[500003936819507e:00 t(SSP)] 6 Gbps > > > > > > phy 4:D:attached:[5000039368197d5a:00 t(SSP)] 6 Gbps > > > > > > phy 5:D:attached:[5000039368197c6e:00 t(SSP)] 6 Gbps > > > > > > phy 6:D:attached:[500003936818770e:00 t(SSP)] 6 Gbps > > > > > > phy 7:D:attached:[5000039368238eba:00 t(SSP)] 6 Gbps > > > > > > phy 8:D:attached:[5000039368232f42:00 t(SSP)] 6 Gbps > > > > > > phy 10:D:attached:[500003936813c31e:00 t(SSP)] 6 Gbps > > > > > > phy 11:D:attached:[5000039368233892:00 t(SSP)] 6 Gbps > > > > > > phy 12:D:attached:[500003936813c2ca:00 t(SSP)] 6 Gbps > > > > > > ... > > > > > > > > > > > > The address of slot 10 matches. There is a disk in the slot - just > > > > > > isn't recognized and attached. > > > > > > > > > > > > Back to the basic question. How can I issue a command to the enclosure > > > > > > to force a re-initialization of the device to recover it without > > > > > > having to physically pull & insert it. Even if the device numbers > > > > > > are not sequential, I need access to the drive... > > > > > > > > > > You can try sending a link reset: > > > > > > > > > > camcontrol smppc ses4 -p 10 -o linkreset > > > > > > > > > > It may or may not work. You can also try disabling the PHY (-o disable) > > > > > and then sending a link reset to re-enable the link. You can also try a > > > > > hard reset (-o hardreset) > > > > > > > > Hi Ken, > > > > > > > > Well, I hadn't tried to actually disable the device. That did bring some > > > > reaction: > > > > > > > > # camcontrol smppc ses4 -p 10 -o disable > > > > # camcontrol smpphylist ses4 > > > > 37 PHYs: > > > > PHY Attached SAS Address > > > > 0 0x5000039368233602 (pass105,da98) > > > > .... > > > > 8 0x5000039368232f42 (pass113,da106) > > > > 9 0x0000000000000000 > > > > 10 0x0000000000000000 > > > > 11 0x5000039368233892 (pass114,da107) > > > > ... > > > > > > > > The device is gone. > > > > > > > > # camcontrol smppc ses4 -p 10 -o hardreset > > > > root@vprzfs01p:/root # camcontrol smpphylist ses4 > > > > 37 PHYs: > > > > PHY Attached SAS Address > > > > 0 0x5000039368233602 (pass105,da98) > > > > .... > > > > 8 0x5000039368232f42 (pass113,da106) > > > > 9 0x0000000000000000 > > > > 10 0x500003936813c31e > > > > 11 0x5000039368233892 (pass114,da107) > > > > ... > > > > > > > > The device is back, but not attached - This msg: > > > > > > > > kernel: mps1: mpssas_alloc_tm freezing simq > > > > kernel: mps1: mpssas_remove_complete on handle 0x0069, IOCStatus= 0x0 > > > > kernel: mps1: mpssas_free_tm releasing simq > > > > kernel: _mapping_add_new_device: failed to add the device with handle 0x0069 to persistent table because there is no free space available - entry 0 > > > > > > That message is harmless, it won't prevent the drive from attaching. > > > > > > > >From a debug statement in the driver: MaxPersistentEntries == 128, but I > > > > have more than 128 devices per LSI card and they normally all show up - > > > > though I do get a bunch of the above messages in dmesg.. > > > > > > You might try turning on some of the debugging in the mps(4) driver and > > > disabling and resetting the link again. > > > > > > Try: > > > > > > sysctl -w dev.mps.0.debug_level=0xf > > > > > > You might get a lot of output, so be prepared to reset it back to 4: > > > > > > sysctl -w dev.mps.0.debug_level=4 > > > > Hi Ken, > > > > I don't see anything obvious. Hopefully you're more familair with the > > code and have better eyes than I do... Here's everything from messages > > after the -o disable. There are some "unknown/unhandled"s showing up. > > Here is where the drive shows up: > > > kernel: mps_intr_locked sc 0xffffff8001353000 writing postindex 243 > > kernel: mps_enqueue_request SMID 653 cm 0xffffff80013ca4a8 ccb 0 > > kernel: mps_intr_locked sc 0xffffff8001353000 starting with replypostindex 243 > > kernel: mps_intr_locked sc 0xffffff8001353000 writing postindex 244 > > kernel: SAS Address from SAS device page0 = 500003936811feae > > kernel: Found device <401,End Device> <6.0Gbps> <0x0078> <4/36> > > kernel: mpssas_rescan_target targetid 255 > > kernel: mpssas_rescan > > kernel: > > kernel: Target id 0xff added > > It finds the device, with target ID 255 (which is a little suspicious) and > queues a rescan, but nothing happens after that. > > You might try doing a manual rescan of that device to see what happens: > > camcontrol rescan X:255:0 > > Where X is the scbus number from camcontrol devlist. > > If that doesn't work, then we need to figure out what the maximum number of > targets supported by the adapter is. To do that, set this in > /boot/loader.conf and reboot: > > hw.mps.debug_level=1 > > That should result in the IOCFacts page getting printed on boot. > > How many drives and other devices are currently attached to that > controller? What controller model is it, and do you have IT or IR > firmware on it? Hi Ken, First to some of your questions. There are 3 LSI cards installed in the box. LSI 2116 - One 16i (not in use) and a pair of 16e. There are 8 d2700 shelves attached to the 16e cards - dual channel. Eight shelves, 25 devices each: 200 disks, dual attached, da0 - da399 for fully populated shelves.. The IOCFacts are included at the end of this email. I think you've seen this diagram before: http://people.freebsd.org/~jwd/zfsnfsserver.jpg I think there is something to your comment about device id 255 being suspicious. After a fresh reboot, it appears that device 255 on both busses is missing. Issuing just the commands: camcontrol rescan 7:255:0 camcontrol rescan 8:255:0 and the drives show up: mps1: handle(0x0062), ioc_status(scsi data underrun)(0x0045), mps1: scsi_status(good)(0x00), scsi_state( )(0x00) mps1: handle(0x0062), ioc_status(scsi data underrun)(0x0045), mps1: scsi_status(good)(0x00), scsi_state( )(0x00) mps1: handle(0x0062), ioc_status(scsi data underrun)(0x0045), mps1: scsi_status(good)(0x00), scsi_state( )(0x00) da390 at mps1 bus 0 scbus7 target 255 lun 0 da390: Fixed Direct Access SCSI-5 device da390: 600.000MB/s transfers da390: Command Queueing enabled da390: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C) GEOM_MULTIPATH: da390 added to Z78 mps2: handle(0x0069), ioc_status(scsi data underrun)(0x0045), mps2: scsi_status(good)(0x00), scsi_state( )(0x00) mps2: handle(0x0069), ioc_status(scsi data underrun)(0x0045), mps2: scsi_status(good)(0x00), scsi_state( )(0x00) mps2: handle(0x0069), ioc_status(scsi data underrun)(0x0045), mps2: scsi_status(good)(0x00), scsi_state( )(0x00) da391 at mps2 bus 0 scbus8 target 255 lun 0 da391: Fixed Direct Access SCSI-5 device da391: 600.000MB/s transfers da391: Command Queueing enabled da391: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C) GEOM_MULTIPATH: da391 added to Z75 I don't know if the 'scsi data underrun' is important here. A camcontrol smpphylist of the two related shelves. Note, there is no device is bay 9. # camcontrol smpphylist ses4 37 PHYs: PHY Attached SAS Address 0 0x500003936812dd36 (pass105,da98) .... 8 0x500003936810b756 (pass113,da106) 9 0x0000000000000000 10 0x500003936811feae (da390,pass409) ... 24 0x500003936819898a (pass127,da120) # camcontrol smpphylist ses12 37 PHYs: PHY Attached SAS Address 0 0x5000c5003c4f5986 (pass308,da293) .... 8 0x5000c5003c4f5ebe (pass316,da301) 9 0x0000000000000000 10 0x5000c5003c375b86 (da391,pass410) ... 24 0x5000c5003c00cb7a (pass330,da315) On a related note, the pass & da devices are reversed for the two devices found via the rescan. I've been scanning the driver code but don't see an obvious place where device id 255 would be dropped. Please let me know if there is something you'd like me to check. This has been a learning experience. Thanks Ken. Thanks, John 16i card: (Not currenlty in use) mps0: port 0x5000-0x50ff mem 0xfbdf0000-0xfbdf3fff,0xfbd80000-0xfbdbffff irq 52 at device 0.0 on pci24 mps0: Doorbell= 0x12000000 mps0: IOCFacts : MsgVersion: 0x200 HeaderVersion: 0x1900 IOCNumber: 0 IOCExceptions: 0x0 MaxChainDepth: 128 WhoInit: ROM BIOS NumberOfPorts: 1 RequestCredit: 7632 ProductID: 0x2213 IOCCapabilities: 1285c FWVersion= 14-0-0-0 IOCRequestFrameSize: 32 MaxInitiators: 30 MaxTargets: 756 MaxSasExpanders: 224 MaxEnclosures: 224 ProtocolFlags: 3 HighPriorityCredit: 120 MaxReplyDescriptorPostQueueDepth: 65504 ReplyFrameSize: 32 MaxVolumes: 0 MaxDevHandle: 1026 MaxPersistentEntries: 128 mps0: Firmware: 14.00.00.00, Driver: 14.00.00.01-fbsd mps0: IOCCapabilities: 1285c 1st 16e card: mps1: port 0x7000-0x70ff mem 0xfbff0000-0xfbff3fff,0xfbf80000-0xfbfbffff irq 48 at device 0.0 on pci33 mps1: Doorbell= 0x12000000 mps1: IOCFacts : MsgVersion: 0x200 HeaderVersion: 0x1900 IOCNumber: 0 IOCExceptions: 0x0 MaxChainDepth: 128 WhoInit: ROM BIOS NumberOfPorts: 1 RequestCredit: 32455 ProductID: 0x2213 IOCCapabilities: 1285c FWVersion= 14-0-0-0 IOCRequestFrameSize: 32 MaxInitiators: 32 MaxTargets: 1024 MaxSasExpanders: 128 MaxEnclosures: 129 ProtocolFlags: 3 HighPriorityCredit: 127 MaxReplyDescriptorPostQueueDepth: 65504 ReplyFrameSize: 32 MaxVolumes: 0 MaxDevHandle: 1200 MaxPersistentEntries: 128 mps1: Firmware: 14.00.00.00, Driver: 14.00.00.01-fbsd mps1: IOCCapabilities: 1285c 2nd 16e card: mps2: port 0x6000-0x60ff mem 0xfbef0000-0xfbef3fff,0xfbe80000-0xfbebffff irq 56 at device 0.0 on pci27 mps2: Doorbell= 0x12000000 mps2: IOCFacts : MsgVersion: 0x200 HeaderVersion: 0x1900 IOCNumber: 0 IOCExceptions: 0x0 MaxChainDepth: 128 WhoInit: ROM BIOS NumberOfPorts: 1 RequestCredit: 32455 ProductID: 0x2213 IOCCapabilities: 1285c FWVersion= 14-0-0-0 IOCRequestFrameSize: 32 MaxInitiators: 32 MaxTargets: 1024 MaxSasExpanders: 128 MaxEnclosures: 129 ProtocolFlags: 3 HighPriorityCredit: 127 MaxReplyDescriptorPostQueueDepth: 65504 ReplyFrameSize: 32 MaxVolumes: 0 MaxDevHandle: 1200 MaxPersistentEntries: 128 mps2: Firmware: 14.00.00.00, Driver: 14.00.00.01-fbsd mps2: IOCCapabilities: 1285c And here is the entire device listing after running the pair of camcontrol rescan commands. Note, the OS lives on the revodrive. All shelf drives are for data. at scbus0 target 0 lun 0 (ada0,pass0) at scbus1 target 0 lun 0 (ada1,pass1) at scbus5 target 0 lun 0 (pass2,cd0) at scbus7 target 144 lun 0 (pass3,da0) at scbus7 target 145 lun 0 (pass4,da1) at scbus7 target 146 lun 0 (pass5,da2) at scbus7 target 147 lun 0 (pass6,da3) at scbus7 target 148 lun 0 (pass7,da4) at scbus7 target 149 lun 0 (pass8,da5) at scbus7 target 150 lun 0 (pass9,da6) at scbus7 target 151 lun 0 (pass10,da7) at scbus7 target 152 lun 0 (pass11,da8) at scbus7 target 153 lun 0 (pass12,da9) at scbus7 target 154 lun 0 (pass13,da10) at scbus7 target 155 lun 0 (pass14,da11) at scbus7 target 156 lun 0 (pass15,da12) at scbus7 target 157 lun 0 (pass16,da13) at scbus7 target 158 lun 0 (pass17,da14) at scbus7 target 159 lun 0 (pass18,da15) at scbus7 target 160 lun 0 (pass19,da16) at scbus7 target 161 lun 0 (pass20,da17) at scbus7 target 162 lun 0 (pass21,da18) at scbus7 target 163 lun 0 (pass22,da19) at scbus7 target 164 lun 0 (pass23,da20) at scbus7 target 165 lun 0 (pass24,da21) at scbus7 target 166 lun 0 (pass25,da22) at scbus7 target 167 lun 0 (pass26,da23) at scbus7 target 168 lun 0 (ses0,pass27) at scbus7 target 169 lun 0 (pass28,da24) at scbus7 target 170 lun 0 (pass29,da25) at scbus7 target 171 lun 0 (pass30,da26) at scbus7 target 172 lun 0 (pass31,da27) at scbus7 target 173 lun 0 (pass32,da28) at scbus7 target 174 lun 0 (pass33,da29) at scbus7 target 175 lun 0 (pass34,da30) at scbus7 target 176 lun 0 (pass35,da31) at scbus7 target 177 lun 0 (pass36,da32) at scbus7 target 178 lun 0 (pass37,da33) at scbus7 target 179 lun 0 (pass38,da34) at scbus7 target 180 lun 0 (pass39,da35) at scbus7 target 181 lun 0 (pass40,da36) at scbus7 target 182 lun 0 (pass41,da37) at scbus7 target 183 lun 0 (pass42,da38) at scbus7 target 184 lun 0 (pass43,da39) at scbus7 target 185 lun 0 (pass44,da40) at scbus7 target 186 lun 0 (pass45,da41) at scbus7 target 187 lun 0 (pass46,da42) at scbus7 target 188 lun 0 (pass47,da43) at scbus7 target 189 lun 0 (pass48,da44) at scbus7 target 190 lun 0 (pass49,da45) at scbus7 target 191 lun 0 (pass50,da46) at scbus7 target 192 lun 0 (pass51,da47) at scbus7 target 193 lun 0 (pass52,da48) at scbus7 target 194 lun 0 (ses1,pass53) at scbus7 target 195 lun 0 (pass54,da49) at scbus7 target 196 lun 0 (pass55,da50) at scbus7 target 197 lun 0 (pass56,da51) at scbus7 target 198 lun 0 (pass57,da52) at scbus7 target 199 lun 0 (pass58,da53) at scbus7 target 200 lun 0 (pass59,da54) at scbus7 target 201 lun 0 (pass60,da55) at scbus7 target 202 lun 0 (pass61,da56) at scbus7 target 203 lun 0 (pass62,da57) at scbus7 target 204 lun 0 (pass63,da58) at scbus7 target 205 lun 0 (pass64,da59) at scbus7 target 206 lun 0 (pass65,da60) at scbus7 target 207 lun 0 (pass66,da61) at scbus7 target 208 lun 0 (pass67,da62) at scbus7 target 209 lun 0 (pass68,da63) at scbus7 target 210 lun 0 (pass69,da64) at scbus7 target 211 lun 0 (pass70,da65) at scbus7 target 212 lun 0 (pass71,da66) at scbus7 target 213 lun 0 (pass72,da67) at scbus7 target 214 lun 0 (pass73,da68) at scbus7 target 215 lun 0 (pass74,da69) at scbus7 target 216 lun 0 (pass75,da70) at scbus7 target 217 lun 0 (pass76,da71) at scbus7 target 218 lun 0 (pass77,da72) at scbus7 target 219 lun 0 (ses2,pass78) at scbus7 target 220 lun 0 (pass79,da73) at scbus7 target 221 lun 0 (pass80,da74) at scbus7 target 222 lun 0 (pass81,da75) at scbus7 target 223 lun 0 (pass82,da76) at scbus7 target 224 lun 0 (pass83,da77) at scbus7 target 225 lun 0 (pass84,da78) at scbus7 target 226 lun 0 (pass85,da79) at scbus7 target 227 lun 0 (pass86,da80) at scbus7 target 228 lun 0 (pass87,da81) at scbus7 target 229 lun 0 (pass88,da82) at scbus7 target 230 lun 0 (pass89,da83) at scbus7 target 231 lun 0 (pass90,da84) at scbus7 target 232 lun 0 (pass91,da85) at scbus7 target 233 lun 0 (pass92,da86) at scbus7 target 234 lun 0 (pass93,da87) at scbus7 target 235 lun 0 (pass94,da88) at scbus7 target 236 lun 0 (pass95,da89) at scbus7 target 237 lun 0 (pass96,da90) at scbus7 target 238 lun 0 (pass97,da91) at scbus7 target 239 lun 0 (pass98,da92) at scbus7 target 240 lun 0 (pass99,da93) at scbus7 target 241 lun 0 (pass100,da94) at scbus7 target 242 lun 0 (pass101,da95) at scbus7 target 243 lun 0 (pass102,da96) at scbus7 target 244 lun 0 (pass103,da97) at scbus7 target 245 lun 0 (ses3,pass104) at scbus7 target 246 lun 0 (pass105,da98) at scbus7 target 247 lun 0 (pass106,da99) at scbus7 target 248 lun 0 (pass107,da100) at scbus7 target 249 lun 0 (pass108,da101) at scbus7 target 250 lun 0 (pass109,da102) at scbus7 target 251 lun 0 (pass110,da103) at scbus7 target 252 lun 0 (pass111,da104) at scbus7 target 253 lun 0 (pass112,da105) at scbus7 target 254 lun 0 (pass113,da106) at scbus7 target 255 lun 0 (da390,pass409) at scbus7 target 256 lun 0 (pass114,da107) at scbus7 target 257 lun 0 (pass115,da108) at scbus7 target 258 lun 0 (pass116,da109) at scbus7 target 259 lun 0 (pass117,da110) at scbus7 target 260 lun 0 (pass118,da111) at scbus7 target 261 lun 0 (pass119,da112) at scbus7 target 262 lun 0 (pass120,da113) at scbus7 target 263 lun 0 (pass121,da114) at scbus7 target 264 lun 0 (pass122,da115) at scbus7 target 265 lun 0 (pass123,da116) at scbus7 target 266 lun 0 (pass124,da117) at scbus7 target 267 lun 0 (pass125,da118) at scbus7 target 268 lun 0 (pass126,da119) at scbus7 target 269 lun 0 (pass127,da120) at scbus7 target 270 lun 0 (ses4,pass128) at scbus7 target 271 lun 0 (pass129,da121) at scbus7 target 272 lun 0 (pass130,da122) at scbus7 target 273 lun 0 (pass131,da123) at scbus7 target 274 lun 0 (pass132,da124) at scbus7 target 275 lun 0 (pass133,da125) at scbus7 target 276 lun 0 (pass134,da126) at scbus7 target 277 lun 0 (pass135,da127) at scbus7 target 278 lun 0 (pass136,da128) at scbus7 target 279 lun 0 (pass137,da129) at scbus7 target 280 lun 0 (pass138,da130) at scbus7 target 281 lun 0 (pass139,da131) at scbus7 target 282 lun 0 (pass140,da132) at scbus7 target 283 lun 0 (pass141,da133) at scbus7 target 284 lun 0 (pass142,da134) at scbus7 target 285 lun 0 (pass143,da135) at scbus7 target 286 lun 0 (pass144,da136) at scbus7 target 287 lun 0 (pass145,da137) at scbus7 target 288 lun 0 (pass146,da138) at scbus7 target 289 lun 0 (pass147,da139) at scbus7 target 290 lun 0 (pass148,da140) at scbus7 target 291 lun 0 (pass149,da141) at scbus7 target 292 lun 0 (pass150,da142) at scbus7 target 293 lun 0 (pass151,da143) at scbus7 target 294 lun 0 (pass152,da144) at scbus7 target 295 lun 0 (pass153,da145) at scbus7 target 296 lun 0 (ses5,pass154) at scbus7 target 297 lun 0 (pass155,da146) at scbus7 target 298 lun 0 (pass156,da147) at scbus7 target 299 lun 0 (pass157,da148) at scbus7 target 300 lun 0 (pass158,da149) at scbus7 target 301 lun 0 (pass159,da150) at scbus7 target 302 lun 0 (pass160,da151) at scbus7 target 303 lun 0 (pass161,da152) at scbus7 target 304 lun 0 (pass162,da153) at scbus7 target 305 lun 0 (pass163,da154) at scbus7 target 306 lun 0 (pass164,da155) at scbus7 target 307 lun 0 (pass165,da156) at scbus7 target 308 lun 0 (pass166,da157) at scbus7 target 309 lun 0 (pass167,da158) at scbus7 target 310 lun 0 (pass168,da159) at scbus7 target 311 lun 0 (pass169,da160) at scbus7 target 312 lun 0 (pass170,da161) at scbus7 target 313 lun 0 (pass171,da162) at scbus7 target 314 lun 0 (pass172,da163) at scbus7 target 315 lun 0 (pass173,da164) at scbus7 target 316 lun 0 (pass174,da165) at scbus7 target 317 lun 0 (pass175,da166) at scbus7 target 318 lun 0 (pass176,da167) at scbus7 target 319 lun 0 (pass177,da168) at scbus7 target 320 lun 0 (pass178,da169) at scbus7 target 321 lun 0 (ses6,pass179) at scbus7 target 322 lun 0 (pass180,da170) at scbus7 target 323 lun 0 (pass181,da171) at scbus7 target 324 lun 0 (pass182,da172) at scbus7 target 325 lun 0 (pass183,da173) at scbus7 target 326 lun 0 (pass184,da174) at scbus7 target 327 lun 0 (pass185,da175) at scbus7 target 328 lun 0 (pass186,da176) at scbus7 target 329 lun 0 (pass187,da177) at scbus7 target 330 lun 0 (pass188,da178) at scbus7 target 331 lun 0 (pass189,da179) at scbus7 target 332 lun 0 (pass190,da180) at scbus7 target 333 lun 0 (pass191,da181) at scbus7 target 334 lun 0 (pass192,da182) at scbus7 target 335 lun 0 (pass193,da183) at scbus7 target 336 lun 0 (pass194,da184) at scbus7 target 337 lun 0 (pass195,da185) at scbus7 target 338 lun 0 (pass196,da186) at scbus7 target 339 lun 0 (pass197,da187) at scbus7 target 340 lun 0 (pass198,da188) at scbus7 target 341 lun 0 (pass199,da189) at scbus7 target 342 lun 0 (pass200,da190) at scbus7 target 343 lun 0 (pass201,da191) at scbus7 target 344 lun 0 (pass202,da192) at scbus7 target 345 lun 0 (pass203,da193) at scbus7 target 346 lun 0 (pass204,da194) at scbus7 target 347 lun 0 (ses7,pass205) at scbus8 target 144 lun 0 (pass206,da195) at scbus8 target 145 lun 0 (pass207,da196) at scbus8 target 146 lun 0 (pass208,da197) at scbus8 target 147 lun 0 (pass209,da198) at scbus8 target 148 lun 0 (pass210,da199) at scbus8 target 149 lun 0 (pass211,da200) at scbus8 target 150 lun 0 (pass212,da201) at scbus8 target 151 lun 0 (pass213,da202) at scbus8 target 152 lun 0 (pass214,da203) at scbus8 target 153 lun 0 (pass215,da204) at scbus8 target 154 lun 0 (pass216,da205) at scbus8 target 155 lun 0 (pass217,da206) at scbus8 target 156 lun 0 (pass218,da207) at scbus8 target 157 lun 0 (pass219,da208) at scbus8 target 158 lun 0 (pass220,da209) at scbus8 target 159 lun 0 (pass221,da210) at scbus8 target 160 lun 0 (pass222,da211) at scbus8 target 161 lun 0 (pass223,da212) at scbus8 target 162 lun 0 (pass224,da213) at scbus8 target 163 lun 0 (pass225,da214) at scbus8 target 164 lun 0 (pass226,da215) at scbus8 target 165 lun 0 (pass227,da216) at scbus8 target 166 lun 0 (pass228,da217) at scbus8 target 167 lun 0 (pass229,da218) at scbus8 target 168 lun 0 (ses8,pass230) at scbus8 target 169 lun 0 (pass231,da219) at scbus8 target 170 lun 0 (pass232,da220) at scbus8 target 171 lun 0 (pass233,da221) at scbus8 target 172 lun 0 (pass234,da222) at scbus8 target 173 lun 0 (pass235,da223) at scbus8 target 174 lun 0 (pass236,da224) at scbus8 target 175 lun 0 (pass237,da225) at scbus8 target 176 lun 0 (pass238,da226) at scbus8 target 177 lun 0 (pass239,da227) at scbus8 target 178 lun 0 (pass240,da228) at scbus8 target 179 lun 0 (pass241,da229) at scbus8 target 180 lun 0 (pass242,da230) at scbus8 target 181 lun 0 (pass243,da231) at scbus8 target 182 lun 0 (pass244,da232) at scbus8 target 183 lun 0 (pass245,da233) at scbus8 target 184 lun 0 (pass246,da234) at scbus8 target 185 lun 0 (pass247,da235) at scbus8 target 186 lun 0 (pass248,da236) at scbus8 target 187 lun 0 (pass249,da237) at scbus8 target 188 lun 0 (pass250,da238) at scbus8 target 189 lun 0 (pass251,da239) at scbus8 target 190 lun 0 (pass252,da240) at scbus8 target 191 lun 0 (pass253,da241) at scbus8 target 192 lun 0 (pass254,da242) at scbus8 target 193 lun 0 (pass255,da243) at scbus8 target 194 lun 0 (ses9,pass256) at scbus8 target 195 lun 0 (pass257,da244) at scbus8 target 196 lun 0 (pass258,da245) at scbus8 target 197 lun 0 (pass259,da246) at scbus8 target 198 lun 0 (pass260,da247) at scbus8 target 199 lun 0 (pass261,da248) at scbus8 target 200 lun 0 (pass262,da249) at scbus8 target 201 lun 0 (pass263,da250) at scbus8 target 202 lun 0 (pass264,da251) at scbus8 target 203 lun 0 (pass265,da252) at scbus8 target 204 lun 0 (pass266,da253) at scbus8 target 205 lun 0 (pass267,da254) at scbus8 target 206 lun 0 (pass268,da255) at scbus8 target 207 lun 0 (pass269,da256) at scbus8 target 208 lun 0 (pass270,da257) at scbus8 target 209 lun 0 (pass271,da258) at scbus8 target 210 lun 0 (pass272,da259) at scbus8 target 211 lun 0 (pass273,da260) at scbus8 target 212 lun 0 (pass274,da261) at scbus8 target 213 lun 0 (pass275,da262) at scbus8 target 214 lun 0 (pass276,da263) at scbus8 target 215 lun 0 (pass277,da264) at scbus8 target 216 lun 0 (pass278,da265) at scbus8 target 217 lun 0 (pass279,da266) at scbus8 target 218 lun 0 (pass280,da267) at scbus8 target 219 lun 0 (ses10,pass281) at scbus8 target 220 lun 0 (pass282,da268) at scbus8 target 221 lun 0 (pass283,da269) at scbus8 target 222 lun 0 (pass284,da270) at scbus8 target 223 lun 0 (pass285,da271) at scbus8 target 224 lun 0 (pass286,da272) at scbus8 target 225 lun 0 (pass287,da273) at scbus8 target 226 lun 0 (pass288,da274) at scbus8 target 227 lun 0 (pass289,da275) at scbus8 target 228 lun 0 (pass290,da276) at scbus8 target 229 lun 0 (pass291,da277) at scbus8 target 230 lun 0 (pass292,da278) at scbus8 target 231 lun 0 (pass293,da279) at scbus8 target 232 lun 0 (pass294,da280) at scbus8 target 233 lun 0 (pass295,da281) at scbus8 target 234 lun 0 (pass296,da282) at scbus8 target 235 lun 0 (pass297,da283) at scbus8 target 236 lun 0 (pass298,da284) at scbus8 target 237 lun 0 (pass299,da285) at scbus8 target 238 lun 0 (pass300,da286) at scbus8 target 239 lun 0 (pass301,da287) at scbus8 target 240 lun 0 (pass302,da288) at scbus8 target 241 lun 0 (pass303,da289) at scbus8 target 242 lun 0 (pass304,da290) at scbus8 target 243 lun 0 (pass305,da291) at scbus8 target 244 lun 0 (pass306,da292) at scbus8 target 245 lun 0 (ses11,pass307) at scbus8 target 246 lun 0 (pass308,da293) at scbus8 target 247 lun 0 (pass309,da294) at scbus8 target 248 lun 0 (pass310,da295) at scbus8 target 249 lun 0 (pass311,da296) at scbus8 target 250 lun 0 (pass312,da297) at scbus8 target 251 lun 0 (pass313,da298) at scbus8 target 252 lun 0 (pass314,da299) at scbus8 target 253 lun 0 (pass315,da300) at scbus8 target 254 lun 0 (pass316,da301) at scbus8 target 255 lun 0 (da391,pass410) at scbus8 target 256 lun 0 (pass317,da302) at scbus8 target 257 lun 0 (pass318,da303) at scbus8 target 258 lun 0 (pass319,da304) at scbus8 target 259 lun 0 (pass320,da305) at scbus8 target 260 lun 0 (pass321,da306) at scbus8 target 261 lun 0 (pass322,da307) at scbus8 target 262 lun 0 (pass323,da308) at scbus8 target 263 lun 0 (pass324,da309) at scbus8 target 264 lun 0 (pass325,da310) at scbus8 target 265 lun 0 (pass326,da311) at scbus8 target 266 lun 0 (pass327,da312) at scbus8 target 267 lun 0 (pass328,da313) at scbus8 target 268 lun 0 (pass329,da314) at scbus8 target 269 lun 0 (pass330,da315) at scbus8 target 270 lun 0 (ses12,pass331) at scbus8 target 271 lun 0 (pass332,da316) at scbus8 target 272 lun 0 (pass333,da317) at scbus8 target 273 lun 0 (pass334,da318) at scbus8 target 274 lun 0 (pass335,da319) at scbus8 target 275 lun 0 (pass336,da320) at scbus8 target 276 lun 0 (pass337,da321) at scbus8 target 277 lun 0 (pass338,da322) at scbus8 target 278 lun 0 (pass339,da323) at scbus8 target 279 lun 0 (pass340,da324) at scbus8 target 280 lun 0 (pass341,da325) at scbus8 target 281 lun 0 (pass342,da326) at scbus8 target 282 lun 0 (pass343,da327) at scbus8 target 283 lun 0 (pass344,da328) at scbus8 target 284 lun 0 (pass345,da329) at scbus8 target 285 lun 0 (pass346,da330) at scbus8 target 286 lun 0 (pass347,da331) at scbus8 target 287 lun 0 (pass348,da332) at scbus8 target 288 lun 0 (pass349,da333) at scbus8 target 289 lun 0 (pass350,da334) at scbus8 target 290 lun 0 (pass351,da335) at scbus8 target 291 lun 0 (pass352,da336) at scbus8 target 292 lun 0 (pass353,da337) at scbus8 target 293 lun 0 (pass354,da338) at scbus8 target 294 lun 0 (pass355,da339) at scbus8 target 295 lun 0 (pass356,da340) at scbus8 target 296 lun 0 (ses13,pass357) at scbus8 target 297 lun 0 (pass358,da341) at scbus8 target 298 lun 0 (pass359,da342) at scbus8 target 299 lun 0 (pass360,da343) at scbus8 target 300 lun 0 (pass361,da344) at scbus8 target 301 lun 0 (pass362,da345) at scbus8 target 302 lun 0 (pass363,da346) at scbus8 target 303 lun 0 (pass364,da347) at scbus8 target 304 lun 0 (pass365,da348) at scbus8 target 305 lun 0 (pass366,da349) at scbus8 target 306 lun 0 (pass367,da350) at scbus8 target 307 lun 0 (pass368,da351) at scbus8 target 308 lun 0 (pass369,da352) at scbus8 target 309 lun 0 (pass370,da353) at scbus8 target 310 lun 0 (pass371,da354) at scbus8 target 311 lun 0 (pass372,da355) at scbus8 target 312 lun 0 (pass373,da356) at scbus8 target 313 lun 0 (pass374,da357) at scbus8 target 314 lun 0 (pass375,da358) at scbus8 target 315 lun 0 (pass376,da359) at scbus8 target 316 lun 0 (pass377,da360) at scbus8 target 317 lun 0 (pass378,da361) at scbus8 target 318 lun 0 (pass379,da362) at scbus8 target 319 lun 0 (pass380,da363) at scbus8 target 320 lun 0 (pass381,da364) at scbus8 target 321 lun 0 (ses14,pass382) at scbus8 target 322 lun 0 (pass383,da365) at scbus8 target 323 lun 0 (pass384,da366) at scbus8 target 324 lun 0 (pass385,da367) at scbus8 target 325 lun 0 (pass386,da368) at scbus8 target 326 lun 0 (pass387,da369) at scbus8 target 327 lun 0 (pass388,da370) at scbus8 target 328 lun 0 (pass389,da371) at scbus8 target 329 lun 0 (pass390,da372) at scbus8 target 330 lun 0 (pass391,da373) at scbus8 target 331 lun 0 (pass392,da374) at scbus8 target 332 lun 0 (pass393,da375) at scbus8 target 333 lun 0 (pass394,da376) at scbus8 target 334 lun 0 (pass395,da377) at scbus8 target 335 lun 0 (pass396,da378) at scbus8 target 336 lun 0 (pass397,da379) at scbus8 target 337 lun 0 (pass398,da380) at scbus8 target 338 lun 0 (pass399,da381) at scbus8 target 339 lun 0 (pass400,da382) at scbus8 target 340 lun 0 (pass401,da383) at scbus8 target 341 lun 0 (pass402,da384) at scbus8 target 342 lun 0 (pass403,da385) at scbus8 target 343 lun 0 (pass404,da386) at scbus8 target 344 lun 0 (pass405,da387) at scbus8 target 345 lun 0 (pass406,da388) at scbus8 target 346 lun 0 (pass407,da389) at scbus8 target 347 lun 0 (ses15,pass408)