Date: Tue, 2 Oct 2012 13:03:04 +0300 From: Nikolay Denev <ndenev@gmail.com> To: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Subject: Re: CAM Target Layer and Linux (continued) Message-ID: <11028C2E-9DB0-4B71-A7B1-98160D5AEA93@gmail.com> In-Reply-To: <72A4B763-D36B-4912-8C20-7373A0562EA1@gmail.com> References: <72A4B763-D36B-4912-8C20-7373A0562EA1@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sep 27, 2012, at 6:33 PM, Nikolay Denev <ndenev@gmail.com> wrote:
> Hi All,
>=20
> With the help of Chuck Tuffli, I'm now able to use CTL to export a =
zvol over FC to a Linux host:
>=20
> LUN Backend Size (Blocks) BS Serial Number Device ID =20=
> 0 block 4185915392 512 FBSDZFS001 ORA_ASM_01 =20=
> lun_type=3D0
> num_threads=3D14
> file=3D/dev/zvol/tank/oracle_asm_01
> 1 block 4185915392 512 FBSDZFS002 ORA_ASM_02 =20=
> lun_type=3D0
> num_threads=3D14
> file=3D/dev/zvol/tank/oracle_asm_02
> 2 block 4185915392 512 FBSDZFS003 ORA_ASM_03 =20=
> lun_type=3D0
> num_threads=3D14
> file=3D/dev/zvol/tank/oracle_asm_03
> 3 block 4185915392 512 FBSDZFS004 ORA_ASM_04 =20=
> lun_type=3D0
> num_threads=3D14
> file=3D/dev/zvol/tank/oracle_asm_04
>=20
> Then we ran some tests using Oracle's ORION benchmark tool from the =
Linux host.
> We ran one test which passed successfully,
> then I've just disabled zfs prefetch -> "vfs.zfs.prefetch_disable=3D1"
> and rerun the test, which failed due to this error.
>=20
> On the FreeBSD side:
>=20
> (0:3:0:1): READ(10). CDB: 28 0 84 f9 58 0 0 4 0 0=20
> (0:3:0:1): Tag: 0x116220, Type: 1
> (0:3:0:1): CTL Status: SCSI Error
> (0:3:0:1): SCSI Status: Check Condition
> (0:3:0:1): SCSI sense: NOT READY asc:4b,0 (Data phase error)
>=20
> Linux reported :
>=20
> sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready
> Add. Sense: Data phase error
>=20
> end_request: I/O error, dev sdr, sector 2230933504
> device-mapper: multipath: Failing path 65:16.
> sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready
> Add. Sense: Data phase error
>=20
> end_request: I/O error, dev sdr, sector 2230934528
>=20
> There are no other suspicious messages in dmesg.
>=20
> Also, ctladm dumpooa does not show anything.
> Here is dumpscructs output :
>=20
> CTL IID to WWPN map start:
> CTL IID to WWPN map end
> CTL Persistent Reservation information start:
> CTL Persistent Reservation information end
> CTL Frontends:
> Frontend CTL ioctl Type 4 pport 0 vport 0 WWNN 0 WWPN 0
> Frontend ctl2cam Type 8 pport 0 vport 0 WWNN 0x5000000995680700 WWPN =
0x5000000995680702
> Frontend CTL internal Type 8 pport 0 vport 0 WWNN 0 WWPN 0
> Frontend isp0 Type 1 pport 0 vport 0 WWNN 0x20000024ff376b98 WWPN =
0x21000024ff376b98
> isp0: max tagged openings: 4096, max dev openings: 4096
> isp0: max_ccbs: 20488, ccb_count: 79
> isp0: ccb_freeq is NOT empty
> isp0: alloc_queue.entries 0, alloc_openings 4096
> isp0: qfrozen_cnt:0:0:0:0:0
> (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs
> (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 =
freed)
> (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 =
returned
> (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs
> (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 =
freed)
> (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 =
returned
> (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs
> (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 =
freed)
> (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 =
returned
> (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs
> (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 =
freed)
> (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 =
returned
> isp1: max tagged openings: 4096, max dev openings: 4096
> isp1: max_ccbs: 20488, ccb_count: 1
> isp1: ccb_freeq is NOT empty
> isp1: alloc_queue.entries 0, alloc_openings 4096
> isp1: qfrozen_cnt:0:0:0:0:0
> (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs
> (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs
> (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs
> (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs
> (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned
> Frontend isp1 Type 1 pport 1 vport 0 WWNN 0x20000024ff376b99 WWPN =
0x21000024ff376b99
> isp0: max tagged openings: 4096, max dev openings: 4096
> isp0: max_ccbs: 20488, ccb_count: 79
> isp0: ccb_freeq is NOT empty
> isp0: alloc_queue.entries 0, alloc_openings 4096
> isp0: qfrozen_cnt:0:0:0:0:0
> (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs
> (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 =
freed)
> (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 =
returned
> (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs
> (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 =
freed)
> (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 =
returned
> (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs
> (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 =
freed)
> (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 =
returned
> (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs
> (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 =
freed)
> (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 =
returned
> isp1: max tagged openings: 4096, max dev openings: 4096
> isp1: max_ccbs: 20488, ccb_count: 1
> isp1: ccb_freeq is NOT empty
> isp1: alloc_queue.entries 0, alloc_openings 4096
> isp1: qfrozen_cnt:0:0:0:0:0
> (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs
> (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs
> (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs
> (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs
> (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned
> CTL Frontend information end
>=20
> zpool status is showing no errors.
>=20
> P.S.:
> The machine is 8 core 2.0Ghz Xeon E5-2650 with 196G of RAM
> Other things to note are that I'm running with "hw.mfi.max_cmds=3D254"
> Also my zpool is on GELI hw encrypted disks: 24 JBOD/RAID0 drives on =
mfi, each encrypted with GELI, and arranged in 3 raidz2 vdevs of 8 =
drives.
> And the machine also acts as a fairly loaded NFS server.
>=20
After a whole day of orion tests without problems, we started an Oracle =
ASM instance from the Linux host and
again got an error, this time it was WRITE error :
(0:3:0:3): WRITE(10). CDB: 2a 0 1 5b 10 0 0 4 0 0=20
(0:3:0:3): Tag: 0x110940, Type: 1
(0:3:0:3): CTL Status: SCSI Error
(0:3:0:3): SCSI Status: Check Condition
(0:3:0:3): SCSI sense: NOT READY asc:4b,0 (Data phase error)
I've tried to track down this "Data phase error" in the CTL code and it =
looks like it is something related to the isp(4) driver:
default:
/*
* XXX KDM the isp(4) driver doesn't =
really
* seem to send errors back for data
* transfers that I can tell. There is =
one
* case where it'll send =
CAM_REQ_CMP_ERR,
* but probably not that many more =
cases.
* So set a generic data phase error =
here,
* like the SXP driver sets.
*/
io->io_hdr.port_status =3D 0xbad1;
ctl_set_data_phase_error(&io->scsiio);
/*
* XXX KDM figure out residual.
*/
break;
But I can't find error statistics for the isp(4) driver anywhere. Any =
ideas how this can be debugged?=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?11028C2E-9DB0-4B71-A7B1-98160D5AEA93>