Date: Tue, 15 Jul 2008 09:45:06 -0400 From: Sven Willenberger <sven@dmv.com> To: Danny Braniss <danny@cs.huji.ac.il> Cc: freebsd-stable@freebsd.org Subject: Re: Using iscsi with multiple targets Message-ID: <1216129506.27608.8.camel@lanshark.dmv.com> In-Reply-To: <E1KIJR3-000EAw-1P@cs1.cs.huji.ac.il> References: <487A40FE.7030102@dmv.com> <E1KIJR3-000EAw-1P@cs1.cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-yOAB+Ws8kJqa7LgyeOFw
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
On Mon, 2008-07-14 at 11:29 +0300, Danny Braniss wrote:
> > FreeBSD 7.0
> >=20
> > I have 2 machines with identical configurations/hardware, let's call th=
em A (master)=20
> > and B (slave). I have installed iscsi-target from ports and have set up=
3 targets=20
> > representing the 3 drives I wish to be connected to from A.
> >=20
> > The Targets file:
> > # extents file start length
> > extent0 /dev/da1 0 465GB
> > extent1 /dev/da2 0 465GB
> > extent2 /dev/da3 0 465GB
> >=20
> > # target flags storage netmask
> > target0 rw extent0 192.168.0.1/24
> > target1 rw extent1 192.168.0.1/24
> > target2 rw extent2 192.168.0.1/24
> >=20
> > I then start up iscsi_target and all is good.
> >=20
> > Now on A I have set up my /etc/iscsi.conf file as follows:
> >=20
> > # cat /etc/iscsi.conf
> > data1 {
> > targetaddress=3D192.168.0.252
> > targetname=3Diqn.1994-04.org.netbsd.iscsi-target:target0
> > initiatorname=3Diqn.2005-01.il.ac.huji.cs::BSD-2-1.sven.local
> > }
> > data2 {
> > targetaddress=3D192.168.0.252
> > targetname=3Diqn.1994-04.org.netbsd.iscsi-target:target1
> > initiatorname=3Diqn.2005-01.il.ac.huji.cs::BSD-2-1.sven.local
> > }
> > data3 {
> > targetaddress=3D192.168.0.252
> > targetname=3Diqn.1994-04.org.netbsd.iscsi-target:target2
> > initiatorname=3Diqn.2005-01.il.ac.huji.cs::BSD-2-1.sven.local
> > }
> >=20
> > So far so good, now come the issues. First of all, it would appear that=
with=20
> > iscontrol one can only start one "named" session at a time; for example
> > /sbin/iscontrol -n data1
> > /sbin/iscontrol -n data2
> > /sbin/isconrtol -n data3
> >=20
> > I guess that is ok, except that each invocation of iscontrol resets the=
other=20
> > sessions. Here is the camcontrol and dmesg output from running the abov=
e 3 commands.
> >=20
> > # camcontrol devlist
> > <AMCC 9550SXU-8L DISK 3.08> at scbus0 target 0 lun 0 (pass0,da0)
> > <AMCC 9550SXU-8L DISK 3.08> at scbus0 target 1 lun 0 (pass1,da1)
> > <AMCC 9550SXU-8L DISK 3.08> at scbus0 target 2 lun 0 (pass2,da2)
> > <AMCC 9550SXU-8L DISK 3.08> at scbus0 target 3 lun 0 (pass3,da3)
> > <NetBSD NetBSD iSCSI 0> at scbus1 target 0 lun 0 (da5,pass5)
> > <NetBSD NetBSD iSCSI 0> at scbus1 target 1 lun 0 (da6,pass6)
> > <NetBSD NetBSD iSCSI 0> at scbus1 target 2 lun 0 (da4,pass4)
> >=20
> >=20
> > [ /sbin/iscontrol -n data1 ]
> > da4 at iscsi0 bus 0 target 0 lun 0
> > da4: <NetBSD NetBSD iSCSI 0> Fixed Direct Access SCSI-3 device
> >=20
> > [ /sbin/iscontrol -n data2 ]
> > (da4:iscsi0:0:0:0): lost device
> > (da4:iscsi0:0:0:0): removing device entry
> > da4 at iscsi0 bus 0 target 0 lun 0
> > da4: <NetBSD NetBSD iSCSI 0> Fixed Direct Access SCSI-3 device
> > da5 at iscsi0 bus 0 target 1 lun 0
> > da5: <NetBSD NetBSD iSCSI 0> Fixed Direct Access SCSI-3 device
> >=20
> > [ /sbin/iscontrol -n data3 ]
> > (da4:iscsi0:0:0:0): lost device
> > (da4:iscsi0:0:0:0): removing device entry
> > (da5:iscsi0:0:1:0): lost device
> > (da5:iscsi0:0:1:0): removing device entry
> > da4 at iscsi0 bus 0 target 2 lun 0
> > da4: <NetBSD NetBSD iSCSI 0> Fixed Direct Access SCSI-3 device
> > da5 at iscsi0 bus 0 target 0 lun 0
> > da5: <NetBSD NetBSD iSCSI 0> Fixed Direct Access SCSI-3 device
> > da6 at iscsi0 bus 0 target 1 lun 0
> > da6: <NetBSD NetBSD iSCSI 0> Fixed Direct Access SCSI-3 device
> >=20
> >=20
> > It would appear that rather than appending the new device to the end of=
the "da"=20
> > devices, it starts to do some type of naming queue after the second dev=
ice. If I am=20
> > to use these devices in any type of automated setup, how can make sure =
that after=20
> > these commands, "da6" will always be target 1 (i.e. /dev/da2 on the sla=
ve machine).
> >=20
> > Next, there is no "startup" script for iscontrol - would that simply ha=
ve to be=20
> > added the system or is there a way with sysctl that it could be done. T=
he plan here=20
> > is use gmirror such that /dev/da1 on A is mirrored with the /dev/da1 on=
B using iscsi.
>=20
> Hi Sven,
> I just tried it here, and it seems that at the end all is ok :-)
> I think the lost/removing/found has something to do to iscontrol calling
> camcontrol rescan - I will check this later, but the end result is that
> you should have all /dev/da's.
> I don't see any reasonable safe way to tie a scsi# (/dev/dan),
> except to label (see glabel) the disk.
> The startup script is, at the moment, not trivial, but I'm attaching
> it, so someone can suggest improvements :-)
> #!/bin/sh
>=20
> # PROVIDE: iscsi
> # REQUIRE: NETWORKING
> # BEFORE: DAEMON
> # KEYWORD: nojail shutdown
>=20
> #
> # Add the following lines to /etc/rc.conf to enable iscsi:
> #
> # iscsi_enable=3D"YES"
> # iscsi_fstab=3D"/etc/fstab.iscsi"
>=20
> . /etc/rc.subr
> . /cs/share/etc/rc.subr
>=20
> name=3Discsi
> rcvar=3D`set_rcvar`
>=20
> command=3D/sbin/iscontrol
>=20
> iscsi_enable=3D${iscsi_enable:-"NO"}
> iscsi_fstab=3D${iscsi_fstab:-"/etc/fstab.iscsi"}
> iscsi_exports=3D${iscsi_exports:-"/etc/exports.iscsi"}
> iscsi_debug=3D${iscsi_debug:-0}
> start_cmd=3D"iscsi_start"
> faststop_cmp=3D"iscsi_stop"
> stop_cmd=3D"iscsi_stop"
>=20
> start_precmd=3D"iscontrol_precmd"
> iscontrol_prog=3D${iscontrol_prog:-"iscontrol"}
> iscontrol_log=3D${iscontrol_log:-"/var/log/$iscontrol_prog"}
> iscontrol_syslog=3D${iscontrol_syslog:-"644 3 100 * JC"}
>=20
> iscontrol_precmd()
> {
> setup_syslog "$iscontrol_prog" "$iscontrol_log" "$iscontrol_syslog"
> }
>=20
> iscsi_wait()
> {
> dev=3D$1
> trap "echo 'wait loop cancelled'; exit 1" 2
> count=3D0
> while true; do
> if [ -c $dev ]; then
> break;
> fi
> if [ $count -eq 0 ]; then
> echo -n Waiting for ${dev}': '
> fi
> count=3D$((${count} + 1))
> if [ $count -eq 6 ]; then
> echo " Failed for dev=3D$dev"
> return 0
> break
> fi
> echo -n '.'
> sleep 5;
> done
> echo "$dev ok."
> return 1
> }
>=20
> iscsi_start()
> {
> #
> # load needed modules
> for m in iscsi_initiator geom_label; do
> kldstat -qm $m || kldload $m
> done
>=20
> sysctl debug.iscsi_initiator=3D$iscsi_debug
> #
> # start iscontrol for each target
> if [ -n "${iscsi_targets}" ]; then
> for target in ${iscsi_targets}; do
> ${command} ${rc_flags} -n ${target}
> done
> fi
>=20
> if [ -f "${iscsi_fstab}" ]; then
> while read spec file type opt t1 t2
> do
> case ${spec} in
> \#*|'')
> ;;
> *)
> if iscsi_wait ${spec}; then
> break;
> fi
> echo type=3D$type spec=3D$spec file=3D$file
> fsck -p ${spec} && mkdir -p ${file} && mount ${spec} ${file}
> chmod 755 ${file}=20
> ;;
> esac
> done < ${iscsi_fstab}=20
> fi
>=20
> if [ -f "${iscsi_exports}" ]; then
> cat ${iscsi_exports} >> /etc/exports
> #/etc/rc.d/mountd reload
> kill -1 `cat /var/run/mountd.pid`
> fi
> }
>=20
> iscsi_stop()
> {
> echo 'iscsi stopping'
> while read spec file type opt t1 t2
> do
> case ${spec} in
> \#*|'')
> ;;
> *)
> echo iscsi: umount $spec
> umount -fv $spec
> ;;
> esac
> done < ${iscsi_fstab}=20
> }
>=20
> load_rc_config $name
> run_rc_command "$1"
Thanks for the script and information. I have found that so long as I
add the targets in the same order each time (i.e. iscontrol -n data1,
then data2, then data3) the actual scsi target will be the same even
though the device number will change. So if I do something like:
camcontrol devlist | grep "scbus1 target 0" | sed 's/^.*da\(.\).*/da\1/'
I can parse out the device number it was assigned.
After some other reports it would appear as though iscsi will not work
for my needs (which basically involves creating a mirrored
pool/filesystem spanning 2 machines) as iscsi will lock when the target
machine goes down. I will continue this in a new thread as it strays
from the question originally asked here.
Sven
--=-yOAB+Ws8kJqa7LgyeOFw
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
iD8DBQBIfKnbSnmnd8q3JGsRAmJyAKCFEY81PrYtt6ti58vHxJxn+gdj1ACgrL/9
XlMrGEX8+VsCqcaaQNafdmo=
=ez5a
-----END PGP SIGNATURE-----
--=-yOAB+Ws8kJqa7LgyeOFw--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1216129506.27608.8.camel>