Date: Fri, 22 Jan 2010 03:05:22 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> To: Daniel Braniss <danny@cs.huji.ac.il> Cc: freebsd-scsi@freebsd.org Subject: Re: performance tuning of iSCSI and Dell MD3000i Message-ID: <4B5907E2.2060206@quip.cz> In-Reply-To: <E1NXs1v-0008L0-Nx@kabab.cs.huji.ac.il> References: <4B57263E.7040809@quip.cz> <E1NXs1v-0008L0-Nx@kabab.cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------010604060208080102020106
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Daniel Braniss wrote:
> Hi Miroslav,
>> Hi,
>>
>> [1]
>> I am using iSCSI Initiator 2.2.3 (on 7-STABLE amd64) and Dell MD3000i as
>> storage for server distributing downloads of large files (10-200MB)
>> through Lighttpd daemon.
>> Is there any tips for performance tuning?
>>
>> My iscsi.conf is:
>> styx_storage_1 {
>> initiatorname = iqn.2005-01.il.ac.huji.cs::dust.example.com
>> TargetName =
>> iqn.1984-05.com.dell:powervault.md3000i.60026b900042587b000000004ae58efc
>> TargetAddress = 192.168.130.101:3260,1
>> tags = 64
>> }
>>
>> Lighttpd is serving about 200-300 client connections at speed 220Mb/s it
>> is about 215Mb/s on iSCSI network interface to MD3000i. (iostat showed
>> 100% busy of da0)
>> Is it really maximum achievable speed on this type of iSCSI storage?
>> (made of 4 SATA drives in RAID5)
>> Or can it be better with some tuned settings?
>>
>
> the limiting factor is mainly the network, I have no experience with the
> MD3000i,
> we have MD3000 and they are very fast, unfourtunately, I can't come up
> with real statistical values, since the raid+zfs are doing very aggresive
> caching and my benchmarks are useless.
> In any case, the values you are getting seem a bit low, I would first measure
> the speed on the server, if your net is GB, then you should expect something
> around 80-100 MB/s.
The network is gigabit. Internet connection is through 3Com 2928 switch,
connection between server and MD3000i is crosslink with Jumbo Frames
(9k) enabled on server and storage array too.
Internet NIC:
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:22:11:64:0d:ff
inet AA.BB.CC.DD netmask 0xffffff80 broadcast AA.BB.CC.EE
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
iSCSI NIC:
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
9000
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:22:11:64:0e:01
inet 192.168.130.2 netmask 0xffffff00 broadcast 192.168.130.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
I can achieve higher speed with sequential read test by dd command. With
dd I can read about 680Mbps (85MB/s), but I think the high concurrency
workload is more stressing disk than network.
Is there some interesting sysctl which I can post you / tune network
performance?
>> Have anyone experiences with MD3000i - if I add more disk drives in to
>> RAID5, will it help? How much?
>>
>>
>> [2]
>> Is somewhere final rc script for iscsi initiator which can be used to
>> mount filesystem from iSCSI target at boot time by standard rc
>> procedure? Why is it not included in the base system if iscsi_initiator
>> is there for a long time?
>
> I guess I should work on that :-(
Is the attached version the more recent one? I found it in the list
http://lists.freebsd.org/pipermail/freebsd-stable/2008-July/043611.html
Is it really necessary to run it after normal mount stage, so iSCSI
devices must be defined in fstab.iscsi and fscked & mounted by
rc.d/iscsi instead of standard system procedures? (I am just asking as I
am not so skilled in rc.d scripting)
Did you talk about it with some freebsd-rc@ guys?
>> [3]
>> And one other thing, I got following error logged in messages after 2
>> days of running some performance tests:
> can you check the M3000i logs at that time? or else let me know
> what tests you are doing, so that i can try it here?
There is nothing in the Event Log of MD3000i in the time of this error.
The test was relatively simple. There is Lighttpd runing on the server
(with server.max-worker = 4 to have higher throughput) and I run
http_load (www/http_load) from another machine connected to the same
switch.
http_load was ran in 4 instances with the following settings:
http_load -parallel 70 -seconds 43200 download.url
So it generates 280 parallel connections on to Lighttpd with URLs from
download.url file.
There is 24376 of lines (URLs to files) in this file.
>> Jan 19 05:36:50 dust kernel:>>> so_recv: have a problem, error=0
>> Jan 19 05:36:50 dust kernel: isc_in: sp->soc=0xffffff0028f14000
>> Jan 19 05:36:50 dust kernel: 0] isc_in: terminated, flags=39f so_count=2
>> so_state=2000 error=2080 proc=0xffffff0005ab1910
>> Jan 19 05:36:50 dust kernel: 0] isc_in: pid=963750 s]i gin_alfeudl(l3f0e)atu
>> Jan 19 05:36:50 dust kernel: r0e]: ifslca_gi=n0:
>> Jan 19 05:36:50 dust kernel: dropped ISC_CON_RUNNING
>> Jan 19 05:36:50 dust iscontrol[96375]: trapped signal 30
>> Jan 19 05:36:52 dust kernel: 0] i_send: len=492 ahs_len=0 ds_len=441
>> buf=444@0xffffff0007068000
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: maxRecvDataSegmentLength=65536
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.maXmitDataSegmentLength=65536
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.maxBurstLength=131072
>> Jan 19 05:36:52 dust kernel: 0] i_setopt:
>> opt.targetAddress='192.168.130.101'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt:
>> opt.targetName='iqn.1984-05.com.dell:powervault.md3000i.60026b900042587b000000004ae58efc'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt:
>> opt.initiatorName='iqn.2005-01.il.ac.huji.cs::dust.example.com'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.headerDigest='None'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.dataDigest='None'
>> Jan 19 05:36:52 dust kernel: 0] i_fullfeature: flag=2
>> Jan 19 05:36:52 dust kernel: 0] ism_restart: restart ...
>> Jan 19 05:36:52 dust kernel: 0] iscsi_requeue: last=30a30e4 n=30a30e4
>> Jan 19 05:36:52 dust kernel: 0] iscsi_requeue: last=30a30e4 n=30a30e5
[...]
>> Jan 19 05:36:52 dust kernel: 0] iscsi_requeue: last=30a30e4 n=30a312a
>> Jan 19 05:36:52 dust kernel: 0] ism_restart: restarted sn.cmd=0x30a312b
>> lastcmd=0x30a30e4
>>
>> Then transfer of files continued at slightly lower speed than before.
>
> it seems that there was a network glitch, but it recovered.
It is connected with direct crosslink cable and there is not "link state
changed to DOWN / UP" message for the NIC so I don't know what network
problem can happened.
Miroslav Lachman
--------------010604060208080102020106
Content-Type: text/plain;
name="rc.d_iscsi.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="rc.d_iscsi.txt"
#!/bin/sh
# PROVIDE: iscsi
# REQUIRE: NETWORKING
# BEFORE: DAEMON
# KEYWORD: nojail shutdown
#
# Add the following lines to /etc/rc.conf to enable iscsi:
#
# iscsi_enable="YES"
# iscsi_fstab="/etc/fstab.iscsi"
. /etc/rc.subr
. /cs/share/etc/rc.subr
name=iscsi
rcvar=`set_rcvar`
command=/sbin/iscontrol
iscsi_enable=${iscsi_enable:-"NO"}
iscsi_fstab=${iscsi_fstab:-"/etc/fstab.iscsi"}
iscsi_exports=${iscsi_exports:-"/etc/exports.iscsi"}
iscsi_debug=${iscsi_debug:-0}
start_cmd="iscsi_start"
faststop_cmp="iscsi_stop"
stop_cmd="iscsi_stop"
start_precmd="iscontrol_precmd"
iscontrol_prog=${iscontrol_prog:-"iscontrol"}
iscontrol_log=${iscontrol_log:-"/var/log/$iscontrol_prog"}
iscontrol_syslog=${iscontrol_syslog:-"644 3 100 * JC"}
iscontrol_precmd()
{
setup_syslog "$iscontrol_prog" "$iscontrol_log" "$iscontrol_syslog"
}
iscsi_wait()
{
dev=$1
trap "echo 'wait loop cancelled'; exit 1" 2
count=0
while true; do
if [ -c $dev ]; then
break;
fi
if [ $count -eq 0 ]; then
echo -n Waiting for ${dev}': '
fi
count=$((${count} + 1))
if [ $count -eq 6 ]; then
echo " Failed for dev=$dev"
return 0
break
fi
echo -n '.'
sleep 5;
done
echo "$dev ok."
return 1
}
iscsi_start()
{
#
# load needed modules
for m in iscsi_initiator geom_label; do
kldstat -qm $m || kldload $m
done
sysctl debug.iscsi_initiator=$iscsi_debug
#
# start iscontrol for each target
if [ -n "${iscsi_targets}" ]; then
for target in ${iscsi_targets}; do
${command} ${rc_flags} -n ${target}
done
fi
if [ -f "${iscsi_fstab}" ]; then
while read spec file type opt t1 t2
do
case ${spec} in
\#*|'')
;;
*)
if iscsi_wait ${spec}; then
break;
fi
echo type=$type spec=$spec file=$file
fsck -p ${spec} && mkdir -p ${file} && mount ${spec} ${file}
chmod 755 ${file}
;;
esac
done < ${iscsi_fstab}
fi
if [ -f "${iscsi_exports}" ]; then
cat ${iscsi_exports} >> /etc/exports
#/etc/rc.d/mountd reload
kill -1 `cat /var/run/mountd.pid`
fi
}
iscsi_stop()
{
echo 'iscsi stopping'
while read spec file type opt t1 t2
do
case ${spec} in
\#*|'')
;;
*)
echo iscsi: umount $spec
umount -fv $spec
;;
esac
done < ${iscsi_fstab}
}
load_rc_config $name
run_rc_command "$1"
--------------010604060208080102020106--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4B5907E2.2060206>