Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 10 Aug 2016 11:54:01 +0200
From:      Ben RUBSON <ben.rubson@gmail.com>
To:        freebsd-fs@freebsd.org
Subject:   [iSCSI] Trying to reach max disk throughput
Message-ID:  <6B32251D-49B4-4E61-A5E8-08013B15C82B@gmail.com>

next in thread | raw e-mail | index | archive | help
Hello,

I'm facing something strange with iSCSI, I can't manage to reach the =
expected disk throughput using one (read or write) thread.



### Target : local disk throughput, one thread :

# dd if=3D/dev/da8 of=3D/dev/null bs=3D$((128*1024)) count=3D81920
10737418240 bytes transferred in 22.127838 secs (485244798 bytes/sec) - =
462MB/s



### Initiator : network throughput to target, one thread :

# iperf -c 192.168.2.2 -t 30 -i 5 -P 1 -l 128KB
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec  19.5 GBytes  33.5 Gbits/sec
[  3]  5.0-10.0 sec  19.7 GBytes  33.9 Gbits/sec
[  3] 10.0-15.0 sec  19.6 GBytes  33.6 Gbits/sec
[  3] 15.0-20.0 sec  19.6 GBytes  33.7 Gbits/sec
[  3] 20.0-25.0 sec  19.8 GBytes  34.0 Gbits/sec
[  3] 25.0-30.0 sec  19.9 GBytes  34.2 Gbits/sec



### Initiator : network latency to target :

# ping -c 10 192.168.2.2=20
64 bytes from 192.168.2.2: icmp_seq=3D0 ttl=3D64 time=3D0.025 ms
64 bytes from 192.168.2.2: icmp_seq=3D1 ttl=3D64 time=3D0.024 ms
64 bytes from 192.168.2.2: icmp_seq=3D2 ttl=3D64 time=3D0.027 ms
64 bytes from 192.168.2.2: icmp_seq=3D3 ttl=3D64 time=3D0.021 ms
64 bytes from 192.168.2.2: icmp_seq=3D4 ttl=3D64 time=3D0.020 ms
64 bytes from 192.168.2.2: icmp_seq=3D5 ttl=3D64 time=3D0.025 ms
64 bytes from 192.168.2.2: icmp_seq=3D6 ttl=3D64 time=3D0.022 ms
64 bytes from 192.168.2.2: icmp_seq=3D7 ttl=3D64 time=3D0.020 ms
64 bytes from 192.168.2.2: icmp_seq=3D8 ttl=3D64 time=3D0.022 ms
64 bytes from 192.168.2.2: icmp_seq=3D9 ttl=3D64 time=3D0.023 ms
round-trip min/avg/max/stddev =3D 0.020/0.023/0.027/0.002 ms



### Initiator : iscsi disk throughput :

## dd if=3D/dev/da8 of=3D/dev/null bs=3D$((128*1024)) count=3D81920
10737418240 bytes transferred in 34.731815 secs (309152234 bytes/sec) - =
295MB/s

With 2 parallel dd jobs : 345MB/s
With 4 parallel dd jobs : 502MB/s



### Questions :

Why such a difference ?
Where are the 167MB/s (462-295) lost ?

All CPUs, on both sides, are above 90% idle during these tests.
I tried to increase net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max.
I also increased SOCKBUF_SIZE in iscsid.h and ctld.h.
And tried HTCP as the TCP algorithm.
But with no chance.

Any idea ?

Many thanks !

Ben




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6B32251D-49B4-4E61-A5E8-08013B15C82B>