Date: Wed, 10 Aug 2016 11:54:01 +0200 From: Ben RUBSON <ben.rubson@gmail.com> To: freebsd-fs@freebsd.org Subject: [iSCSI] Trying to reach max disk throughput Message-ID: <6B32251D-49B4-4E61-A5E8-08013B15C82B@gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, I'm facing something strange with iSCSI, I can't manage to reach the = expected disk throughput using one (read or write) thread. ### Target : local disk throughput, one thread : # dd if=3D/dev/da8 of=3D/dev/null bs=3D$((128*1024)) count=3D81920 10737418240 bytes transferred in 22.127838 secs (485244798 bytes/sec) - = 462MB/s ### Initiator : network throughput to target, one thread : # iperf -c 192.168.2.2 -t 30 -i 5 -P 1 -l 128KB [ ID] Interval Transfer Bandwidth [ 3] 0.0- 5.0 sec 19.5 GBytes 33.5 Gbits/sec [ 3] 5.0-10.0 sec 19.7 GBytes 33.9 Gbits/sec [ 3] 10.0-15.0 sec 19.6 GBytes 33.6 Gbits/sec [ 3] 15.0-20.0 sec 19.6 GBytes 33.7 Gbits/sec [ 3] 20.0-25.0 sec 19.8 GBytes 34.0 Gbits/sec [ 3] 25.0-30.0 sec 19.9 GBytes 34.2 Gbits/sec ### Initiator : network latency to target : # ping -c 10 192.168.2.2=20 64 bytes from 192.168.2.2: icmp_seq=3D0 ttl=3D64 time=3D0.025 ms 64 bytes from 192.168.2.2: icmp_seq=3D1 ttl=3D64 time=3D0.024 ms 64 bytes from 192.168.2.2: icmp_seq=3D2 ttl=3D64 time=3D0.027 ms 64 bytes from 192.168.2.2: icmp_seq=3D3 ttl=3D64 time=3D0.021 ms 64 bytes from 192.168.2.2: icmp_seq=3D4 ttl=3D64 time=3D0.020 ms 64 bytes from 192.168.2.2: icmp_seq=3D5 ttl=3D64 time=3D0.025 ms 64 bytes from 192.168.2.2: icmp_seq=3D6 ttl=3D64 time=3D0.022 ms 64 bytes from 192.168.2.2: icmp_seq=3D7 ttl=3D64 time=3D0.020 ms 64 bytes from 192.168.2.2: icmp_seq=3D8 ttl=3D64 time=3D0.022 ms 64 bytes from 192.168.2.2: icmp_seq=3D9 ttl=3D64 time=3D0.023 ms round-trip min/avg/max/stddev =3D 0.020/0.023/0.027/0.002 ms ### Initiator : iscsi disk throughput : ## dd if=3D/dev/da8 of=3D/dev/null bs=3D$((128*1024)) count=3D81920 10737418240 bytes transferred in 34.731815 secs (309152234 bytes/sec) - = 295MB/s With 2 parallel dd jobs : 345MB/s With 4 parallel dd jobs : 502MB/s ### Questions : Why such a difference ? Where are the 167MB/s (462-295) lost ? All CPUs, on both sides, are above 90% idle during these tests. I tried to increase net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max. I also increased SOCKBUF_SIZE in iscsid.h and ctld.h. And tried HTCP as the TCP algorithm. But with no chance. Any idea ? Many thanks ! Ben
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6B32251D-49B4-4E61-A5E8-08013B15C82B>