Date: Mon, 5 Oct 2015 22:32:04 +0000 (UTC) From: Pallav Bose <pallav_bose@yahoo.com> To: "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org> Subject: Poor SSD performance with Dell PERC H730 Mini attached to the mfi(4) driver on FreeBSD 10.2 Message-ID: <363271937.243741.1444084324524.JavaMail.yahoo@mail.yahoo.com> References: <363271937.243741.1444084324524.JavaMail.yahoo@mail.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, Hello, I have two Dell PowerEdge R430 servers - dell13g-mfi and dell13g-mrsas - both with PERC H730 Mini RAID controllers. I have FreeBSD 10.2 running on each server. One server is using the mfi(4) driver, and the other the mrsas(4) driver. I'm observing very poor SSD throughput on the server using the mfi(4) driver. I saw similarly poor throughput on FreeBSD 8.0 with a patched mfi(4) driver and posted a question (https://lists.freebsd.org/pipermail/freebsd-questions/2015-September/268349.html), so I tried FreeBSD 10.2 and see similar results. dell13g-mfi info: root@dell13g-mfi:~ # uname -a FreeBSD dell13g-mfi 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@dell13g-mfi:~ # cat /boot/loader.conf kern.geom.label.gptid.enable="0" zfs_load="YES" root@dell13g-mfi:~ # pciconf -bclveV mfi0 mfi0@pci0:1:0:0: class=0x010400 card=0x1f491028 chip=0x005d1000 rev=0x02 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = 'MegaRAID SAS-3 3108 [Invader]' class = mass storage subclass = RAID bar [10] = type I/O Port, range 32, base 0x2000, size 256, enabled bar [14] = type Memory, range 64, base 0x91d00000, size 65536, enabled bar [1c] = type Memory, range 64, base 0x91c00000, size 1048576, enabled cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0 cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8) speed 8.0(8.0) ASPM disabled(L0s) cap 03[d0] = VPD cap 05[a8] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message cap 11[c0] = MSI-X supports 97 messages Table in map 0x14[0xe000], PBA in map 0x14[0xf000] ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0019[1e0] = PCIe Sec 1 lane errors 0 ecap 0004[1c0] = Power Budgeting 1 ecap 000e[148] = ARI 1 PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error dell13g-mrsas info: root@dell13g-mrsas:~ # uname -a FreeBSD dell13g-mrsas 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@dell13g-mrsas:~ # cat /boot/loader.conf kern.geom.label.gptid.enable="0" zfs_load="YES" hw.mfi.mrsas_enable=1 root@dell13g-mrsas:~ # pciconf -bclveV mrsas0 mrsas0@pci0:1:0:0: class=0x010400 card=0x1f491028 chip=0x005d1000 rev=0x02 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = 'MegaRAID SAS-3 3108 [Invader]' class = mass storage subclass = RAID bar [10] = type I/O Port, range 32, base 0x2000, size 256, enabled bar [14] = type Memory, range 64, base 0x91d00000, size 65536, enabled bar [1c] = type Memory, range 64, base 0x91c00000, size 1048576, enabled cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0 cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8) speed 8.0(8.0) ASPM disabled(L0s) cap 03[d0] = VPD cap 05[a8] = MSI supports 1 message, 64 bit, vector masks cap 11[c0] = MSI-X supports 97 messages, enabled Table in map 0x14[0xe000], PBA in map 0x14[0xf000] ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0019[1e0] = PCIe Sec 1 lane errors 0 ecap 0004[1c0] = Power Budgeting 1 ecap 000e[148] = ARI 1 PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error A simple read from one of the SSDs shows very poor performance in case of mfi(4). root@dell13g-mfi:~ # dd if=/dev/mfid1 of=/dev/null bs=1m count=100 100+0 records in 100+0 records out 104857600 bytes transferred in 19.956666 secs (5254264 bytes/sec) Compare that with mrsas(4). root@dell13g-mrsas:~ # dd if=/dev/da1 of=/dev/null bs=1m count=100 100+0 records in 100+0 records out 104857600 bytes transferred in 0.248799 secs (421455329 bytes/sec) High CPU usage by swi6 on the server using mfi(4). root@dell13g-mfi:~ # top -PSHI last pid: 1779; load averages: 0.55, 0.50, 0.44 up 0+06:39:27 14:49:45 244 processes: 7 running, 223 sleeping, 14 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 64.6% interrupt, 35.4% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 4: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle CPU 5: 0.0% user, 0.0% nice, 0.0% system, 5.8% interrupt, 94.2% idle Mem: 3256K Active, 25M Inact, 588M Wired, 30G Free ARC: 207M Total, 51M MFU, 147M MRU, 16K Anon, 1864K Header, 7317K Other Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 96K CPU0 0 395:01 100.00% idle{idle: cpu0} 11 root 155 ki31 0K 96K CPU3 3 361:02 100.00% idle{idle: cpu3} 11 root 155 ki31 0K 96K CPU2 2 357:53 100.00% idle{idle: cpu2} 11 root 155 ki31 0K 96K RUN 4 358:00 99.85% idle{idle: cpu4} 11 root 155 ki31 0K 96K CPU5 5 385:17 99.27% idle{idle: cpu5} 11 root 155 ki31 0K 96K CPU1 1 362:22 54.88% idle{idle: cpu1} 12 root -52 - 0K 224K WAIT 1 161:05 48.19% intr{swi6: task queue} 12 root -88 - 0K 224K WAIT 5 13:57 4.05% intr{irq264: mfi0} Interrupt rate is also on the higher side in the absence of disk activity. mfi(4): root@dell13g-mfi:~ # vmstat -i interrupt total rate irq9: acpi0 1 0 irq18: ehci0 ehci1 74619 3 cpu0:timer 779089 32 irq264: mfi0 2645728 110 irq265: bge0 117996 4 cpu2:timer 2862930 119 cpu4:timer 2888148 120 cpu3:timer 2650936 110 cpu1:timer 2567105 106 cpu5:timer 971503 40 Total 15558055 647 Compare the above with mrsas(4): root@dell13g-mrsas:~ # vmstat -i interrupt total rate irq9: acpi0 1 0 irq18: ehci0 ehci1 77600 3 cpu0:timer 557554 22 irq264: mrsas0 10682 0 irq265: mrsas0 8230 0 irq266: mrsas0 8278 0 irq267: mrsas0 8328 0 irq268: mrsas0 8466 0 irq269: mrsas0 8849 0 irq270: bge0 134170 5 cpu4:timer 153721 6 cpu2:timer 145912 6 cpu3:timer 148140 6 cpu1:timer 154170 6 cpu5:timer 108449 4 Total 1532550 63 Procstat output. root@dell13g-mfi:~ # procstat -kk 12 PID TID COMM TDNAME KSTACK 12 100009 intr swi1: netisr 0 mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe 12 100010 intr swi4: clock mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe 12 100011 intr swi4: clock 12 100012 intr swi4: clock 12 100013 intr swi4: clock 12 100014 intr swi4: clock 12 100015 intr swi4: clock 12 100016 intr swi3: vm 12 100023 intr swi6: task queue <running> 12 100024 intr swi6: Giant task mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe 12 100026 intr swi5: fast taskq 12 100033 intr irq264: mfi0 mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe 12 100034 intr irq18: ehci0 ehc mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe 12 100047 intr swi0: uart uart Dtrace output. root@dell13g-mfi:~ # kldload dtraceall root@dell13g-mfi:~ # dtrace -n 'profile:::profile-276hz { @pc[stack()]=count(); }' dtrace: description 'profile:::profile-276hz ' matched 1 probe ^C kernel`copyin+0x4b 1 kernel`sched_idletd+0x1bb 1 kernel`spinlock_exit+0x2d kernel`mfi_tbolt_complete_cmd+0x230 kernel`mfi_intr_tbolt+0x60 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 1 kernel`__mtx_lock_sleep+0x210 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 5 kernel`__mtx_lock_sleep+0x221 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 7 kernel`__mtx_lock_sleep+0x21c kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 10 kernel`get_tsc+0x4 kernel`mfi_data_cb+0x24c kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 13 kernel`get_tsc+0x1 kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 17 kernel`__mtx_lock_sleep+0x218 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 22 kernel`DELAY+0xd8 kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 32 kernel`DELAY+0xd0 kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 38 kernel`__mtx_lock_sleep+0x215 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 39 kernel`DELAY+0xf1 kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 43 kernel`DELAY+0xe8 kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 47 kernel`__mtx_lock_sleep+0x212 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 47 kernel`DELAY+0xd2 kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 187 kernel`get_tsc kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 463 kernel`get_tsc+0x6 kernel`mfi_data_cb+0x24c kernel`bus_dmamap_load+0xd1 kernel`mfi_mapcmd+0x9a kernel`mfi_startio+0x3a0 kernel`mfi_wait_command+0x8a kernel`mfi_tbolt_sync_map_info+0x75 kernel`mfi_handle_map_sync+0x4c kernel`taskqueue_run_locked+0xe5 kernel`taskqueue_run+0x81 kernel`intr_event_execute_handlers+0xab kernel`ithread_loop+0x96 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 654 kernel`acpi_cpu_c1+0x6 kernel`sched_idletd+0x1d5 kernel`fork_exit+0x9a kernel`0xffffffff80d30d2e 13283 storcli doesn't work on either of the servers. root@dell13g-mrsas:~ # storcli show all Status Code = 0 Status = Success Description = None Number of Controllers = 0 Host Name = dell13g-mrsas Operating System = FreeBSD10.2-RELEASE root@dell13g-mrsas:~ # storcli64 show all Status Code = 0 Status = Success Description = None Number of Controllers = 0 Host Name = dell13g-mrsas Operating System = FreeBSD10.2-RELEASE root@dell13g-mfi:~ # storcli show all Status Code = 0 Status = Success Description = None Number of Controllers = 0 Host Name = dell13g-mfi Operating System = FreeBSD10.2-RELEASE root@dell13g-mfi:~ # storcli64 show all Status Code = 0 Status = Success Description = None Number of Controllers = 0 Host Name = dell13g-mfi Operating System = FreeBSD10.2-RELEASE Can someone help me debug this problem? Regards, Pallav
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?363271937.243741.1444084324524.JavaMail.yahoo>