Date: Thu, 31 Oct 2013 08:20:02 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Venkata Duvvuru <VenkatKumar.Duvvuru@emulex.com> Cc: "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Subject: Re: Freebsd-10.0-CURRENT problem in the bottom half Message-ID: <CAJ-Vmom0vbqcK5AOUXqiTxED13k-xaHC9tw2fWe=Ry0XRXqicg@mail.gmail.com> In-Reply-To: <BF3270C86E8B1349A26C34E4EC1C44CB2C72A773@CMEXMB1.ad.emulex.com> References: <BF3270C86E8B1349A26C34E4EC1C44CB2C72A773@CMEXMB1.ad.emulex.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
Can you please try 10-STABLE or 11-CURRENT? 10-CURRENT indicates that
you're a little behind in the source tree(s).
There's been a bit of work recently that may improve things in general for =
you.
Thanks!
-a
On 31 October 2013 07:00, Venkata Duvvuru
<VenkatKumar.Duvvuru@emulex.com> wrote:
> Hi,
> In Freebsd 10.0-current with Emulex's OCE driver, I observe that the bott=
om half is hogging all the CPU which is leading to system sluggishness. I u=
sed the same hardware to check the behavior on 9.1-RELEASE, everything is f=
ine, bottom half is not taking more than 10% of the CPU even at the line ra=
te speed. But with half the throughput of line rate in Freebsd-10.0-current=
all the CPUs peak and "top -aSCHIP" shows that it's all bottom half who is=
eating the CPU. I also tried with Intel's 10g NIC to see if it has got the=
same problem and interestingly it is also having the same problem, where b=
ottom half is hogging all the CPU.
>
> We did some profiling to check where the problem is. I'm pasting the prof=
iling results for reference here. Observation from the results is that uma=
_zalloc_arg=3D=3D> __mtx_lock_sleep is taking quite a bit of the CPU. We sl=
eep at a few places in the bottom half and I think the bottom half scheduli=
ng is what is causing the problem. Please let me know if you see something =
very obvious from the profiling data.
>
> I'm seeing similar problem with ixgbe driver as well uma_zalloc_arg=3D=3D=
> __mtx_lock_sleep is taking quite a bit of the CPU.
>
> Top command results are also pasted below for both 10.0-current (where th=
e problem is seen) and 9.1-RELEASE (where the problem is not seen).
>
> 42.99% [372357] __mtx_lock_sleep @ /boot/kernel/kernel
> 100.0% [372357] __mtx_lock_flags
> 80.37% [299245] sosend_generic
> 100.0% [299245] soo_write
> 17.27% [64322] uma_zalloc_arg
> 50.43% [32438] m_copym
> 45.85% [29490] m_getm2
> 03.72% [2393] tcp_output
> 00.00% [1] oce_alloc_rx_bufs @ /boot/kernel/oce.ko
> 02.35% [8762] _sleep @ /boot/kernel/kernel
> 100.0% [8762] sbwait
> 00.00% [11] in_matroute
> 100.0% [11] rtalloc1_fib
> 00.00% [10] uma_zfree_arg
> 60.00% [6] m_freem
> 40.00% [4] sbdrop_internal
> 00.00% [3] rtalloc1_fib
> 100.0% [3] rtalloc_ign_fib
> 00.00% [3] tcp_output
> 100.0% [3] tcp_usr_send
> 00.00% [1] _cv_wait
> 100.0% [1] usb_process
>
> 34.98% [303027] trash_ctor @ /boot/kernel/kernel
> 87.44% [264956] mb_ctor_clust
> 100.0% [264956] uma_zalloc_arg
> 100.0% [264956] m_getm2
> 12.51% [37921] mb_ctor_mbuf
> 100.0% [37921] uma_zalloc_arg
> 48.28% [18308] m_getm2
> 48.18% [18272] m_copym
> 03.54% [1341] tcp_output
> 00.05% [150] mb_ctor_pack
> 100.0% [150] uma_zalloc_arg
> 96.67% [145] m_getm2
> 03.33% [5] oce_alloc_rx_bufs @ /boot/kernel/oce.ko
>
> 04.36% [37786] __rw_wlock_hard @ /boot/kernel/kernel
> 100.0% [37786] _rw_wlock_cookie
> 99.88% [37741] tcp_usr_send
> 100.0% [37741] sosend_generic
> 00.11% [43] in_pcblookup_hash
> 100.0% [43] tcp_input
>
>
> Top -aSCHIP with Freebsd-10.0-CURRENT (throughput 4.5 Gbps)
> PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND
> 0 root -92 0 0K 960K - 11 4:20 100.00% [kernel{=
oce1 taskq}]
> 2325 root 103 0 70524K 6240K CPU5 5 0:39 100.00% iperf -c =
12.0.238.24 -t 180 -i 3 -P 10{iperf}
> 0 root -92 0 0K 960K CPU8 8 5:17 99.37% [kernel{o=
ce1 taskq}]
> 0 root -92 0 0K 960K CPU10 10 4:27 99.37% [kernel{o=
ce1 taskq}]
> 0 root -92 0 0K 960K CPU9 9 3:35 98.88% [kernel{o=
ce1 taskq}]
> 0 root -92 0 0K 960K CPU7 7 7:25 98.29% [kernel{o=
ce1 taskq}]
> 11 root 155 ki31 0K 256K RUN 12 14:55 93.90% [idle{idl=
e: cpu12}]
> 11 root 155 ki31 0K 256K CPU13 13 14:48 91.70% [idle{idl=
e: cpu13}]
> 11 root 155 ki31 0K 256K CPU14 14 18:35 89.79% [idle{idl=
e: cpu14}]
> 11 root 155 ki31 0K 256K RUN 15 20:55 87.99% [idle{idl=
e: cpu15}]
> 11 root 155 ki31 0K 256K RUN 4 19:35 73.68% [idle{idl=
e: cpu4}]
> 11 root 155 ki31 0K 256K RUN 0 19:47 66.89% [idle{idl=
e: cpu0}]
> 11 root 155 ki31 0K 256K CPU2 2 18:53 64.89% [idle{idl=
e: cpu2}]
> 11 root 155 ki31 0K 256K CPU6 6 19:55 61.18% [idle{idl=
e: cpu6}]
> 11 root 155 ki31 0K 256K CPU1 1 18:46 55.66% [idle{idl=
e: cpu1}]
> 11 root 155 ki31 0K 256K RUN 5 19:41 42.48% [idle{idl=
e: cpu5}]
> 2325 root 52 0 70524K 6240K sbwait 6 0:15 38.77% iperf -c 1=
2.0.238.24 -t 180 -i 3 -P 10{iperf}
> 11 root 155 ki31 0K 256K CPU3 3 19:06 36.38% [idle{idl=
e: cpu3}]
> 2325 root 49 0 70524K 6240K CPU1 1 0:14 35.06% iperf -c 1=
2.0.238.24 -t 180 -i 3 -P 10{iperf}
> 2325 root 44 0 70524K 6240K sbwait 0 0:13 33.59% iperf -c 1=
2.0.238.24 -t 180 -i 3 -P 10{iperf}
> 2325 root 42 0 70524K 6240K sbwait 2 0:13 33.15% iperf -c 1=
2.0.238.24 -t 180 -i 3 -P 10{iperf}
> 2325 root 41 0 70524K 6240K sbwait 6 0:12 29.98% iperf -c 1=
2.0.238.24 -t 180 -i 3 -P 10{iperf}
> 2325 root 36 0 70524K 6240K sbwait 15 0:09 25.10% iperf -c 1=
2.0.238.24 -t 180 -i 3 -P 10{iperf}
> 2325 root 34 0 70524K 6240K CPU13 13 0:09 24.46% iperf -c 1=
2.0.238.24 -t 180 -i 3 -P 10{iperf}
>
> Top -aSCHIP with Freebsd-9.1-RELEASE (throughput 9.6 Gbps)
> last pid: 16802; load averages: 0.68, 0.15, 0.05 =
=
up 3+06:17:33 07:35:29
> 187 processes: 15 running, 128 sleeping, 44 waiting
> CPU 0: 0.0% user, 0.0% nice, 13.8% system, 0.4% interrupt, 85.8% idle
> CPU 1: 0.0% user, 0.0% nice, 9.4% system, 0.8% interrupt, 89.8% idle
> CPU 2: 0.0% user, 0.0% nice, 10.2% system, 0.0% interrupt, 89.8% idle
> CPU 3: 0.0% user, 0.0% nice, 12.6% system, 0.0% interrupt, 87.4% idle
> CPU 4: 0.0% user, 0.0% nice, 13.4% system, 0.0% interrupt, 86.6% idle
> CPU 5: 0.4% user, 0.0% nice, 11.0% system, 0.0% interrupt, 88.6% idle
> CPU 6: 0.0% user, 0.0% nice, 16.9% system, 0.0% interrupt, 83.1% idle
> CPU 7: 0.0% user, 0.0% nice, 32.3% system, 0.0% interrupt, 67.7% idle
> CPU 8: 0.0% user, 0.0% nice, 21.7% system, 0.0% interrupt, 78.3% idle
> CPU 9: 0.0% user, 0.0% nice, 24.4% system, 0.0% interrupt, 75.6% idle
> CPU 10: 0.4% user, 0.0% nice, 6.3% system, 0.0% interrupt, 93.3% idle
> CPU 11: 0.0% user, 0.0% nice, 16.1% system, 0.0% interrupt, 83.9% idle
> Mem: 21M Active, 2201M Inact, 1094M Wired, 1236M Buf, 8552M Free
> Swap: 15G Total, 15G Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND
> 11 root 155 ki31 0K 192K CPU1 1 78.1H 95.75% [idle{idle:=
cpu1}]
> 11 root 155 ki31 0K 192K RUN 10 78.1H 95.07% [idle{idle:=
cpu10}]
> 11 root 155 ki31 0K 192K CPU4 4 78.2H 94.29% [idle{idle:=
cpu4}]
> 11 root 155 ki31 0K 192K CPU2 2 78.2H 94.19% [idle{idle:=
cpu2}]
> 11 root 155 ki31 0K 192K CPU5 5 78.2H 94.09% [idle{idle:=
cpu5}]
> 11 root 155 ki31 0K 192K CPU3 3 78.2H 94.09% [idle{idle:=
cpu3}]
> 11 root 155 ki31 0K 192K CPU6 6 78.1H 86.77% [idle{idle:=
cpu6}]
> 11 root 155 ki31 0K 192K CPU11 11 78.1H 86.57% [idle{idle:=
cpu11}]
> 11 root 155 ki31 0K 192K CPU0 0 77.5H 86.28% [idle{idle:=
cpu0}]
> 11 root 155 ki31 0K 192K RUN 8 78.1H 81.79% [idle{idle:=
cpu8}]
> 11 root 155 ki31 0K 192K CPU9 9 78.1H 78.47% [idle{idle:=
cpu9}]
> 11 root 155 ki31 0K 192K RUN 7 78.1H 76.86% [idle{idle:=
cpu7}]
> 0 root -92 0 0K 784K - 7 0:08 26.37% [kernel{oce=
2 taskq}]
> 0 root -92 0 0K 784K CPU4 9 0:06 22.36% [kernel{oce=
2 taskq}]
> 0 root -92 0 0K 784K - 8 0:07 20.65% [kernel{oce=
2 taskq}]
> 0 root -92 0 0K 784K CPU3 6 0:06 15.48% [kernel{oce=
2 taskq}]
> 0 root -92 0 0K 784K - 0 0:04 10.99% [kernel{oce=
2 taskq}]
> 0 root -92 0 0K 784K - 11 0:04 10.89% [kernel{oce=
2 taskq}]
> 16801 root 26 0 65196K 5292K sbwait 4 0:03 10.50% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 23 0 65196K 5292K sbwait 0 0:02 6.79% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 23 0 65196K 5292K sbwait 10 0:01 6.05% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 24 0 65196K 5292K sbwait 5 0:02 5.96% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 24 0 65196K 5292K sbwait 2 0:01 5.66% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 22 0 65196K 5292K sbwait 10 0:01 4.79% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 23 0 65196K 5292K sbwait 11 0:01 4.59% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 23 0 65196K 5292K sbwait 10 0:01 3.96% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 22 0 65196K 5292K sbwait 10 0:01 3.96% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> 16801 root 21 0 65196K 5292K sbwait 3 0:01 3.27% iperf -c 12=
.0.238.24 -t 180 -i 3 -P 10{iperf}
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org=
"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmom0vbqcK5AOUXqiTxED13k-xaHC9tw2fWe=Ry0XRXqicg>