Date: Sat, 28 Mar 2015 21:10:26 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Adrian Chadd <adrian@freebsd.org> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: irq cpu binding Message-ID: <20150328181026.GB23643@zxy.spb.ru> In-Reply-To: <CAJ-VmokpWNy-arZ_=4hNbyAvzBYgDWu4F=vYSDwqPgpjaGHk=w@mail.gmail.com> References: <20150328112035.GZ23643@zxy.spb.ru> <CAJ-Vmo=sxtLgn1HABbWchu1Hm8kZ4Qy%2BscPeD7ezkkBzr%2BQijA@mail.gmail.com> <20150328154031.GA23643@zxy.spb.ru> <CAJ-VmokpWNy-arZ_=4hNbyAvzBYgDWu4F=vYSDwqPgpjaGHk=w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Mar 28, 2015 at 10:43:07AM -0700, Adrian Chadd wrote:
> On 28 March 2015 at 08:40, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> > On Sat, Mar 28, 2015 at 08:20:08AM -0700, Adrian Chadd wrote:
> >
> >> On 28 March 2015 at 04:20, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> >> > Can someone describe how on FreeBSD/amd64 do interrupt handling?
> >> > Can be interrupt handler (hardware interrupt) direct dispatch to
> >> > specific CPU core (and only to this core)?
> >> > Can be all work be only on this core (ithread, device driver interrupt
> >> > handler, finalise)?
> >>
> >> Yes - you can use cpuset on the interrupt to get them bound that way.
> >>
> >> John and I are trying to make that whole process more automated and
> >> NUMA friendly. I'm debugging some of his work at the moment.
> >
> > cpuset don't work as expected -- I see irq handling on other cpu.
>
> Well, when you see "irq handling on other cpu", what do you mean?
> How are you using cpuset to move things around?
I have dual Xenon and dual 82599EB (handled bu second socket).
irq270: ix0:que 0 293090597 2483
irq271: ix0:que 1 290379344 2460
irq272: ix0:que 2 292648245 2479
irq273: ix0:link 1 0
irq274: ix1:que 0 294816977 2498
irq275: ix1:que 1 292665696 2479
irq276: ix1:que 2 294411404 2494
irq277: ix1:link 2 0
First, I do from 'cpuset -l 6 -x 270' to 'cpuset -l 11 -x 276'.
I try 'cpuset -l 0 pmcstat -S CPU_CLK_UNHALTED_CORE -O sample.out -c 2 -l 10'
and next pmcstat -R sample.out -G out.txt.
And I see many ixgbe_msix_que in out.txt (I save out.txt, you can see this, plot flame graph, etc).
After this I see in `ps -axdHO lwp | grep ix` more then one irq handler:
# ps -axdHO lwp | grep ix
94661 100976 0 S+ 0:00.00 | | `-- grep ix
12 100064 - WL 840:32.50 - [intr/irq270: ix0:qu]
12 100066 - WL 843:33.09 - [intr/irq271: ix0:qu]
12 100068 - RL 828:25.61 - [intr/irq272: ix0:qu]
12 100070 - WL 0:00.00 - [intr/irq273: ix0:li]
12 100072 - RL 858:21.86 - [intr/irq274: ix1:qu]
12 100074 - RL 858:43.72 - [intr/irq275: ix1:qu]
12 100076 - WL 843:01.04 - [intr/irq276: ix1:qu]
12 100078 - WL 0:00.00 - [intr/irq277: ix1:li]
0 100065 - RLs 70:03.18 [kernel/ix0 que]
0 100067 - RLs 71:06.67 [kernel/ix0 que]
0 100069 - DLs 68:23.48 [kernel/ix0 que]
0 100071 - DLs 0:01.48 [kernel/ix0 linkq]
0 100073 - DLs 50:25.37 [kernel/ix1 que]
0 100075 - DLs 49:08.89 [kernel/ix1 que]
0 100077 - DLs 48:18.12 [kernel/ix1 que]
0 100079 - DLs 0:00.00 [kernel/ix1 linkq]
I am don't know what thread binded by 'cpuset -x'. I think intr/irqNNN (as more CPU spending).
I do 'cpuset -l 6 -t 100065' .. 'cpuset -l 11 -t 100077'.
After this I do `cpuset -l 0 pmcstat -S CPU_CLK_UNHALTED_CORE -O sample2.out -c 2 -l 10` and
`pmcstat -R sample2.out -G out2.txt` (I save out2.txt).
I still see ixgbe_msix_que:
06.94% [17981] tcp_output @ /boot/kernel/kernel
100.0% [17981] tcp_do_segment
100.0% [17981] tcp_input
100.0% [17981] ip_input
100.0% [17981] netisr_dispatch_src
100.0% [17981] ether_demux
100.0% [17981] ether_nh_input
100.0% [17981] netisr_dispatch_src
97.56% [17542] ixgbe_rxeof @ /boot/kernel/if_ixgbe.ko
75.54% [13252] ixgbe_msix_que
100.0% [13252] intr_event_execute_handlers @ /boot/kernel/kernel
100.0% [13252] ithread_loop
100.0% [13252] fork_exit
24.46% [4290] ixgbe_handle_que @ /boot/kernel/if_ixgbe.ko
100.0% [4290] taskqueue_run_locked @ /boot/kernel/kernel
100.0% [4290] taskqueue_thread_loop
100.0% [4290] fork_exit
02.44% [439] tcp_lro_flush
96.58% [424] ixgbe_rxeof @ /boot/kernel/if_ixgbe.ko
100.0% [424] ixgbe_msix_que
100.0% [424] intr_event_execute_handlers @ /boot/kernel/kernel
100.0% [424] ithread_loop
100.0% [424] fork_exit
03.42% [15] tcp_lro_rx
100.0% [15] ixgbe_rxeof @ /boot/kernel/if_ixgbe.ko
100.0% [15] ixgbe_msix_que
100.0% [15] intr_event_execute_handlers @ /boot/kernel/kernel
100.0% [15] ithread_loop
100.0% [15] fork_exit
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150328181026.GB23643>