Date: Fri, 11 Apr 2014 17:23:13 -0700 From: hiren panchasara <hiren.panchasara@gmail.com> To: Patrick Kelsey <kelsey@ieee.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Adrian Chadd <adrian@freebsd.org> Subject: Re: netisr observations Message-ID: <CALCpEUFVtcs9HhEcZ=AEvEudpsc-=QyXjMg8MXFFdLiUHUf-kQ@mail.gmail.com> In-Reply-To: <CAD44qMUVLLw0UNTgaTZ74=Ktq46ROT9E%2BssrHznHPhqujScBkA@mail.gmail.com> References: <CALCpEUHhUkZ9b=2ynaN5-MkxOObs%2BO4RTsUhmhcMeC-WDnAxKg@mail.gmail.com> <CAJ-Vmo=TUVwuoWJeTCYvC-2sYvLRh%2BevACukS%2BaNHOaz9hwkrA@mail.gmail.com> <CAD44qMUVLLw0UNTgaTZ74=Ktq46ROT9E%2BssrHznHPhqujScBkA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Apr 11, 2014 at 11:30 AM, Patrick Kelsey <kelsey@ieee.org> wrote:
>
> The output of netstat -Q shows IP dispatch is set to default, which is
> direct (NETISR_DISPATCH_DIRECT). That means each IP packet will be
> processed on the same CPU that the Ethernet processing for that packet was
> performed on, so CPU selection for IP packets will not be based on flowid.
> The output of netstat -Q shows Ethernet dispatch is set to direct
> (NETISR_DISPATCH_DIRECT if you wind up reading the code), so the Ethernet
> processing for each packet will take place on the same CPU that the driver
> receives that packet on.
>
> For the igb driver with queues autoconfigured and msix enabled, as the
> sysctl output shows you have, the driver will create a number of queues
> subject to device limitations, msix message limitations, and the number of
> CPUs in the system, establish a separate interrupt handler for each one, and
> bind each of those interrupt handlers to a separate CPU. It also creates a
> separate single-threaded taskqueue for each queue. Each queue interrupt
> handler sends work to its associated taskqueue when the interrupt fires.
> Those taskqueues are where the Ethernet packets are received and processed
> by the driver. The question is where those taskqueue threads will be run.
> I don't see anything in the driver that makes an attempt to bind those
> taskqueue threads to specific CPUs, so really the location of all of the
> packet processing is up to the scheduler (i.e., arbitrary).
>
> The summary is:
>
> 1. the hardware schedules each received packet to one of its queues and
> raises the interrupt for that queue
> 2. that queue interrupt is serviced on the same CPU all the time, which is
> different from the CPUs for all other queues on that interface
> 3. the interrupt handler notifies the corresponding task queue, which runs
> its task in a thread on whatever CPU the scheduler chooses
> 4. that task dispatches the packet for Ethernet processing via netisr, which
> processes it on whatever the current CPU is
> 5. Ethernet processing dispatches that packet for IP processing via netisr,
> which processes it on whatever the current CPU is
I really appreciate you taking time and explaining this. Thank you.
I am specially confused with ip "Queued" column from netstat -Q
showing 203888563 only for cpu3. Does this mean that cpu3 queues
everything and then distributes among other cpus? Where does this
queuing on cpu3 happens out of 5 stages you mentioned above?
This value gets populated in snwp->snw_queued field for each cpu
inside sysctl_netisr_work().
>
> You might want to try changing the default netisr dispatch policy to
> 'deferred' (sysctl net.isr.dispatch=deferred). If you do that, the Ethernet
> processing will still happen on an arbitrary CPU chosen by the scheduler,
> but the IP processing should then get mapped to a CPU based on the flowid
> assigned by the driver. Since igb assigns flowids based on received queue
> number, all IP (and above) processing for that packet should then be
> performed on the same CPU the queue interrupt was bound to.
I will give this a try and see how things behave.
I was also thinking about net.isr.bindthreads. netisr_start_swi() does
intr_event_bind() if we have it bindthreads set to 1. What would that
gain me, if anything?
Would it stop moving intr{swi1: netisr 3} on to different cpus (as I
am seeing in 'top' o/p) and bind it to a single cpu?
I've came across a thread discussing some side-effects of this though:
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-January/037597.html
Thanks a ton, again.
cheers,
Hiren
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALCpEUFVtcs9HhEcZ=AEvEudpsc-=QyXjMg8MXFFdLiUHUf-kQ>