From owner-freebsd-net@FreeBSD.ORG Sat Apr 12 00:23:14 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6839C9AA; Sat, 12 Apr 2014 00:23:14 +0000 (UTC) Received: from mail-qa0-x22c.google.com (mail-qa0-x22c.google.com [IPv6:2607:f8b0:400d:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1839C10B8; Sat, 12 Apr 2014 00:23:14 +0000 (UTC) Received: by mail-qa0-f44.google.com with SMTP id hw13so5915331qab.17 for ; Fri, 11 Apr 2014 17:23:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=aolNYDhR0bYYU4OdhozqDTB5II36pR7yiTLntVUrSck=; b=Z/bsZl2+JnyeI3w6fJIqWYgATnj7ArxgtSDO4m6Cbe9u2GXn572y5iQamlabo+JTHK Mg4cGxQ/cAGarL3lZbi4cyCP6Imxncc/v93VFUO+UDMiHBFCB/Z99G49QJ8iENuCuRP0 BqF4ebFn8xUIulX2Y3075pb0+sOsLcKJTBX72k3Z9sRCnbDjQAPLf1p3qRgpweKoKteA lGEAxU1ZJZD7sCc+T6fC+Qr7n9higZptsLQlxHpZsiMx93rmddWLTii/h2X9Mhg5q2lq IhK/dIevpA5tO3t8Vb/NrkDDNHigTiAJfAYNevczsPivKU2RJR/ecc8JTFtoSbCcSYxT a63g== MIME-Version: 1.0 X-Received: by 10.140.41.200 with SMTP id z66mr6364421qgz.102.1397262193302; Fri, 11 Apr 2014 17:23:13 -0700 (PDT) Received: by 10.140.101.151 with HTTP; Fri, 11 Apr 2014 17:23:13 -0700 (PDT) In-Reply-To: References: Date: Fri, 11 Apr 2014 17:23:13 -0700 Message-ID: Subject: Re: netisr observations From: hiren panchasara To: Patrick Kelsey Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" , Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Apr 2014 00:23:14 -0000 On Fri, Apr 11, 2014 at 11:30 AM, Patrick Kelsey wrote: > > The output of netstat -Q shows IP dispatch is set to default, which is > direct (NETISR_DISPATCH_DIRECT). That means each IP packet will be > processed on the same CPU that the Ethernet processing for that packet was > performed on, so CPU selection for IP packets will not be based on flowid. > The output of netstat -Q shows Ethernet dispatch is set to direct > (NETISR_DISPATCH_DIRECT if you wind up reading the code), so the Ethernet > processing for each packet will take place on the same CPU that the driver > receives that packet on. > > For the igb driver with queues autoconfigured and msix enabled, as the > sysctl output shows you have, the driver will create a number of queues > subject to device limitations, msix message limitations, and the number of > CPUs in the system, establish a separate interrupt handler for each one, and > bind each of those interrupt handlers to a separate CPU. It also creates a > separate single-threaded taskqueue for each queue. Each queue interrupt > handler sends work to its associated taskqueue when the interrupt fires. > Those taskqueues are where the Ethernet packets are received and processed > by the driver. The question is where those taskqueue threads will be run. > I don't see anything in the driver that makes an attempt to bind those > taskqueue threads to specific CPUs, so really the location of all of the > packet processing is up to the scheduler (i.e., arbitrary). > > The summary is: > > 1. the hardware schedules each received packet to one of its queues and > raises the interrupt for that queue > 2. that queue interrupt is serviced on the same CPU all the time, which is > different from the CPUs for all other queues on that interface > 3. the interrupt handler notifies the corresponding task queue, which runs > its task in a thread on whatever CPU the scheduler chooses > 4. that task dispatches the packet for Ethernet processing via netisr, which > processes it on whatever the current CPU is > 5. Ethernet processing dispatches that packet for IP processing via netisr, > which processes it on whatever the current CPU is I really appreciate you taking time and explaining this. Thank you. I am specially confused with ip "Queued" column from netstat -Q showing 203888563 only for cpu3. Does this mean that cpu3 queues everything and then distributes among other cpus? Where does this queuing on cpu3 happens out of 5 stages you mentioned above? This value gets populated in snwp->snw_queued field for each cpu inside sysctl_netisr_work(). > > You might want to try changing the default netisr dispatch policy to > 'deferred' (sysctl net.isr.dispatch=deferred). If you do that, the Ethernet > processing will still happen on an arbitrary CPU chosen by the scheduler, > but the IP processing should then get mapped to a CPU based on the flowid > assigned by the driver. Since igb assigns flowids based on received queue > number, all IP (and above) processing for that packet should then be > performed on the same CPU the queue interrupt was bound to. I will give this a try and see how things behave. I was also thinking about net.isr.bindthreads. netisr_start_swi() does intr_event_bind() if we have it bindthreads set to 1. What would that gain me, if anything? Would it stop moving intr{swi1: netisr 3} on to different cpus (as I am seeing in 'top' o/p) and bind it to a single cpu? I've came across a thread discussing some side-effects of this though: http://lists.freebsd.org/pipermail/freebsd-hackers/2012-January/037597.html Thanks a ton, again. cheers, Hiren