From owner-freebsd-net@FreeBSD.ORG Fri Apr 11 18:30:47 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB05E8BE; Fri, 11 Apr 2014 18:30:47 +0000 (UTC) Received: from mail-lb0-x231.google.com (mail-lb0-x231.google.com [IPv6:2a00:1450:4010:c04::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 229B41CCC; Fri, 11 Apr 2014 18:30:46 +0000 (UTC) Received: by mail-lb0-f177.google.com with SMTP id z11so3744392lbi.22 for ; Fri, 11 Apr 2014 11:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=N8tW2BGPT1qKRq8wvR80x/auqRYVlgEB9X1AsS8cNuA=; b=rBJxbQXuR7mH3fcuL2NtTjSaff9fy7VFpTZWsw6FhKXw/3M7w5yCSXsApl+dXHUbaE YvTwxs122AbAtnEhphDrF2YUokMz4i/SQxKNhdb9aNRz07OLfn6nhKJtUx2bgc0gWwNI OiQfjSoXjKW0P6ss749mLf6MPf9hP+83XFC5qgheb4SKXpwM40Pf5wSN+emz7OPerYCE wnRXqcb9GyzL0e+HqMm6y2M+PeY9yp038lbG4/cztBEolCOSfHh3FVACBvYm49ONwzTp 1MXo/qK4zDXLesbLK8T18Q8bV/advHCB3BYwlKNFfHnKwYBRdLB4tYML0QTTCKytDmdB +BuQ== MIME-Version: 1.0 X-Received: by 10.152.44.234 with SMTP id h10mr106068lam.68.1397241044572; Fri, 11 Apr 2014 11:30:44 -0700 (PDT) Sender: pkelsey@gmail.com Received: by 10.112.141.196 with HTTP; Fri, 11 Apr 2014 11:30:44 -0700 (PDT) In-Reply-To: References: Date: Fri, 11 Apr 2014 14:30:44 -0400 X-Google-Sender-Auth: jcs4ZdF4AHAVEZhiKCkt9rQrdG0 Message-ID: Subject: Re: netisr observations From: Patrick Kelsey To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: "freebsd-net@freebsd.org" , hiren panchasara X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 18:30:47 -0000 On Fri, Apr 11, 2014 at 2:48 AM, Adrian Chadd wrote: > [snip] > > So, hm, the thing that comes to mind is the flowid. What's the various > flowid's for flows? Are they all mapping to CPU 3 somehow The output of netstat -Q shows IP dispatch is set to default, which is direct (NETISR_DISPATCH_DIRECT). That means each IP packet will be processed on the same CPU that the Ethernet processing for that packet was performed on, so CPU selection for IP packets will not be based on flowid. The output of netstat -Q shows Ethernet dispatch is set to direct (NETISR_DISPATCH_DIRECT if you wind up reading the code), so the Ethernet processing for each packet will take place on the same CPU that the driver receives that packet on. For the igb driver with queues autoconfigured and msix enabled, as the sysctl output shows you have, the driver will create a number of queues subject to device limitations, msix message limitations, and the number of CPUs in the system, establish a separate interrupt handler for each one, and bind each of those interrupt handlers to a separate CPU. It also creates a separate single-threaded taskqueue for each queue. Each queue interrupt handler sends work to its associated taskqueue when the interrupt fires. Those taskqueues are where the Ethernet packets are received and processed by the driver. The question is where those taskqueue threads will be run. I don't see anything in the driver that makes an attempt to bind those taskqueue threads to specific CPUs, so really the location of all of the packet processing is up to the scheduler (i.e., arbitrary). The summary is: 1. the hardware schedules each received packet to one of its queues and raises the interrupt for that queue 2. that queue interrupt is serviced on the same CPU all the time, which is different from the CPUs for all other queues on that interface 3. the interrupt handler notifies the corresponding task queue, which runs its task in a thread on whatever CPU the scheduler chooses 4. that task dispatches the packet for Ethernet processing via netisr, which processes it on whatever the current CPU is 5. Ethernet processing dispatches that packet for IP processing via netisr, which processes it on whatever the current CPU is You might want to try changing the default netisr dispatch policy to 'deferred' (sysctl net.isr.dispatch=deferred). If you do that, the Ethernet processing will still happen on an arbitrary CPU chosen by the scheduler, but the IP processing should then get mapped to a CPU based on the flowid assigned by the driver. Since igb assigns flowids based on received queue number, all IP (and above) processing for that packet should then be performed on the same CPU the queue interrupt was bound to. Unfortunately, I don't have a system with igb interfaces to try that on. -Patrick