From owner-freebsd-net@freebsd.org Mon Jul 16 14:32:04 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BF87102A387 for ; Mon, 16 Jul 2018 14:32:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 23656735FF for ; Mon, 16 Jul 2018 14:32:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id CE24B102A386; Mon, 16 Jul 2018 14:32:03 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A864D102A384 for ; Mon, 16 Jul 2018 14:32:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 465EB735F7 for ; Mon, 16 Jul 2018 14:32:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5E653130BC for ; Mon, 16 Jul 2018 14:32:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w6GEW26d041317 for ; Mon, 16 Jul 2018 14:32:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w6GEW2fA041313 for net@FreeBSD.org; Mon, 16 Jul 2018 14:32:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 203856] [igb] PPPoE RX traffic is limitied to one queue Date: Mon, 16 Jul 2018 14:32:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: eugen@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2018 14:32:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203856 --- Comment #11 from Eugene Grosbein --- There seems to be common mis-understanging in how hardware receive queues w= ork in igb(4) chipsets. First, one should read Intel's datasheet on the NIC. For example of 82576-b= ased NIC this is https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/825= 76eb-gigabit-ethernet-controller-datasheet.pdf The section 7.1.1.7 of the datasheet states that NIC "supports a single hash function, as defined by Microsoft RRS". Reading on, one learns this means t= hat only frames containing IPv4 or IPv6 packets are hashed using their IP addre= sses as hash function arguments and, optionally, TCP port numbers. This means that all incoming PPPoE ethernet frames are NOT hashed by such N= IC in hardware, as any other frames carrying no plain IPv4 nor IPv6 packets. T= his is the reason why all incoming PPPoE ethernet frames get to the same (zero) queue.=20 The igb(4) driver has nothing to do with this problem, and mentioned "patch" cannot solve the problem too. However, there are other ways. Most performant way for production use is usage of several igb NICs combined with lagg(4) logical channel connected to managed switch that is configured= to distribute traffic flows between ports of the logical channel based on sour= ce MAC address of a frame. This is useful for mass-servicing of clients when o= ne has multiple PPPoE clients generating flows of PPPoE frames each using dist= inct MAC address. This way is not really useful for PPPoE client receiving all frames from single PPPoE server. There is another way. By default, FreeBSD kernel performs all processing of received PPPoE frame within driver interrupt context: decapsulation, option= al decompression/decryption, network address translation, routing lookups, pac= ket filtering and so on. This can result in overloaded single CPU core in defau= lt configuration when sysctl net.isr.dispatch=3Ddirect. Since FreeBSD 8 we have netisr(8) network dispatch service allowing any NIC driver just enqueue received ethernet frame and cease its following processing freeing this CPU core. Other kernel threads using other CPU cores will then dequeue received frames to complete decapsilation etc. loading all CPU cores evenly. So, one just should make sure it has "net.isr.maxthreads" and "net.isr.numthreads" greater than 1 and switch net.isr.dispatch to "deferre= d" value that permits NIC drivers to use netisr(9) queues to distribute load between CPU cores. --=20 You are receiving this mail because: You are the assignee for the bug.=