From owner-freebsd-net@FreeBSD.ORG Wed Sep 7 14:53:06 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DDB0106564A for ; Wed, 7 Sep 2011 14:53:06 +0000 (UTC) (envelope-from melifaro@yandex-team.ru) Received: from forward10.mail.yandex.net (forward10.mail.yandex.net [IPv6:2a02:6b8:0:202::5]) by mx1.freebsd.org (Postfix) with ESMTP id B76BE8FC08 for ; Wed, 7 Sep 2011 14:53:05 +0000 (UTC) Received: from smtpcorp2.mail.yandex.net (smtpcorp2.mail.yandex.net [77.88.61.36]) by forward10.mail.yandex.net (Yandex) with ESMTP id A5ACA1023425; Wed, 7 Sep 2011 18:53:03 +0400 (MSD) Received: from smtpcorp2.mail.yandex.net (localhost [127.0.0.1]) by smtpcorp2.mail.yandex.net (Yandex) with ESMTP id 92D55740223; Wed, 7 Sep 2011 18:53:03 +0400 (MSD) Received: from dhcp170-36-red.yandex.net (dhcp170-36-red.yandex.net [95.108.170.36]) by smtpcorp2.mail.yandex.net (nwsmtp/Yandex) with ESMTP id r3LCtMFm; Wed, 7 Sep 2011 18:53:03 +0400 Message-ID: <4E678521.7080006@yandex-team.ru> Date: Wed, 07 Sep 2011 18:52:17 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.16) Gecko/20110120 Thunderbird/3.0.11 MIME-Version: 1.0 To: freebsd-net@freebsd.org, jfvogel@gmail.com, melifaro@ipfw.ru Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 07 Sep 2011 15:29:47 +0000 Cc: Subject: igb/ixgbe RSS/RX queues for non-IP traffic X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 14:53:06 -0000 Hello list! (CC'in Jack Vogel since this is intel drivers/hardware specific question) There are some techniques for assigning network traffic to different NIC RX queues (bound to different CPUs/cores). The main problem for all techniques is avoiding any possible packet reordering in single flow. Most obvious (and most used) are described in Microsoft NDIS driver specification: Hashing is done on various tuples like: * IPv46 src/dst + TCP/UDP src/dst port for TCP/UDP * IPv46 src/dst for general IPv46 traffic igb's have a bit more options like assigning selected ethertype or TCP SYN to different queue (See section 7.1.1 of 82576EB datasheet) ixgbe (82599) goes even more far: flow director functionality permits 8K 'perfect match' filters allowing to select any subset in: * vlan * IPv46 src/dst * L4 proto * TCP/UDP/SCTP protocol ports and even 'flexible 2-byte tuple anywhere in the first 64 bytes of the packet' There are other possibilities to assign traffic to different queues in ixbge like using .1q priority filed (DCB functionality) More information can be found in sections 7.1.2 (RX queues assignment), 7.1.2.7(Flow Director), 7.7.1 (DCB) of 82599 datasheet However, there are many setups where PC can be used as platform for routing/dispatching non-IP traffic. PPPoE server is a typical example. People have to do some tricks (link aggregation, non-direct ISR) to get traffic dispatched by more than single CPU/core, but this much less efficient. I'm a bit curious: why some generic hashing mechanism based on something like 'flexible 2-byte tuple anywhere in the first 64 bytes of the packet' were not added? It can be (at least from my point of view) easily based on (for example) flow director functionality. Particularly I'm trying to figure out how can I use all this variety of filters to get MPLS traffic split to different RX queues. Maybe someone can point me to the right direction? -- Alexander V. Chernikov Yandex NOC