From owner-freebsd-net@FreeBSD.ORG Mon Aug 26 17:18:26 2013 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0BB05F39; Mon, 26 Aug 2013 17:18:26 +0000 (UTC) (envelope-from gibbs@FreeBSD.org) Received: from aslan.scsiguy.com (www.scsiguy.com [70.89.174.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D259C222C; Mon, 26 Aug 2013 17:18:25 +0000 (UTC) Received: from [192.168.6.166] (207-225-98-3.dia.static.qwest.net [207.225.98.3]) (authenticated bits=0) by aslan.scsiguy.com (8.14.7/8.14.5) with ESMTP id r7QHINuA058863 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 26 Aug 2013 17:18:24 GMT (envelope-from gibbs@FreeBSD.org) From: "Justin T. Gibbs" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Flow ID, LACP, and igb Date: Mon, 26 Aug 2013 11:18:18 -0600 Message-Id: To: net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) X-Mailer: Apple Mail (2.1508) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (aslan.scsiguy.com [70.89.174.89]); Mon, 26 Aug 2013 17:18:24 +0000 (UTC) Cc: jfv@FreeBSD.org, Alan Somers X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Aug 2013 17:18:26 -0000 Hi Net, I'm an infrequent traveler through the networking code and would appreciate some feedback on some proposed solutions to issues Spectra has seen with outbound LACP traffic. lacp_select_tx_port() uses the flow ID if it is available in the outbound mbuf to select the outbound port. The igb driver uses the msix queue of the inbound packet to set a packet's flow ID. This doesn't provide enough bits of information to yield a high quality flow ID. If, for example, the switch controlling inbound packet distribution does a poor job, the outbound packet distribution will also be poorly distributed. The majority of the adapters supported by this driver will compute the Toeplitz RSS hash. Using this data seems to work quite well in our tests (3 member LAGG group). Is there any reason we shouldn't use the RSS hash for flow ID? We also tried disabling the use of flow ID and doing the hash directly in the driver. Unfortunately, the current hash is pretty weak. It multiplies by 33, which yield very poor distributions if you need to mod the result by 3 (e.g. LAGG group with 3 members). Alan modified the driver to use the FNV hash, which is already in the kernel, and this yielded much better results. He is still benchmarking the impact of this change. Assuming we can get decent flow ID data, this should only impact outbound UDP, since the stack doesn't provide a flow ID in this case. Are there other checksums we should be looking at in addition to FNV? Thanks, Justin