From owner-freebsd-net@FreeBSD.ORG Mon Aug 26 17:31:39 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6CAFF978; Mon, 26 Aug 2013 17:31:39 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qc0-x229.google.com (mail-qc0-x229.google.com [IPv6:2607:f8b0:400d:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EB7472330; Mon, 26 Aug 2013 17:31:38 +0000 (UTC) Received: by mail-qc0-f169.google.com with SMTP id k8so818484qcq.28 for ; Mon, 26 Aug 2013 10:31:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:cc:content-type; bh=abG1iumbPc++xGdNRWmaRzpETXkxhFeXpGYx/ArwDZE=; b=SoS8SnZf9kKYWnVWgd7mScweWAR99mFWDA0I63io+FYJWnrSoG/sIACoJxku7bGGwa +gJog+mmwCYqr7u/s8q5N77F0c+AIGqt4Fle8aFgM6SNwZ2sibmixY5ZCsojhc5S7LbK bcQy4nxLqedvoQSw/9RafqZr872y/WHrMvBdoZsoZOFlwffjVGtPlF6AHnVIYgPrQCSW hALsv8xNHNAXocPIIRzhY68Bt7PD0FUbTUl3q0TStj+3GYDkkhcG8lKJvRi5Uw8E9GLx L2uvt+MlBkF4m9cKv7cYSMrN7qnpj4fHyntoD3e+hUALsP++RrRLOr9ZVI721BNQhrr2 784g== MIME-Version: 1.0 X-Received: by 10.224.23.134 with SMTP id r6mr17387677qab.34.1377538298027; Mon, 26 Aug 2013 10:31:38 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.49.39.101 with HTTP; Mon, 26 Aug 2013 10:31:37 -0700 (PDT) Date: Mon, 26 Aug 2013 11:31:37 -0600 X-Google-Sender-Auth: eWRpuz1COx_oc4nkaa1z_5hvPNE Message-ID: Subject: Re: Flow ID, LACP, and igb From: Alan Somers To: "Justin T. Gibbs" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: jfv@freebsd.org, Alan Somers , net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Aug 2013 17:31:39 -0000 On Mon, Aug 26, 2013 at 11:18 AM, Justin T. Gibbs wrote: > Hi Net, > > I'm an infrequent traveler through the networking code and would > appreciate some feedback on some proposed solutions to issues Spectra > has seen with outbound LACP traffic. > > lacp_select_tx_port() uses the flow ID if it is available in the outbound > mbuf to select the outbound port. The igb driver uses the msix queue of > the inbound packet to set a packet's flow ID. This doesn't provide enough > bits of information to yield a high quality flow ID. If, for example, the > switch controlling inbound packet distribution does a poor job, the > outbound > packet distribution will also be poorly distributed. > It's actually worse than this. If two inbound TCP packets get sent to the same queue on different igb ports, then they will have the same flowid. That could happen even if the switch is distributing packets just fine. > > The majority of the adapters supported by this driver will compute > the Toeplitz RSS hash. Using this data seems to work quite well > in our tests (3 member LAGG group). Is there any reason we shouldn't > use the RSS hash for flow ID? > > We also tried disabling the use of flow ID and doing the hash directly in > the driver. Unfortunately, the current hash is pretty weak. It multiplies > by 33, which yield very poor distributions if you need to mod the result > by 3 (e.g. LAGG group with 3 members). Alan modified the driver to use > the FNV hash, which is already in the kernel, and this yielded much better > results. He is still benchmarking the impact of this change. Assuming we > can get decent flow ID data, this should only impact outbound UDP, since > the > stack doesn't provide a flow ID in this case. > It also affects outbound TCP packets for streams that originated on the host. For example, it affects tcp-mounted NFS clients. > > Are there other checksums we should be looking at in addition to FNV? > s/checksums/hashes/ > > Thanks, > Justin > >