From owner-freebsd-net@FreeBSD.ORG Thu Apr 23 12:14:41 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1952D96D for ; Thu, 23 Apr 2015 12:14:41 +0000 (UTC) Received: from mail-pd0-x230.google.com (mail-pd0-x230.google.com [IPv6:2607:f8b0:400e:c02::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DD5F31476 for ; Thu, 23 Apr 2015 12:14:40 +0000 (UTC) Received: by pdea3 with SMTP id a3so16939187pde.3 for ; Thu, 23 Apr 2015 05:14:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=Vbn5VdIHRhsBmqEpyR0jptfqx1ZM8DeaGkFUC/MtYxM=; b=eFbZAppU2J+P4dEZiYaYLa1y0QtVR3Q+QbepEwtR9hGtvv7xJscIhzJBsk1e8V+Z2Q gSlu2BoivNHq03c7TwJdwCKsEE6genDoeP7Zz6ZRsfhoOlStDRyCltuK0lEZOP0p2cR/ Z0+5Y7ySVmQVsMe4+l5ZRFPBLFpbYa3ahIaUbHOQ7McAnEgErMNbUdiy0JLkd4V0wft0 c+UcUDancH9vzJtkGIrzZ9H8MxPKDZCp294FKAi5g/XkeuZHkArjL1cy0fe5jh5zB9H9 Tom8XMtwafqdvijsNTJxZKECEAADYYLI0+R396ouLxYNPDC9y9zac0E3fubojElQVHKc nBFQ== X-Received: by 10.68.114.68 with SMTP id je4mr4646812pbb.99.1429791280159; Thu, 23 Apr 2015 05:14:40 -0700 (PDT) Received: from ox (c-24-6-44-228.hsd1.ca.comcast.net. [24.6.44.228]) by mx.google.com with ESMTPSA id do6sm7995089pdb.72.2015.04.23.05.14.38 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 23 Apr 2015 05:14:38 -0700 (PDT) Date: Thu, 23 Apr 2015 05:14:33 -0700 From: Navdeep Parhar To: Scott Larson Cc: freebsd-net@freebsd.org Subject: Re: net.inet.ip.forwarding impact on throughput Message-ID: <20150423121433.GA15890@ox> Mail-Followup-To: Scott Larson , freebsd-net@freebsd.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2015 12:14:41 -0000 On Tue, Apr 21, 2015 at 12:47:45PM -0700, Scott Larson wrote: > We're in the process of migrating our network into the future with 40G > at the core, including our firewall/traffic routers with 40G interfaces. An > issue which this exposed and threw me for a week turns out to be directly > related to net.inet.ip.forwarding and I'm looking to just get some insight > on what exactly is occurring as a result of using it. Enabling forwarding disables LRO and TSO and that probably accounts for a large part of the difference in throughput that you've observed. The number of packets passing through the stack (and not the amount of data passing through) is the dominant bottleneck. fastforwarding _should_ make a difference, but only if packets actually take the fast-forward path. Check the counters available via netstat: # netstat -sp ip | grep forwarded Regards, Navdeep > What I am seeing is when that knob is set to 0, an identical pair of > what will be PF/relayd servers with direct DAC links between each other > using Chelsio T580s can sustain around 38Gb/s on iperf runs. However the > moment I set that knob to 1, that throughput collapses down into the 3 to > 5Gb/s range. As the old gear this is replacing is all GigE I'd never > witnessed this. Twiddling net.inet.ip.fastforwarding has no apparent effect. > I've not found any docs going in depth on what deeper changes enabling > forwarding does to the network stack. Does it ultimately put a lower > priority on traffic where the server functioning as the packet router is > the final endpoint in exchange for having more resources available to route > traffic across interfaces as would generally be the case? > > > *[image: userimage]Scott Larson[image: los angeles] > Lead > Systems Administrator[image: wdlogo] [image: > linkedin] [image: facebook] > [image: twitter] > [image: instagram] > T 310 823 8238 x1106 > <310%20823%208238%20x1106> | M 310 904 8818 <310%20904%208818>* > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"