From owner-freebsd-net@FreeBSD.ORG Tue Apr 21 20:50:17 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 95493D1C for ; Tue, 21 Apr 2015 20:50:17 +0000 (UTC) Received: from mail-wg0-f70.google.com (mail-wg0-f70.google.com [74.125.82.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3786A1AA8 for ; Tue, 21 Apr 2015 20:50:16 +0000 (UTC) Received: by wgtl5 with SMTP id l5so48321500wgt.1 for ; Tue, 21 Apr 2015 13:50:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=aL8zuIacG2/6CQBKPFBsa8wLU6AJ3hhh1bUy0IIEhTI=; b=jJssfWU2eQBPzavAuMXJu9NEhpJwOAzv61HamYejCaJ97SZNs+2R/wAhV7LS/5d+Xq vKR55VJx2Ov+qyuAw3fKLrQiCyMbILDYp6xta9nFgup+3tMTFWFUKDAlw4SJmcUzwwnX 3X/52amWingXz2XCf5JEs/dsBVMNglOTuhAI/Qyj4S7+BeNPykjWQshQO/tPZ87zBBsR VxXvhH4kvn5ilCpORgiGfSkJhGJpYvgHEpKabBNAYwTBeChwxC8d7eKyMkbiJmlBmmyt eMtseVh/VFp9RJeuthcKnjSy3FVkCleYiojD6BupK+z8IJW1pY0YdOjkdcTdY6FSbKG2 ONnQ== X-Gm-Message-State: ALoCoQkjpWgmW04mevB61oLK0kIBlfmZlywuGo/abDIjFm8SsrKvdyj15fo0Gr0+4NlKnT4cWADr MIME-Version: 1.0 X-Received: by 10.112.40.9 with SMTP id t9mr21912988lbk.55.1429645665099; Tue, 21 Apr 2015 12:47:45 -0700 (PDT) Received: by 10.25.78.149 with HTTP; Tue, 21 Apr 2015 12:47:45 -0700 (PDT) Date: Tue, 21 Apr 2015 12:47:45 -0700 Message-ID: Subject: net.inet.ip.forwarding impact on throughput From: Scott Larson To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2015 20:50:17 -0000 We're in the process of migrating our network into the future with 40G at the core, including our firewall/traffic routers with 40G interfaces. An issue which this exposed and threw me for a week turns out to be directly related to net.inet.ip.forwarding and I'm looking to just get some insight on what exactly is occurring as a result of using it. What I am seeing is when that knob is set to 0, an identical pair of what will be PF/relayd servers with direct DAC links between each other using Chelsio T580s can sustain around 38Gb/s on iperf runs. However the moment I set that knob to 1, that throughput collapses down into the 3 to 5Gb/s range. As the old gear this is replacing is all GigE I'd never witnessed this. Twiddling net.inet.ip.fastforwarding has no apparent effect. I've not found any docs going in depth on what deeper changes enabling forwarding does to the network stack. Does it ultimately put a lower priority on traffic where the server functioning as the packet router is the final endpoint in exchange for having more resources available to route traffic across interfaces as would generally be the case? *[image: userimage]Scott Larson[image: los angeles] Lead Systems Administrator[image: wdlogo] [image: linkedin] [image: facebook] [image: twitter] [image: instagram] T 310 823 8238 x1106 <310%20823%208238%20x1106> | M 310 904 8818 <310%20904%208818>*