From owner-freebsd-stable@freebsd.org Thu Dec 17 19:24:38 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 147B6A4AD6D for ; Thu, 17 Dec 2015 19:24:38 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "A1-33714", Issuer "A1-33714" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E1A0A1818 for ; Thu, 17 Dec 2015 19:24:37 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (localhost [127.0.0.1]) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPS id tBHJOYTT012713 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 17 Dec 2015 14:24:34 -0500 (EST) (envelope-from ken@mithlond.kdm.org) Received: (from ken@localhost) by mithlond.kdm.org (8.15.2/8.14.9/Submit) id tBHJOYKo012712; Thu, 17 Dec 2015 14:24:34 -0500 (EST) (envelope-from ken) Date: Thu, 17 Dec 2015 14:24:34 -0500 From: "Kenneth D. Merry" To: Mike Tancsa Cc: FreeBSD-STABLE Mailing List Subject: Re: traffic shaping on RELENG_10 ? Message-ID: <20151217192433.GA12383@mithlond.kdm.org> References: <5671C78A.4030807@sentex.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5671C78A.4030807@sentex.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [127.0.0.1]); Thu, 17 Dec 2015 14:24:34 -0500 (EST) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mithlond.kdm.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Dec 2015 19:24:38 -0000 On Wed, Dec 16, 2015 at 15:20:26 -0500, Mike Tancsa wrote: > I need to do some simple traffic shaping, but whenever I try and use > altq, I dont seem to have any luck-- I mean zero. It took me a while to get ALTQ working. More below.. > I like the management of pf via pf.conf, but miss the simplicity of > dummynet and ipfw. Has anyone used pf and ipfw together in RELENG10 ? > Any tips / caveats ? Or am I better off re-writing the pf rules into > ipfw and be done with it. I'm using ipfw for firewall and in-kernel NAT, and ALTQ (configured via pf) for traffic shaping. Step 1 was getting an ethernet card that supported ALTQ. I tried enabling IGB_LEGACY_TX in the igb driver, but it led to panics. I tried configuring ng_iface to provide the single queue that ALTQ needs, but wasn't able to figure it out. It ended up being easier to buy a $22 ethernet card that uses the em(4) driver. Hopefully this will help someone in a similar situation reading the list archives... This mostly works for me. I have 10Mb upstream. One of the keys was getting outbound DNS traffic to have a real-time priority (it's in the "other" category below) so that web browsers didn't time out when there was heavy upstream traffic. (e.g. uploading gigabytes of a photo library to my iCloud Photo Library...) One thing that was non-obvious with the in-kernel NAT versus the userland natd(8) was that you have to set 'sysctl net.inet.ip.fw.one_pass=0'. Otherwise the rules after the NAT step in the ipfw configuration are skipped. Using the in-kernel NAT was necessary to allow tagging packets from certain internal hosts so that they could be routed to different ALTQ rules after the NAT step. The tags are lost if you use the userland natd. I use ipfw count rules to tag packets, and then make decisions about whether to drop them later. Although the priorities are specified, I believe the hfsc schedule does not use them. So pay no attention to that. And I don't currently have any hosts tagged in the lpweb (low priority web) category. I'm not sure that would be productive in any case without a change in the ALTQ setup to limit the bandwidth for that queue or something like that. And, even then, if I have a large upload going on, I want it to go as fast as possible when it isn't contending with other traffic. Here's my pf.conf: altq on em0 hfsc bandwidth 9600Kb queue { lpweb, mail, web, hphost, ssh, icmp, femtocell, other } queue lpweb bandwidth 20% priority 0 hfsc (upperlimit 99%) queue mail bandwidth 10% priority 1 hfsc (upperlimit 99%) queue web bandwidth 20% priority 2 hfsc (upperlimit 99%) queue hphost bandwidth 10% priority 3 hfsc (upperlimit 90% realtime 20%) queue other bandwidth 20% priority 4 hfsc (default upperlimit 99% realtime 20%) queue icmp bandwidth 5% priority 5 hfsc (upperlimit 15%) queue ssh bandwidth 10% priority 6 hfsc (upperlimit 99% realtime 15%) queue femtocell bandwidth 5% priority 7 hfsc (upperlimit 50% realtime 20%) And here are some snippets from my firewall script. Obviously this isn't complete: fwcmd="ipfw" pfcmd="pfctl" oif="em0" iif="igb0" oip="my_external_ip_address" Here is where I tag outbound packets from certain hosts. The tags will follow the packets through the in-kernel NAT so that I can then put them on the correct ALTQ on the outside interface: # Streaming boxes ${fwcmd} add count tag 10 ip from 10.0.0.0/24{30-39} to any via ${iif} # Mark packets coming from the Verizon Femtocell ${fwcmd} add count tag 30 all from 10.0.0.44 to any via ${iif} In-kernel NAT: ${fwcmd} nat 123 config log if ${oif} same_ports unreg_only # This is necessary with the in-kernel NAT to re-inject packets once they've # gone through the NAT. sysctl net.inet.ip.fw.one_pass=0 # ALTQ queues have to be setup via pf ${pfcmd} -f $PF_FILE # Use count rules to tag the traffic in different queues. # Femtocell and similar super-high priority, but lower bandwidth # machines. ${fwcmd} add count altq femtocell ip from any to any tagged 30-39 # Tags 10-19 are high priority machines. ${fwcmd} add count altq hphost ip from any to any tagged 10-19 # ${WORK} VPN server. ssh connections to work run over the VPN. ${fwcmd} add count altq ssh ip from any to ${WORK_VPN_SERVER} dst-port 443 # Low priority machines. This would primarily be needed if there # were a machine taking up a lot of bandwidth. ${fwcmd} add count altq lpweb ip from any to any dst-port 443 tagged 20-29 # IMAP, outbound mail ${fwcmd} add count altq mail ip from ${oip} 25,143,993 to any ${fwcmd} add count altq mail ip from ${oip} to any dst-port 25 # Normal web access. ${fwcmd} add count altq web ip from any to any dst-port 80,443 # SSH ${fwcmd} add count altq ssh ip from any to any dst-port 22 # ICMP ${fwcmd} add count altq icmp icmp from any to any # Explicitly add DNS to the other queue. ${fwcmd} add count altq other udp from any to any dst-port 53 # Anything else, especially DNS. This is the default. ${fwcmd} add count altq other ip from any to any via ${oif} ${fwcmd} enable altq Ken -- Kenneth Merry ken@FreeBSD.ORG