From owner-freebsd-amd64@FreeBSD.ORG Sun Jun 12 04:50:01 2011 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10F8F1065673 for ; Sun, 12 Jun 2011 04:50:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id DF7CA8FC22 for ; Sun, 12 Jun 2011 04:50:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p5C4o0N2052700 for ; Sun, 12 Jun 2011 04:50:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p5C4o0DM052699; Sun, 12 Jun 2011 04:50:00 GMT (envelope-from gnats) Resent-Date: Sun, 12 Jun 2011 04:50:00 GMT Resent-Message-Id: <201106120450.p5C4o0DM052699@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Ben Baron Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F2E6106566C for ; Sun, 12 Jun 2011 04:42:04 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 151DA8FC08 for ; Sun, 12 Jun 2011 04:42:04 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p5C4g3bL089889 for ; Sun, 12 Jun 2011 04:42:03 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p5C4g3nN089888; Sun, 12 Jun 2011 04:42:03 GMT (envelope-from nobody) Message-Id: <201106120442.p5C4g3nN089888@red.freebsd.org> Date: Sun, 12 Jun 2011 04:42:03 GMT From: Ben Baron To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Sun, 12 Jun 2011 04:59:43 +0000 Cc: Subject: amd64/157785: amd64 + jail + ipfw + natd = very slow outbound traffic from jail (5KB/s) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jun 2011 04:50:01 -0000 >Number: 157785 >Category: amd64 >Synopsis: amd64 + jail + ipfw + natd = very slow outbound traffic from jail (5KB/s) >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Jun 12 04:50:00 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Ben Baron >Release: 8.2-RELEASE >Organization: Be Five Enterprises, Inc. >Environment: FreeBSD localhost 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >Description: I have a newly set up FreeBSD 8.2 server with one public network interface setup with a jail for housing Apache. I was using ipfw + natd as my solution to firewall and nat traffic between the machine and jail (actually had multiple jails for different processes). The setup was working fine, however transfers out from my Apache jail were limited to about 5 KB/sec (~40kbit/sec) (I didn't test other jails but I assume they were the same speed, also didn't test jail to jail). Traffic into the jail from the internet was full speed. Traffic out from the host was also full speed. Only traffic out from the jail was affected. I researched the issue online and determined that this is a known issue with ipfw + natd and jails when using amd64 processors (in my case an Intel Core i7). I read conflicting statements on the FreeBSD forums about whether this also affects i386 (it looks like it doesn't, but one person did comment that they had the same issue on an i386 machine IIRC). >How-To-Repeat: 1. Do a fresh install of FreeBSD 8.2 2. Create an lo1 interface with the IP 10.0.0.2/24 3. Use ezjail to create a jail with 10.0.0.2 IP address 4. Install Apache in jail 5. Setup ipfw with the following settings (edited from my original settings to remove some sensitive information): # Interface: em1 # List of Ports: # Host SSH: 22 # JAIL WEB SERVER: 80, 443 IPF="ipfw -q add" ipfw -q -f flush # Replace with your interface PIF="em1" # Host IP inet="XXX.XXX.XXX.XXX" # Jail IPs apache_jail="10.0.0.2" # Loopback $IPF 100 allow all from any to any via lo0 $IPF 110 deny all from any to 127.0.0.0/8 $IPF 120 deny all from 127.0.0.0/8 to any $IPF 130 deny tcp from any to any frag # Allow SSH to host $IPF 300 allow tcp from any to me 22 established # Allow out NATed traffic from Apache Jail. This is DNATed traffic. $IPF 500 divert natd ip from $apache_jail to not me out via $PIF $IPF 570 skipto 10000 ip from any to any diverted # Allow remaining all outgoing traffic $IPF 600 allow all from me to any out # Allow NATed traffic to Jail. This is SNATed traffic. natd (NAT daemon) will take care of these. $IPF 700 divert natd ip from not me to any in via $PIF $IPF 710 skipto 10000 ip from any to any diverted # Deny ICMP $IPF 800 allow icmp from any to any # Log an deny the traffic $IPF 900 deny log logamount 10000 ip from any to any # This is where all diverted NAT traffic are skipped to after being processed. $IPF 10000 allow ip from any to any 6. Setup natd with the following settings: # Apache Jail redirect_port tcp 10.0.0.2:80 80 redirect_port tcp 10.0.0.2:443 443 >Fix: The only fix I was able to find was to switch to pf. After switching to pf for my firewall and nat solution, I'm getting full speed traffic both ways from the jail. >Release-Note: >Audit-Trail: >Unformatted: