From owner-freebsd-net@freebsd.org Sun Oct 25 20:50:07 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C9938450 for ; Sun, 25 Oct 2015 20:50:07 +0000 (UTC) (envelope-from James@Lodge.me.uk) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0134.outbound.protection.outlook.com [157.56.112.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B63113CB for ; Sun, 25 Oct 2015 20:50:06 +0000 (UTC) (envelope-from James@Lodge.me.uk) Received: from VI1PR06MB1037.eurprd06.prod.outlook.com (10.162.123.156) by VI1PR06MB1037.eurprd06.prod.outlook.com (10.162.123.156) with Microsoft SMTP Server (TLS) id 15.1.306.13; Sun, 25 Oct 2015 20:33:53 +0000 Received: from VI1PR06MB1037.eurprd06.prod.outlook.com ([10.162.123.156]) by VI1PR06MB1037.eurprd06.prod.outlook.com ([10.162.123.156]) with mapi id 15.01.0306.003; Sun, 25 Oct 2015 20:33:53 +0000 From: James Lodge To: "freebsd-net@freebsd.org" Subject: Re: Jail - PF - NAT - Network Performance Thread-Topic: Jail - PF - NAT - Network Performance Thread-Index: AQHRD0CmbA4uAEGug0il9nIvPZfXmJ58cg8AgAAIzKKAAB/YgIAACJuI Date: Sun, 25 Oct 2015 20:33:53 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=James@Lodge.me.uk; x-originating-ip: [46.101.56.132] x-microsoft-exchange-diagnostics: 1; VI1PR06MB1037; 5:FXe/y3ZxZkVosdltjiKL0GN1qT8aJxbstSd5lhtr1zv9G+lD61rGE9h67w6d7DtrdQcAQrJpo8M7tIoms8552NRWG+eDgvPz1LN/34RXWj49tU/NJVN59GwMKWa53TfZqCz8AibnWuwfHtdemQDXbA==; 24:yCd59+Ip1tV0sYRtlmE8LXvnhYdXHv7e6JDAM1wHC1smPFRS6cTEQQSGBoaD7gUJNZpqOZlDmvFS/otj7bEhMTgeeb2Esft9O7RfBynEDpo=; 20:QLmnhmScTI+TlYv3Lk/9AunXku0iMqXMowSnvMxfEkak048ov257e7tNaZ0kqjexS73XeMOraZyNw/i8EfAlwA== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:VI1PR06MB1037; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001)(102215026); SRVR:VI1PR06MB1037; BCL:0; PCL:0; RULEID:; SRVR:VI1PR06MB1037; x-forefront-prvs: 074040B844 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(189002)(24454002)(199003)(54356999)(87936001)(74482002)(2351001)(122556002)(2501003)(5003600100002)(40100003)(5002640100001)(101416001)(33656002)(5008740100001)(450100001)(50986999)(76176999)(92566002)(93886004)(102836002)(97736004)(5001920100001)(81156007)(86362001)(19580405001)(77096005)(19580395003)(11100500001)(76576001)(10400500002)(2900100001)(5007970100001)(80792005)(2950100001)(5004730100002)(66066001)(105586002)(74316001)(189998001)(5001960100002)(110136002)(106356001)(106116001)(107886002); DIR:OUT; SFP:1102; SCL:1; SRVR:VI1PR06MB1037; H:VI1PR06MB1037.eurprd06.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: Lodge.me.uk does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Lodge.me.uk X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Oct 2015 20:33:53.2995 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: ded56ae9-7c77-4cf6-bbfd-39e6a505742d X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR06MB1037 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Oct 2015 20:50:07 -0000 > On 25 Oct 2015, at 18:47, James Lodge wrote: > >> On 25 Oct 2015, at 17:46, James Lodge wrote: >> I currently have a FreeBSD 10.1 host running on Digital Ocean. I have mu= ltiple jails and I'm not using vimage. >> >> >> I'm using PF on the host to NAT traffic from said jails and all is worki= ng as expected. I have a jail running OpenVPN and clients can connect and t= raffic is routed to the Internet down the tunnel via PF/NAT. The issue I'm = seeing is download speeds to the client from the Internet on the external s= ide on PF. Upload always seem reasonable, but download is always woeful. I'= m using a Windows machine as the client if that make any odds. >> >> Yeah, there=1B$B!G=1B(Bs an issue with checksums and pf/Xen. >> Disabling TSO should work for you (sudo sysctl net.inet.tcp.tso=3D0), an= d the problem should be completely fixed in the >next release (10.3 or 11.0= ) >> >> Regards, >> Kristof > > > Thanks Kristof for the quick reply, > > I was hoping it would be that simple, but Digital Ocean use KVM (from wha= t I know) as their hypervisor so disabling TSO and LRO seems to have no not= iced increase in performance. > >Can you run a tcpdump while you=1B$B!G=1B(Bre seeing the problem? >The issue I know about is related to the TCP checksum, so I=1B$B!G=1B(Bd e= xpect to see many invalid checksums. > >Regards, >Kristof There are discussions/threads regarding KVM/TSO with the same symptoms, but= I would expect to see the same poor performance through PF be it local or = remotely via OpenVPN, but I don't. Local I see perfect performance in and o= ut from the host and jails. =20 Running wireshark locally on the windows client and downloading from the In= ternet via PF, I see a lot of retransmissions and duplicate ACKs.=20 28 0.824146000 80.249.99.148 10.8.0.10 TCP 1422 [TCP Retransmission] 80=1B$= B"*=1B(B57292 [ACK] Seq=3D9577 Ack=3D1 Win=3D14 Len=3D1368 33 0.872164000 10.8.0.10 80.249.99.148 TCP 66 [TCP Dup ACK 29#1] 57292=1B$B= "*=1B(B80 [ACK] Seq=3D1 Ack=3D13681 Win=3D256 Len=3D0 SLE=3D15049 SRE=3D164= 17 I don't see any retransmissions or duplicate ACKs when downloading from the= web server jail via its private IP down the tunnel.=20 if I run tcpdump i tun0 -n at the same time as seen the retransmissions and= duplicate ACKS I get..... 20:26:58.567017 IP 10.8.0.10.60339 > 80.249.99.148.80: Flags [.], ack 43776= 0, win 256, options [nop,nop,sack 1 {440496:443232}], length 0 20:26:58.568680 IP 80.249.99.148.80 > 10.8.0.10.60339: Flags [.], seq 43776= 0:439128, ack 1, win 14, length 1368 20:26:58.580530 IP 10.8.0.10.60339 > 80.249.99.148.80: Flags [.], ack 43912= 8, win 256, options [nop,nop,sack 1 {440496:443232}], length 0 20:26:58.585141 IP 80.249.99.148.80 > 10.8.0.10.60339: Flags [.], seq 44323= 2:444600, ack 1, win 14, length 1368 20:26:58.596466 IP 10.8.0.10.60339 > 80.249.99.148.80: Flags [.], ack 43912= 8, win 256, options [nop,nop,sack 1 {440496:444600}], length 0 Regards James=20 =