From owner-freebsd-net@freebsd.org Thu Feb 16 22:25:02 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A032CE2F53 for ; Thu, 16 Feb 2017 22:25:02 +0000 (UTC) (envelope-from meyer.sydney@googlemail.com) Received: from mail-wr0-x230.google.com (mail-wr0-x230.google.com [IPv6:2a00:1450:400c:c0c::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E15412DEB for ; Thu, 16 Feb 2017 22:25:01 +0000 (UTC) (envelope-from meyer.sydney@googlemail.com) Received: by mail-wr0-x230.google.com with SMTP id i10so20680530wrb.0 for ; Thu, 16 Feb 2017 14:25:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=klutlttWKWpmCpVc/2xb0gGNd3SQOq2mndapZg26f5E=; b=qY6T9aAwRD0R/w8vqqCcD9i8FQuFdnGoRYs4PIeqG3ekuZWEcf4njqDIlcj2tsPIM6 xwZBIVVhIsBSWts5h6M1/S67l7cApb7OnQEqTU5/5HyvUmLiwIduxsqrdlPyuOmozZm0 bKYAsadcgTyuaxIyJGhFDieT1+RBvaaCw8NC/loeoEW3C5vY4PJqC4kak5Lzv2+DuK2d a5sYbDe5XZIP/DI6h5fvwGn/SMSgVt1aO0oGTOmMPjBzgoeYdY1go4SXLR/Be0SRmeFF Kz5i+mUFODqKORR/nIRQga9apEn8ve32dtNjgH/KnW4s+LrYcTYjLNVBtFOYDjgR/fjJ 623w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=klutlttWKWpmCpVc/2xb0gGNd3SQOq2mndapZg26f5E=; b=Pbxa3Lffqpx5Z9qZfv8FHtgVFkz8F7WMiRlpryTB6RKZlVtKk8q2Dttx47USVKW7zo qPjvCDo21IXsdkVABcvQDxRDgPEDCudIFEyHn6juf7hPpyt7B1SKb7TrnSchQiKtcZYt BvvjyI0FoLwjzzMZk57QJGtBAPyMmuDN11gFWShoE/RZX/cr4SlzixPUs1R70C/aJAe/ jY9uRbM4neASnxd9d8w+GC8UzCnNX758V1kw1phL+CJTJJctvHqsJTbpelKtMv2qqfee tbcrDGW9UHCNiyz34Y3YMgPdXYqkPQtskvcHlvsCP+EQtDvKsHNIVBMIvldOc54JJL1E nsFg== X-Gm-Message-State: AMke39mB82wbfcHAFy1B8QZ9Lt6GCWkErG4tGEE++SD1LoQVQ8YEJCb4GKtvf2WUoYPf4Q== X-Received: by 10.223.148.35 with SMTP id 32mr4507657wrq.18.1487283899620; Thu, 16 Feb 2017 14:24:59 -0800 (PST) Received: from ?IPv6:2a02:a03f:a4e:be01:c0c4:fff1:8d9b:caa5? ([2a02:a03f:a4e:be01:c0c4:fff1:8d9b:caa5]) by smtp.googlemail.com with ESMTPSA id b87sm1879351wmi.0.2017.02.16.14.24.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Feb 2017 14:24:59 -0800 (PST) From: Sydney Meyer Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Re: Slow Download Speeds from AWS S3 Date: Thu, 16 Feb 2017 23:24:57 +0100 References: <18e49da9-47eb-1803-e223-b4385e7d8690@multiplay.co.uk> <30F9FC69-CEBE-4CBE-93A9-DEDBD053C6C6@googlemail.com> <673aca75-7bb5-73e7-d23b-b12a25816c8a@multiplay.co.uk> To: freebsd-net@freebsd.org In-Reply-To: <673aca75-7bb5-73e7-d23b-b12a25816c8a@multiplay.co.uk> Message-Id: <5BE01E68-A8EF-4572-9B48-C0C58683F085@googlemail.com> X-Mailer: Apple Mail (2.3259) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2017 22:25:02 -0000 Hi Steve, > On 16 Feb 2017, at 18:18, Steven Hartland = wrote: >=20 > It does seem to be related to TCP Receive Window Size. >=20 > When I tested on a 11.0-RELEASE box I got 30MB/s out of the box and = the increase of recvspace was only needed on the original 10.2 however = the 11 box is only 1.2ms from AWS where as the 10.2 was 17ms away, so = likely explains the difference. Yes, i saw this too.. when i did some tests with a VM from within the = same AWS region, i did not see the troughput issues as with connections = with a higher latency (probably should have mentioned this in the first = post, retrospectively).. this would also explain the vmware guest = bridged vs nat'ed difference then.. >=20 > It seems that when talking to hosts that don't support Timestamps = FreeBSD's performance is going to be poor due to lack of auto scaling :( I did some tests with Debian 8.x (Linux Kernel 3.16.x) and there the = default for /proc/sys/net/ipv4/tcp_timestamps appears to be enabled, at = least with Debian's 3.16 Linux Kernel. I can see the TS val field now in = tcpdump which is lacking in traffic from AWS S3, regardless from region. >=20 > Reading the thread you linked is seems like we need to implement auto = scaling based on RTT estimations similar to linux. Well, i guess the issue described in the related thread is indeed the = case here. I will adjust this buffer size manually on this machine, as it has no = other major purpose than pulling archives from S3 anyway. But thanks again, learned something new (to me) about FreeBSDs = networking stack. Updated the Forums Thread accordingly here: https://forums.freebsd.org/threads/59756/ Cheers, Sydney. >=20 > Regards > Steve >=20 > On 16/02/2017 16:55, Sydney Meyer via freebsd-net wrote: >> Hi Steve, >>=20 >> increasing the buffer size did indeed enhance throughput. >>=20 >> I am obviously not an expert in this field, but i don't understand = why or if the TCP Receive Window Size shouldn't increase automatically. >>=20 >> I found this thread on the ML and i'm reading myself a bit more into = the topic right now. >>=20 >> = https://lists.freebsd.org/pipermail/freebsd-net/2014-August/039495.html >>=20 >> Thank you for your tip, however.. >>=20 >> Sydney >>=20 >>> On 16 Feb 2017, at 16:35, Steven Hartland = wrote: >>>=20 >>> Window scaling and receive buffer scaling is the most likely cause. >>>=20 >>> Check what the sysctl net.inet.tcp.recvspace is set to, then try = increasing it e.g. >>> sysctl net.inet.tcp.recvspace=3D655360 >>>=20 >>> This jumped the transfer rate with a wget and your test URL from = 3MB/s to 30MB/s here. >>>=20 >>> Regards >>> Steve >>>=20 >>> On 16/02/2017 14:34, Sydney Meyer via freebsd-net wrote: >>>> Hello, >>>>=20 >>>> I'm seeing continuous slow download speeds from Amazon S3, but only = on FreeBSD. Other OSes saturate the connection without problems. >>>>=20 >>>> This happens with 10.3-RELEASE and 11.0-RELEASE and only with AWS = S3 in different regions (Ireland, London, Frankfurt, US Standard have = been tested) whilst using fetch, curl, et. al. >>>>=20 >>>> I have tested this on multiple machines, bare metal, bhyve, Xen and = VMware VMs, routed setups and direct pppoe links.. all the same. >>>>=20 >>>> Anyone seeing similar issues? >>>>=20 >>>> Here's a url to try: >>>>=20 >>>> = http://s3.eu-central-1.amazonaws.com/4f48caf1d8bcbef8/c5b38f8b3625d2b6/zer= ofile.raw >>>>=20 >>>> Also, when doing double NAT, i.e. VMware Fusion FreeBSD Guest with = "Share with my Mac"-Interface, the machine is doing completely fine, as = in saturates the link, 90Mbps otherwise between 12-15Mbps.. >>>>=20 >>>> I also switched the FreeBSD Routers with Linux-based ones and with = the isp-provided router box, with the same result. >>>>=20 >>>> I have launched VMs with Digital Ocean to "rule out" my ISP and = there seems to be the same issue. Downloading from S3 is multiple times = slower than any other connection to services outside of S3 or on any = other OS. >>>>=20 >>>> It seems like other people are seeing the same issue: >>>>=20 >>>> https://forums.freebsd.org/threads/59756/#post-343064 >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"