From owner-freebsd-bugs@freebsd.org Fri Feb 5 05:31:59 2021 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0832A53D3E9 for ; Fri, 5 Feb 2021 05:31:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DX3vL6g0Gz4WRS for ; Fri, 5 Feb 2021 05:31:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id E4A9553D3E8; Fri, 5 Feb 2021 05:31:58 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E474A53D264 for ; Fri, 5 Feb 2021 05:31:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DX3vL65kSz4WRR for ; Fri, 5 Feb 2021 05:31:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C464019521 for ; Fri, 5 Feb 2021 05:31:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1155VwRS086115 for ; Fri, 5 Feb 2021 05:31:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1155Vw9F086114 for bugs@FreeBSD.org; Fri, 5 Feb 2021 05:31:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 253261] AWS EC2 disk I/O performance is extremely bad Date: Fri, 05 Feb 2021 05:31:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: raj@gusw.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 05:31:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253261 Bug ID: 253261 Summary: AWS EC2 disk I/O performance is extremely bad Product: Base System Version: 12.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: raj@gusw.net I am trying to move all my AWS systems back from Amazon Linux to FreeBSD. B= ut I encounter massive slowness. People have reported about slow ENA network driver before.=20 Even worse is the performance of the NVMe nvd disks. This is so bad that my attempt to go with FreeBSD is now completely dead. And it makes me really s= ad. FreeBSD was once "the power to serve" and now it feels like I'm running Ult= rix on a VAX for the fun of it but for any real work I need Linux. So sad! The issue can be reproduced by anyone on AWS. Just get a FreeBSD 12 (latest) instance and allocate a gp2 of 1 TB or a gp3 of smaller size for which you specify 3000 IOPS and 250 MB/s IO capacity. Let's say that disk is /dev/nvd2 Now do this: dd if=3D/dev/zero of=3D/dev/nvd2 bs=3D100M status=3Dprogress and you see that it's writing with the whopping speed of 70 MB/s.=20 That used to be good, but it is no longer good. Compare Amazon Linux doing = the same thing at 300 MB/s. Now, when you put a file system over it, zfs or ufs, then instantly the performance gets better: newfs /dev/nvd2 mount /dev/nvd2 /mnt dd if=3D/dev/zero of=3D/mnt/test bs=3D100M status=3Dprogress now that works at about 250 MB/s. Decent. So, problem solved? No! It turns out if I create a PostgreSQL database over this setup, then ag= ain there is massive delay on the read and write and throughput will drop to ev= en worse than 70 MB/s. Creating one index takes 10 times as long as that same = on the Linux system. The dd issue is probably the best manifestation for fixing at least part of= the problem.=20 Is this about tuning? Even if it was, there is a severe problem with how the system is set up by default. But I'm afraid it's not even about tuning so b= ad is it. --=20 You are receiving this mail because: You are the assignee for the bug.=