From owner-freebsd-performance@freebsd.org Fri Feb 5 20:45:47 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 917AA529EA4 for ; Fri, 5 Feb 2021 20:45:47 +0000 (UTC) (envelope-from raj@gusw.net) Received: from gateway22.websitewelcome.com (gateway22.websitewelcome.com [192.185.46.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXS9k4JMGz4Wtn for ; Fri, 5 Feb 2021 20:45:46 +0000 (UTC) (envelope-from raj@gusw.net) Received: from cm16.websitewelcome.com (cm16.websitewelcome.com [100.42.49.19]) by gateway22.websitewelcome.com (Postfix) with ESMTP id 62E1D4315 for ; Fri, 5 Feb 2021 14:45:45 -0600 (CST) Received: from host2097.hostmonster.com ([67.20.114.243]) by cmsmtp with SMTP id 87z7lWWcJHPnU87z7l4yhw; Fri, 05 Feb 2021 14:45:45 -0600 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=schadow.us; s=default; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version :Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=d0ldUDxfnA3zcNLR66p3o2wd/Kxdm9BGqXr/2JnB144=; b=OGXdj3IaQO+YNElVNf4uS/wH2B 0PcNQKUDN/K53y6mWSmUZ2SEiJHHnGAI1KNsjivSMov+iz5p25OO+KQqYu4UA6heKSYZTZ7XfzrNt +cMOY6gbsYTHrdP0Kg9bDPTIN; Received: from [177.143.140.27] (port=49230 helo=[10.0.4.7]) by host2097.hostmonster.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1l87z6-002b7C-MK for freebsd-performance@freebsd.org; Fri, 05 Feb 2021 13:45:44 -0700 Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> From: Gunther Schadow Message-ID: <3fde2934-1e18-5ea4-84d6-21200eaf4b20@gusw.net> Date: Fri, 5 Feb 2021 15:45:42 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2097.hostmonster.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gusw.net X-BWhitelist: no X-Source-IP: 177.143.140.27 X-Source-L: No X-Exim-ID: 1l87z6-002b7C-MK X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: ([10.0.4.7]) [177.143.140.27]:49230 X-Source-Auth: ebiz+schadow.us X-Email-Count: 1 X-Source-Cap: cHJhZ21hdDE7cHJhZ21hdDE7aG9zdDIwOTcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-Rspamd-Queue-Id: 4DXS9k4JMGz4Wtn X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=schadow.us header.s=default header.b=OGXdj3Ia; dmarc=none; spf=softfail (mx1.freebsd.org: 192.185.46.225 is neither permitted nor denied by domain of raj@gusw.net) smtp.mailfrom=raj@gusw.net X-Spamd-Result: default: False [-3.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_X_SOURCE(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[schadow.us:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; HAS_X_ANTIABUSE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:46606, ipnet:192.185.0.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[schadow.us:s=default]; FREEFALL_USER(0.00)[raj]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[gusw.net]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[192.185.46.225:from]; RWL_MAILSPIKE_VERYGOOD(0.00)[192.185.46.225:from]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 20:45:47 -0000 Gordon Bergling wrote: > Can you verify your feelings by numbers? Yes, like I said >> Not by a few % points, but by factors if not an order of magnitude! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253261 Do this: dd if=/dev/zero of=/dev/nvd2 bs=100M status=progress and you see that it's writing with the "whopping" speed of 70 MB/s. That used to be good, but it is no longer good. Compare Amazon Linux doing the same thing at 300 MB/s. Now, when you put a file system over it, zfs or ufs, then instantly the performance gets better: newfs /dev/nvd2 mount /dev/nvd2 /mnt dd if=/dev/zero of=/mnt/test bs=100M status=progress now that works at about 250 MB/s. Decent. So, problem solved? No! It turns out if I create a PostgreSQL database over this setup, then again there is massive delay on the read and write and throughput will drop to even worse than 70 MB/s. Creating one index takes 10 times as long as that same on the Linux system. PS: no need to point out that Linux uses buffer cache for direct write to device and BSD doesn't. Those effects will not make a difference when you write (or read) more than the buffer cache size (e.g., a few GBs).