From owner-freebsd-performance@freebsd.org Wed Feb 10 16:06:00 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5516F54ADD5 for ; Wed, 10 Feb 2021 16:06:00 +0000 (UTC) (envelope-from raj@gusw.net) Received: from gateway36.websitewelcome.com (gateway36.websitewelcome.com [192.185.200.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DbPkb3WlPz4fmP for ; Wed, 10 Feb 2021 16:05:59 +0000 (UTC) (envelope-from raj@gusw.net) Received: from cm17.websitewelcome.com (cm17.websitewelcome.com [100.42.49.20]) by gateway36.websitewelcome.com (Postfix) with ESMTP id CE02D400F0ACC for ; Wed, 10 Feb 2021 10:05:53 -0600 (CST) Received: from host2097.hostmonster.com ([67.20.114.243]) by cmsmtp with SMTP id 9s01lJdupDT649s01l6ko9; Wed, 10 Feb 2021 10:05:53 -0600 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=schadow.us; s=default; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version :Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=M7yf190a0SLJ204Kb5/pMnDDzLPJWUsy9RBAXJ+74aA=; b=lMrhVxoYo4djBNBxJWDv+qoCrK NdKU58FiAVU+/1ycuHoDim3wl/PyyO+hwOs0HXL8rr6l6ayt2AoyHqhk7WzEE2qgcNr/OX9/aTG66 bVbS1cjS5VHbUL8jdHw1Ql52r; Received: from [179.218.210.156] (port=56483 helo=[192.168.121.116]) by host2097.hostmonster.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1l9s01-001SqK-0s for freebsd-performance@freebsd.org; Wed, 10 Feb 2021 09:05:53 -0700 Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org References: <5BC0DEF9-3D58-4FFC-9E20-311B0520A25A@longcount.org> From: Gunther Schadow Message-ID: <96d43dbe-ed50-8ba4-f676-673ee99725bd@gusw.net> Date: Wed, 10 Feb 2021 11:05:48 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <5BC0DEF9-3D58-4FFC-9E20-311B0520A25A@longcount.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2097.hostmonster.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gusw.net X-BWhitelist: no X-Source-IP: 179.218.210.156 X-Source-L: No X-Exim-ID: 1l9s01-001SqK-0s X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: ([192.168.121.116]) [179.218.210.156]:56483 X-Source-Auth: ebiz+schadow.us X-Email-Count: 1 X-Source-Cap: cHJhZ21hdDE7cHJhZ21hdDE7aG9zdDIwOTcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-Rspamd-Queue-Id: 4DbPkb3WlPz4fmP X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=schadow.us header.s=default header.b=lMrhVxoY; dmarc=none; spf=softfail (mx1.freebsd.org: 192.185.200.11 is neither permitted nor denied by domain of raj@gusw.net) smtp.mailfrom=raj@gusw.net X-Spamd-Result: default: False [-3.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_X_SOURCE(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[schadow.us:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; HAS_X_ANTIABUSE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[192.185.200.11:from]; ASN(0.00)[asn:46606, ipnet:192.185.192.0/19, country:US]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_NEUTRAL(0.00)[192.185.200.11:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[schadow.us:s=default]; FREEFALL_USER(0.00)[raj]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[gusw.net]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[192.185.200.11:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[192.185.200.11:from]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2021 16:06:00 -0000 I think we got enough feed back now from other professionals to suggest that it would do the FreeBSD project good to acknowledge the issue and create some sort of work in progress / project statement, perhaps a wiki, where people can flock to to look for workarounds, current statue, just to not feel so lost and lonely and wondering if anybody cares at all. Such a meeting-point could at least be something of a self-help group, for emotional support and positive thinking =8). I have a slight sign of hope also: in my latest Amazon Linux db server deployment the performance is only half of what I got in my previous db server deployment. Go figure. But it's a fresh launch and I can't figure out what else I had optimized with the previous Amazon Linux box. So, while not improving anything, it at least closes the inequality between Linux and FreeBSD in the socialistic way ;) Now I am trying the FreeBSD install again with the same disk setup. Because one thing I clearly know is that it makes a huge difference of having one large EBS device and a single partition / file system vs. the same large EBS device with many partitions and file systems, separating tables, indexes, temporary sort space, etc. so that there is less random access contention on a single file system. Why that is important, I wonder, actually. And it's not the underlying EBS that matters so much. Like in bare metal world, the approach would be separate disks vs. striping (RAID-0) to get more spindles involved instead of having disk seek. But none of that should mater that much any more with SSD drives (which EBS gp2 and gp3 are!) Fortunately now Amazon has gp3 volumes also which support 3000 io transactions per second (IOps) and 250 or up to 1000 MB/s throughput despite being as small as 4 GB. So what I'm doing now is instead of a partitioned 1 TB gp2 volume with many partitions, I make myself over 20 separate smaller gp3 volumes, the advantage of this is that I can individually resize those without colliding with the neighboring partition. Tomorrow I should have better comparison numbers for my database on both Linux and FreeBSD configured the exact same way, and it might be closer now. But sadly only in the socialist way. regards, -Gunther On 2/6/2021 9:55 AM, Mark Saad wrote: > On Feb 6, 2021, at 7:07 AM, Łukasz Wąsikowski wrote: > All > So what I was getting at, is do we have good data on what the issue is ? Can we make a new wiki page on the FreeBSD wiki to track what works what and doesn’t . Does one exist ? > > To be clear we should check if the issue something that aws is doing with their xen platform , kvm/qemu one or common to all ? Also does that same issue appear on google and Microsoft’s platforms? This will at least get some bounds on the problem and what if any fixes may exist . > There are some commercial FreeBSD products running on aws . Maybe the vendors know some stuff that can help ? > > > Thoughts ? > > --- > Mark Saad | nonesuch@longcount.org > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >