From owner-freebsd-performance@freebsd.org Fri Feb 5 14:53:47 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5661E549BF0 for ; Fri, 5 Feb 2021 14:53:47 +0000 (UTC) (envelope-from raj@gusw.net) Received: from gateway20.websitewelcome.com (gateway20.websitewelcome.com [192.185.46.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXJMZ2jJGz3MY9 for ; Fri, 5 Feb 2021 14:53:46 +0000 (UTC) (envelope-from raj@gusw.net) Received: from cm10.websitewelcome.com (cm10.websitewelcome.com [100.42.49.4]) by gateway20.websitewelcome.com (Postfix) with ESMTP id 0E27C400D0A87 for ; Fri, 5 Feb 2021 08:46:29 -0600 (CST) Received: from host2097.hostmonster.com ([67.20.114.243]) by cmsmtp with SMTP id 82UQl0hyhuDoA82UQl2Ab1; Fri, 05 Feb 2021 08:53:43 -0600 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=schadow.us; s=default; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version :Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=cB3IiyyxoxsNvEkS4hdteIF6Z1owUxEaNR+bZzc/kJI=; b=TjmGVJVRau6lYKvY0qTRiGM89y zmyU6P7sgPS2UciKsVirCt+2XHccPSxACCF5bHHwZe0yPMVHY7NV+Dt2/xjbwWE1StvJwV+NKCge6 OCubbSIr8RBUHtsLJTE0UFeRQ; Received: from [177.143.140.27] (port=63252 helo=[10.0.4.7]) by host2097.hostmonster.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1l82UQ-002uSu-Dn for freebsd-performance@freebsd.org; Fri, 05 Feb 2021 07:53:42 -0700 Subject: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org References: From: Gunther Schadow Message-ID: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> Date: Fri, 5 Feb 2021 09:53:33 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2097.hostmonster.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gusw.net X-BWhitelist: no X-Source-IP: 177.143.140.27 X-Source-L: No X-Exim-ID: 1l82UQ-002uSu-Dn X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: ([10.0.4.7]) [177.143.140.27]:63252 X-Source-Auth: ebiz+schadow.us X-Email-Count: 1 X-Source-Cap: cHJhZ21hdDE7cHJhZ21hdDE7aG9zdDIwOTcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-Rspamd-Queue-Id: 4DXJMZ2jJGz3MY9 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=schadow.us header.s=default header.b=TjmGVJVR; dmarc=none; spf=softfail (mx1.freebsd.org: 192.185.46.107 is neither permitted nor denied by domain of raj@gusw.net) smtp.mailfrom=raj@gusw.net X-Spamd-Result: default: False [-3.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_X_SOURCE(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[schadow.us:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; HAS_X_ANTIABUSE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:46606, ipnet:192.185.0.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[192.185.46.107:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[schadow.us:s=default]; FREEFALL_USER(0.00)[raj]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[gusw.net]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[192.185.46.107:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[192.185.46.107:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[192.185.46.107:from]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 14:53:47 -0000 Hi, I've been with FreeBSD since 386BSD 0.0new. Always tried to run everything on it. I saw us lose the epic race against Linux over the stupid BSDI lawsuit. But now I'm afraid I am witnessing the complete fading of FreeBSD from relevance in the marketplace as the performance of FreeBSD on AWS EC2 (and as I see in the chatter from other "cloud" platforms) falls far behind that of Linux. Not by a few % points, but by factors if not an order of magnitude! The motto "the power to serve" meant that FreeBSD was the most solid and consistently performing system for heavy multi-tasking network and disk operation. A single thread was allowed to do better on another OS without us feeling shame, but overall you could rely on FreeBSD being your best choice to overall server performance. The world has changed. We used to run servers on bare metal in a cage in physical data center. I did that. A year or two of instability with the FreeBSD drivers for new beefy hardware didn't scare me off. Now the cost and flexibility calculations today changed the market away from bare metal to those "cloud" service providers, Amazon AWS (>38% market share), Azure (19% market share), and many others. I still remember searching for "hosting" providers who would offer FreeBSD (or any BSD) as an option and it was hard to find. On Amazon AWS we have the FreeBSD image ready to launch, that is good. But the problem is, it's disk (and network?) performance is bad (to horrible) and it's really sad and embarrassing. Leaving FreeBSD beaten far behind and for realistic operations, it's impossible to use, despite being so much better organized than Linux. I have put significant investment into a flexible scalable FreeBSD image only to find now that I just cannot justify using FreeBSD when Linux out of the box is several times faster. There have been few problem reports about this over many years, and they all end the same way: either no response, or defensive response ("your measures are invalid"), with the person reporting the problem eventually walking away with no solution. Disinterest. I can link to those instances. Examples: https://lists.freebsd.org/pipermail/freebsd-performance/2009-February/003677.html https://forums.freebsd.org/threads/aws-disk-i-o-performance-xbd-vs-nvd.74751/ https://forums.freebsd.org/threads/aws-ec2-ena-poor-network-performance-low-pps.77093/#post-492744 https://forums.freebsd.org/threads/poor-php-and-python-performances.72427/ https://forums.freebsd.org/threads/freebsd-was-once-the-power-to-server-but-in-an-aws-world-we-have-fallen-way-waaay-behind-and-there-seems-no-interest-to-fix-it.78738/page-2 My intention is not to rant, vent, proselytize to Linux (I hate Linux) but to see what is wrong with FreeBSD? And how it can be fixed? Why does it seem nobody is interested in getting the dismal AWS EC2 performance resolved? This looks to me like a vicious cycle: FreeBSD on AWS is bad so nobody will use it for any real work, and because nobody uses it there is no interest in making it work well. In addition there is no interest on the side of FreeBSD people to make it better. It's got to be the lack of interest, not of anyone not having access to the AWS EC2 hardware. What can be done? I am trying to run a company, so I cannot justify playing with this for much longer shooting in the dark. If I wasn't the boss myself, my boss would have long told me to quit this nonsense and use Linux. If I saw interest, I could justify holding out just a little longer. But I don't see any encouraging feedback. Is there anyone at all in the FreeBSD dev or FreeBSD.org as an organization interested in actually being competitive in the AWS EC2 space (and other virtualization "clouds")? If so, how many? How can this be fixed? How can I help? I cannot justify spending too much more of my own time on it, but I could help making resources available or paying for someone who has both a sense of great urgency to redeem FreeBSD and the know-how to make it happen. regards, -Gunther From owner-freebsd-performance@freebsd.org Fri Feb 5 15:42:28 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1B3E54AABE for ; Fri, 5 Feb 2021 15:42:28 +0000 (UTC) (envelope-from claudiu.vasadi@gmail.com) Received: from mail-oi1-x229.google.com (mail-oi1-x229.google.com [IPv6:2607:f8b0:4864:20::229]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXKRm2tXSz3Pym for ; Fri, 5 Feb 2021 15:42:28 +0000 (UTC) (envelope-from claudiu.vasadi@gmail.com) Received: by mail-oi1-x229.google.com with SMTP id k25so7816384oik.13 for ; Fri, 05 Feb 2021 07:42:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=NA/F+X37ASPUkJv8DoG0VoSrP5I3Yikfgwef2R+hs8c=; b=FBGT8Z+RXidz0euqviOZWKJqZF/tXHgi5t9M/4ddIZiAaiCEvn61u6uBfaaLeKSyR8 cYCBE8oG5kXZe/9FQ7vVW6uaibXkpb0a/o8vGDiYVbND+kd31hSggSXS+awnkWvrN128 xtfQHtDWmF3pajFI4rcwWR0ME0lJTY1XT5HZYTwdwmXZVO2iWUkj8gIzaUfhP3BpUPsL kRfXjzsC41fQZiS+2UgKLs/6GDsDLQDknNBAiAlO4tC5DaOam0wzbbIsGSOos0KWuYDf SynfgMDOZbMSnaK2VGTFZn951hMknPLilRjc719bvvrZY+/rHyJuaoVS04S88trx52hF OXzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=NA/F+X37ASPUkJv8DoG0VoSrP5I3Yikfgwef2R+hs8c=; b=rimwh+JY4lQGLDezt0VHnw4Y9BTI1owuoAYhhv7K07SjCwoRrEUo38udErroTydwqv WqSvUlv9oKseU2A7dX873HcDm07bIBn3zoOFEVyb8M1y9QV2ITUzLh9NDfZjQC2knJ/G VsfaBAol6Yhl6rbom3/YKP2LzZuM3df0Bpcq8RokuC7uU/OSBR8LkcTBttcog9a80323 eSV2BB0MDYLVEOovRsfbnZqR4LXDH+Hq2YWswNFLHF/aPZvXtGF0j9YhKhPmfG5Nx4Ab Q2Fp4nLJmhgNY3Z9QuEWWu6sX8mTbkDD7OKIUNOkvRXYOqQtWkbjIcEpM4mnrL3Qw4Sc i8Wg== X-Gm-Message-State: AOAM531PKhYCTq4wzEQMh/VtL6DsJ8/mJQQTxS79ZMSqmOcQdOBik3hn k2TnNbsIiQBj33/Hs6I2BolYA7cFS58Jmp2g58ZhmK3ipNU= X-Google-Smtp-Source: ABdhPJwzbh7WD4S9kpF73UxYuT9JBwbv1Fh2ZvBAHz6KYRXgooa67+N4sgLIPyV2uwPbIxqNFM+FvrS04qzOB7EFztQ= X-Received: by 2002:aca:ef89:: with SMTP id n131mr3449101oih.4.1612539747419; Fri, 05 Feb 2021 07:42:27 -0800 (PST) MIME-Version: 1.0 References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> In-Reply-To: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> From: claudiu vasadi Date: Fri, 5 Feb 2021 16:42:16 +0100 Message-ID: Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: Gunther Schadow Cc: freebsd-performance@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DXKRm2tXSz3Pym X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=FBGT8Z+R; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of claudiuvasadi@gmail.com designates 2607:f8b0:4864:20::229 as permitted sender) smtp.mailfrom=claudiuvasadi@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::229:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::229:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::229:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 15:42:28 -0000 FreeBSD got killed a long time ago. thank the leadership, or lack thereof, for it. I was in your shoes and I had no choice but to ditch it all together eventually. want pf? openbsd. want something else, go linux :shrug: several on the forum (old and new) feel the same and had to make the same decision in the end..... it's just sad. From owner-freebsd-performance@freebsd.org Fri Feb 5 18:17:05 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AD88854ED03 for ; Fri, 5 Feb 2021 18:17:05 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXNt94VrPz3sXk; Fri, 5 Feb 2021 18:17:05 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from localhost (p200300d5d711ab71c412b6d6dff2b461.dip0.t-ipconnect.de [IPv6:2003:d5:d711:ab71:c412:b6d6:dff2:b461]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: gbe) by smtp.freebsd.org (Postfix) with ESMTPSA id 22980255BA; Fri, 5 Feb 2021 18:17:05 +0000 (UTC) (envelope-from gbe@freebsd.org) Date: Fri, 5 Feb 2021 19:17:04 +0100 From: Gordon Bergling To: Gunther Schadow Cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems Message-ID: References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="bYq1GYnMzmh5VPU/" Content-Disposition: inline In-Reply-To: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> X-Url: X-Operating-System: FreeBSD 12.2-STABLE amd64 X-Host-Uptime: 7:15PM up 7:01, 4 users, load averages: 0.25, 0.25, 0.19 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 18:17:05 -0000 --bYq1GYnMzmh5VPU/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Sorry, for top posting. Can you verify your feelings by numbers? --Gordon On Fri, Feb 05, 2021 at 09:53:33AM -0500, Gunther Schadow wrote: > Hi, I've been with FreeBSD since 386BSD 0.0new. Always tried to run > everything on it. I saw us lose the epic race against Linux over the > stupid BSDI lawsuit. But now I'm afraid I am witnessing the complete > fading of FreeBSD from relevance in the marketplace as the performance > of FreeBSD on AWS EC2 (and as I see in the chatter from other "cloud" > platforms) falls far behind that of Linux. Not by a few % points, but > by factors if not an order of magnitude! >=20 > The motto "the power to serve" meant that FreeBSD was the most solid > and consistently performing system for heavy multi-tasking network > and disk operation. A single thread was allowed to do better on another > OS without us feeling shame, but overall you could rely on FreeBSD > being your best choice to overall server performance. >=20 > The world has changed. We used to run servers on bare metal in a cage > in physical data center. I did that. A year or two of instability with > the FreeBSD drivers for new beefy hardware didn't scare me off. >=20 > Now the cost and flexibility calculations today changed the market > away from bare metal to those "cloud" service providers, Amazon AWS > (>38% market share), Azure (19% market share), and many others. I > still remember searching for "hosting" providers who would > offer FreeBSD (or any BSD) as an option and it was hard to find. On > Amazon AWS we have the FreeBSD image ready to launch, that is good. >=20 > But the problem is, it's disk (and network?) performance is bad (to > horrible) and it's really sad and embarrassing. Leaving FreeBSD beaten > far behind and for realistic operations, it's impossible to use, despite > being so much better organized than Linux. I have put significant > investment into a flexible scalable FreeBSD image only to find now that I > just cannot justify using FreeBSD when Linux out of the box is several > times faster. >=20 > There have been few problem reports about this over many years, and > they all end the same way: either no response, or defensive response > ("your measures are invalid"), with the person reporting the problem > eventually walking away with no solution. Disinterest. I can link to > those instances. Examples: >=20 > https://lists.freebsd.org/pipermail/freebsd-performance/2009-February/003= 677.html > https://forums.freebsd.org/threads/aws-disk-i-o-performance-xbd-vs-nvd.74= 751/ > https://forums.freebsd.org/threads/aws-ec2-ena-poor-network-performance-l= ow-pps.77093/#post-492744 > https://forums.freebsd.org/threads/poor-php-and-python-performances.72427/ > https://forums.freebsd.org/threads/freebsd-was-once-the-power-to-server-b= ut-in-an-aws-world-we-have-fallen-way-waaay-behind-and-there-seems-no-inter= est-to-fix-it.78738/page-2 >=20 > My intention is not to rant, vent, proselytize to Linux (I hate Linux) > but to see what is wrong with FreeBSD? And how it can be fixed? Why does > it seem nobody is interested in getting the dismal AWS EC2 performance > resolved? This looks to me like a vicious cycle: FreeBSD on AWS is > bad so nobody will use it for any real work, and because nobody uses it > there is no interest in making it work well. In addition there is no inte= rest > on the side of FreeBSD people to make it better. It's got to be the lack > of interest, not of anyone not having access to the AWS EC2 hardware. >=20 > What can be done? I am trying to run a company, so I cannot justify playi= ng > with this for much longer shooting in the dark. If I wasn't the boss myse= lf, > my boss would have long told me to quit this nonsense and use Linux. > If I saw interest, I could justify holding out just a little longer. But > I don't see any encouraging feedback. Is there anyone at all in the FreeB= SD > dev or FreeBSD.org as an organization interested in actually being compet= itive > in the AWS EC2 space (and other virtualization "clouds")? If so, how many? > How can this be fixed? How can I help? I cannot justify spending too much > more of my own time on it, but I could help making resources available > or paying for someone who has both a sense of great urgency to redeem > FreeBSD and the know-how to make it happen. >=20 > regards, > -Gunther >=20 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= =2Eorg" --=20 --bYq1GYnMzmh5VPU/ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAEBCgB9FiEEYbWI0KY5X7yH/Fy4OQX2V8rP09wFAmAdi59fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDYx QjU4OEQwQTYzOTVGQkM4N0ZDNUNCODM5MDVGNjU3Q0FDRkQzREMACgkQOQX2V8rP 09wjCwf+IGiZAmGqnXPRCOG80wunh8jvo8HNY0lZHqkR5GTDyU9Vro6EYUKz4MOC Dfs8wm0mLHsjVfgFivoSmH3xsSaA+J9MvjvHEoubOpsYkH5FcbzW4KZNtlokF1K3 mDeyXBC+a1zoyEY8ZpMjsHBYBeg9HL1CrLfOuVcLtG30KQZ7JA9zhtbAgSE8u0J6 35imG/eow2KMRxY/RJWyA558ZGQSGwcOXOS7rJUUkRZwpHj+cUSMLL6yUuYxCZhH gOcRX/3SB+tlsXKnBGWEkby6dP8OL0vYDdC5nuLIfKaSWtRYalLO7/hMxGCwCDdT JDh11D/d9PWdlSQ5B+fgTJX0Rrvmcw== =JhlO -----END PGP SIGNATURE----- --bYq1GYnMzmh5VPU/-- From owner-freebsd-performance@freebsd.org Fri Feb 5 20:45:47 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 917AA529EA4 for ; Fri, 5 Feb 2021 20:45:47 +0000 (UTC) (envelope-from raj@gusw.net) Received: from gateway22.websitewelcome.com (gateway22.websitewelcome.com [192.185.46.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXS9k4JMGz4Wtn for ; Fri, 5 Feb 2021 20:45:46 +0000 (UTC) (envelope-from raj@gusw.net) Received: from cm16.websitewelcome.com (cm16.websitewelcome.com [100.42.49.19]) by gateway22.websitewelcome.com (Postfix) with ESMTP id 62E1D4315 for ; Fri, 5 Feb 2021 14:45:45 -0600 (CST) Received: from host2097.hostmonster.com ([67.20.114.243]) by cmsmtp with SMTP id 87z7lWWcJHPnU87z7l4yhw; Fri, 05 Feb 2021 14:45:45 -0600 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=schadow.us; s=default; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version :Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=d0ldUDxfnA3zcNLR66p3o2wd/Kxdm9BGqXr/2JnB144=; b=OGXdj3IaQO+YNElVNf4uS/wH2B 0PcNQKUDN/K53y6mWSmUZ2SEiJHHnGAI1KNsjivSMov+iz5p25OO+KQqYu4UA6heKSYZTZ7XfzrNt +cMOY6gbsYTHrdP0Kg9bDPTIN; Received: from [177.143.140.27] (port=49230 helo=[10.0.4.7]) by host2097.hostmonster.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1l87z6-002b7C-MK for freebsd-performance@freebsd.org; Fri, 05 Feb 2021 13:45:44 -0700 Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> From: Gunther Schadow Message-ID: <3fde2934-1e18-5ea4-84d6-21200eaf4b20@gusw.net> Date: Fri, 5 Feb 2021 15:45:42 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2097.hostmonster.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gusw.net X-BWhitelist: no X-Source-IP: 177.143.140.27 X-Source-L: No X-Exim-ID: 1l87z6-002b7C-MK X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: ([10.0.4.7]) [177.143.140.27]:49230 X-Source-Auth: ebiz+schadow.us X-Email-Count: 1 X-Source-Cap: cHJhZ21hdDE7cHJhZ21hdDE7aG9zdDIwOTcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-Rspamd-Queue-Id: 4DXS9k4JMGz4Wtn X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=schadow.us header.s=default header.b=OGXdj3Ia; dmarc=none; spf=softfail (mx1.freebsd.org: 192.185.46.225 is neither permitted nor denied by domain of raj@gusw.net) smtp.mailfrom=raj@gusw.net X-Spamd-Result: default: False [-3.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_X_SOURCE(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[schadow.us:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; HAS_X_ANTIABUSE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:46606, ipnet:192.185.0.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[schadow.us:s=default]; FREEFALL_USER(0.00)[raj]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[gusw.net]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[192.185.46.225:from]; RWL_MAILSPIKE_VERYGOOD(0.00)[192.185.46.225:from]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 20:45:47 -0000 Gordon Bergling wrote: > Can you verify your feelings by numbers? Yes, like I said >> Not by a few % points, but by factors if not an order of magnitude! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253261 Do this: dd if=/dev/zero of=/dev/nvd2 bs=100M status=progress and you see that it's writing with the "whopping" speed of 70 MB/s. That used to be good, but it is no longer good. Compare Amazon Linux doing the same thing at 300 MB/s. Now, when you put a file system over it, zfs or ufs, then instantly the performance gets better: newfs /dev/nvd2 mount /dev/nvd2 /mnt dd if=/dev/zero of=/mnt/test bs=100M status=progress now that works at about 250 MB/s. Decent. So, problem solved? No! It turns out if I create a PostgreSQL database over this setup, then again there is massive delay on the read and write and throughput will drop to even worse than 70 MB/s. Creating one index takes 10 times as long as that same on the Linux system. PS: no need to point out that Linux uses buffer cache for direct write to device and BSD doesn't. Those effects will not make a difference when you write (or read) more than the buffer cache size (e.g., a few GBs). From owner-freebsd-performance@freebsd.org Fri Feb 5 23:00:14 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1D6C652C9A7 for ; Fri, 5 Feb 2021 23:00:14 +0000 (UTC) (envelope-from jguojun@gmail.com) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXW8s2yHWz4fvd for ; Fri, 5 Feb 2021 23:00:13 +0000 (UTC) (envelope-from jguojun@gmail.com) Received: by mail-ot1-x32e.google.com with SMTP id v1so8462219ott.10 for ; Fri, 05 Feb 2021 15:00:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=uas030oI5eW4w3MeXn6n/gFjH+WN/Lg9K8jI2p/bgbk=; b=YemOL0+OVfCB1Mqk5AVn9S/Oj23/6nAtVmyNaMAwd2/oycTikaLoP0BUL/vKrFAzjz KDnayMBNd+EAw84EhtYuzOyM0pAuNDAUPRqcOoBI53KIPxW8svLdlT8g3iBOy4QiMDky OKv4p/+QHShovHGVvQ/qrYrpZqDuXqMg/32+vNb7nrbmKJPqgcZnR9srlzyBcmC7kPm1 T3Ef0HlsSwTVd8XOLcsODDYaCTeQmWoQ+RM7IDYnh9nBFCU2oTImO6n7PbtTIOKrnK6g 6COYNCqXY2fDUsoII1nmJ3S0w1houLgJK8LsoFsONGBQ4735M8kAFYvf5aIHioxOy2Pv HXKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=uas030oI5eW4w3MeXn6n/gFjH+WN/Lg9K8jI2p/bgbk=; b=FDPao0QSJrQ+BVCgbrdX28ld9XHTH6YrCHMknx8GYalOSfId7f6yedyb0J+pG3l11C N1kb2UDw3nQzCXjKVrxRsWtEpsf9IC5N0pzAiF5QdMMhb1kQW92K1M+pLe7S7X2qg7bp 7aEUY2na6RZVuFSGmFGvtlMgJHAhBaTbBRXa9flVwVg9JwFVXRhX7m/QhkBN/C8ve3JV 4qn0zjKd5hg4wpUUr5fgblGB1pNxCsF+FP6zVmYJa/JDbGdzl6AP4UHyu6cR1dBXjwCE Xkj7fvgyYoGuDpiVSJhuJixare86AnGJ0iLCtwXDNldNm6USwCbo+vjMKZEIwOdlNqYZ QQHw== X-Gm-Message-State: AOAM531ulAf9IOOmomvkP32EFhb5xSfjgnbTLp+YQ7qwnhPzu18dBReT Kx68JkLUVNhY477tu2TT8ss9QeYbOfPA X-Google-Smtp-Source: ABdhPJx1HsfIqPJphPz9dS7QfI0Ot3CE7+15RVmyBnEk92OL9vs99gus87ssYCTXRU+d8q7iJZjmkA== X-Received: by 2002:a9d:2da2:: with SMTP id g31mr5023042otb.222.1612566011456; Fri, 05 Feb 2021 15:00:11 -0800 (PST) Received: from ?IPv6:2600:1702:2771:1060:e23f:49ff:fe45:91e6? ([2600:1702:2771:1060:e23f:49ff:fe45:91e6]) by smtp.gmail.com with ESMTPSA id m15sm2057273otl.11.2021.02.05.15.00.10 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 05 Feb 2021 15:00:10 -0800 (PST) Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> <3fde2934-1e18-5ea4-84d6-21200eaf4b20@gusw.net> From: "Jin Guojun[VFF]" Message-ID: <4d53a12a-99a5-6bab-a30a-74f680b77edb@gmail.com> Date: Fri, 5 Feb 2021 15:00:04 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <3fde2934-1e18-5ea4-84d6-21200eaf4b20@gusw.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4DXW8s2yHWz4fvd X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=YemOL0+O; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jguojun@gmail.com designates 2607:f8b0:4864:20::32e as permitted sender) smtp.mailfrom=jguojun@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::32e:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::32e:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32e:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 23:00:14 -0000 On 2021-02-05 12:45, Gunther Schadow wrote: > Gordon Bergling wrote: >> Can you verify your feelings by numbers? > > Yes, like I said > >>> Not by a few % points, but by factors if not an order of magnitude! > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253261 > > Do this: > > dd if=/dev/zero of=/dev/nvd2 bs=100M status=progress > > and you see that it's writing with the "whopping" speed of 70 MB/s. > > That used to be good, but it is no longer good. Compare Amazon Linux > doing > the same thing at 300 MB/s. > > Now, when you put a file system over it, zfs or ufs, then instantly the > performance gets better: > > newfs /dev/nvd2 > mount /dev/nvd2 /mnt > dd if=/dev/zero of=/mnt/test bs=100M status=progress > > now that works at about 250 MB/s. Decent. So, problem solved? It is not clear if this compares Apple to Apple. What disk drives and CPUs are on FreeBSD, and what are disk drive(s) and CPU(s) on AWS? Knowing the drive brand and models will tell approximately the disk throughput. Agree, 70MB/s is slow for modern disks, but your information does not provide clue why this could be slow. Can this setup get 250MB/s on FreeBSD 11.4? or 300MB/s with Ubuntu 16.04 on the same hardware? -Jin From owner-freebsd-performance@freebsd.org Fri Feb 5 23:14:35 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D07D752CD2E for ; Fri, 5 Feb 2021 23:14:35 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXWTR5XMHz4h2y for ; Fri, 5 Feb 2021 23:14:35 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: seanc) by smtp.freebsd.org (Postfix) with ESMTPSA id AA6C227F9E for ; Fri, 5 Feb 2021 23:14:35 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: by mail-pl1-f178.google.com with SMTP id e9so4338226plh.3 for ; Fri, 05 Feb 2021 15:14:35 -0800 (PST) X-Gm-Message-State: AOAM532AHMFAihOFcw4ItUMSevtA/+b20Tx6k4ycEcXAUP0zw5m6MdpR h8wLLyCaBuEepBQDUDZo5SmiMUiv20308Q/DfrxTOw== X-Google-Smtp-Source: ABdhPJyHQT4v679u6NhJeU5OjOcLu4Y4x7mZXec5D3bHfA9vN2OKZ3eTcYdPvwl31YNNS7rtfzJO9S5bt+w3Cwt/moo= X-Received: by 2002:a17:902:e808:b029:de:5a8d:c654 with SMTP id u8-20020a170902e808b02900de5a8dc654mr6202444plg.80.1612566874492; Fri, 05 Feb 2021 15:14:34 -0800 (PST) MIME-Version: 1.0 References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> <3fde2934-1e18-5ea4-84d6-21200eaf4b20@gusw.net> <4d53a12a-99a5-6bab-a30a-74f680b77edb@gmail.com> In-Reply-To: <4d53a12a-99a5-6bab-a30a-74f680b77edb@gmail.com> From: Sean Chittenden Date: Fri, 5 Feb 2021 15:14:18 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: "Jin Guojun[VFF]" Cc: freebsd-performance@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 23:14:35 -0000 To be clear, this is a known issue that needs attention: it is not a benchmarking setup problem. Network throughput has a similar problem and needs similar attention. Cloud is not a fringe server workload. -sc On Fri, Feb 5, 2021 at 3:00 PM Jin Guojun[VFF] wrote: > On 2021-02-05 12:45, Gunther Schadow wrote: > > Gordon Bergling wrote: > >> Can you verify your feelings by numbers? > > > > Yes, like I said > > > >>> Not by a few % points, but by factors if not an order of magnitude! > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253261 > > > > Do this: > > > > dd if=/dev/zero of=/dev/nvd2 bs=100M status=progress > > > > and you see that it's writing with the "whopping" speed of 70 MB/s. > > > > That used to be good, but it is no longer good. Compare Amazon Linux > > doing > > the same thing at 300 MB/s. > > > > Now, when you put a file system over it, zfs or ufs, then instantly the > > performance gets better: > > > > newfs /dev/nvd2 > > mount /dev/nvd2 /mnt > > dd if=/dev/zero of=/mnt/test bs=100M status=progress > > > > now that works at about 250 MB/s. Decent. So, problem solved? > > It is not clear if this compares Apple to Apple. > > What disk drives and CPUs are on FreeBSD, and what are disk drive(s) and > CPU(s) on AWS? > > Knowing the drive brand and models will tell approximately the disk > throughput. Agree, 70MB/s is slow for modern disks, but your information > does not provide clue why this could be slow. > > Can this setup get 250MB/s on FreeBSD 11.4? or 300MB/s with Ubuntu 16.04 > on the same hardware? > > -Jin > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@freebsd.org Sat Feb 6 00:10:05 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 92C3C52DD77 for ; Sat, 6 Feb 2021 00:10:05 +0000 (UTC) (envelope-from raj@gusw.net) Received: from gateway31.websitewelcome.com (gateway31.websitewelcome.com [192.185.143.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXXjS42Tmz4ktl for ; Sat, 6 Feb 2021 00:10:04 +0000 (UTC) (envelope-from raj@gusw.net) Received: from cm14.websitewelcome.com (cm14.websitewelcome.com [100.42.49.7]) by gateway31.websitewelcome.com (Postfix) with ESMTP id 72E0691CA35 for ; Fri, 5 Feb 2021 18:10:01 -0600 (CST) Received: from host2097.hostmonster.com ([67.20.114.243]) by cmsmtp with SMTP id 8BAnlg3o1svw98BAnlmi12; Fri, 05 Feb 2021 18:10:01 -0600 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=schadow.us; s=default; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version :Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=HvJxILZdTKeUkf9JJ74agOTFQGy/z0WEpsPdTvzHzGc=; b=mdjJK2cmx9r6VRbRxaV/xGKNYT YOx0on7d2axPKXliIwbl303vM8pizIgFPGbvhRIEBhEhjhDRYS8yHQZfpzCnxH6KrWPt6biz/p+Yx a5DotDxhGPZFY+0qfOJo7RkHE; Received: from 191-246-21-95.3g.claro.net.br ([191.246.21.95]:9440 helo=[192.168.43.239]) by host2097.hostmonster.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1l8BAm-000PPW-N1 for freebsd-performance@freebsd.org; Fri, 05 Feb 2021 17:10:00 -0700 Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> <3fde2934-1e18-5ea4-84d6-21200eaf4b20@gusw.net> <4d53a12a-99a5-6bab-a30a-74f680b77edb@gmail.com> From: Gunther Schadow Message-ID: <5007f6a8-5e15-2ac0-f190-0a60e2865114@gusw.net> Date: Fri, 5 Feb 2021 19:09:58 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2097.hostmonster.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gusw.net X-BWhitelist: no X-Source-IP: 191.246.21.95 X-Source-L: No X-Exim-ID: 1l8BAm-000PPW-N1 X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: 191-246-21-95.3g.claro.net.br ([192.168.43.239]) [191.246.21.95]:9440 X-Source-Auth: ebiz+schadow.us X-Email-Count: 1 X-Source-Cap: cHJhZ21hdDE7cHJhZ21hdDE7aG9zdDIwOTcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-Rspamd-Queue-Id: 4DXXjS42Tmz4ktl X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=schadow.us header.s=default header.b=mdjJK2cm; dmarc=none; spf=softfail (mx1.freebsd.org: 192.185.143.4 is neither permitted nor denied by domain of raj@gusw.net) smtp.mailfrom=raj@gusw.net X-Spamd-Result: default: False [-2.86 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_X_SOURCE(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[schadow.us:+]; NEURAL_HAM_SHORT(-0.56)[-0.563]; HAS_X_ANTIABUSE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[192.185.143.4:from]; ASN(0.00)[asn:46606, ipnet:192.185.128.0/18, country:US]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[191.246.21.95:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[schadow.us:s=default]; FREEFALL_USER(0.00)[raj]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[gusw.net]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[192.185.143.4:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[192.185.143.4:from]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Feb 2021 00:10:05 -0000 Hi Sean and Brendan On 2/5/2021 6:14 PM, Sean Chittenden wrote: > To be clear, this is a known issue that needs attention: it is not a > benchmarking setup problem. Network throughput has a similar problem and > needs similar attention. Cloud is not a fringe server workload. -sc I am so glad you're saying that, because I was afraid I'd have to argue again and again to make people see there is a problem. But I come here anyway with some numbers: This is the Amazon Linux system which I fell back to in desperation, launched, added ZFS, compiled and set up PostgreSQL-13.2 (whatever newest) and am now pg_dumpall -h db-old |dd bs=10M status=progress |psql -d postgres we had sucked the data tables at 32 MB/s that is about the same speed as I got on the FreeBSD. I assume that might be network bound. Now it's regenerating the indexes and I see the single EBS g3 volume on fire: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme1n1 0.00 7.00 781.00 824.00 99508.50 76675.00 219.54 1.98 1.66 1.75 1.57 0.41 66.40 nvme1n1 0.00 3.00 1740.00 456.00 222425.00 33909.50 233.46 4.43 2.42 2.29 2.91 0.46 100.80 nvme1n1 0.00 0.00 1876.00 159.00 239867.00 16580.00 252.04 3.56 2.20 2.09 3.47 0.49 100.00 nvme1n1 0.00 0.00 1883.00 151.00 240728.00 15668.00 252.11 3.49 2.15 2.10 2.83 0.49 100.00 nvme1n1 0.00 0.00 1884.00 152.00 240593.50 15688.00 251.75 3.54 2.19 2.13 3.00 0.49 100.00 nvme1n1 0.00 1.00 1617.00 431.00 206680.00 50047.50 250.71 4.50 2.63 2.49 3.13 0.48 98.40 nvme1n1 0.00 1.00 1631.00 583.00 208331.50 47909.00 231.47 4.75 2.54 2.49 2.66 0.45 100.00 nvme1n1 0.00 0.00 1892.00 148.00 241128.50 15440.00 251.54 3.20 2.01 1.96 2.73 0.49 100.00 On FreeBSD I haven't got the numbers saved now, but with all the simultaneous reading and writing activity going on, the system got down to just about 40 MB/s read and 40 MB/s write, if lucky. There is heavy read of base tables, then sorting in temporary space (resad/write), then write to the WAL and to the index. This stuff would bring FreeBSD to its knees. Here on Linux I'm not even trying to be smart. With the FreeBSD attempt I had already taken the different read and write streams to different EBS disks each having a different ZFS pool. That helped a little bit. But not much. On this Linux thing here I didn't even bother and it's going decently fast. There is still some 20% iowait, which could probably be optimized by doing what I did for FreeBSD, separate devices, or, I might try to just make a single ZFS volume striped from 5 smaller 100 GB drives rather than a single 500 GB drive. This was just to provide some numbers that I have here. But I absolutely maintain that this has got to be a well known problem and that it's not about the finer subtleties of how to make a valid benchmark comparison. regards, -Gunther From owner-freebsd-performance@freebsd.org Sat Feb 6 00:23:02 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8D2C352E1D4 for ; Sat, 6 Feb 2021 00:23:02 +0000 (UTC) (envelope-from brendan.d.gregg@gmail.com) Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXY0P4Ckyz4lYy for ; Sat, 6 Feb 2021 00:23:01 +0000 (UTC) (envelope-from brendan.d.gregg@gmail.com) Received: by mail-io1-xd2f.google.com with SMTP id s24so9042575iob.6 for ; Fri, 05 Feb 2021 16:23:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=l/n+L0h8UzdRk/xGt5yM+NoJsAuE3ZNyqXHWaKdIjx4=; b=oL3psAXC9cGoH0ry80uOW5vZLNM0MuY5ztRxDMXudxFkn4cOds0OqLkbg8Ng1MSgj/ Pd+8fYVnpkTF8m8CZKJGd3P7k3tDbRw/dpCoyDetQyQZ1O2Pg9iSgVoeqRl49Vq617IS xb4suEHe7xjCi/EihqzvbkNbrzGg0XyTuAcZgfyHdxRTp+I3+H/n11zS5ZClWV4dOylD YWBfObA4Q+TpDic28Id2PyW2YGh0urN7n6p55Tud1lP0OeCHcWQaQn1mCwFdXdvgKznB k6s/Zj3FuEuKiS6mQ0z8swoT9e7etf1U45zGjhDAvdyLOGiEEjwDJcZ6h0kOvaUfEfBc r0vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=l/n+L0h8UzdRk/xGt5yM+NoJsAuE3ZNyqXHWaKdIjx4=; b=WgsgBU4iMAGwBryoLa9UyZCcE1RWxiWchQ0lbY3oaDSTjdw+hRFIe6/1sFOemo2QPL 58fiFsqCSiAJQnRPm9RxePD75GffbIkY6bnlZUcpB4RvI8OrnxrUVTzgIq7guzFtfiy7 iQNbZ+c24S38IU/kOAgtkn6Zd8jR4MKldYPpF1H3RHQ1SeAgNoy2DgQDlo8mULTRzxEO jb1me/bZplehpfLOiYAmS1J6Lrhf8lsnUh6PLYwk7qg6UFIWOTtiyLiPw6+OOYec9jE4 xEnsM/9FkaLw0e4hYwVI6Llt0fJwT/ilCpMlrzhiWbAVqrXJ1yY2+pie4RQoNRwfHoZt O2TA== X-Gm-Message-State: AOAM532a/FDlvQY+OG7pLD9gkMEN8RwPbOWGvbkA9OFmVo7g/9BcQ6dT O3piQlHxv8Ahot29oLM3x2TYYUI2EHdiOXmcY9cW3lgLDaVCpw== X-Google-Smtp-Source: ABdhPJxKv3eoW8tf27sQt2YDvNGbRD8OPGzveIYPG6BKNslhU7bV4KKTOomILfOF64CBS8vTto9QxDj1uZG0oLI2GUw= X-Received: by 2002:a05:6638:14d5:: with SMTP id l21mr7519278jak.54.1612570980463; Fri, 05 Feb 2021 16:23:00 -0800 (PST) MIME-Version: 1.0 References: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> In-Reply-To: <98fc52d4-caf1-8d48-5cb2-94643a490d4f@gusw.net> From: Brendan Gregg Date: Sat, 6 Feb 2021 11:22:34 +1100 Message-ID: Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: Gunther Schadow Cc: freebsd-performance@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DXY0P4Ckyz4lYy X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=oL3psAXC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of brendandgregg@gmail.com designates 2607:f8b0:4864:20::d2f as permitted sender) smtp.mailfrom=brendandgregg@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d2f:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d2f:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2f:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Feb 2021 00:23:02 -0000 G'Day Gunther, I can at least share some experiences from the other side, and how they can apply to FreeBSD. On Sat, Feb 6, 2021 at 1:53 AM Gunther Schadow wrote: > > Hi, I've been with FreeBSD since 386BSD 0.0new. Always tried to run > everything on it. I saw us lose the epic race against Linux over the > stupid BSDI lawsuit. But now I'm afraid I am witnessing the complete > fading of FreeBSD from relevance in the marketplace as the performance > of FreeBSD on AWS EC2 (and as I see in the chatter from other "cloud" > platforms) falls far behind that of Linux. Not by a few % points, but > by factors if not an order of magnitude! > > The motto "the power to serve" meant that FreeBSD was the most solid > and consistently performing system for heavy multi-tasking network > and disk operation. A single thread was allowed to do better on another > OS without us feeling shame, but overall you could rely on FreeBSD > being your best choice to overall server performance. > > The world has changed. We used to run servers on bare metal in a cage > in physical data center. I did that. A year or two of instability with > the FreeBSD drivers for new beefy hardware didn't scare me off. > > Now the cost and flexibility calculations today changed the market > away from bare metal to those "cloud" service providers, Amazon AWS > (>38% market share), Azure (19% market share), and many others. I > still remember searching for "hosting" providers who would > offer FreeBSD (or any BSD) as an option and it was hard to find. On > Amazon AWS we have the FreeBSD image ready to launch, that is good. > > But the problem is, it's disk (and network?) performance is bad (to > horrible) and it's really sad and embarrassing. Leaving FreeBSD beaten > far behind and for realistic operations, it's impossible to use, despite > being so much better organized than Linux. I have put significant > investment into a flexible scalable FreeBSD image only to find now that I > just cannot justify using FreeBSD when Linux out of the box is several > times faster. > > There have been few problem reports about this over many years, and > they all end the same way: either no response, or defensive response > ("your measures are invalid"), with the person reporting the problem > eventually walking away with no solution. Disinterest. I can link to > those instances. Examples: > > https://lists.freebsd.org/pipermail/freebsd-performance/2009-February/003677.html > https://forums.freebsd.org/threads/aws-disk-i-o-performance-xbd-vs-nvd.74751/ > https://forums.freebsd.org/threads/aws-ec2-ena-poor-network-performance-low-pps.77093/#post-492744 > https://forums.freebsd.org/threads/poor-php-and-python-performances.72427/ > https://forums.freebsd.org/threads/freebsd-was-once-the-power-to-server-but-in-an-aws-world-we-have-fallen-way-waaay-behind-and-there-seems-no-interest-to-fix-it.78738/page-2 > > My intention is not to rant, vent, proselytize to Linux (I hate Linux) > but to see what is wrong with FreeBSD? And how it can be fixed? Why does > it seem nobody is interested in getting the dismal AWS EC2 performance > resolved? This looks to me like a vicious cycle: FreeBSD on AWS is > bad so nobody will use it for any real work, and because nobody uses it > there is no interest in making it work well. In addition there is no interest > on the side of FreeBSD people to make it better. It's got to be the lack > of interest, not of anyone not having access to the AWS EC2 hardware. I think the better question is how many full time staff work on EC2 FreeBSD performance, and how to create more roles. Large companies have performance engineering teams to reduce cost, as do latency-sensitive companies of any size. I'd estimate there are well over 100 staff with the title "performance engineer" who work on Linux: most of whom work on Linux as part of another product. There is a mentality with these large companies, whether it makes sense or not, to run their own datacenters. So most of the performance engineers on Linux are looking at bare metal performance and not EC2. Fortunately for Linux, there are a few of us who do work on EC2. I work at Netflix on the streaming side, and myself and a colleague (Amer) are performance engineers who work on Linux EC2 a lot (as well as other things). Other teams work on Linux EC2 performance from time to time (e.g., BaseOS and Titus). If you were to add up all our collective time, it probably works out to 2 full time engineers working on Linux EC2 performance. Our focus is LTS releases, so we aren't finding issues every day (we likely would if we were looking at mainline), but we do find some and get them fixed. So who is the Netflix of EC2 FreeBSD? What large company, with a performance team (or is large enough to create one) runs (or may consider running) FreeBSD on EC2? I'd identify the company and management, and help them with a proposal and job description for a performance engineer. My Systems Performance book (2nd Ed) documents methodologies that are applicable to BSD (and includes various mentions of BSD), and could help a new engineer get started. There are some who do work on FreeBSD EC2 sometimes, e.g.: https://twitter.com/cperciva/status/1211125881264934917 Work like this is great, but performance work is endless and needs full-time attention. So I wouldn't say the problem is a lack of interest, but a lack of full-time roles. Who can create them? (Yes, Netflix does create full-time roles for FreeBSD bare metal performance on the OCA team. But their focus is bare metal.) As for the low-level performance issues: EC2 has been switching to the Nitro hypervisor, and I imagine there may be driver work to make sure it's using it correctly. I summarized the switch as: http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html Brendan > > What can be done? I am trying to run a company, so I cannot justify playing > with this for much longer shooting in the dark. If I wasn't the boss myself, > my boss would have long told me to quit this nonsense and use Linux. > If I saw interest, I could justify holding out just a little longer. But > I don't see any encouraging feedback. Is there anyone at all in the FreeBSD > dev or FreeBSD.org as an organization interested in actually being competitive > in the AWS EC2 space (and other virtualization "clouds")? If so, how many? > How can this be fixed? How can I help? I cannot justify spending too much > more of my own time on it, but I could help making resources available > or paying for someone who has both a sense of great urgency to redeem > FreeBSD and the know-how to make it happen. > > regards, > -Gunther > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Fri Feb 5 23:35:12 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EAD1852D480 for ; Fri, 5 Feb 2021 23:35:12 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXWxD67fDz4j0Y for ; Fri, 5 Feb 2021 23:35:12 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-qk1-x735.google.com with SMTP id a12so8682825qkh.10 for ; Fri, 05 Feb 2021 15:35:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; h=content-transfer-encoding:from:mime-version:subject:date:message-id :references:cc:in-reply-to:to; bh=FCJCJ63ebkHUTBbGZaRbHG4x94P0G9eokMrRUU2wY/U=; b=KU0u/sL8p79JZ19SNM4NqDVOQEylOh6YtfBPNnmtXu/cbqYNvsRecszJ0iz/sNIYbs dx+RzS2NqB72G5ScklcRC7zQ/W72cFfL59hv3NazmjGCXujI0RhC882l4akQbHFkOn6s PanLAdcHSW7evIHXsi1lYUkJx/2dbujeVyHv8QqcpdTiWkZx64m3cxRL/yVA7pQT2pvB 2Evn38G+Cb9soCoi1D7oaCWMu/sfFk+3/wUA81cINZFJBOaboOZ1Ji9yUb7luT2vxZSJ icPXk6QrB34MST/a3iOp5JTp6np/VSEZlBUgDhuLtlnoC2uVSkIR/9ryX6NBAyyIplFk 4gWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:content-transfer-encoding:from:mime-version :subject:date:message-id:references:cc:in-reply-to:to; bh=FCJCJ63ebkHUTBbGZaRbHG4x94P0G9eokMrRUU2wY/U=; b=KcEnARJxhozZb2DWhasDoH5ET6Lb6mVsudoFah3Aq2+C6pV48XT2y3KsdnI4A6iGYh hPy0BtSoZeIbD7ebTCuGps6ZCWH+ZAEx5QFE8SU5O4QXH381Y49X5EDgTrZFspVlP+EN K3plZSFBaRTHt6ILNZD2OAZ3YvrlMxwpfuOZDLvjiwMHUlNNimUZJKV3lTxwE9mIxrHL j0H/rw1rm+osBFA/AkDRmzVZPzD/aqDPfHuinqTV2Jz3k7TKw+TDw8VN4Xf6IyANTXpm JMMVULwUMeIZVTr8PtBTqYw92BNSODh16Yw5y90kfMAZ3EOpqrgaMX7XaSNV3O+7kbSd m4vw== X-Gm-Message-State: AOAM531n8KHz3Yg6qzbNh71JHzgSMFWyRuS8lI9Hclq4rLWF23tNgcxH 51wm/Kumlf4XNn0lnRo72A7BRg== X-Google-Smtp-Source: ABdhPJyXLkvPr1+M4O5y3tsr0Pd8qc7gSK/ig4asBOud9BabiLlUWoc4A9B4LH+pHJiV0tno90tTlg== X-Received: by 2002:a37:b204:: with SMTP id b4mr6822914qkf.72.1612568111884; Fri, 05 Feb 2021 15:35:11 -0800 (PST) Received: from [192.168.1.50] (pool-96-232-87-29.nycmny.fios.verizon.net. [96.232.87.29]) by smtp.gmail.com with ESMTPSA id k132sm10554636qke.77.2021.02.05.15.35.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 05 Feb 2021 15:35:11 -0800 (PST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Mark Saad Mime-Version: 1.0 (1.0) Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems Date: Fri, 5 Feb 2021 18:35:10 -0500 Message-Id: <5D3DADA0-33E0-4EA1-A97E-C65C3FC859C3@longcount.org> References: Cc: "Jin Guojun[VFF]" , freebsd-performance@freebsd.org In-Reply-To: To: Sean Chittenden X-Mailer: iPhone Mail (18C66) X-Rspamd-Queue-Id: 4DXWxD67fDz4j0Y X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Mailman-Approved-At: Sat, 06 Feb 2021 10:05:53 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2021 23:35:13 -0000 > On Feb 5, 2021, at 6:14 PM, Sean Chittenden wrote: >=20 > =EF=BB=BFTo be clear, this is a known issue that needs attention: it is no= t a > benchmarking setup problem. Network throughput has a similar problem and > needs similar attention. Cloud is not a fringe server workload. -sc >=20 >=20 >> On Fri, Feb 5, 2021 at 3:00 PM Jin Guojun[VFF] wrote:= >>=20 >>> On 2021-02-05 12:45, Gunther Schadow wrote: >>> Gordon Bergling wrote: >>>> Can you verify your feelings by numbers? >>>=20 >>> Yes, like I said >>>=20 >>>>> Not by a few % points, but by factors if not an order of magnitude! >>>=20 >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253261 >>>=20 >>> Do this: >>>=20 >>> dd if=3D/dev/zero of=3D/dev/nvd2 bs=3D100M status=3Dprogress >>>=20 >>> and you see that it's writing with the "whopping" speed of 70 MB/s. >>>=20 >>> That used to be good, but it is no longer good. Compare Amazon Linux >>> doing >>> the same thing at 300 MB/s. >>>=20 >>> Now, when you put a file system over it, zfs or ufs, then instantly the >>> performance gets better: >>>=20 >>> newfs /dev/nvd2 >>> mount /dev/nvd2 /mnt >>> dd if=3D/dev/zero of=3D/mnt/test bs=3D100M status=3Dprogress >>>=20 >>> now that works at about 250 MB/s. Decent. So, problem solved? >>=20 >> It is not clear if this compares Apple to Apple. >>=20 >> What disk drives and CPUs are on FreeBSD, and what are disk drive(s) and >> CPU(s) on AWS? >>=20 >> Knowing the drive brand and models will tell approximately the disk >> throughput. Agree, 70MB/s is slow for modern disks, but your information >> does not provide clue why this could be slow. >>=20 >> Can this setup get 250MB/s on FreeBSD 11.4? or 300MB/s with Ubuntu 16.04 >> on the same hardware? >>=20 >> -Jin >>=20 >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to " >> freebsd-performance-unsubscribe@freebsd.org" >>=20 So silly question ; how does FreeBSD work on Google gcp, Microsoft Azure , D= igital Ocean? Do they all suffer the same issue or is this just an Amazon issue ? Also just a bit of advice; Contrary to popular belief Amazon does not actu= ally sell magic beans .=20 --- Mark Saad | nonesuch@longcount.org > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.= org" From owner-freebsd-performance@freebsd.org Sat Feb 6 12:07:29 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7CB30547584 for ; Sat, 6 Feb 2021 12:07:29 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [5.196.167.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXrdF2ChTz4X6y; Sat, 6 Feb 2021 12:07:28 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [IPv6:2001:41d0:a:71bf::1]) by mail.freebsd.systems (Postfix) with ESMTP id 48D097573; Sat, 6 Feb 2021 13:07:27 +0100 (CET) X-Virus-Scanned: amavisd-new at freebsd.systems Received: from mail.freebsd.systems ([5.196.167.1]) by mail.freebsd.systems (scan.freebsd.systems [5.196.167.1]) (amavisd-new, port 10026) with ESMTP id pJNGWGEwUulH; Sat, 6 Feb 2021 13:07:27 +0100 (CET) Received: from [192.168.168.3] (89-70-50-99.dynamic.chello.pl [89.70.50.99]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.freebsd.systems (Postfix) with ESMTPSA id CE0C977B3; Sat, 6 Feb 2021 13:07:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wasikowski.net; s=default; t=1612613247; bh=GXDecP5YtRRZGuZEju2Euh/AxglH//jCsbJhgixVzNs=; h=To:Cc:References:From:Date:In-Reply-To; b=kvL0q2lb2I17AMmSCTWzykIU8Gerv3e014sWshhn21nCHottd91rvqQ0bocfpw8l/ l0Yl8Rux55ku3r+CJm79oGVzI12gdLbaadCUeKrVnx7VZd49D5h8d070Ynll73r01r g4cN8Ib8nORtf7SqlXjZlFywjhaSOIZhgqxZf+YoEZdeaVU1RrOa1jCDbfN8d3SQc+ dmGXfrAhMcejI+3cXf7XI2NFPrFRpTbQSaq/PJmybortt1m1e1hxt8U5zbA20mRgDw d1yn4msB7L5iDF5lEg5h/Q4Yul8FBHOz30Vg41TuFTjYlx6qMlInxvMq4DE0KZk1N/ mz9/85Lgbuo1A== Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: Mark Saad , Sean Chittenden Cc: freebsd-performance@freebsd.org References: <5D3DADA0-33E0-4EA1-A97E-C65C3FC859C3@longcount.org> From: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= Message-ID: <7f0edef6-5dfd-f7fe-9bee-0cfa2f96ee27@wasikowski.net> Date: Sat, 6 Feb 2021 12:07:25 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: <5D3DADA0-33E0-4EA1-A97E-C65C3FC859C3@longcount.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: pl Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4DXrdF2ChTz4X6y X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Feb 2021 12:07:29 -0000 W dniu 2021-02-06 o 00:35, Mark Saad pisze: > Also just a bit of advice; Contrary to popular belief Amazon does not actually sell magic beans . AWS has 32% market share. I don't know if they beans are magic or not, but this is the biggest cloud today. If FreeBSD want to be on this train, it has to perform at least as good as competitors. It's simply matter of survival for this project. In 2007 about 90% of my machines were running FreeBSD (30-40 servers). Now I run FreeBSD on 9 boxes, and 130 is running Linux. And I'm not the only one out there who was forced to do this transition. -- Best regards, Łukasz Wąsikowski From owner-freebsd-performance@freebsd.org Sat Feb 6 16:45:24 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E566852E819 for ; Sat, 6 Feb 2021 16:45:24 +0000 (UTC) (envelope-from vladilen.kozin@gmail.com) Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXynw273Kz4pb7 for ; Sat, 6 Feb 2021 16:45:24 +0000 (UTC) (envelope-from vladilen.kozin@gmail.com) Received: by mail-io1-xd2e.google.com with SMTP id q7so10629397iob.0 for ; Sat, 06 Feb 2021 08:45:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=Bim6trVaPAJRsksyZ+UYHNOGfj8jorPTrUBhCJPyxAA=; b=QlsJ8l5gICbeIswXVfoShR4NbjDWXKkzMtCPbkAZ8yk1IVJXSBwr08PK5Mcd30BCL0 3xzC5FwYZYMlhShbN5jhx09upLT7q8gnXM4W8vb+A81tGgfK2G7khr6hqtwed9QpB/z0 X9FwnXo0Yx1cSc/sobeEmurUiu0lKgnq39YGpaBv+HgQLU+bPfW5MX4vUENer2GItCUB AOMq78kFzkJl55JhAA4g3OTwl90z43tJChqInkvEz76IMaTPNI2bDdrq/IvdBr40Hj/D SOmBcLlhYIFHOLYjI6T3STbP2+S5nfSN81ZNlUXickG2CQ349IWRJynJBdoQZyVhczPj 4m7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Bim6trVaPAJRsksyZ+UYHNOGfj8jorPTrUBhCJPyxAA=; b=brQ0pbV3JSu1iY7pXH5yZZRNx2RGLZjvIgGteNALUf1WtOP8zycjwIIlr2JVB9WwuC xJHgmK2TlJ4oeER4nxrrpIbN1VoeH+zyB/qkka1klfZLpDGqJJPlZHNXd+CJZTOV9qSn DoXoEhtkXmPbAH6B8gl7USCJjlvInLpw5m46ccvoC1fDrJPhdy39yuOZAJiqSnuxpA8V IOzh3Lvw43JZ2IZPV+gYeIEyrqZl+Fmn7fSaLAxWbmkZdG+nBElhPH5swiqa2E8bIYtq /z9jXGMF8o6uBDacF9wXFHJ2WCMVUMM9G4fxsgip0tes5lPMT2fAZBcLargz57Ke1U2V NR1g== X-Gm-Message-State: AOAM532dRn+4sC9CdPNNAKZJtaykog9H7bhnaoCLP0uG6HYal6PbfyLq KfnxCKVgEycHTS3+xz6oNNOtXO6jbklEjBFe3HTH15eAXGs= X-Google-Smtp-Source: ABdhPJyhVyydPEG2EOCjHlAg/l+4gL7vxg1bhnnhT78RH3Sz4E4BllQD/TGSu2tUftkYacC7zh8MlhUXL8SVKf7ouko= X-Received: by 2002:a02:856d:: with SMTP id g100mr10317835jai.10.1612629923138; Sat, 06 Feb 2021 08:45:23 -0800 (PST) MIME-Version: 1.0 From: Vladilen Kozin Date: Sat, 6 Feb 2021 16:45:12 +0000 Message-ID: Subject: Tuning and monitoring write intensive server To: freebsd-performance@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DXynw273Kz4pb7 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=QlsJ8l5g; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of vladilenkozin@gmail.com designates 2607:f8b0:4864:20::d2e as permitted sender) smtp.mailfrom=vladilenkozin@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d2e:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[0.0.0.0:email,0.0.0.2:email]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d2e:from:127.0.2.255]; DBL_PROHIBIT(0.00)[0.0.0.0:email,0.0.0.2:email]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2e:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Feb 2021 16:45:25 -0000 Hello list. I have a rather niche and specific task I'm trying to tune for my recently acquired homelab server. Easily parallelized, extremely disk IO (write) intensive. If having read this you think it belongs in freebsd-questions, kindly tell me so. I've not used FreeBSD for 20 years, so I'm not up to speed with the lay of the land, any guidance be welcome. Machine is SuperMicro CSE-216, 24 SAS 10K spinners, backplane daughter card connected to LSI SAS9211-8i HBA with two wires, two XEON CPUs 12 cores each for a total of 24, 128GB RAM. Surprisingly both Ubuntu Server and Fedora server choked at install trying to write to those disks. Seems like they have mostly standard block size but maybe they were somehow write protected (server is obviously not new off of Ebay). Anyhow, FreeBSD ate it like a champ so here I am. Freshly installed 12.2-release, all disks UFS, no RAID so each disk is its own thing. Like I mentioned very specific task, be running as many processes as there are cores (so 24 each consuming 2 threads - does FBSD allow hyperthreading?) each writing to its dedicated disk - no other process will ever touch another process's disk. This is what I'm trying to optimize for. So, given that info what are the things you think I should optimize (if anything), what tools do I have available for that and should try on FreeBSD. E.g. maybe I should be particular how I partition or format those disks, or the way I mount them, or the way I run my processes etc. ATM I've followed the Handbook - nothing at all fancy there. FWIW here're the disks in question: sudo diskinfo -v da1 da1 512 # sectorsize 900185481216 # mediasize in bytes (838G) 1758174768 # mediasize in sectors 0 # stripesize 0 # stripeoffset 109441 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. IBM-207x ST900MM0006 # Disk descr. id1,enc@n500304800169193d/type@0/slot@2/elmdesc@Slot_02 # Physical path No # TRIM/UNMAP support 10500 # Rotation rate in RPM Not_Zoned # Zone Mode Related, what metrics I should and can collect and track and what tools does FreeBSD give me to do so. Great if those are end user ready tools with UI etc, but I am good enough programmer to script things to get what I want - no issue. Task non-specific question is how to best manage and monitor that SuperMicro and again what tools FreeBSD gives me. But that probably belongs in freebsd-questions. SM IPMI gave me a head start but it is clunky to say the least. ATM I'm going over SSH but I probably do want to monitor server health somehow. Thank you -- Best regards Vlad Kozin From owner-freebsd-performance@freebsd.org Sat Feb 6 14:55:41 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B03D652BA34 for ; Sat, 6 Feb 2021 14:55:41 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DXwMJ69VSz4hlG for ; Sat, 6 Feb 2021 14:55:40 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-qt1-x836.google.com with SMTP id c1so7303418qtc.1 for ; Sat, 06 Feb 2021 06:55:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; h=content-transfer-encoding:from:mime-version:date:subject:message-id :cc:to; bh=Tp1ObV/0OfrCUISgeOsM4miWhq51U0lmy8ctbYBxApM=; b=gUQWDQvm1CLQsLjnfeMRP7xusMC9oItOEY1BQmr8LfM0qofum8aZQb/exoFS1pe28f N1pJSa1x38ZwY8qWTshvlAogfxaZp3ffkYNOTxRT9yqwOCkUaotBmLEor7sTsk4NQVgj 5qSwFbCia2ues4OvNXCnReqVAoi4bgi6NBCN3rL5hSmHK3/UFoZS9gD0HVLobD2YF16a IlXcZ+uSAhAgXs6OOcFs0pnlai6t4A9moAj1w0UJTJkOpvZVKJnjDTcPbJZ4TVgL73zh pnZIj8LFLmij3zmHQZZzsvi4DJQMYvr0VtG44PZsAtAbYrCV1xR84ZJvXC68hzxtTx2G VoBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:content-transfer-encoding:from:mime-version:date :subject:message-id:cc:to; bh=Tp1ObV/0OfrCUISgeOsM4miWhq51U0lmy8ctbYBxApM=; b=NhguY7V/sNfZm1abXwDF7P68jep+/3TwJ1ySXmzJ4+Bv9DomsE/bXMMzIkaQNnN3nV eiau/Hok0qBAgGMmH/bzdE1iSEh/0Z56XtB1DIglBXoPtQOkSwy0aTMzXVu9xhCFcNex a8Ur/pYBKH5jQPK8L9cTCayxqDjzgFOixXtUKP7/zeGf70mnClS0CDOzpASt8vVdJo+B GX9ng4o8Qnxk/2R0vozu+uI0oAJaIdL2h9UKTQfX1yWrBtsSmxgh/vKxIvilj9c2WupJ wiFNBFqdudDcWUTgiACB5Rh0a4OOaU8ig9ORt4ZsO99sbraBJgpJbkjpJGW5nZX/vA/u jI9A== X-Gm-Message-State: AOAM533w4uoHNDP64ai541nvKLDFopZPSKmUiScbxUKAwtLYZf6xx9ly rB+LB7NwWT+Xyz78pt2G2/KXzA== X-Google-Smtp-Source: ABdhPJyVFlq3J7HXQ2kSZZyEvaz7FhjkRtvqEKl2pzc0oWBfWeCAH9Bhp7vNOGjpiusP569Lq7ayEg== X-Received: by 2002:ac8:7b89:: with SMTP id p9mr9072207qtu.276.1612623339986; Sat, 06 Feb 2021 06:55:39 -0800 (PST) Received: from [192.168.1.50] (pool-96-232-87-29.nycmny.fios.verizon.net. [96.232.87.29]) by smtp.gmail.com with ESMTPSA id l26sm4979015qtp.49.2021.02.06.06.55.38 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 06 Feb 2021 06:55:39 -0800 (PST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Mark Saad Mime-Version: 1.0 (1.0) Date: Sat, 6 Feb 2021 09:55:38 -0500 Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems Message-Id: <5BC0DEF9-3D58-4FFC-9E20-311B0520A25A@longcount.org> Cc: Sean Chittenden , freebsd-performance@freebsd.org To: =?utf-8?Q?=C5=81ukasz_W=C4=85sikowski?= X-Mailer: iPhone Mail (18C66) X-Rspamd-Queue-Id: 4DXwMJ69VSz4hlG X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount.org header.s=google header.b=gUQWDQvm; dmarc=none; spf=pass (mx1.freebsd.org: domain of nonesuch@longcount.org designates 2607:f8b0:4864:20::836 as permitted sender) smtp.mailfrom=nonesuch@longcount.org X-Spamd-Result: default: False [-2.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[longcount.org:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::836:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.232.87.29:received]; FAKE_REPLY(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[longcount.org:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[longcount.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::836:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::836:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-performance] X-Mailman-Approved-At: Sat, 06 Feb 2021 18:09:41 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Feb 2021 14:55:41 -0000 On Feb 6, 2021, at 7:07 AM, =C5=81ukasz W=C4=85sikowski wrote: >=20 All So what I was getting at, is do we have good data on what the issue is ? C= an we make a new wiki page on the FreeBSD wiki to track what works what and = doesn=E2=80=99t . Does one exist ? To be clear we should check if the issue something that aws is doing with th= eir xen platform , kvm/qemu one or common to all ? Also does that same issue= appear on google and Microsoft=E2=80=99s platforms? This will at least get= some bounds on the problem and what if any fixes may exist .=20 There are some commercial FreeBSD products running on aws . Maybe the vendor= s know some stuff that can help ?=20 Thoughts ?=20 --- Mark Saad | nonesuch@longcount.org