From owner-freebsd-performance@freebsd.org Wed Feb 10 16:48:04 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 674CA54BE5A for ; Wed, 10 Feb 2021 16:48:04 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DbQg81zGhz4jvx for ; Wed, 10 Feb 2021 16:48:04 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: seanc) by smtp.freebsd.org (Postfix) with ESMTPSA id 0F1C2DC54 for ; Wed, 10 Feb 2021 16:48:03 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: by mail-pj1-f43.google.com with SMTP id d2so1467069pjs.4 for ; Wed, 10 Feb 2021 08:48:03 -0800 (PST) X-Gm-Message-State: AOAM531liOMLIBnB3/gRCcZChzP79osxISrVIPEYJnE18q3/4rWThAh5 zgiF5oMYTGSm8C0Y/BXM2B2/x67F8Ao0YsjBFWqwPg== X-Google-Smtp-Source: ABdhPJyd1UN2fl32sG2u5Wql7XKP1q2ZhndhMpoPS8PnUX2G+q6K2Fwu2utolaSpoNRH15mHGGkIz3PIaWgQ9gLd0x4= X-Received: by 2002:a17:90b:3886:: with SMTP id mu6mr3832052pjb.153.1612975682070; Wed, 10 Feb 2021 08:48:02 -0800 (PST) MIME-Version: 1.0 References: <5BC0DEF9-3D58-4FFC-9E20-311B0520A25A@longcount.org> <96d43dbe-ed50-8ba4-f676-673ee99725bd@gusw.net> In-Reply-To: <96d43dbe-ed50-8ba4-f676-673ee99725bd@gusw.net> From: Sean Chittenden Date: Wed, 10 Feb 2021 08:47:45 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2021 16:48:04 -0000 Expect to see something about this on this year's Community Survey and the Core Team will do something with this information. Cloud has been an increasingly important workload for FreeBSD users. Or at least, our community is moving a fair number of workloads to virtualized metal, and running on the cloud is in need of attention as or more than some of our other subsystems. Many of the people who have the skills to jump in and diagnose and fix this type of problem have been sucked up with other professional commitments and aren't available. Which is to say, we need to increase the depth of our bench for performance-related work. But if someone's reading this thinking, "well I'll wait until Core...." don't wait! Jump in. If people are looking for a single starting point, I'd suggest trying to conjure up an iflib-backed ethernet driver and reducing the administrative friction of spinning up one of our images in the Cloud. "Secure by default" or conservative image defaults aren't doing anyone any favors in the cloud era. Or begin porting the io_uring kernel interface and begin working down the stack into the driver layer. $0.02. -sc On Wed, Feb 10, 2021 at 8:06 AM Gunther Schadow wrote: > I think we got enough feed back now from other professionals to suggest > that it > would do the FreeBSD project good to acknowledge the issue and create som= e > sort > of work in progress / project statement, perhaps a wiki, where people can > flock > to to look for workarounds, current statue, just to not feel so lost and > lonely > and wondering if anybody cares at all. Such a meeting-point could at leas= t > be > something of a self-help group, for emotional support and positive > thinking =3D8). > > I have a slight sign of hope also: in my latest Amazon Linux db server > deployment > the performance is only half of what I got in my previous db server > deployment. > Go figure. But it's a fresh launch and I can't figure out what else I had > optimized > with the previous Amazon Linux box. So, while not improving anything, it > at least > closes the inequality between Linux and FreeBSD in the socialistic way ;) > > Now I am trying the FreeBSD install again with the same disk setup. > Because one > thing I clearly know is that it makes a huge difference of having one > large EBS > device and a single partition / file system vs. the same large EBS device > with > many partitions and file systems, separating tables, indexes, temporary > sort space, > etc. so that there is less random access contention on a single file > system. > > Why that is important, I wonder, actually. And it's not the underlying EB= S > that > matters so much. Like in bare metal world, the approach would be separate > disks > vs. striping (RAID-0) to get more spindles involved instead of having dis= k > seek. > But none of that should mater that much any more with SSD drives (which > EBS gp2 and > gp3 are!) > > Fortunately now Amazon has gp3 volumes also which support 3000 io > transactions > per second (IOps) and 250 or up to 1000 MB/s throughput despite being as > small as > 4 GB. So what I'm doing now is instead of a partitioned 1 TB gp2 volume > with many > partitions, I make myself over 20 separate smaller gp3 volumes, the > advantage of > this is that I can individually resize those without colliding with the > neighboring > partition. > > Tomorrow I should have better comparison numbers for my database on both > Linux and > FreeBSD configured the exact same way, and it might be closer now. But > sadly only > in the socialist way. > > regards, > -Gunther > > On 2/6/2021 9:55 AM, Mark Saad wrote: > > > On Feb 6, 2021, at 7:07 AM, =C5=81ukasz W=C4=85sikowski > wrote: > > All > > So what I was getting at, is do we have good data on what the issue > is ? Can we make a new wiki page on the FreeBSD wiki to track what works > what and doesn=E2=80=99t . Does one exist ? > > > > To be clear we should check if the issue something that aws is doing > with their xen platform , kvm/qemu one or common to all ? Also does that > same issue appear on google and Microsoft=E2=80=99s platforms? This will= at least > get some bounds on the problem and what if any fixes may exist . > > There are some commercial FreeBSD products running on aws . Maybe the > vendors know some stuff that can help ? > > > > > > Thoughts ? > > > > --- > > Mark Saad | nonesuch@longcount.org > > > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" >