From owner-freebsd-performance@freebsd.org Sun Feb 7 15:55:20 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C585552FEC2 for ; Sun, 7 Feb 2021 15:55:20 +0000 (UTC) (envelope-from "") Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DYYdh0f59z4lxx for ; Sun, 7 Feb 2021 15:55:19 +0000 (UTC) (envelope-from "") Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id 4959A160065 for ; Sun, 7 Feb 2021 16:55:17 +0100 (CET) Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4DYYdc25jFz9rxR; Sun, 7 Feb 2021 16:55:16 +0100 (CET) From: Walter von Entferndt To: freebsd-performance@freebsd.org Subject: Re: Tuning and monitoring write intensive server Date: Sun, 07 Feb 2021 16:50:49 +0100 Message-ID: <2002412.uJW0cDvVUg@t450s.local.lan> X-Face: #$[hC+4[4W*mS3hB&izisyT_#E]^Aq+7Isv`2Tu5q*1~jR@&['74B>Ibyrk]GTJ!j$ NjX=#L2#k2X7OnaaRM_Pd5`>`8OJ3; +I2 References: MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Rspamd-Queue-Id: 4DYYdh0f59z4lxx X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=posteo.net (policy=none); spf=none (mx1.freebsd.org: domain of mout01.posteo.de has no SPF policy when checking 185.67.36.141) smtp.helo=mout01.posteo.de X-Spamd-Result: default: False [-2.70 / 15.00]; RCVD_TLS_ALL(0.00)[]; CTE_CASE(0.50)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[185.67.36.141:from]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8495, ipnet:185.67.36.0/23, country:DE]; FREEMAIL_CC(0.00)[gmail.com]; MAILMAN_DEST(0.00)[freebsd-performance]; DMARC_POLICY_SOFTFAIL(0.10)[posteo.net : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Feb 2021 15:55:20 -0000 At Sonntag, 7. Februar 2021, 13:00:04 CET Vladilen Kozin wrote: > [1 dedicated disk/ufs per thread, no redundency,...] RTFM tuning(7), zpool(8), zfs(8), gjournal(8), gstripe(8), gsched(8). - Obviously striping the disks will be beneficial, but it seems you don't want that (not enough disks?) & know what you're doing. I suppose your special task can tolerate data loss intentionally (no redundency). - Having the intent log on a dedicated, fast medium (SSD or NVD) would gain performance. Either ZFS can do that, or you can use gjournal(8). - Inserting an I/O scheduler might improve performance, too (gsched(8)). Yes, UFS is likely faster than ZFS on such a setup, but ZFS offers many advantages in terms of administration, fault tolerance & reliability. You can fetch my scripts to insert the scheduler (rc(8) script) & fs_summarize.awk to estimate the parameters for newfs(8) from the forums in the thread "Useful scripts". I.e. run the AWK script on some samples of your working data, then adjust the appropiate knobs to newfs(8). Note that ZFS automagically adjusts to the I/O chunk size. To monitor the I/O, use systat(1). Additionally, you can find a plethora of ports(7) for this, use psearch(1) or portfind(1) (install 1st). -- =|o) "Stell' Dir vor es geht und keiner kriegt's hin." (Wolfgang Neuss) From owner-freebsd-performance@freebsd.org Sun Feb 7 18:12:03 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 073AF53350A for ; Sun, 7 Feb 2021 18:12:03 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DYcgP4kqNz4tbv for ; Sun, 7 Feb 2021 18:12:01 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:66:5032:f0d4:21ac] ([IPv6:2607:f3e0:0:4:66:5032:f0d4:21ac]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 117IC0iw048478 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sun, 7 Feb 2021 13:12:00 -0500 (EST) (envelope-from mike@sentex.net) Subject: Re: Tuning and monitoring write intensive server To: Walter von Entferndt , freebsd-performance@freebsd.org References: <2002412.uJW0cDvVUg@t450s.local.lan> From: mike tancsa Message-ID: Date: Sun, 7 Feb 2021 13:12:00 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: <2002412.uJW0cDvVUg@t450s.local.lan> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4DYcgP4kqNz4tbv X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; SPAMHAUS_ZRD(0.00)[2607:f3e0:0:3::19:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f3e0:0:3::19:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Feb 2021 18:12:03 -0000 On 2/7/2021 10:50 AM, Walter von Entferndt wrote: > > - Inserting an I/O scheduler might improve performance, too (gsched(8)). > Yes, UFS is likely faster than ZFS on such a setup, but ZFS offers many > advantages in terms of administration, fault tolerance & reliability. Especially if the data is compressible, zfs can be a big win.     --Mike From owner-freebsd-performance@freebsd.org Mon Feb 8 10:21:51 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6166C52CAEF for ; Mon, 8 Feb 2021 10:21:51 +0000 (UTC) (envelope-from vladilen.kozin@gmail.com) Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DZ2BQ31Dkz3FyW for ; Mon, 8 Feb 2021 10:21:50 +0000 (UTC) (envelope-from vladilen.kozin@gmail.com) Received: by mail-io1-xd29.google.com with SMTP id n14so14400486iog.3 for ; Mon, 08 Feb 2021 02:21:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=a8Y2w0e0BcoYkmUY7dQhA+NpN8gI86NBBIrJ0mX9Phc=; b=LD22ZO1XhRpy65JHevx68izhdO6g1pm2eBZWDPArn96GJ7OfegA7xfMOKCt4sB0ywP tY856NbXcq8PnWchv/HKs2sABX0AofyUlW/BL7BUnOSR9S92oerKC+XB5MRWXIXTPYLd qaM7NOBbSex3JWhUwDPbuc7Ki9v8afMIWBvXZRtHta48Z4ToPLbX6PKEKrMVl7NFyH3a B9rxRLP/fbdm+l3tcHnApVXUcG7wpUCM/5/xOjvQW60Nr4hvFmAOgLU3NyvUi9md4QcY DfzCEQAXM9fuDuXNNbMmCZQkllZIuc7qQur3zLz9KzQoRSoL70ZD30x6Vctq7nyBrmot raFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=a8Y2w0e0BcoYkmUY7dQhA+NpN8gI86NBBIrJ0mX9Phc=; b=VIRjFI2b+EA+nyHjQfkBx19dJaDrSCJBMGV4N2s5zs6Tm3dEZKE6BzlJFu/eeGehEa oe890ZMvb7BGf+YWWweILpMvXqdg2IGxnA4kMrqZeEuf+MSV6ClZ3UxVgDIL0ZvcNFw2 CtXN/WMjQcMJ4yBWLp2aecgXZxpTvSojXgM49GpU1+IOJaiRwmKAlkc1M9c5yklCRkeo 4MIoAip3yJv6VeZFRZpkDS2qb9z3EDZnoWRGMtolyJk4RY8Lfk48MO9QXPRMzg6pZ7IZ A4PtIyNSGvMHEXS6ghQMxABnVaCiaWTIsCQTkDj0DTJhMKyVRIO9XyzjpQwMh/Yf2ccv KwLw== X-Gm-Message-State: AOAM533ZgXqsQe5CCK0SvkzXMcCaPgl6YlMtdRyAcg/oGWT0qfsaM7IJ j8ndIgOgUelfvVjotR6TB+FPZ6VeQ1iYL5giD70= X-Google-Smtp-Source: ABdhPJwWlr1Ie8BbFJdpN7rDO89roXrK1Z9tma2Iad+aOg4peRDMQm2nZObU/7hn9OFA2dPX4dJR7Wy2yyOIQRdtj7g= X-Received: by 2002:a02:856d:: with SMTP id g100mr17080535jai.10.1612779709515; Mon, 08 Feb 2021 02:21:49 -0800 (PST) MIME-Version: 1.0 References: <2002412.uJW0cDvVUg@t450s.local.lan> In-Reply-To: <2002412.uJW0cDvVUg@t450s.local.lan> From: Vladilen Kozin Date: Mon, 8 Feb 2021 10:21:38 +0000 Message-ID: Subject: Re: Tuning and monitoring write intensive server To: Walter von Entferndt Cc: freebsd-performance@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DZ2BQ31Dkz3FyW X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=LD22ZO1X; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of vladilenkozin@gmail.com designates 2607:f8b0:4864:20::d29 as permitted sender) smtp.mailfrom=vladilenkozin@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.979]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d29:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d29:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d29:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Feb 2021 10:21:51 -0000 These are excellent pointers and get me on my way! Thank you very much Walter. Invaluable. On Sun, 7 Feb 2021 at 15:55, Walter von Entferndt wrote: > > At Sonntag, 7. Februar 2021, 13:00:04 CET Vladilen Kozin > wrote: > > [1 dedicated disk/ufs per thread, no redundency,...] > > RTFM tuning(7), zpool(8), zfs(8), gjournal(8), gstripe(8), gsched(8). > > - Obviously striping the disks will be beneficial, but it seems you > don't want that (not enough disks?) & know what you're doing. I suppose > your special task can tolerate data loss intentionally (no redundency). > > - Having the intent log on a dedicated, fast medium (SSD or NVD) would > gain performance. Either ZFS can do that, or you can use gjournal(8). > > - Inserting an I/O scheduler might improve performance, too (gsched(8)). > Yes, UFS is likely faster than ZFS on such a setup, but ZFS offers many > advantages in terms of administration, fault tolerance & reliability. > > You can fetch my scripts to insert the scheduler (rc(8) script) & > fs_summarize.awk to estimate the parameters for newfs(8) from the forums > in the thread "Useful scripts". I.e. run the AWK script on some samples > of your working data, then adjust the appropiate knobs to newfs(8). > Note that ZFS automagically adjusts to the I/O chunk size. > > To monitor the I/O, use systat(1). Additionally, you can find a > plethora of ports(7) for this, use psearch(1) or portfind(1) (install > 1st). > -- > =|o) "Stell' Dir vor es geht und keiner kriegt's hin." (Wolfgang Neuss) > > -- Best regards Vlad Kozin From owner-freebsd-performance@freebsd.org Mon Feb 8 10:31:57 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1C81552CDEB for ; Mon, 8 Feb 2021 10:31:57 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DZ2Q41jBCz3GQY for ; Mon, 8 Feb 2021 10:31:56 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-ed1-f50.google.com with SMTP id t5so17390798eds.12 for ; Mon, 08 Feb 2021 02:31:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=AqY3ObotmVbV6CNBXlExtXbNuF7KDrD6qRApUF4uGE8=; b=R3vxHM94DCrDnoScB8MpPnpF1r3Xh9qwgTwtQTqxfWW8t6TQjphwwJwqmJDGA9lcWf nc4pDWi6CfO/3VDs1mlk2+n4eu2vZbX8Eyddj1jna03pvaoQPdopwjRX9A8WgvOg+SDy 0AEKJ7zLmgduKalDFW2zMr9DniPU3iSUzbervjmpE9ykKHAf5E8T4sLh5jce/kUzCW8t un07kxesVyJHPdGbjT+4CtgN6LGnyboxN28v7STKjiUhwxgWEpI3VZ9E7JkNFx+SuyjX cNNaFpVCqICr4KnQad4az/lgfQJqbrs7W48ab8HE32edfIHSUXTvuEntAp9HmTyYL46L 4COw== X-Gm-Message-State: AOAM532gjPL4EheEwhZvUdTI6cGc2l9sIQADdfvNhvVdA/57BkyjqviG bcmVpI59MkB42P4IppBsJO3We+/c2vtvO2hF X-Google-Smtp-Source: ABdhPJxvzUHkjXnOqmXbBY0dbbQ4MYacVITgO8K/txbbUqhtRB0o92SfbW115JZPU7NBY1h10U3mvg== X-Received: by 2002:a50:bc15:: with SMTP id j21mr16374387edh.187.1612780314708; Mon, 08 Feb 2021 02:31:54 -0800 (PST) Received: from [192.168.0.202] (ip5f5af629.dynamic.kabel-deutschland.de. [95.90.246.41]) by smtp.gmail.com with ESMTPSA id cb21sm9232716edb.57.2021.02.08.02.31.53 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 08 Feb 2021 02:31:53 -0800 (PST) Subject: Re: Tuning and monitoring write intensive server To: Vladilen Kozin , Walter von Entferndt Cc: freebsd-performance@freebsd.org References: <2002412.uJW0cDvVUg@t450s.local.lan> From: Mateusz Piotrowski <0mp@FreeBSD.org> Message-ID: <1bc8a9f7-866a-db31-86b8-03d875196aa9@FreeBSD.org> Date: Mon, 8 Feb 2021 11:32:29 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4DZ2Q41jBCz3GQY X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mpp302@gmail.com designates 209.85.208.50 as permitted sender) smtp.mailfrom=mpp302@gmail.com X-Spamd-Result: default: False [-2.99 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.99)[-0.985]; FREEMAIL_TO(0.00)[gmail.com,posteo.net]; FORGED_SENDER(0.30)[0mp@FreeBSD.org,mpp302@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[95.90.246.41:received]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[0mp@FreeBSD.org,mpp302@gmail.com]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.208.50:from]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; SPAMHAUS_ZRD(0.00)[209.85.208.50:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.50:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.50:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Feb 2021 10:31:57 -0000 On 08/02/2021 11:21, Vladilen Kozin wrote: > These are excellent pointers and get me on my way! Thank you very much > Walter. Invaluable. > As a general note perhaps you may be interested in the stats(7) manual page. And the dwatch utility. From owner-freebsd-performance@freebsd.org Wed Feb 10 16:06:00 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5516F54ADD5 for ; Wed, 10 Feb 2021 16:06:00 +0000 (UTC) (envelope-from raj@gusw.net) Received: from gateway36.websitewelcome.com (gateway36.websitewelcome.com [192.185.200.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DbPkb3WlPz4fmP for ; Wed, 10 Feb 2021 16:05:59 +0000 (UTC) (envelope-from raj@gusw.net) Received: from cm17.websitewelcome.com (cm17.websitewelcome.com [100.42.49.20]) by gateway36.websitewelcome.com (Postfix) with ESMTP id CE02D400F0ACC for ; Wed, 10 Feb 2021 10:05:53 -0600 (CST) Received: from host2097.hostmonster.com ([67.20.114.243]) by cmsmtp with SMTP id 9s01lJdupDT649s01l6ko9; Wed, 10 Feb 2021 10:05:53 -0600 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=schadow.us; s=default; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version :Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=M7yf190a0SLJ204Kb5/pMnDDzLPJWUsy9RBAXJ+74aA=; b=lMrhVxoYo4djBNBxJWDv+qoCrK NdKU58FiAVU+/1ycuHoDim3wl/PyyO+hwOs0HXL8rr6l6ayt2AoyHqhk7WzEE2qgcNr/OX9/aTG66 bVbS1cjS5VHbUL8jdHw1Ql52r; Received: from [179.218.210.156] (port=56483 helo=[192.168.121.116]) by host2097.hostmonster.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1l9s01-001SqK-0s for freebsd-performance@freebsd.org; Wed, 10 Feb 2021 09:05:53 -0700 Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org References: <5BC0DEF9-3D58-4FFC-9E20-311B0520A25A@longcount.org> From: Gunther Schadow Message-ID: <96d43dbe-ed50-8ba4-f676-673ee99725bd@gusw.net> Date: Wed, 10 Feb 2021 11:05:48 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <5BC0DEF9-3D58-4FFC-9E20-311B0520A25A@longcount.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2097.hostmonster.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - gusw.net X-BWhitelist: no X-Source-IP: 179.218.210.156 X-Source-L: No X-Exim-ID: 1l9s01-001SqK-0s X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: ([192.168.121.116]) [179.218.210.156]:56483 X-Source-Auth: ebiz+schadow.us X-Email-Count: 1 X-Source-Cap: cHJhZ21hdDE7cHJhZ21hdDE7aG9zdDIwOTcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-Rspamd-Queue-Id: 4DbPkb3WlPz4fmP X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=schadow.us header.s=default header.b=lMrhVxoY; dmarc=none; spf=softfail (mx1.freebsd.org: 192.185.200.11 is neither permitted nor denied by domain of raj@gusw.net) smtp.mailfrom=raj@gusw.net X-Spamd-Result: default: False [-3.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_X_SOURCE(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[schadow.us:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; HAS_X_ANTIABUSE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[192.185.200.11:from]; ASN(0.00)[asn:46606, ipnet:192.185.192.0/19, country:US]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_NEUTRAL(0.00)[192.185.200.11:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[schadow.us:s=default]; FREEFALL_USER(0.00)[raj]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-performance@freebsd.org]; DMARC_NA(0.00)[gusw.net]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[192.185.200.11:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[192.185.200.11:from]; MAILMAN_DEST(0.00)[freebsd-performance] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2021 16:06:00 -0000 I think we got enough feed back now from other professionals to suggest that it would do the FreeBSD project good to acknowledge the issue and create some sort of work in progress / project statement, perhaps a wiki, where people can flock to to look for workarounds, current statue, just to not feel so lost and lonely and wondering if anybody cares at all. Such a meeting-point could at least be something of a self-help group, for emotional support and positive thinking =8). I have a slight sign of hope also: in my latest Amazon Linux db server deployment the performance is only half of what I got in my previous db server deployment. Go figure. But it's a fresh launch and I can't figure out what else I had optimized with the previous Amazon Linux box. So, while not improving anything, it at least closes the inequality between Linux and FreeBSD in the socialistic way ;) Now I am trying the FreeBSD install again with the same disk setup. Because one thing I clearly know is that it makes a huge difference of having one large EBS device and a single partition / file system vs. the same large EBS device with many partitions and file systems, separating tables, indexes, temporary sort space, etc. so that there is less random access contention on a single file system. Why that is important, I wonder, actually. And it's not the underlying EBS that matters so much. Like in bare metal world, the approach would be separate disks vs. striping (RAID-0) to get more spindles involved instead of having disk seek. But none of that should mater that much any more with SSD drives (which EBS gp2 and gp3 are!) Fortunately now Amazon has gp3 volumes also which support 3000 io transactions per second (IOps) and 250 or up to 1000 MB/s throughput despite being as small as 4 GB. So what I'm doing now is instead of a partitioned 1 TB gp2 volume with many partitions, I make myself over 20 separate smaller gp3 volumes, the advantage of this is that I can individually resize those without colliding with the neighboring partition. Tomorrow I should have better comparison numbers for my database on both Linux and FreeBSD configured the exact same way, and it might be closer now. But sadly only in the socialist way. regards, -Gunther On 2/6/2021 9:55 AM, Mark Saad wrote: > On Feb 6, 2021, at 7:07 AM, Łukasz Wąsikowski wrote: > All > So what I was getting at, is do we have good data on what the issue is ? Can we make a new wiki page on the FreeBSD wiki to track what works what and doesn’t . Does one exist ? > > To be clear we should check if the issue something that aws is doing with their xen platform , kvm/qemu one or common to all ? Also does that same issue appear on google and Microsoft’s platforms? This will at least get some bounds on the problem and what if any fixes may exist . > There are some commercial FreeBSD products running on aws . Maybe the vendors know some stuff that can help ? > > > Thoughts ? > > --- > Mark Saad | nonesuch@longcount.org > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@freebsd.org Wed Feb 10 16:48:04 2021 Return-Path: Delivered-To: freebsd-performance@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 674CA54BE5A for ; Wed, 10 Feb 2021 16:48:04 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DbQg81zGhz4jvx for ; Wed, 10 Feb 2021 16:48:04 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: seanc) by smtp.freebsd.org (Postfix) with ESMTPSA id 0F1C2DC54 for ; Wed, 10 Feb 2021 16:48:03 +0000 (UTC) (envelope-from seanc@freebsd.org) Received: by mail-pj1-f43.google.com with SMTP id d2so1467069pjs.4 for ; Wed, 10 Feb 2021 08:48:03 -0800 (PST) X-Gm-Message-State: AOAM531liOMLIBnB3/gRCcZChzP79osxISrVIPEYJnE18q3/4rWThAh5 zgiF5oMYTGSm8C0Y/BXM2B2/x67F8Ao0YsjBFWqwPg== X-Google-Smtp-Source: ABdhPJyd1UN2fl32sG2u5Wql7XKP1q2ZhndhMpoPS8PnUX2G+q6K2Fwu2utolaSpoNRH15mHGGkIz3PIaWgQ9gLd0x4= X-Received: by 2002:a17:90b:3886:: with SMTP id mu6mr3832052pjb.153.1612975682070; Wed, 10 Feb 2021 08:48:02 -0800 (PST) MIME-Version: 1.0 References: <5BC0DEF9-3D58-4FFC-9E20-311B0520A25A@longcount.org> <96d43dbe-ed50-8ba4-f676-673ee99725bd@gusw.net> In-Reply-To: <96d43dbe-ed50-8ba4-f676-673ee99725bd@gusw.net> From: Sean Chittenden Date: Wed, 10 Feb 2021 08:47:45 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: FreeBSD on Amazon AWS EC2 long standing performance problems To: freebsd-performance@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Feb 2021 16:48:04 -0000 Expect to see something about this on this year's Community Survey and the Core Team will do something with this information. Cloud has been an increasingly important workload for FreeBSD users. Or at least, our community is moving a fair number of workloads to virtualized metal, and running on the cloud is in need of attention as or more than some of our other subsystems. Many of the people who have the skills to jump in and diagnose and fix this type of problem have been sucked up with other professional commitments and aren't available. Which is to say, we need to increase the depth of our bench for performance-related work. But if someone's reading this thinking, "well I'll wait until Core...." don't wait! Jump in. If people are looking for a single starting point, I'd suggest trying to conjure up an iflib-backed ethernet driver and reducing the administrative friction of spinning up one of our images in the Cloud. "Secure by default" or conservative image defaults aren't doing anyone any favors in the cloud era. Or begin porting the io_uring kernel interface and begin working down the stack into the driver layer. $0.02. -sc On Wed, Feb 10, 2021 at 8:06 AM Gunther Schadow wrote: > I think we got enough feed back now from other professionals to suggest > that it > would do the FreeBSD project good to acknowledge the issue and create som= e > sort > of work in progress / project statement, perhaps a wiki, where people can > flock > to to look for workarounds, current statue, just to not feel so lost and > lonely > and wondering if anybody cares at all. Such a meeting-point could at leas= t > be > something of a self-help group, for emotional support and positive > thinking =3D8). > > I have a slight sign of hope also: in my latest Amazon Linux db server > deployment > the performance is only half of what I got in my previous db server > deployment. > Go figure. But it's a fresh launch and I can't figure out what else I had > optimized > with the previous Amazon Linux box. So, while not improving anything, it > at least > closes the inequality between Linux and FreeBSD in the socialistic way ;) > > Now I am trying the FreeBSD install again with the same disk setup. > Because one > thing I clearly know is that it makes a huge difference of having one > large EBS > device and a single partition / file system vs. the same large EBS device > with > many partitions and file systems, separating tables, indexes, temporary > sort space, > etc. so that there is less random access contention on a single file > system. > > Why that is important, I wonder, actually. And it's not the underlying EB= S > that > matters so much. Like in bare metal world, the approach would be separate > disks > vs. striping (RAID-0) to get more spindles involved instead of having dis= k > seek. > But none of that should mater that much any more with SSD drives (which > EBS gp2 and > gp3 are!) > > Fortunately now Amazon has gp3 volumes also which support 3000 io > transactions > per second (IOps) and 250 or up to 1000 MB/s throughput despite being as > small as > 4 GB. So what I'm doing now is instead of a partitioned 1 TB gp2 volume > with many > partitions, I make myself over 20 separate smaller gp3 volumes, the > advantage of > this is that I can individually resize those without colliding with the > neighboring > partition. > > Tomorrow I should have better comparison numbers for my database on both > Linux and > FreeBSD configured the exact same way, and it might be closer now. But > sadly only > in the socialist way. > > regards, > -Gunther > > On 2/6/2021 9:55 AM, Mark Saad wrote: > > > On Feb 6, 2021, at 7:07 AM, =C5=81ukasz W=C4=85sikowski > wrote: > > All > > So what I was getting at, is do we have good data on what the issue > is ? Can we make a new wiki page on the FreeBSD wiki to track what works > what and doesn=E2=80=99t . Does one exist ? > > > > To be clear we should check if the issue something that aws is doing > with their xen platform , kvm/qemu one or common to all ? Also does that > same issue appear on google and Microsoft=E2=80=99s platforms? This will= at least > get some bounds on the problem and what if any fixes may exist . > > There are some commercial FreeBSD products running on aws . Maybe the > vendors know some stuff that can help ? > > > > > > Thoughts ? > > > > --- > > Mark Saad | nonesuch@longcount.org > > > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" >