From owner-freebsd-stable@freebsd.org Fri Mar 5 23:12:13 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 919E856AF88 for ; Fri, 5 Mar 2021 23:12:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dsk5m1NX7z3spW for ; Fri, 5 Mar 2021 23:12:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 125NC2fJ007114 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 6 Mar 2021 01:12:05 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 125NC2fJ007114 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 125NC1S6007113; Sat, 6 Mar 2021 01:12:01 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 6 Mar 2021 01:12:01 +0200 From: Konstantin Belousov To: Christos Chatzaras Cc: FreeBSD-STABLE Mailing List , Kirk McKusick Subject: Re: Filesystem operations slower in 13.0 than 12.2 Message-ID: References: <202103051842.125IgNl9013402@nuc.oldach.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4Dsk5m1NX7z3spW X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.97 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_SOFTFAIL(0.00)[~all:c]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.973]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-stable]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2021 23:12:13 -0000 On Sat, Mar 06, 2021 at 12:27:55AM +0200, Christos Chatzaras wrote: > I did some more tests. Finally I see similar results (with the exception of one "portsnap extract" test). Also with 13.0 I can't trigger a bug that I describe here: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250576 > > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > > Command: /usr/bin/time -l rm -fr /usr/ports /usr/src (these tests done with exactly the same hardware - I upgrade 12.2p4 to 13.0-RC1 for the 2nd test) > > FreeBSD 12.2p4 > > 12.67 real 0.36 user 1.94 sys > 13.18 real 0.41 user 1.81 sys > 12.16 real 0.36 user 1.85 sys > FreeBSD 13.0-RC1 > > 16.71 real 0.63 user 3.02 sys > 14.53 real 0.48 user 2.98 sys > 13.97 real 0.70 user 2.85 sys > > Command: /usr/bin/time -l tar xf src.tar (these tests done with 2 different idle servers but with same 4TB HDDs models) > > FreeBSD 12.2p4 > > 37.35 real 1.03 user 3.34 sys > > FreeBSD 13.0-RC1 > > 44.97 real 1.15 user 3.34 sys > > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > > Command: /usr/bin/time -l tar xf ports.tar (these tests done with 2 different idle servers but with same 4TB HDDs models) > > FreeBSD 12.2p4 > > 50.80 real 1.55 user 4.62 sys > > FreeBSD 13.0-RC1 > > 59.93 real 1.69 user 4.73 sys > > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > > > Command: /usr/bin/time -l portsnap extract (these tests done with 2 different idle servers but with same 4TB HDDs models) > > FreeBSD 12.2p4 > > 99.45 real 34.90 user 59.63 sys > 100.00 real 34.91 user 59.97 sys > 82.95 real 35.98 user 60.68 sys > > FreeBSD 13.0-RC1 > > 217.43 real 75.67 user 110.97 sys > 125.50 real 63.00 user 96.47 sys > 118.93 real 62.91 user 96.28 sys I trimmed the data above to show the interesting numbers more compact. In the portsnap results for 13RC1, the variance is too high to conclude anything, I think. There was (is) bugs in FreeBSD UFS SU < 13 - some LoR existed in SU code, where it needed to lock a containing directory to provide posix guarantees for fsync(), while owning the vnode lock. I do not believe it is observable in a real-world uses - in some situations UFS SU in < 13 did not performed necessary fsync() of the directory, related to the previous item The end result was that after sucessfull fsync() followed by a system failure e.g. power or panic, the parent directory for the synced vnode would not be synced and the vnode dirent' is not written to the permanent store. This volatiles posix requirement that after fsync, the data can be read, since you plain cannot open the file. During the development of the patch to fix both LoR and related ommission of fsync, a mistake was made resulting in much more aggessive syncing of directories. It was not exactly that, but approximately, on most of metadata operations that created or removed directory entry, the directory was fully synced. This resulted in the significant slow down, which was eliminated around BETA4..RC1. I.e. most of fixes come to BETA4, but minor parts were only discovered later and ready for RC1. There are still more fsync(dir) in 13RC1 than it is in any 12, by the nature of the bug and its fix, but the current belief is that all fsync calls left in the flow are required for correctness.