From owner-freebsd-stable@freebsd.org Sun Mar 14 02:00:19 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2AB5D56DEE6 for ; Sun, 14 Mar 2021 02:00:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DyjS30W8Kz4Vws for ; Sun, 14 Mar 2021 02:00:18 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x72b.google.com with SMTP id f124so28448524qkj.5 for ; Sat, 13 Mar 2021 18:00:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Nw0jle3MHJ3qYTlCxSq2yuQ4ewD+TAR7eiLF8iLYLe4=; b=xe9PC3J9EDwFIeQNsXV/3HvFQ1hCCZnTklUA1uj/cMdlJPHxkl7QsKVWQJJP+qPw9e dc9RI8bvLMLhiE4DTYlbupESkHGSHzz4eFiI2x5rV9uZxcNI2H53wL3LCZ1BMAxO1vR5 XxCxxrwQiuTAQzcmN/lJEpcQdUdDpDoSb2St6mRsIzA0U7bZGKfrUMsAokY6AYPFxRdK AlyTUtfa0H5k482gP+IMNHnjxg5aMeuov8hl/Jqi1n9lho9JcV5sl3uSpZudXwd6DaEM vH4E17Hd+OWz6rmoDW2cRSDNzsPF2LIapF95LqIpJtb7vC2OL3mCkIFSfgaSiq4jqPDc VwyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Nw0jle3MHJ3qYTlCxSq2yuQ4ewD+TAR7eiLF8iLYLe4=; b=ugM8Th8DHJdGKH6IU1cAOH3QPJpFzbnRtFQ6f8Xx4ZTrrF6G5Fm2u940enkJkx4/9Y HYJnOM+szdTnjHz72+AcFiIwx6Wt0WqPMFgivwzHcvZ5SFQvU3QIN/U0Kn/pwho4NuX7 YP8evWLlV+3fjluZ2c31bHBgG6bV2UoRKMa7okRDqxouQ4fl+2F9YdzYG3XUWdgjC30N O1io6XbCY5IdKqmPXTXOzZulaNxAgd1Dvqz3+vCzZEucbxi3KErOppA6fui/joGTvBkJ 6im3rDWxqj3S5YyfJW3FGDpdQkWVke7LWiB2P74QtIDO8QJ5k8hlUYP+zgCSKkKhaD42 PB5g== X-Gm-Message-State: AOAM532PpkdXdQG3f9Y0l5zRIV5E6pUtmAp8b1AaN2Eiil6AgtkKE7u0 py9UlvhyirQZFXP7xm5xS1ESug+nngpntAmA1YO57w== X-Google-Smtp-Source: ABdhPJxlJ2oC33y9q9Z+/rOx+HjZzllJrT8rsACV4AgW9o1cb/1YKpi3CKsa4owqE6Qg/oEkBfBk3kP0FfEKT4G/inQ= X-Received: by 2002:ae9:e010:: with SMTP id m16mr2560289qkk.44.1615687217951; Sat, 13 Mar 2021 18:00:17 -0800 (PST) MIME-Version: 1.0 References: <12705C29-53EA-4484-8291-C409AF4B3DE5.ref@yahoo.com> <12705C29-53EA-4484-8291-C409AF4B3DE5@yahoo.com> In-Reply-To: From: Warner Losh Date: Sat, 13 Mar 2021 19:00:07 -0700 Message-ID: Subject: Re: Filesystem operations slower in 13.0 than 12.2 To: Kevin Oberman Cc: Mark Millard , Konstantin Belousov , FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 4DyjS30W8Kz4Vws X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 02:00:19 -0000 On Sat, Mar 13, 2021 at 6:37 PM Kevin Oberman wrote: > I have been dealing with this for a long time since head back in September > through 13-stable of Mar-4. I have seen no improvement over this time. It > seems (my perception without supporting data) that it got worse in the > timeframe of BETA-3 tag. I was running stable, so not quite BETA-3. It also > does not help that I have also been bitten by the P-State related freeze > issue which has some similarities. disabling p-states has almost eliminated > this issue, though, with only three occurrences since I disabled them in > late January. > > As a result, I don't think it is a recent change, but a problem that has > existed for at least 3 months. This was made worse by two hardware issues > that kept the system unavailable for most of the time between buying it > last spring and getting the keyboard replaced in January. (Both the > mainboard and the disk drive had already been replaced.) There was another > slow I/O issue that I had assumed was the same as mine, but was reportedly > fixed with BETA-4. A few are still seeing slow I/O, so I assume that there > were different issues with I/O. Since CometLake systems seem pretty > uncommon, it might be related to that. > It was a change from last fall, or set of changes. RC1 or defintely RC2 has fixes to regain performance lost. If BETA4 was the last one you evaluated, perhaps you could do a couple tests with RC2 now that it's out to see if it is the same thing? Warner > Kevin Oberman, Part time kid herder and retired Network Engineer > E-mail: rkoberman@gmail.com > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 > > > On Sat, Mar 13, 2021 at 4:36 PM Warner Losh wrote: > >> >> >> On Sat, Mar 13, 2021 at 5:33 PM Kevin Oberman >> wrote: >> >>> Just spent a little time looking at my issue and have a few more notes: >>> >> >> What version did you evaluate? There's a number of changes lately that >> could have a big impact on this... >> >> Warner >> >> >>> Seems to only occur on large r/w operations from/to the same disk. "sp >>> big-file /other/file/on/same/disk" or tar/untar operations on large >>> files. >>> Hit this today updating firefox. >>> >>> I/O starts at >40MB/s. Dropped to about 1.5MB/s. If I tried doing other >>> things while it was running slowly, the disk would appear to lock up. >>> E.g. >>> pwd(1) seemed to completely lock up the system, but I could still ping it >>> and, after about 30 seconds, things came back to life. It was also not >>> instantaneous. Disc activity dropped to <1MB/s for a few seconds before >>> everything froze. >>> >>> During the untar of firefox, I saw; this several times. I also looked at >>> my >>> console where I found these errors during : >>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 55043, size: 8192 >>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 51572, size: 4096 >>> >>> I should note that some operations continue just fine while this is going >>> on until I do something that freezes the system. I assume that this >>> eliminates the disk drive and low-level driver. Is vfs a possible issue. >>> It >>> had some serious work in the past few months by markj. That does not >>> explain why more people are not seeing this. >>> >>> I have been seeing this since at least September 2020, so it goes back a >>> way. As this CometLake system will not run graphics on 12, I can't >>> confirm >>> operation before 13. >>> -- >>> Kevin Oberman, Part time kid herder and retired Network Engineer >>> E-mail: rkoberman@gmail.com >>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >>> >>> >>> On Fri, Mar 5, 2021 at 10:47 PM Mark Millard via freebsd-stable < >>> freebsd-stable@freebsd.org> wrote: >>> >>> > >>> > Konstantin Belousov kostikbel at gmail.com wrote on >>> > Fri Mar 5 23:12:13 UTC 2021 : >>> > >>> > > On Sat, Mar 06, 2021 at 12:27:55AM +0200, Christos Chatzaras wrote: >>> > . . . >>> > > > Command: /usr/bin/time -l portsnap extract (these tests done with 2 >>> > different idle servers but with same 4TB HDDs models) >>> > > > >>> > > > FreeBSD 12.2p4 >>> > > > >>> > > > 99.45 real 34.90 user 59.63 sys >>> > > > 100.00 real 34.91 user 59.97 sys >>> > > > 82.95 real 35.98 user 60.68 sys >>> > > > >>> > > > FreeBSD 13.0-RC1 >>> > > > >>> > > > 217.43 real 75.67 user 110.97 sys >>> > > > 125.50 real 63.00 user 96.47 sys >>> > > > 118.93 real 62.91 user 96.28 sys >>> > > . . . >>> > > In the portsnap results for 13RC1, the variance is too high to >>> conclude >>> > > anything, I think. >>> > >>> > I'll note that there are other reports of wide variance >>> > in transfer rates observed during an overall operation >>> > such as "make extract". The one I'm thinking of is: >>> > >>> > >>> https://lists.freebsd.org/pipermail/freebsd-stable/2021-March/093251.html >>> > >>> > which is an update to earlier reports, but based on more recent >>> > stable/13. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253968 >>> > comment 4 has some more notes about the context. The "make extract" >>> > for firefox likely is not as complicated as the portsnap extract >>> > example's execution structure. >>> > >>> > Might be something to keep an eye on if there are on-going >>> > examples of over time. >>> > >>> > === >>> > Mark Millard >>> > marklmi at yahoo.com >>> > ( dsl-only.net went >>> > away in early 2018-Mar) >>> > >>> > _______________________________________________ >>> > freebsd-stable@freebsd.org mailing list >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> > To unsubscribe, send any mail to " >>> freebsd-stable-unsubscribe@freebsd.org" >>> > >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org >>> " >>> >>