From owner-freebsd-stable@freebsd.org Sun Mar 14 18:10:02 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 04E795ADFFC for ; Sun, 14 Mar 2021 18:10:02 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dz6yx098Vz4YhJ; Sun, 14 Mar 2021 18:10:00 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-oi1-x235.google.com with SMTP id x135so27834409oia.9; Sun, 14 Mar 2021 11:10:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=v5i//1ex05dIve5lGMFOLX50py+eLaRkniWR+fprGrU=; b=cgTbAb3J9694e3mgvpcJBH22cHY8hM0Vm1NBGXIxXXX5YcTWUwQQ9GuNmykREcEo2A COvkBDw3J0byevhnWDOvvpqvE9q9MOWVneyTwna+uKgIz2FCO2I+Ynv7gVoNHvjGmHYL KannCmnxf5LFhYfq+zrGNkKBHNNmW75sEKSMMd07sOfmSh1srvun53SWghJMhyNGxgcP vfWxX01zA9loHzCcbGZIJP+Hnz3o8UHYB9BDCWezSqiPxVa/mqyfYkf6K2X8kGihxRw/ gwQ6ib0OTiaZ5zCdvmt3PMPSsRhqIODnN89u6ku/GQXMwQFEp0qdSCpWHsGQSZGvP71i MvcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=v5i//1ex05dIve5lGMFOLX50py+eLaRkniWR+fprGrU=; b=RqCR6K3DwlBIB0IvOOE03icu6XBeCXU7pdN6EsVkCIdklOgytlR3eTbNljsEcI5MPQ LrSXiIIB1OnukI7jcCQHBBmX/yilc8N5uVAYsgCpTF6s/Tknn9sfZa3tVRArVPpbmCvz Qr2zhEYjVcbltH0q8BX/zvtoPP5c5DkwsH8lXDV0/hhqSZyKe2mNlIrodF8fJ/+MiJAc +XqVvrVUZ3a5UdMaZ+ivW0m9sqe1KRvO8poehiltJ9DLygYh3+V2r4DtwG9mEqzlE2SF qfLasxTNUaA6Stq/MEfbq7lGXCP5tRyhuaPOSd5Kax0ODZU+ngHX57eGwfyDGFg8f+Zn gxMg== X-Gm-Message-State: AOAM531wtQ2w03k6frl/YX2ZhWvWXoDwnuUkEp46Sbm/TMgg6tKCEvd/ WfrumOjhe9HVMLXsLcgruYkUZUf5AGdLQnOM20I= X-Google-Smtp-Source: ABdhPJx/4F49AFPIDS+bbhz7/tkuWQ78VY9MXfbWylQZGezSIMIKv77hZzn0nincM3R9mTxjpN81BrrL/14X8vaKCno= X-Received: by 2002:aca:ac04:: with SMTP id v4mr16080835oie.57.1615745399741; Sun, 14 Mar 2021 11:09:59 -0700 (PDT) MIME-Version: 1.0 References: <12705C29-53EA-4484-8291-C409AF4B3DE5.ref@yahoo.com> <12705C29-53EA-4484-8291-C409AF4B3DE5@yahoo.com> In-Reply-To: From: Kevin Oberman Date: Sun, 14 Mar 2021 11:09:43 -0700 Message-ID: Subject: Re: Filesystem operations slower in 13.0 than 12.2 To: Warner Losh Cc: Mark Millard , Konstantin Belousov , FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 4Dz6yx098Vz4YhJ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=cgTbAb3J; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2607:f8b0:4864:20::235 as permitted sender) smtp.mailfrom=kob6558@gmail.com X-Spamd-Result: default: False [-3.70 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::235:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::235:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::235:from]; FREEMAIL_CC(0.00)[yahoo.com,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-stable]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 18:10:02 -0000 On Sat, Mar 13, 2021 at 9:43 PM Kevin Oberman wrote: > No improvement with stable/13-n244880-cec3990d347. It may be worse. > worse. An attempt to unpack firefox-86.0.1,2 saw disk rates in the range > of 800K to 2 MB/s range and with repeated 30 second freezes. I have no idea > what made it so much worse, but I'm forced to start wondering if it could > be a hardware issue. The disk drive was already replaced once due to a bad > bearing. Went from a WD Black to a Seagate. Since it just keeps getting > worse, I must consider that possibility. It is odd, though, that it was > suddenly worse with the updated system. > > I think I will try going back to n244765-a00bf7d9bba (March 4) and see if > it improves. If it does, I can likely eliminate bad hardware. > -- > Kevin Oberman, Part time kid herder and retired Network Engineer > E-mail: rkoberman@gmail.com > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 > > > On Sat, Mar 13, 2021 at 6:00 PM Warner Losh wrote: > >> >> >> On Sat, Mar 13, 2021 at 6:37 PM Kevin Oberman >> wrote: >> >>> I have been dealing with this for a long time since head back in >>> September through 13-stable of Mar-4. I have seen no improvement over this >>> time. It seems (my perception without supporting data) that it got worse in >>> the timeframe of BETA-3 tag. I was running stable, so not quite BETA-3. It >>> also does not help that I have also been bitten by the P-State related >>> freeze issue which has some similarities. disabling p-states has almost >>> eliminated this issue, though, with only three occurrences since I disabled >>> them in late January. >>> >>> As a result, I don't think it is a recent change, but a problem that >>> has existed for at least 3 months. This was made worse by two hardware >>> issues that kept the system unavailable for most of the time between buying >>> it last spring and getting the keyboard replaced in January. (Both the >>> mainboard and the disk drive had already been replaced.) There was another >>> slow I/O issue that I had assumed was the same as mine, but was reportedly >>> fixed with BETA-4. A few are still seeing slow I/O, so I assume that there >>> were different issues with I/O. Since CometLake systems seem pretty >>> uncommon, it might be related to that. >>> >> >> It was a change from last fall, or set of changes. RC1 or defintely RC2 >> has fixes to regain performance lost. If BETA4 was the last one you >> evaluated, perhaps you could do a couple tests with RC2 now that it's out >> to see if it is the same thing? >> >> Warner >> >> >>> Kevin Oberman, Part time kid herder and retired Network Engineer >>> E-mail: rkoberman@gmail.com >>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >>> >>> >>> On Sat, Mar 13, 2021 at 4:36 PM Warner Losh wrote: >>> >>>> >>>> >>>> On Sat, Mar 13, 2021 at 5:33 PM Kevin Oberman >>>> wrote: >>>> >>>>> Just spent a little time looking at my issue and have a few more notes: >>>>> >>>> >>>> What version did you evaluate? There's a number of changes lately that >>>> could have a big impact on this... >>>> >>>> Warner >>>> >>>> >>>>> Seems to only occur on large r/w operations from/to the same disk. "sp >>>>> big-file /other/file/on/same/disk" or tar/untar operations on large >>>>> files. >>>>> Hit this today updating firefox. >>>>> >>>>> I/O starts at >40MB/s. Dropped to about 1.5MB/s. If I tried doing other >>>>> things while it was running slowly, the disk would appear to lock up. >>>>> E.g. >>>>> pwd(1) seemed to completely lock up the system, but I could still ping >>>>> it >>>>> and, after about 30 seconds, things came back to life. It was also not >>>>> instantaneous. Disc activity dropped to <1MB/s for a few seconds before >>>>> everything froze. >>>>> >>>>> During the untar of firefox, I saw; this several times. I also looked >>>>> at my >>>>> console where I found these errors during : >>>>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 55043, size: 8192 >>>>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 51572, size: 4096 >>>>> >>>>> I should note that some operations continue just fine while this is >>>>> going >>>>> on until I do something that freezes the system. I assume that this >>>>> eliminates the disk drive and low-level driver. Is vfs a possible >>>>> issue. It >>>>> had some serious work in the past few months by markj. That does not >>>>> explain why more people are not seeing this. >>>>> >>>>> I have been seeing this since at least September 2020, so it goes back >>>>> a >>>>> way. As this CometLake system will not run graphics on 12, I can't >>>>> confirm >>>>> operation before 13. >>>>> -- >>>>> Kevin Oberman, Part time kid herder and retired Network Engineer >>>>> E-mail: rkoberman@gmail.com >>>>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >>>>> >>>>> >>>>> On Fri, Mar 5, 2021 at 10:47 PM Mark Millard via freebsd-stable < >>>>> freebsd-stable@freebsd.org> wrote: >>>>> >>>>> > >>>>> > Konstantin Belousov kostikbel at gmail.com wrote on >>>>> > Fri Mar 5 23:12:13 UTC 2021 : >>>>> > >>>>> > > On Sat, Mar 06, 2021 at 12:27:55AM +0200, Christos Chatzaras wrote: >>>>> > . . . >>>>> > > > Command: /usr/bin/time -l portsnap extract (these tests done >>>>> with 2 >>>>> > different idle servers but with same 4TB HDDs models) >>>>> > > > >>>>> > > > FreeBSD 12.2p4 >>>>> > > > >>>>> > > > 99.45 real 34.90 user 59.63 sys >>>>> > > > 100.00 real 34.91 user 59.97 sys >>>>> > > > 82.95 real 35.98 user 60.68 sys >>>>> > > > >>>>> > > > FreeBSD 13.0-RC1 >>>>> > > > >>>>> > > > 217.43 real 75.67 user 110.97 sys >>>>> > > > 125.50 real 63.00 user 96.47 sys >>>>> > > > 118.93 real 62.91 user 96.28 sys >>>>> > > . . . >>>>> > > In the portsnap results for 13RC1, the variance is too high to >>>>> conclude >>>>> > > anything, I think. >>>>> > >>>>> > I'll note that there are other reports of wide variance >>>>> > in transfer rates observed during an overall operation >>>>> > such as "make extract". The one I'm thinking of is: >>>>> > >>>>> > >>>>> https://lists.freebsd.org/pipermail/freebsd-stable/2021-March/093251.html >>>>> > >>>>> > which is an update to earlier reports, but based on more recent >>>>> > stable/13. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253968 >>>>> > comment 4 has some more notes about the context. The "make extract" >>>>> > for firefox likely is not as complicated as the portsnap extract >>>>> > example's execution structure. >>>>> > >>>>> > Might be something to keep an eye on if there are on-going >>>>> > examples of over time. >>>>> > >>>>> > === >>>>> > Mark Millard >>>>> > marklmi at yahoo.com >>>>> > ( dsl-only.net went >>>>> > away in early 2018-Mar) >>>>> > >>>>> >>>> Backing off to Mar. 4 was not an improvement. My untar did seem better for a couple of minutes, but then the display froze again for 30 seconds and disk performance dropped to <1M. then things got really bad and behaved in a manner that was baffling to me. The screen froze again, but stayed frozen after half a minute. I clicked on a couple of buttons in Firefox to no effect and then hit ctrl-q to quit. After the long pause, I pressed the power button to try to force a shutdown. Suddenly, it started unwinding everything I had done during the freeze. My browser did the updates from my mouse clicks including quitting. It then switched to a different workspace from ctrl-alt-right and did a clean shutdown. ???? Do I also have a graphics issue? Examining log files show no indication that anything was happening. SMART shows no errors and reasonable values for everything. No indication of a HW problem. The system performs well unless I do something that tries a bulk disk data move. Building world takes about 75 minutes. I just have a very hard time building big ports. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683