From owner-freebsd-stable@freebsd.org Sun Mar 14 05:43:54 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 135B35737BF for ; Sun, 14 Mar 2021 05:43:54 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DypQ16qk4z4mhf; Sun, 14 Mar 2021 05:43:53 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-ot1-x329.google.com with SMTP id r24so5540293otp.12; Sat, 13 Mar 2021 21:43:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=W9cKpU1yMVdPgu9q5YCoDd3FhxPSGONzJS/jr8eC29c=; b=t385N8wKHgkr1LMVEubDzp6+ujvZQOBTmRpsp5f9mCAkdYyTfOuMiviH4YJvE53hvJ YiEA4mjrIX2kuBOme0Wycz9EM1Orn/Pvk7hYfFDJb3HQqZU3XEj3ugj2fS8fRLjkcaLw qjeOZgouE25YRXpu1tGYZy8Y2QeTGFqU5DDmva/URAgPOms2mY082d0/TtG9KlVTvpgm uSaLbkz1bohSEIlHh6rHW9wsyLXxTQPcDZ9L0a72cicZxZseo7Oj6dyMEGZNcuoBZAZI czYg7bVnjlS0QX1AaEgB55iVAVUNyiav0qtDCg5ha2fcVUbtEI3bhc6E3ruqaEIm4nO9 jssw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=W9cKpU1yMVdPgu9q5YCoDd3FhxPSGONzJS/jr8eC29c=; b=jWMjoRLL4DGEKeM2byqTWxcCZW7HWoP98tVoSULkOuAVdZmTtt201rWh1BApglAkeg D6q9jSQMECb0COVUjKs6PtEplP71Myg/28ZoZ63HBIYgi9yG5sFeN8ObzqJo5kKKyHeJ VBZQsZJUFF6FpZjz80QXNmSe1uWiCfiK8vle3/Uwg3uBg3DzLNF+aUIxoK1c8GRbJLMP KgAXKZPMulFbGJsvF1LU7iNa5Is28Fufd1Fd6pR1pepUT37t00L0t+yESEQG72zK2fCF PI/9xlSUATH1kuLrNMfTxIAtZMKwl/bR7JblB0UMTwPLNooUTbdKc+mWEjd53CUVv2IZ qE0Q== X-Gm-Message-State: AOAM532s8YfnO0P85ezvYUs6gUc5t3fpij2JoRz1u4SymMQcnB3GtJXB AyhQaSB7HPjCqPdrb5wBu/fKW2oo+IW6T3xDvjQ= X-Google-Smtp-Source: ABdhPJzhP1fPwj1AZcDznw+J1uNQK0uGhMVaxwVvyRbgsMpOXScUGsRqFOD3UXFUq2U2epUnLGumk1LYaJXxoBcVZzQ= X-Received: by 2002:a05:6830:1e14:: with SMTP id s20mr9940298otr.199.1615700632700; Sat, 13 Mar 2021 21:43:52 -0800 (PST) MIME-Version: 1.0 References: <12705C29-53EA-4484-8291-C409AF4B3DE5.ref@yahoo.com> <12705C29-53EA-4484-8291-C409AF4B3DE5@yahoo.com> In-Reply-To: From: Kevin Oberman Date: Sat, 13 Mar 2021 21:43:36 -0800 Message-ID: Subject: Re: Filesystem operations slower in 13.0 than 12.2 To: Warner Losh Cc: Mark Millard , Konstantin Belousov , FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 4DypQ16qk4z4mhf X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 05:43:54 -0000 No improvement with stable/13-n244880-cec3990d347. It may be worse. worse. An attempt to unpack firefox-86.0.1,2 saw disk rates in the range of 800K to 2 MB/s range and with repeated 30 second freezes. I have no idea what made it so much worse, but I'm forced to start wondering if it could be a hardware issue. The disk drive was already replaced once due to a bad bearing. Went from a WD Black to a Seagate. Since it just keeps getting worse, I must consider that possibility. It is odd, though, that it was suddenly worse with the updated system. I think I will try going back to n244765-a00bf7d9bba (March 4) and see if it improves. If it does, I can likely eliminate bad hardware. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 On Sat, Mar 13, 2021 at 6:00 PM Warner Losh wrote: > > > On Sat, Mar 13, 2021 at 6:37 PM Kevin Oberman wrote: > >> I have been dealing with this for a long time since head back in >> September through 13-stable of Mar-4. I have seen no improvement over this >> time. It seems (my perception without supporting data) that it got worse in >> the timeframe of BETA-3 tag. I was running stable, so not quite BETA-3. It >> also does not help that I have also been bitten by the P-State related >> freeze issue which has some similarities. disabling p-states has almost >> eliminated this issue, though, with only three occurrences since I disabled >> them in late January. >> >> As a result, I don't think it is a recent change, but a problem that has >> existed for at least 3 months. This was made worse by two hardware issues >> that kept the system unavailable for most of the time between buying it >> last spring and getting the keyboard replaced in January. (Both the >> mainboard and the disk drive had already been replaced.) There was another >> slow I/O issue that I had assumed was the same as mine, but was reportedly >> fixed with BETA-4. A few are still seeing slow I/O, so I assume that there >> were different issues with I/O. Since CometLake systems seem pretty >> uncommon, it might be related to that. >> > > It was a change from last fall, or set of changes. RC1 or defintely RC2 > has fixes to regain performance lost. If BETA4 was the last one you > evaluated, perhaps you could do a couple tests with RC2 now that it's out > to see if it is the same thing? > > Warner > > >> Kevin Oberman, Part time kid herder and retired Network Engineer >> E-mail: rkoberman@gmail.com >> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >> >> >> On Sat, Mar 13, 2021 at 4:36 PM Warner Losh wrote: >> >>> >>> >>> On Sat, Mar 13, 2021 at 5:33 PM Kevin Oberman >>> wrote: >>> >>>> Just spent a little time looking at my issue and have a few more notes: >>>> >>> >>> What version did you evaluate? There's a number of changes lately that >>> could have a big impact on this... >>> >>> Warner >>> >>> >>>> Seems to only occur on large r/w operations from/to the same disk. "sp >>>> big-file /other/file/on/same/disk" or tar/untar operations on large >>>> files. >>>> Hit this today updating firefox. >>>> >>>> I/O starts at >40MB/s. Dropped to about 1.5MB/s. If I tried doing other >>>> things while it was running slowly, the disk would appear to lock up. >>>> E.g. >>>> pwd(1) seemed to completely lock up the system, but I could still ping >>>> it >>>> and, after about 30 seconds, things came back to life. It was also not >>>> instantaneous. Disc activity dropped to <1MB/s for a few seconds before >>>> everything froze. >>>> >>>> During the untar of firefox, I saw; this several times. I also looked >>>> at my >>>> console where I found these errors during : >>>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 55043, size: 8192 >>>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 51572, size: 4096 >>>> >>>> I should note that some operations continue just fine while this is >>>> going >>>> on until I do something that freezes the system. I assume that this >>>> eliminates the disk drive and low-level driver. Is vfs a possible >>>> issue. It >>>> had some serious work in the past few months by markj. That does not >>>> explain why more people are not seeing this. >>>> >>>> I have been seeing this since at least September 2020, so it goes back a >>>> way. As this CometLake system will not run graphics on 12, I can't >>>> confirm >>>> operation before 13. >>>> -- >>>> Kevin Oberman, Part time kid herder and retired Network Engineer >>>> E-mail: rkoberman@gmail.com >>>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >>>> >>>> >>>> On Fri, Mar 5, 2021 at 10:47 PM Mark Millard via freebsd-stable < >>>> freebsd-stable@freebsd.org> wrote: >>>> >>>> > >>>> > Konstantin Belousov kostikbel at gmail.com wrote on >>>> > Fri Mar 5 23:12:13 UTC 2021 : >>>> > >>>> > > On Sat, Mar 06, 2021 at 12:27:55AM +0200, Christos Chatzaras wrote: >>>> > . . . >>>> > > > Command: /usr/bin/time -l portsnap extract (these tests done with >>>> 2 >>>> > different idle servers but with same 4TB HDDs models) >>>> > > > >>>> > > > FreeBSD 12.2p4 >>>> > > > >>>> > > > 99.45 real 34.90 user 59.63 sys >>>> > > > 100.00 real 34.91 user 59.97 sys >>>> > > > 82.95 real 35.98 user 60.68 sys >>>> > > > >>>> > > > FreeBSD 13.0-RC1 >>>> > > > >>>> > > > 217.43 real 75.67 user 110.97 sys >>>> > > > 125.50 real 63.00 user 96.47 sys >>>> > > > 118.93 real 62.91 user 96.28 sys >>>> > > . . . >>>> > > In the portsnap results for 13RC1, the variance is too high to >>>> conclude >>>> > > anything, I think. >>>> > >>>> > I'll note that there are other reports of wide variance >>>> > in transfer rates observed during an overall operation >>>> > such as "make extract". The one I'm thinking of is: >>>> > >>>> > >>>> https://lists.freebsd.org/pipermail/freebsd-stable/2021-March/093251.html >>>> > >>>> > which is an update to earlier reports, but based on more recent >>>> > stable/13. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253968 >>>> > comment 4 has some more notes about the context. The "make extract" >>>> > for firefox likely is not as complicated as the portsnap extract >>>> > example's execution structure. >>>> > >>>> > Might be something to keep an eye on if there are on-going >>>> > examples of over time. >>>> > >>>> > === >>>> > Mark Millard >>>> > marklmi at yahoo.com >>>> > ( dsl-only.net went >>>> > away in early 2018-Mar) >>>> > >>>> > _______________________________________________ >>>> > freebsd-stable@freebsd.org mailing list >>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> > To unsubscribe, send any mail to " >>>> freebsd-stable-unsubscribe@freebsd.org" >>>> > >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to " >>>> freebsd-stable-unsubscribe@freebsd.org" >>>> >>>