From owner-freebsd-stable@freebsd.org Sun Mar 14 01:37:23 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AE6DD56D828 for ; Sun, 14 Mar 2021 01:37:23 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dyhxb454Bz4TQk; Sun, 14 Mar 2021 01:37:23 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-ot1-x32d.google.com with SMTP id 75so5449824otn.4; Sat, 13 Mar 2021 17:37:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=U20PeC57cLjQ4SqXGqXFHwMp63bEBBeljU3hFF4TYQ4=; b=p0CShxxxp6fpELpznGBQkwwgZU87yE3w4FzyVZ3OLMaKLJIVT/BOotr4m+V15UN3QM OW6r0IVQtTx3Nh4kjhIya58ZlopwQDeP8zKm0QxrTEU2jo8bVaMk/9sXhB63dzG+JQOc kIGAY03WGE7dbAK98Y6ty6MmZ0kV9J8OVLD+XRiEmKfiTsohnCF2/thqBqRqdoJOk1yV Av5wAb8Igy0ME5mdNTDKz3Cd4b145CdoUBLL4D1CrNV2Z/57Z+wNFycG7KTHW6HKOAad UBV1VKLSPhI/JNimfqk/bS79qGoqcVJGcU2QucPYNNJ3/pBFvmjQtf4Cf2AuK9mxROLU gX3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=U20PeC57cLjQ4SqXGqXFHwMp63bEBBeljU3hFF4TYQ4=; b=FGyGiny9aN0gbv/Lp56ZNeyLJfZx15imB1KwWHPG518y2Cn4L44v8/5pldgGctWMSX 89Jv+QVGQXe+rg4Y1Xmw1OGeeP8+8bC09NrPuVb8NyTznfDkxGbThi779Jmr7a3ly1CE r95WJaVt6+2QfzlFaxMxV6/660j8S6BpHqhHhGO+qNnE/Ejt9u+7sW9dYlY6cc6Vp2MH TCv5XRQAhrlUg54cR+JjT9rSojwysrxOCCifnz/iIXmVX4D5VqRNSpDq2zxC+bpNa+pm wXShPYFTUH/S6obTrbHTbS3MemYBpiNi0UfBd30z7Lhxdc15P4iNZi9Rz8fq40DbiMYP IxmA== X-Gm-Message-State: AOAM5318hts6M6Kra5TXoFWpv4+5M34xxpIQDmPUZXKKNiHr39ozSxqd Z5241pYSHEkRDekZQF/45SAwtcUMZanPjTy67cw= X-Google-Smtp-Source: ABdhPJy1y+R/hp6LZMcaXVORxKudab63rCNtdUCADfqswp2ztypNppKkJudrCsC/g83Zqd2CU5YonbeNcXBu5UDoS6Y= X-Received: by 2002:a05:6830:1e14:: with SMTP id s20mr9348816otr.199.1615685842568; Sat, 13 Mar 2021 17:37:22 -0800 (PST) MIME-Version: 1.0 References: <12705C29-53EA-4484-8291-C409AF4B3DE5.ref@yahoo.com> <12705C29-53EA-4484-8291-C409AF4B3DE5@yahoo.com> In-Reply-To: From: Kevin Oberman Date: Sat, 13 Mar 2021 17:37:06 -0800 Message-ID: Subject: Re: Filesystem operations slower in 13.0 than 12.2 To: Warner Losh Cc: Mark Millard , Konstantin Belousov , FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 4Dyhxb454Bz4TQk X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 01:37:23 -0000 I have been dealing with this for a long time since head back in September through 13-stable of Mar-4. I have seen no improvement over this time. It seems (my perception without supporting data) that it got worse in the timeframe of BETA-3 tag. I was running stable, so not quite BETA-3. It also does not help that I have also been bitten by the P-State related freeze issue which has some similarities. disabling p-states has almost eliminated this issue, though, with only three occurrences since I disabled them in late January. As a result, I don't think it is a recent change, but a problem that has existed for at least 3 months. This was made worse by two hardware issues that kept the system unavailable for most of the time between buying it last spring and getting the keyboard replaced in January. (Both the mainboard and the disk drive had already been replaced.) There was another slow I/O issue that I had assumed was the same as mine, but was reportedly fixed with BETA-4. A few are still seeing slow I/O, so I assume that there were different issues with I/O. Since CometLake systems seem pretty uncommon, it might be related to that. Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 On Sat, Mar 13, 2021 at 4:36 PM Warner Losh wrote: > > > On Sat, Mar 13, 2021 at 5:33 PM Kevin Oberman wrote: > >> Just spent a little time looking at my issue and have a few more notes: >> > > What version did you evaluate? There's a number of changes lately that > could have a big impact on this... > > Warner > > >> Seems to only occur on large r/w operations from/to the same disk. "sp >> big-file /other/file/on/same/disk" or tar/untar operations on large files. >> Hit this today updating firefox. >> >> I/O starts at >40MB/s. Dropped to about 1.5MB/s. If I tried doing other >> things while it was running slowly, the disk would appear to lock up. E.g. >> pwd(1) seemed to completely lock up the system, but I could still ping it >> and, after about 30 seconds, things came back to life. It was also not >> instantaneous. Disc activity dropped to <1MB/s for a few seconds before >> everything froze. >> >> During the untar of firefox, I saw; this several times. I also looked at >> my >> console where I found these errors during : >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 55043, size: 8192 >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 51572, size: 4096 >> >> I should note that some operations continue just fine while this is going >> on until I do something that freezes the system. I assume that this >> eliminates the disk drive and low-level driver. Is vfs a possible issue. >> It >> had some serious work in the past few months by markj. That does not >> explain why more people are not seeing this. >> >> I have been seeing this since at least September 2020, so it goes back a >> way. As this CometLake system will not run graphics on 12, I can't confirm >> operation before 13. >> -- >> Kevin Oberman, Part time kid herder and retired Network Engineer >> E-mail: rkoberman@gmail.com >> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >> >> >> On Fri, Mar 5, 2021 at 10:47 PM Mark Millard via freebsd-stable < >> freebsd-stable@freebsd.org> wrote: >> >> > >> > Konstantin Belousov kostikbel at gmail.com wrote on >> > Fri Mar 5 23:12:13 UTC 2021 : >> > >> > > On Sat, Mar 06, 2021 at 12:27:55AM +0200, Christos Chatzaras wrote: >> > . . . >> > > > Command: /usr/bin/time -l portsnap extract (these tests done with 2 >> > different idle servers but with same 4TB HDDs models) >> > > > >> > > > FreeBSD 12.2p4 >> > > > >> > > > 99.45 real 34.90 user 59.63 sys >> > > > 100.00 real 34.91 user 59.97 sys >> > > > 82.95 real 35.98 user 60.68 sys >> > > > >> > > > FreeBSD 13.0-RC1 >> > > > >> > > > 217.43 real 75.67 user 110.97 sys >> > > > 125.50 real 63.00 user 96.47 sys >> > > > 118.93 real 62.91 user 96.28 sys >> > > . . . >> > > In the portsnap results for 13RC1, the variance is too high to >> conclude >> > > anything, I think. >> > >> > I'll note that there are other reports of wide variance >> > in transfer rates observed during an overall operation >> > such as "make extract". The one I'm thinking of is: >> > >> > >> https://lists.freebsd.org/pipermail/freebsd-stable/2021-March/093251.html >> > >> > which is an update to earlier reports, but based on more recent >> > stable/13. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253968 >> > comment 4 has some more notes about the context. The "make extract" >> > for firefox likely is not as complicated as the portsnap extract >> > example's execution structure. >> > >> > Might be something to keep an eye on if there are on-going >> > examples of over time. >> > >> > === >> > Mark Millard >> > marklmi at yahoo.com >> > ( dsl-only.net went >> > away in early 2018-Mar) >> > >> > _______________________________________________ >> > freebsd-stable@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> > To unsubscribe, send any mail to " >> freebsd-stable-unsubscribe@freebsd.org" >> > >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> >