From owner-freebsd-stable@freebsd.org Sun Mar 14 00:33:17 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4E97956969E for ; Sun, 14 Mar 2021 00:33:17 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ot1-x32a.google.com (mail-ot1-x32a.google.com [IPv6:2607:f8b0:4864:20::32a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DygWc1V8dz3thk; Sun, 14 Mar 2021 00:33:15 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-ot1-x32a.google.com with SMTP id f73-20020a9d03cf0000b02901b4d889bce0so3381991otf.12; Sat, 13 Mar 2021 16:33:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=mVx8qiEcHu9B3gCKzW9g+7WvZ7ujYB9P9F7Ni765YXo=; b=Wd1X7OTuRoZLmKh5seIdEBOa2a+hLoUd+cp/jCf565IBBkkCagVKtQD2TM3r0mdAGv S0oihO5NzP2PUPJEJyZSv/PLj5yZ7UJHiR32Xlx3dqSQY6GnPArw0/eDG1JPZL16EVnY md7ztNEVmKacU7iI60mQCUC0jX3K6pdWgt28TmNqe5dNw/VG925+AofQDbHQdVQ+GV7I lljdet5U3E9wlq0Gt5txG3VGkxTrpbGYUazjxaZLogRbwVY5s1nAf2p/Yt/r0IeoOmK0 Ys3nGMCnEHMhoBlXEaxcKW2neDji8rtA50r30zWHPcc3Dtp5+5ilxvWId3nm6KwAqCdE UeaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=mVx8qiEcHu9B3gCKzW9g+7WvZ7ujYB9P9F7Ni765YXo=; b=q6cZW8vzZrm1PfGyI9mXf3OU4lzHfRQ+UcbxHC545i2E3a3WexAVYesJtbkmaRLMlu he5CXyVFSRgyi6CVstP1qO435xqIAYS4KD3IJkmicdWv72OFwgnZf1WOsZVA5GiKUux+ G0o4GNsG/zBcAEcSLweKsBS+55n/2ahmbf+EmFIWjjTz5p4kbrJGkw3c461tZ4MHYXnX DkVmZofw0yeTMhKwTjABlsQX3Tz/UdXMafxNw8SWVwsKzgwDfPBUE1IAz+dz4NkyIMTs vUmL2+EB9MT0rih4FFGC0Gi65/fokxTfXaewQB6RGPXDAYLPPQg8E+81PF3BNtHdcTqe 5Rgg== X-Gm-Message-State: AOAM5309sFxEyDYgODOMY97xmLqwcwm60KaUK6p3I2yFDTGJ/gR2Q7A1 VA5x1qqbWxku6NSBTaCtaMwIa+CauNPldqz/s6Q= X-Google-Smtp-Source: ABdhPJzOzuIHaOHopUICMxPuk6Dhuuuuv9BEqmi8F3+wtAApdSwDtllTC4ATCeFfKQ2XHBTZx7alJAKw4hWVlyttC+s= X-Received: by 2002:a05:6830:1e14:: with SMTP id s20mr9188455otr.199.1615681994589; Sat, 13 Mar 2021 16:33:14 -0800 (PST) MIME-Version: 1.0 References: <12705C29-53EA-4484-8291-C409AF4B3DE5.ref@yahoo.com> <12705C29-53EA-4484-8291-C409AF4B3DE5@yahoo.com> In-Reply-To: <12705C29-53EA-4484-8291-C409AF4B3DE5@yahoo.com> From: Kevin Oberman Date: Sat, 13 Mar 2021 16:32:58 -0800 Message-ID: Subject: Re: Filesystem operations slower in 13.0 than 12.2 To: Mark Millard Cc: Konstantin Belousov , FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 4DygWc1V8dz3thk X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Wd1X7OTu; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2607:f8b0:4864:20::32a as permitted sender) smtp.mailfrom=kob6558@gmail.com X-Spamd-Result: default: False [-3.70 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[yahoo.com]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::32a:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::32a:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32a:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-stable] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 00:33:17 -0000 Just spent a little time looking at my issue and have a few more notes: Seems to only occur on large r/w operations from/to the same disk. "sp big-file /other/file/on/same/disk" or tar/untar operations on large files. Hit this today updating firefox. I/O starts at >40MB/s. Dropped to about 1.5MB/s. If I tried doing other things while it was running slowly, the disk would appear to lock up. E.g. pwd(1) seemed to completely lock up the system, but I could still ping it and, after about 30 seconds, things came back to life. It was also not instantaneous. Disc activity dropped to <1MB/s for a few seconds before everything froze. During the untar of firefox, I saw; this several times. I also looked at my console where I found these errors during : swap_pager: indefinite wait buffer: bufobj: 0, blkno: 55043, size: 8192 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 51572, size: 4096 I should note that some operations continue just fine while this is going on until I do something that freezes the system. I assume that this eliminates the disk drive and low-level driver. Is vfs a possible issue. It had some serious work in the past few months by markj. That does not explain why more people are not seeing this. I have been seeing this since at least September 2020, so it goes back a way. As this CometLake system will not run graphics on 12, I can't confirm operation before 13. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 On Fri, Mar 5, 2021 at 10:47 PM Mark Millard via freebsd-stable < freebsd-stable@freebsd.org> wrote: > > Konstantin Belousov kostikbel at gmail.com wrote on > Fri Mar 5 23:12:13 UTC 2021 : > > > On Sat, Mar 06, 2021 at 12:27:55AM +0200, Christos Chatzaras wrote: > . . . > > > Command: /usr/bin/time -l portsnap extract (these tests done with 2 > different idle servers but with same 4TB HDDs models) > > > > > > FreeBSD 12.2p4 > > > > > > 99.45 real 34.90 user 59.63 sys > > > 100.00 real 34.91 user 59.97 sys > > > 82.95 real 35.98 user 60.68 sys > > > > > > FreeBSD 13.0-RC1 > > > > > > 217.43 real 75.67 user 110.97 sys > > > 125.50 real 63.00 user 96.47 sys > > > 118.93 real 62.91 user 96.28 sys > > . . . > > In the portsnap results for 13RC1, the variance is too high to conclude > > anything, I think. > > I'll note that there are other reports of wide variance > in transfer rates observed during an overall operation > such as "make extract". The one I'm thinking of is: > > https://lists.freebsd.org/pipermail/freebsd-stable/2021-March/093251.html > > which is an update to earlier reports, but based on more recent > stable/13. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253968 > comment 4 has some more notes about the context. The "make extract" > for firefox likely is not as complicated as the portsnap extract > example's execution structure. > > Might be something to keep an eye on if there are on-going > examples of over time. > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >