From nobody Mon Feb 14 07:41:25 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F1D531956496 for ; Mon, 14 Feb 2022 07:41:51 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jxx4b0wQQz4RWX for ; Mon, 14 Feb 2022 07:41:51 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-wr1-x433.google.com with SMTP id i14so25259562wrc.10 for ; Sun, 13 Feb 2022 23:41:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vRlEjlrG2f72/EIihLIRtYk7P1DHaWht3tfgVu4xVVE=; b=K9AG9a4rWpLD1LwZGjFqDwpdKD2rV6oBj0ybhNVcQtxgKK7Jlg846CbAfhefkX1BtV YIyXt+MINPBl76lHP1QiTugytzvA3kDCg8RQokO2db8g06YXLM5vPruRZv3qUomca91G /FZf41+KPcA7IGN/6qplZG1bYnjHb0RPd9iA3shAGLt5YZE/b0Smm+V7wBkgWIfikB3v m6mU7rBYLeyyOinInj1s/buL7Sg5+cLD2tuH2zwpkxngmOXLMKluuEbCFugtNG+kCqKh a38jlHuaR29TBbl4tdTXCGCw2INQ3rerXsdPbeUqIvr/717ouFPxKAuUlb3i0Sx7AgBt dtDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vRlEjlrG2f72/EIihLIRtYk7P1DHaWht3tfgVu4xVVE=; b=jLetzjTxrUVHqi72gRnEEy4ugiw/7r2hPPecWQEGJN766ugOT87Ni8f9z0f/bSfpr/ BUAG2t/VVOierQ7R41GWvEYBaQ/zOJ1ZUPufcqqwQWNox20HolFrExwyO3473T+cJfWv C4IcsgYrTJoLYTCbkHsy/i94m4ye7xSuCTsKtEuaMaNfgB2uywdkP2H7HySMLidecrQJ e+ERiGbU4k+CPh1hOSHg3zbKspnHFC6w2V4Saj50li0LxAcPq2ldBXN4NbY6rixxdJn7 PltoAvIVrxX9N5V/W/YOszdPXau5RdmvFVdm9kZSw6KDHzQ2gABq+wzcWi198u9w7Nbr 7iaw== X-Gm-Message-State: AOAM533AjrzvB0z3uBPZV455mYVy3nmAiQPtpooYYggvnLrKzjaEjqVv dx2tgrkG0MiorKrC+p1l2ZtnYHRjm1zFg2h41+Cm5Yr0sxImP/u/ X-Google-Smtp-Source: ABdhPJxt8rswGU6SXNJF53GsrVSxusk4CxNOQuVSKpUn8PMF69cvvtk5IjfLO6ZNK9q8h2E2GONgyehdKz3eDX1Fj/k= X-Received: by 2002:a05:6000:38b:: with SMTP id u11mr4893835wrf.63.1644824510136; Sun, 13 Feb 2022 23:41:50 -0800 (PST) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Kevin Oberman Date: Sun, 13 Feb 2022 23:41:25 -0800 Message-ID: Subject: Re: swap/page problem To: tech-lists Cc: "freebsd-questions@freebsd.org" Content-Type: multipart/alternative; boundary="000000000000a5050905d7f58c91" X-Rspamd-Queue-Id: 4Jxx4b0wQQz4RWX X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=K9AG9a4r; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2a00:1450:4864:20::433 as permitted sender) smtp.mailfrom=kob6558@gmail.com X-Spamd-Result: default: False [-3.70 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::433:from]; MLMMJ_DEST(0.00)[questions]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --000000000000a5050905d7f58c91 Content-Type: text/plain; charset="UTF-8" On Thu, Feb 10, 2022 at 5:14 PM tech-lists wrote: > Hi, > > On Thu, Feb 10, 2022 at 02:28:50PM -0800, Kevin Oberman wrote: > >During a large build (llvm13), my system ground to a near halt with almost > >everything suspended. After several minutes, the system slowly recovered. > >When I looked at the messages log, I found 57 kernel messages spread over > >3.75 hours, in the form of: > >wap_pager: indefinite wait buffer: bufobj: 0, blkno: 862845, size: 20480 > > > >The block numbers and sizes varied. bufobj was always '0'. I had > >significant swap available swap space,as far as I could tell. I have 20GB > >of RAM and 24GB of swap. I am running stable 48937-3c6b6246f2f from > January > >13. > > > >I know that the LLVM build is huge, but I've not seen this before. What, > >exactly, is this message telling me? Am I out of RAM and swap? I couldd > add > >another 24GB of swap, though it would be on spinning rust, not SSD. > > I've seen this problem before and came to the conclusion > after reading threads on the lists and asking questions that > it wasn't a swap problem as primary issue but maybe theres some contention > between processes which affects llvm in particular and makes it eat swap. > Also, as you noted, swap doesn't really run out. > > I use poudriere to build and have parallel jobs set to 1 now > with make jobs enabled and the problem doesn't happen. If you're just > using the ports tree in the traditional way, try make -j1. > If parallel jobs is unset it would use hw.ncpu which here is 8 which would > produce the problem you descrive when compiling llvm13. > > These sysctls are now set: > > vfs.read_max=128 # default 64 - speeds up disk i/o > vfs.aio.max_buf_aio=8192 > vfs.aio.max_aio_queue_per_proc=65536 > vfs.aio.max_aio_per_proc=8192 > vfs.aio.max_aio_queue=65536 > vm.pageout_oom_seq=120 > vm.pfault_oom_attempts=-1 > > those last two may be especially helpful for your situation. > > My context here is amd64 i7-4770K (so 8 cpus with HT) clocked to 4.3GHz > and the disk used for building poers is SSD wiht 16GB swap (partition). > RAM is 32GB > > -- > J. > Thanks for the suggestions. The AIO ones look like those I recommended for vbox, though they are no longer needed for that as vbox has been modified to no longer use AIO. In any case, I'll see what happens. One oddity is that the problem seems to occur when my system reports "critical temperature detected". Shortly after, I see the swap_pager messages start. I have concluded that there is absolutely no issue with either RAM or swap space. Whatever is happening, it is tied to the problems that I have been seeing since I got my Lenovo L-13: P-States disabled, weird CPU frequency behavior, strange thermal control issues. I'm really regretting getting this laptop. Thanks again for the suggestions! -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000a5050905d7f58c91 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Thu, Feb 10, 2022 at 5:14 PM= tech-lists <tech-lists@zyxst.ne= t> wrote:
Hi,

On Thu, Feb 10, 2022 at 02:28:50PM -0800, Kevin Oberman wrote:
>During a large build (llvm13), my system ground to a near halt with alm= ost
>everything suspended. After several minutes, the system slowly recovere= d.
>When I looked at the messages log, I found 57 kernel messages spread ov= er
>3.75 hours, in the form of:
>wap_pager: indefinite wait buffer: bufobj: 0, blkno: 862845, size: 2048= 0
>
>The block numbers and sizes varied. bufobj was always '0'. I ha= d
>significant swap available swap space,as far as I could tell. I have 20= GB
>of RAM and 24GB of swap. I am running stable 48937-3c6b6246f2f from Jan= uary
>13.
>
>I know that the LLVM build is huge, but I've not seen this before. = What,
>exactly, is this message telling me? Am I out of RAM and swap? I couldd= add
>another 24GB of swap, though it would be on spinning rust, not SSD.

I've seen this problem before and came to the conclusion
after reading threads on the lists and asking questions that
it wasn't a swap problem as primary issue but maybe theres some content= ion
between processes which affects llvm in particular and makes it eat swap. Also, as you noted, swap doesn't really run out.

I use poudriere to build and have parallel jobs set to 1 now
with make jobs enabled and the problem doesn't happen. If you're ju= st
using the ports tree in the traditional way, try make -j1.
If parallel jobs is unset it would use hw.ncpu which here is 8 which would<= br> produce the problem you descrive when compiling llvm13.

These sysctls are now set:

vfs.read_max=3D128=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0# default 64 - speeds up disk i/o
vfs.aio.max_buf_aio=3D8192
vfs.aio.max_aio_queue_per_proc=3D65536
vfs.aio.max_aio_per_proc=3D8192
vfs.aio.max_aio_queue=3D65536
vm.pageout_oom_seq=3D120
vm.pfault_oom_attempts=3D-1

those last two may be especially helpful for your situation.

My context here is amd64 i7-4770K (so 8 cpus with HT) clocked to 4.3GHz
and the disk used for building poers is SSD wiht 16GB swap (partition).
RAM is 32GB

--
J.
Thanks for the suggestions. The AIO ones look = like those I recommended for vbox, though they are no longer needed for tha= t as vbox has been modified to no longer use AIO. In any case, I'll see= what happens.

One oddity is that the pr= oblem seems to occur when my system reports "critical temperature dete= cted". Shortly after, I see the swap_pager messages start. I have conc= luded that there is absolutely no issue with either RAM or swap space. What= ever is happening, it is tied to the problems that I have been seeing since= I got my Lenovo L-13: P-States disabled, weird CPU frequency behavior, str= ange thermal control issues. I'm really regretting getting this laptop.=

Thanks again for the suggestions!<= br>
--
Kevin Ob= erman, Part time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
--000000000000a5050905d7f58c91--