From owner-freebsd-arm@freebsd.org Sun Mar 22 17:42:45 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 262EB26819F for ; Sun, 22 Mar 2020 17:42:45 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48llGC08s7z3HFH for ; Sun, 22 Mar 2020 17:42:42 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id 02MHgwKg084979 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 22 Mar 2020 10:42:59 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id 02MHgwYa084978; Sun, 22 Mar 2020 10:42:58 -0700 (PDT) (envelope-from fbsd) Date: Sun, 22 Mar 2020 10:42:57 -0700 From: bob prohaska To: freebsd-arm@freebsd.org Subject: Re: Upgrading u-boot on an rpi3 Message-ID: <20200322174257.GA84906@www.zefox.net> References: <20200318054243.GA67865@www.zefox.net> <4B4CF1DF-F3C0-4ED3-AAC0-4FC0A8182787@yahoo.com> <20200318172339.GB67865@www.zefox.net> <456B1ED8-B335-405B-AB7B-B65968631323@yahoo.com> <20200319001353.GA70624@www.zefox.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200319001353.GA70624@www.zefox.net> User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Queue-Id: 48llGC08s7z3HFH X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [3.85 / 15.00]; ARC_NA(0.00)[]; WWW_DOT_DOMAIN(0.50)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(0.06)[ip: (0.24), ipnet: 50.1.16.0/20(0.12), asn: 7065(-0.03), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.89)[0.892,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(1.00)[0.998,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MID_RHS_WWW(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 17:42:45 -0000 I'd lke to make some "notes to self" for upgrading u-boot on a self-hosted rpi3. Here's what I think got done: Build and install the u-boot-rpi3 port using -DBATCH. Copy the resulting /usr/local/share/u-boot/u-boot-rpi3/u-boot.bin to /boot/msdos/u-boot.bin Following buildworld/installworld copy and rename /boot/loader.efi to /boot/msdos/EFI/BOOT/bootaa64.efi Have I overlooked anything important? As an aside, the Pi3 is now at r359195 and seems to work normally. It does seem to become unresponsive for long periods during svnlite up or any process involving storage I/O despite top reporting ~98% idle. However, it hasn't crashed yet. Thanks for reading, and any comments! bob prohaska On Wed, Mar 18, 2020 at 05:13:53PM -0700, bob prohaska wrote: > On Wed, Mar 18, 2020 at 11:02:43AM -0700, Mark Millard wrote: > > > > > > On 2020-Mar-18, at 10:23, bob prohaska wrote: > > > > > On Tue, Mar 17, 2020 at 11:42:09PM -0700, Mark Millard wrote: > > >> > > >> > > > > >> > > >> Those last 2 lines above indicate that it found > > >> your microsd card media and its bootaa64.efi just > > >> fine. > > >> > > >> How old is this file? > > > > > > Rather ancient: > > > > > > -rwxr-xr-x 1 root wheel 637000 Oct 10 2018 /boot/msdos/EFI/BOOT/bootaa64.efi > > > > > > I have a newer version on a 12.x snapshot: > > > -rwxr-xr-x 1 root wheel 609960 Nov 1 02:29 /mnt/EFI/BOOT/bootaa64.efi > > > Is it prudent to simply substitute the newer version for the older? > > > > You may want to extract a more modern one from a snapshot > > if that does not work. > > > > Turns out that the version of bootaa64.efi from the 12.x snapshot did > the trick. > > > >> Have you been updating > > >> it via copying /boot/loader.efi to it as > /boot/loader.efi is updated? > > > Not following here. Loader.efi appears to be a file and seems to update > > > during normal build/install cycles. It's unclear where bootaa64.efi comes > > > from; there's only one copy in the filesystem after repeated OS update cycles. > > > > For the ARM boards involved, efi/boot/bootaa64.efi is a > > copy of /boot/loader.efi . The loader copy used in booting > > is placed on the msdosfs, not on ufs/zfs. > > > > Ahh, now bells are ringing. IIRC there were some messages issued > to this effect during either make or make install for u-boot-rpi3, > at least formerly. > > > > > Example from the RPi4 context: > > > > # file /boot/loader.efi > > /boot/loader.efi: MS-DOS executable PE32+ executable (EFI application) Aarch64, for MS Windows > > > > # file /boot/efi/EFI/BOOT/bootaa64.efi > > /boot/efi/EFI/BOOT/bootaa64.efi: MS-DOS executable PE32+ executable (EFI application) Aarch64, for MS Windows > > > > Thank you very much! > > bob prohaska > > From owner-freebsd-arm@freebsd.org Sun Mar 22 19:15:02 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 816B0269EBE for ; Sun, 22 Mar 2020 19:15:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-25.consmr.mail.gq1.yahoo.com (sonic303-25.consmr.mail.gq1.yahoo.com [98.137.64.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48lnJh4BQHz4YWG for ; Sun, 22 Mar 2020 19:15:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 16x28XYVM1nA_EbetMskCzESB7XI4DjH.J6VNVL5Iu6tF4GdFqlRh37Xg51NacB bnS4FIGkkepkQirvA2d_bzd36YD3jWfwlhzijDAUwa8bqwPxQ5Od4Wan5jLD.xY.IrL2fDaQLZYK TChpYMtjdjfRYidPX8SNHExTdVME09L_tlVA7nAEwiRz8Se3qseZe1f0pXjeCj_IYPv2NDtScsIO pnd0tvlckjC9DuckJ2pcbj7I_LgE5_XGv82yMHFJQKbjTnJBT11aBfGW7uyWOgGIR5CANN6B8rjM 0dMkNwpGdiafaStSeBdlOEltTiH16I3PqzY4cRC4bdzEsgK.Y4TDhUtpMEMzOLsSAyjhcHnPDQo. rHKJLdYm9Ym4zool2Zs33gjPhj1eGbZs5ZfwR7h2XumDo.zNabLtrQeRw42Yeh9oDD4FDUdQT0UI QJ2CpAuboA.fuxGUFEtXGV0TCh6Ja4jU.uQST7dagGgkF3opZ2mQC3XRcWPE_5jAuumpOTxVpMMA 4erzxDD8_S8QNXahCFUozUJWq.E6sBHD_aUdzEH2vhRqsPZUkZMhtRkOCY3hpmfWU1GRtgHPzkob yUfKo6CWrwzn1vzZxa1I7o41bqU8W34Et8SOw3VyC4vU2VcChhCIA4IScTlRkLC4hWwK5Q152NbU kYD1UGLtf1iLZTUhJH54JugFsgWxJzLh80GEIaXwSbSBD7RLXizpEKN8sxJ5rk2iE50c.GvC01UE YClzUsF2qrPRr3wFCiRUjWZeF9t8LpErUloNp6eFGaOI4nvCMWrrLGDQ8cQWmfkPa8Vjr0t1khF7 rOmYE0yoStsVrVYoXwZybLP2d1U9tTtcNmjZkFU.dz0xLv.oNTg7gB_kDdpSrEJbA72T_F6LIOXx lHrV9tU8BypAynfQXpLe8UbW.eOCMspvI0Z9fLQ6zx6ldXC99aXl5bObsKbL2ApRYlPqPYR6VdrE U4OMIRiZ3OYpQxkqyTZmC3R.MISgHXDG8_pM2eGxbGu1zfJeFO8PRoCBsEIdkVCH6DDUMCMDtuOM 6QcBBFqarSh0ZlCsTDcd1qvFGhWuDJdgu8AOO0OXMY_Hm24xNo0XGRmXRSiuCOH9i2.V9TRuxUZR 46c7YLF.ljv.jRSAWys8aOCtx_gtf0iMUvbr6axblyk7E1Br7qqp6jqERCVNEl0i3UZ3.Ii._BXz BjXhJHOOBzdcMfB8OQnZZnG5etrOv_HgmUnPZOd.jQVEDb91aMyjzZLsxPuB4EojyxFw_vKULFU_ 7g81ebKjAGQC.czY9Iu_DxF_6aolHlKTa3K5gViBrPdzuXHZrNR.BPZpLQsEFTzwCmTUsXj_Ijv5 6G4t5tk3ELMWuQBO4sXUDfgMwIX8uc7zdpxnViwbx1Q2xzn0mNXyZ5eB6DYvzso2tHZJjZTqhfwb GICX3TuV_PvtvDQ-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Sun, 22 Mar 2020 19:14:58 +0000 Received: by smtp402.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 377e139d6e366f7fdbbf75c661fc54b4; Sun, 22 Mar 2020 19:14:55 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Upgrading u-boot on an rpi3 From: Mark Millard In-Reply-To: <20200322174257.GA84906@www.zefox.net> Date: Sun, 22 Mar 2020 12:14:54 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <7ADF6512-A998-420A-8FD9-C7AA8F08E20D@yahoo.com> References: <20200318054243.GA67865@www.zefox.net> <4B4CF1DF-F3C0-4ED3-AAC0-4FC0A8182787@yahoo.com> <20200318172339.GB67865@www.zefox.net> <456B1ED8-B335-405B-AB7B-B65968631323@yahoo.com> <20200319001353.GA70624@www.zefox.net> <20200322174257.GA84906@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48lnJh4BQHz4YWG X-Spamd-Bar: / X-Spamd-Result: default: False [0.59 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; URIBL_BLOCKED(0.00)[dsl-only.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (0.65), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; NEURAL_SPAM_MEDIUM(0.59)[0.591,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.50)[0.501,0]; RCVD_IN_DNSWL_NONE(0.00)[206.64.137.98.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 19:15:02 -0000 On 2020-Mar-22, at 10:42, bob prohaska wrote: > I'd lke to make some "notes to self" for upgrading u-boot on > a self-hosted rpi3. Here's what I think got done: U-boot is over specific terminology: it is 1 of 3 things. . . Port's sysutils/u-boot-rpi3 (after installation) : cp -aRx /usr/local/share/u-boot/u-boot-rpi3/u-boot.bin /boot/msdos/ The following 2 ( sysutils/rpi-firmware and /boot/loader.efi ) are not part of u-boot but are involved/required for booting . . . Port's sysutils/rpi-firmware (after installation) : (If you have tailored materials in /boot/msdos/config.txt , there is the need to preserve your content. So the below 2 copies might not be the right thing to do up front: it would involved more of a merge activity than just copying.) cp -aRx /usr/local/share/rpi-firmware/* /boot/msdos/ Deal with /boot/msdos/config.txt having the right content, such as: cp -aRx /boot/msdos/config_rpi3.txt /boot/msdos/config.txt FreeBSD's /boot/loader.efi (after installworld): cp -aRx /boot/loader.efi /boot/msdos/EFI/BOOT/bootaa64.efi So . . . > Build and install the u-boot-rpi3 port using -DBATCH. > Copy the resulting > /usr/local/share/u-boot/u-boot-rpi3/u-boot.bin > to > /boot/msdos/u-boot.bin Yep. > Following buildworld/installworld copy and rename > /boot/loader.efi > to > /boot/msdos/EFI/BOOT/bootaa64.efi Yep. > Have I overlooked anything important? When sysutils/rpi-firmware has updates it too is involved. I'll note that sysutils/rpi-firmware has the armstub8*.bin materials and they are not from the same places as the rest of the sysutils/rpi-firmware materials: different upstream place. So either upstream can lead to a sysutils/rpi-firmware update. > As an aside, the Pi3 is now at r359195 and > seems to work normally. It does seem to become > unresponsive for long periods during svnlite up > or any process involving storage I/O despite top > reporting ~98% idle. However, it hasn't crashed yet. Interesting. If "gstat -spod" is already running first, does it show anything interesting for the unresponsive time periods? === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Sun Mar 22 22:32:14 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6799226F9A5 for ; Sun, 22 Mar 2020 22:32:14 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48lshD6gd4z4Gyp for ; Sun, 22 Mar 2020 22:32:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ZBJrO9AVM1kGEGR9YnS9.iZ78CD38RVwS.fmRSViqvLeAubZQC.NRWmAKcIp8By _7rs_IKRFbeGLFkCvIJBk5ol3.jDQtjqMPn62N9AmADEHFjZUzOzIah3fZ01hkrty9ScGrSlV_mC 4tVWwN27BAs8CQDOqp5Qd2mXrVFhi3rHU1cz_b.5TzxTJTajHSk3NidrwYK6HUMfdi7BkieQlAd5 jtdz7JPAD6PQIzGlFNfx8IFJI1SqQPdFqr.LD1lGhD3KfiwAHQ46HdA9w8EBjh5LtNAlhJ.R1p6K ssfImW5F_3nteJi0WS7C2D6vVOd75wPuNYxp01XAY7sPyDRVhn2vo1.mDXZLQaS3xUsFv96KSDaf AyOqttLq6BqbLazwp0kAKI2gWZ_ZRrNklwFnQZRRlBOCaDhP5ILn_I2HQy8VyLoP2FfgrsFVD86_ loINoU.b2xn1A9J8eRVzNmY8BXE0zAePRQ3GseyNfQ_33aqceTAGVZw_SgLQq.DfoWtZqJqvS7pT YY3Qet43ZKYaHhloUhuaV3cK39fpVvX0DcdksyUbOcTSih9IMVAIH2ziGxRJA40k1M3Fte71QNWK AB9E_6_Miwv_hw2ww0Qr.rX6VJcb6XFbW4vDpQektBbOiYDWuE5sBmJBfHFENCQm6_XI.yi2CgEZ 3SkggDEHvsgFpBROgS.LNJF4GDMqJITG6uQ3tE9LS1Y9SsULtCuHh4gIKsl.kQL3ISOvY0xck5Fg GAr6BGMssACw5Qsk2F253eUOMDOxyNgGXbX1l76bkIOQ.J95.0k1EHPUQuJFfNNi2vvoYLO99BEJ CDNj.NlzCrA1q5O1sfk8yYLYa5pX7McaxOZqzSry_THAg2SJ_H0BrAMMcpw14tDvhwwrHIpkwa4C s6Wcxo4lm3kQ.y0kgcz.l.gcRu2gGQfFV__94UrEOEvp58EdZeBvHdImwvsawsnOLmfa4g.8ykq2 NU0crdFlAeazIlvADNL1vScE_tRqNmJ.mzACRd_QV5UAf.DEmls8STThtwtNNdfF7oWNuanVLqGp 3tOh8yP07yHxx3GASUSsuGIe8tB0tqpG4uW7LMQXhMXtmQHuVqMpRibz.6Dwk9Sxw2RIuB8M4NU9 QzpuwTjFHV4J68nHYl_v9LCJRX0hK1L_2DeRhOwAVsIk7TiiGU7b3rRQBxedk6vAo7tjLru9sQm. 7Td8YVjRzUmDDb88siHRdWvdWSMBskdrfm37VAAmp6y6siqA6vE8SehH0eQ517lzAXsnttNxoog7 lE7VTtfzV9MrhKzWNOJ86BuzBztGygF9LrExjUg0iBsU0v8XBuheXOtdXEktqS05zmB3O03_8EWw OU826tgxZC86MHWSbM26iYk0ffCcguVIHYNuw0EAqR.xwETR4Qw4NWL6WJEG7STPh34b_Bo0b.x6 ks5PCLeXG.SapWY6QWxeHJzD5GJyAKdRfEEuRu0JenOP3XOePzIQomF6A6w-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Sun, 22 Mar 2020 22:32:11 +0000 Received: by smtp414.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 44f8f4dc2e6f40362941de399693019c; Sun, 22 Mar 2020 22:32:09 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Upgrading u-boot on an rpi3 From: Mark Millard In-Reply-To: <7ADF6512-A998-420A-8FD9-C7AA8F08E20D@yahoo.com> Date: Sun, 22 Mar 2020 15:32:07 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <23A16E57-7902-4734-825D-5A40CF07CE4F@yahoo.com> References: <20200318054243.GA67865@www.zefox.net> <4B4CF1DF-F3C0-4ED3-AAC0-4FC0A8182787@yahoo.com> <20200318172339.GB67865@www.zefox.net> <456B1ED8-B335-405B-AB7B-B65968631323@yahoo.com> <20200319001353.GA70624@www.zefox.net> <20200322174257.GA84906@www.zefox.net> <7ADF6512-A998-420A-8FD9-C7AA8F08E20D@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48lshD6gd4z4Gyp X-Spamd-Bar: + X-Spamd-Result: default: False [1.44 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; URIBL_BLOCKED(0.00)[dsl-only.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (4.76), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; NEURAL_SPAM_MEDIUM(0.96)[0.958,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.99)[0.987,0]; RCVD_IN_DNSWL_NONE(0.00)[147.65.137.98.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 22:32:14 -0000 On 2020-Mar-22, at 12:14, Mark Millard wrote: > On 2020-Mar-22, at 10:42, bob prohaska wrote: > >> I'd lke to make some "notes to self" for upgrading u-boot on >> a self-hosted rpi3. Here's what I think got done: > > U-boot is over specific terminology: it is 1 of 3 things. . . > > > Port's sysutils/u-boot-rpi3 (after installation) : > > cp -aRx /usr/local/share/u-boot/u-boot-rpi3/u-boot.bin /boot/msdos/ > > > > The following 2 ( sysutils/rpi-firmware and > /boot/loader.efi ) are not part of u-boot but > are involved/required for booting . . . > > > Port's sysutils/rpi-firmware (after installation) : > > (If you have tailored materials in /boot/msdos/config.txt , > there is the need to preserve your content. So the below 2 > copies might not be the right thing to do up front: it > would involved more of a merge activity than just copying.) > > cp -aRx /usr/local/share/rpi-firmware/* /boot/msdos/ > > Deal with /boot/msdos/config.txt having the right content, > such as: > > cp -aRx /boot/msdos/config_rpi3.txt /boot/msdos/config.txt > > > FreeBSD's /boot/loader.efi (after installworld): > > cp -aRx /boot/loader.efi /boot/msdos/EFI/BOOT/bootaa64.efi > > So . . . > >> Build and install the u-boot-rpi3 port using -DBATCH. >> Copy the resulting >> /usr/local/share/u-boot/u-boot-rpi3/u-boot.bin >> to >> /boot/msdos/u-boot.bin > > Yep. I probably should have reported that I was ignoring the -DBATCH part. I do not know the specifics of why it was mentioned and I've never explicitly used it. >> Following buildworld/installworld copy and rename >> /boot/loader.efi >> to >> /boot/msdos/EFI/BOOT/bootaa64.efi > > Yep. > >> Have I overlooked anything important? > > When sysutils/rpi-firmware has updates it too > is involved. > > I'll note that sysutils/rpi-firmware has the > armstub8*.bin materials and they are not > from the same places as the rest of the > sysutils/rpi-firmware materials: different > upstream place. So either upstream can lead > to a sysutils/rpi-firmware update. > >> As an aside, the Pi3 is now at r359195 and >> seems to work normally. It does seem to become >> unresponsive for long periods during svnlite up >> or any process involving storage I/O despite top >> reporting ~98% idle. However, it hasn't crashed yet. > > Interesting. If "gstat -spod" is already running > first, does it show anything interesting for the > unresponsive time periods? === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Mon Mar 23 16:48:02 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1E8E3267E0F; Mon, 23 Mar 2020 16:48:02 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48mL0d6N62z4Btp; Mon, 23 Mar 2020 16:48:01 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-8.local (unknown [IPv6:2601:648:8881:1e90:b012:b1e1:5871:b934]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 7532A1EB45; Mon, 23 Mar 2020 16:48:01 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: head -r358966 on aarch64 fails to build base/gcc6: fatal error: bracket nesting level exceeded maximum of 256 To: Mark Millard , FreeBSD Toolchain , freebsd-arm References: <879B19CB-5EBB-4114-8C13-199E1C2E491D.ref@yahoo.com> <879B19CB-5EBB-4114-8C13-199E1C2E491D@yahoo.com> From: John Baldwin Autocrypt: addr=jhb@FreeBSD.org; keydata= mQGiBETQ+XcRBADMFybiq69u+fJRy/0wzqTNS8jFfWaBTs5/OfcV7wWezVmf9sgwn8TW0Dk0 c9MBl0pz+H01dA2ZSGZ5fXlmFIsee1WEzqeJzpiwd/pejPgSzXB9ijbLHZ2/E0jhGBcVy5Yo /Tw5+U/+laeYKu2xb0XPvM0zMNls1ah5OnP9a6Ql6wCgupaoMySb7DXm2LHD1Z9jTsHcAQMD /1jzh2BoHriy/Q2s4KzzjVp/mQO5DSm2z14BvbQRcXU48oAosHA1u3Wrov6LfPY+0U1tG47X 1BGfnQH+rNAaH0livoSBQ0IPI/8WfIW7ub4qV6HYwWKVqkDkqwcpmGNDbz3gfaDht6nsie5Z pcuCcul4M9CW7Md6zzyvktjnbz61BADGDCopfZC4of0Z3Ka0u8Wik6UJOuqShBt1WcFS8ya1 oB4rc4tXfSHyMF63aPUBMxHR5DXeH+EO2edoSwViDMqWk1jTnYza51rbGY+pebLQOVOxAY7k do5Ordl3wklBPMVEPWoZ61SdbcjhHVwaC5zfiskcxj5wwXd2E9qYlBqRg7QeSm9obiBCYWxk d2luIDxqaGJARnJlZUJTRC5vcmc+iGAEExECACAFAkTQ+awCGwMGCwkIBwMCBBUCCAMEFgID AQIeAQIXgAAKCRBy3lIGd+N/BI6RAJ9S97fvbME+3hxzE3JUyUZ6vTewDACdE1stFuSfqMvM jomvZdYxIYyTUpC5Ag0ERND5ghAIAPwsO0B7BL+bz8sLlLoQktGxXwXQfS5cInvL17Dsgnr3 1AKa94j9EnXQyPEj7u0d+LmEe6CGEGDh1OcGFTMVrof2ZzkSy4+FkZwMKJpTiqeaShMh+Goj XlwIMDxyADYvBIg3eN5YdFKaPQpfgSqhT+7El7w+wSZZD8pPQuLAnie5iz9C8iKy4/cMSOrH YUK/tO+Nhw8Jjlw94Ik0T80iEhI2t+XBVjwdfjbq3HrJ0ehqdBwukyeJRYKmbn298KOFQVHO EVbHA4rF/37jzaMadK43FgJ0SAhPPF5l4l89z5oPu0b/+5e2inA3b8J3iGZxywjM+Csq1tqz hltEc7Q+E08AAwUIAL+15XH8bPbjNJdVyg2CMl10JNW2wWg2Q6qdljeaRqeR6zFus7EZTwtX sNzs5bP8y51PSUDJbeiy2RNCNKWFMndM22TZnk3GNG45nQd4OwYK0RZVrikalmJY5Q6m7Z16 4yrZgIXFdKj2t8F+x613/SJW1lIr9/bDp4U9tw0V1g3l2dFtD3p3ZrQ3hpoDtoK70ioIAjjH aIXIAcm3FGZFXy503DOA0KaTWwvOVdYCFLm3zWuSOmrX/GsEc7ovasOWwjPn878qVjbUKWwx Q4QkF4OhUV9zPtf9tDSAZ3x7QSwoKbCoRCZ/xbyTUPyQ1VvNy/mYrBcYlzHodsaqUDjHuW+I SQQYEQIACQUCRND5ggIbDAAKCRBy3lIGd+N/BCO8AJ9j1dWVQWxw/YdTbEyrRKOY8YZNwwCf afMAg8QvmOWnHx3wl8WslCaXaE8= Message-ID: Date: Mon, 23 Mar 2020 09:48:01 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <879B19CB-5EBB-4114-8C13-199E1C2E491D@yahoo.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 16:48:02 -0000 On 3/20/20 11:02 PM, Mark Millard wrote: > While trying to build base/gcc6 on aarch64 (implicitly targeting aarch64: > self hosted), it failed with: > > . . . > c++: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated] > /wrkdirs/usr/ports/base/gcc6/work/gcc-6.5.0/gcc/config/aarch64/aarch64.md:817:10873: fatal error: bracket nesting level exceeded maximum of 256 > /wrkdirs/usr/ports/base/gcc6/work/gcc-6.5.0/gcc/config/aarch64/aarch64.md:817:10873: note: use -fbracket-depth=N to increase maximum nesting level > 116 warnings and 1 error generated. > gmake[2]: *** [Makefile:1086: insn-attrtab.o] Error 1 > gmake[2]: *** Waiting for unfinished jobs.... > . . . > > amd64 (implicitly targeting amd64: self hosted) did not have the problem. > > (These were just build-ability tests, no intent to install as stands.) > > base/binutils did not have such problems. (Actually installed on 32-bit > powerpc so more ports can build.) I think the devel/freebsd-gcc* ports have a workaround for this when being built on aarch64. We probably need the same fix for base/gcc when the build host is aarch64. -- John Baldwin From owner-freebsd-arm@freebsd.org Mon Mar 23 17:58:27 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1793426A2AF for ; Mon, 23 Mar 2020 17:58:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-55.consmr.mail.gq1.yahoo.com (sonic307-55.consmr.mail.gq1.yahoo.com [98.137.64.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48mMYr3NLvz4bN6 for ; Mon, 23 Mar 2020 17:58:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 3JlIrisVM1nLueQRsWlpqe23IcyHrHX3DbyXMOJHBWivOS7985Ha2uotlLOKhiX qTjeXSUDSUJXmIwZEI0MeojUQQY4OAeVx8Hig0DdNzzx_mLKMgNVuquORWJFPwziSy.a77ICcLnL F5.aLD4a0qzlyftDBJ8zE9kY5F0GIY.TmpzIHp.CUWkdzu.3j2XeMaTZBBUtIq5DCdkRLLb_W__I Tn57D9oXHckCD0Dxll8D.1LwXo07_N33BQYMk1_eggtR8Dr3ru7uoKJvzI.KnJCkIxe2AifC0PlD _xlomInVh5_6_cdQIZa8EmgselUl5iRWQzDUA6PIwkfd0tpeR9jJav8LY7Yd1w.rNLmbD9w74fAv rlxgIeSi8HBTmn5n59obIOvsbskE4UnawBTSrxuSYMDE9MfrkLMOMnvePlSM0TwXw4wz6v3umsAf uViqsBxXoa8pkgcIqJRoj2chg9UUUdr_sc_WTiTIAFL8f.AwSLkFnjxtEQzS.1cK.bNXhY_HztMu lj2tEfeionsQQGmvn1_18.cVzoNcWQLqFv4tkWKPKaw9w2ylHJ8ALUm5Hulp3.elKOweDFm2PibC Fq99B2irwvXnXC8YM.afPSiykS_BvaOY_xb2llnOW4IxTLYc4H5rgwLoaUYaxQUXYuZKJJieGFig RIrygdoKZ9yS_qnheDzGqIvghFYl14l1VYLscDyiERfQ8eT9r.SE745xQQlOWzNq5MEMwksx8Cwo EOyx7kz0lkfQa_42ro06bfRElmRQYh_qIcclGEgcVPy4Zlp9jl2YCHaAaAtjNUa9o8VOdgL9ThZV SpLPLTGH51ZFGxT6yJ2O7V3y2IMYwX6JJlPGtr.uvdYg60LiItEpgxMaKBRcbD39DdfokNwAkiPP _HfoZLQ.wVu8gNFxkXb7VD9yDVpvY75GBI5JdPpkbSHCPrDEL5KnwMohJABYa3ZN4Zac8FFmtvbc 2x5MbMUlijwbd_boq6XE1hcLzwiFEEpLNTHEJdqQxmD5024WWSzER.AN0aNBgmvSkTEzyHBstumJ 8FDGGUbTzCB08DJ4GQpzXQWJDV2ptYo8z_FiDEm6Zu39GUpEtffeR.C.016es07TWGrJc6eCHvKD L5O1e.jMHd_Ffbn0m9iHtlVYDRuciKytbYo6gcePkIvkOmAP5O9ILMRodvIoRdOKEWjHybmxejfC ctAIyY5D7XdtZX0LVhBLfFZUgUmcRHfTRmRZk4W21cf38Li.uAxhmEaQCMH0Rr6dnTohz0VnC6Lv x7LHkI9KM3fJ8_Gxndj6KxZGFWijMarIVwP5QWQ93Yb42K8tatfSvgoIKG7k28lJVwwsJmmF5WgO Bv6Dx2XClTBbJAhS0ZjmA8MY9NugwWXfs558wby6L2BeCsfSvwP9dN.pbzx4VPY6N_Q1cWCChlH6 e7hTEH7xaunpmf2KMgfh5OYmmxZVGJjysFU3XdpTyf8C31RRtWzh5nJcT Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Mon, 23 Mar 2020 17:58:22 +0000 Received: by smtp403.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID bb58a9524206733e3493d3ff70e37150; Mon, 23 Mar 2020 17:58:18 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: head -r358966 on aarch64 fails to build base/gcc6: fatal error: bracket nesting level exceeded maximum of 256 From: Mark Millard In-Reply-To: Date: Mon, 23 Mar 2020 10:58:17 -0700 Cc: FreeBSD Toolchain , freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <1FFE6B7C-ED7D-4BA6-B276-F401E0913BAF@yahoo.com> References: <879B19CB-5EBB-4114-8C13-199E1C2E491D.ref@yahoo.com> <879B19CB-5EBB-4114-8C13-199E1C2E491D@yahoo.com> To: John Baldwin X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48mMYr3NLvz4bN6 X-Spamd-Bar: + X-Spamd-Result: default: False [1.38 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_SPAM_MEDIUM(0.92)[0.925,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCVD_IN_DNSWL_NONE(0.00)[31.64.137.98.list.dnswl.org : 127.0.5.0]; NEURAL_SPAM_LONG(0.96)[0.957,0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (4.19), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 17:58:27 -0000 On 2020-Mar-23, at 09:48, John Baldwin wrote: > On 3/20/20 11:02 PM, Mark Millard wrote: >> While trying to build base/gcc6 on aarch64 (implicitly targeting = aarch64: >> self hosted), it failed with: >>=20 >> . . . >> c++: warning: treating 'c' input as 'c++' when in C++ mode, this = behavior is deprecated [-Wdeprecated] >> = /wrkdirs/usr/ports/base/gcc6/work/gcc-6.5.0/gcc/config/aarch64/aarch64.md:= 817:10873: fatal error: bracket nesting level exceeded maximum of 256 >> = /wrkdirs/usr/ports/base/gcc6/work/gcc-6.5.0/gcc/config/aarch64/aarch64.md:= 817:10873: note: use -fbracket-depth=3DN to increase maximum nesting = level >> 116 warnings and 1 error generated. >> gmake[2]: *** [Makefile:1086: insn-attrtab.o] Error 1 >> gmake[2]: *** Waiting for unfinished jobs.... >> . . . >>=20 >> amd64 (implicitly targeting amd64: self hosted) did not have the = problem. >>=20 >> (These were just build-ability tests, no intent to install as = stands.) >>=20 >> base/binutils did not have such problems. (Actually installed on = 32-bit >> powerpc so more ports can build.) >=20 > I think the devel/freebsd-gcc* ports have a workaround for this when = being built > on aarch64. We probably need the same fix for base/gcc when the build = host is > aarch64. Looks like in devel/freebsd-gcc* such code is like: .if ${TARGETARCH} =3D=3D "armv6" || ${TARGETARCH} =3D=3D "aarch64" . if ${COMPILER_TYPE} =3D=3D clang MAKE_ARGS+=3DCXXFLAGS=3D-fbracket-depth=3D512 . endif .endif There is no armv6 flavor in FLAVORS, nor armv7. But having armv6 above but not armv7 looks a little odd. (When I later tried base/gcc6 on an armv7 it also had the problem.) Also, TARGETARCH seems to be the target, not the host. All hosts using clang get the larger bracket depth for handling the listed arm targets, if I read the above right. base/gcc6 does not use FLAVOR and has: TARGETARCH=3D ${ARCH:S/amd64/x86_64/} So, again, it looks like explicitly covering "armv7" would be appropriate for base/gcc* examples that can handle armv7 reasonably. (Unsure for gcc6.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Mon Mar 23 21:26:39 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 73F52270037 for ; Mon, 23 Mar 2020 21:26:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48mSB46KF9z4nrL for ; Mon, 23 Mar 2020 21:26:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: Zzsxy4QVM1kUQikPqaQYLA_zk_bM238VGEwM_DOqa5rR9kts1lP8IQdTe6ZsVT8 nPur7sjusySi2fgyVPsdY1VyoBGmzAJ4LLQsqXJm.jSjqZKiqawfu9ApDLMbA5dK4wTIFIi90.bo LIFe3Qgv5Nmlb.dyitHftytjkP9w0fdyBCJR__0Mhi5nP.cGziPBzhf37IqON0itfnOWB6t0U5Uw vBOY0OUnQfBKab2Whg1ZgrmRzel_sh_lMhPA5DxHDrjWHnrbkE0w45lQ7GB3Vsnv_HnEBq2pCdlz par9x8ezjCpMd0sp5CIR1ZfZEKEqNczAWwFVWhCFXvo8FOSyEZri547ytcq_htbjh7qkVExPjRIP ICmtrHcF9wO9UOXF8hhS7G0bqX8XMKenfpb9kcn9d0kbNE1mRwqRwZ3deURtCzxobm0mZInZcSes xanmz4gKt.86l4z7sKmzB9R.0Sqiw1N.A6ZFyljAI8DL2GsqwvuzekJ8xZcOx5yBXpcJgy9VVm99 Ze72vxBi_nh2lsWMCC1d5vp0NXr4qtGBJyGySG0cQ8L4qO.1dDJnVrkl2g0qvR3KDyLyqp3CpiJB wmQUZ.M_jqeK5E4imlFcFg6Gghb2UNvwdgxNpDwbgdlPdCYob_alhBbPXhSCBcuj3DUqQ_J_5lqV KVszGvgzRDXCPgJH9hfLFhGYSPGkH6Lj7pk6ZBYmldcq8qgsRbAcoClsQlxnN_FDMvxBl1VamU2y Psq5SHtSt8ICaD1HpZAhS2_ibizavJXnfqUtzyhKQS4G4q9g6T.D3f.YdWYz7wnn700gJ3P17niT xkhKo6SILlYAoqoKuBFPSM6fdjXMPAtl49I.ENe_QCdGJBjFKPC24dznwapBxg_I9kKixGKVVnGB P0iGvk.HxEUzO5iLt4I4KWWx9iVKZBnY4fngbOGaZGxL5XzafRlDOgrlhTdCf77msEybeYONd494 V712hUOhsJqL9y9Y.4PVW.h4gzh25NK5HAYdbutYHjTQPgUqMmGLEbxP2jT2q53SSNCAF3CMKooO 73jTXs_dSwf9ldQT5o7lY6Zwlp.Y9oHB3sWOVYoU6m.YWIUxlaJGrx0DtEJDDQpH_JieXuTBMjwN QeNA9JgqN_rkJq8KU8.x6Om_O1UXWR2QJbW0K.TiA1YBtIKuPKShGTse8DjIh8vbWU0kwPbXPjKU KcsDOoOlLvwjjoH7zHFowowxFJde5d2eubMSmO_K525z1LuaN1.7NNmcaTHUow4a3gAJVsonAIBn DedZSzmqImGYmB7eyi4LnUcK3k3LjaAhGWKlvTZ38aI8a6zpGMyGUrn8vnZ5xu2IDWVdrJKWjMDz OV5vx6i9_vdLGLLieUQwMZ0pivK21NGGQYFZbRjXRPKCsEO21uONmNHb_p0XT0xKJmhpC8HMKuye jxEJ8iwpnURpG9Vfp9Rhppm8ARcZJ8Q-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Mon, 23 Mar 2020 21:26:33 +0000 Received: by smtp424.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID ef3749e4e6381ba8706626ddf7cbee4e; Mon, 23 Mar 2020 21:26:31 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: amd64->armv7 cross-build failure for security/ca_root_nss: It failed in memcpy () from /libexec/ld-elf.so.1 From: Mark Millard In-Reply-To: <1EC37157-CBA2-4334-92C1-E845F63DB5CA@yahoo.com> Date: Mon, 23 Mar 2020 14:26:30 -0700 Cc: FreeBSD ports Content-Transfer-Encoding: quoted-printable Message-Id: References: <1EC37157-CBA2-4334-92C1-E845F63DB5CA@yahoo.com> To: freebsd-arm , FreeBSD Hackers X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48mSB46KF9z4nrL X-Spamd-Bar: + X-Spamd-Result: default: False [1.30 / 15.00]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (3.27), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.65), country: US(-0.05)]; NEURAL_SPAM_MEDIUM(0.88)[0.885,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.91)[0.914,0]; RCVD_IN_DNSWL_NONE(0.00)[147.69.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[147.69.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 21:26:39 -0000 On 2020-Mar-16, at 21:48, Mark Millard wrote: > Context: head -r358966 attempting to update ports > to -r528535 . Also, 50+ ports built just fine > but the below has been repeatable in my context. >=20 > The original failure was under devel/poudriere-devel (with > nxb-bin/ materials used). But part of the below is from > exploring with various steps in a handier context. >=20 > The original error message was: >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D> Building for ca_root_nss-3.51 > ## Untrusted certificates omitted from this bundle: 2 > openssl x509 failed with exit code 11 at = /wrkdirs/usr/ports/security/ca_root_nss/work/MAca-bundle.pl line 78. > *** Error code 255 >=20 > The original source that reported the message was: >=20 > sub printcert_info($$) > { > my (undef, $certdata) =3D @_; > return unless $certdata; > open(OUT, "|openssl x509 -text -inform DER -fingerprint") > || die "could not pipe to openssl x509"; > print OUT $certdata; > close(OUT) or die "openssl x509 failed with exit code $?"; > } >=20 > The die produced: >=20 > -rw-r--r-- 1 root wheel 7909376 Mar 17 03:18:04 2020 = qemu_openssl.core >=20 > gdb reported for it: >=20 > Core was generated by `openssl'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0xf501adb4 in memcpy () from /libexec/ld-elf.so.1 >=20 > and: >=20 > (gdb) info threads > Id Target Id Frame=20 > * 1 LWP 1592 "x509" 0xf501adb4 in memcpy () from = /libexec/ld-elf.so.1 >=20 > and: >=20 > gdb) bt > #0 0xf501adb4 in memcpy () from /libexec/ld-elf.so.1 > #1 0xf5004cd0 in do_copy_relocations () from /libexec/ld-elf.so.1 >=20 > and (from a disass): >=20 > =3D> 0xf501adb4 <+436>: strd r4, [r3], #8 >=20 > (It was not clear what code context to supply so > I stuck to showing the instruction with the register > used such that SIGSEGV could result from the use: r3 .) >=20 > Finally the registers were listed as holding: >=20 > (gdb) info reg > r0 0xf4f5d57c 4109751676 > r1 0x14 20 > r2 0x93000 602112 > r3 0x1 1 > r4 0x10 16 > r5 0x9fffdfa4 2684346276 > r6 0xf4fe2404 4110296068 > r7 0xf4fe2004 4110295044 > r8 0x93000 602112 > r9 0x93000 602112 > r10 0x9fffdfe0 2684346336 > r11 0x0 0 > r12 0x9fffdf80 2684346240 > sp 0x9fffdf80 0x9fffdf80 > lr 0xf5004cd0 4110437584 > pc 0xf501adb4 0xf501adb4 > cpsr 0x60000010 1610612752 >=20 > Yep: r3=3D=3D1 would do it. >=20 >=20 > Note: I've otherwise ignored here seeing lots of: >=20 > qemu: unsupported syscall: 574 (calling anyway) >=20 > notices while doing things for extracting > this information. >=20 >=20 > I'll note that I had no such SIGSEGV when > ca_root_nss 3.50 built back at OSVERSION=3D1300077 > on 2020-Feb-16: it built and worked fine back > then. >=20 >=20 >=20 > I'm not sure when I'll have time to do more with this > or if I will again just abandon qemu-user-static for > a time. (Insufficient time to allocate to do more?) > Hopefully the basic information is useful to someone > at some point. >=20 > I'm not claiming that I know qemu-user-static is the > problem, or openssl, or whatever. Just that the > combination is broken in my context. >=20 > Having security/ca_root_nss blocked, blocks > cross-building lots of other things, including > devel/llvm10 . Using: # poudriere bulk -j FBSDFSSDjailArmV7 -i -w ports-mgmt/portmaster to allow trying things from inside the poudriere build environment, I tried: # openssl=20 qemu: uncaught target signal 11 (Segmentation fault) - core dumped # file `which openssl` /usr/bin/openssl: ELF 32-bit LSB executable, ARM, EABI5 version 1 = (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for = FreeBSD 13.0 (1300084), FreeBSD-style, not stripped The backtrace was again memcpy and do_copy_relocations. (So "x509" had nothing to do with the inability to run the original failed command.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Mon Mar 23 21:48:56 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C6ED8271078 for ; Mon, 23 Mar 2020 21:48:56 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48mSgq0k7qz3yNw for ; Mon, 23 Mar 2020 21:48:54 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.110.112]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1jGUwG-0004Qi-5Z for freebsd-arm@freebsd.org; Mon, 23 Mar 2020 22:48:52 +0100 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: freebsd-arm@freebsd.org Date: Mon, 23 Mar 2020 22:48:44 +0100 Subject: arm64 pkg build broken since January MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-As-Hash: bdb49c4ff80bd276e321aade33e76e02752072e2 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50 autolearn=disabled version=3.4.2 X-Scan-Signature: c74461a82029b6293650421ecb57b64a X-Rspamd-Queue-Id: 48mSgq0k7qz3yNw X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 195.190.28.88 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.70)[-0.696,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.190.28.64/27]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[klop.ws]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.79)[-0.786,0]; IP_SCORE(-0.71)[ip: (-0.64), ipnet: 195.190.28.0/24(-0.25), asn: 47172(-2.71), country: NL(0.03)]; RCVD_IN_DNSWL_NONE(0.00)[88.28.190.195.list.dnswl.org : 127.0.10.0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:47172, ipnet:195.190.28.0/24, country:NL]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 21:48:56 -0000 Hi, AFAIK the pkg build of arm64/aarch64 is broken for quite some time. ports-mgmt/pkg does not build on the build cluster since January 13th http://thunderx1.nyi.freebsd.org/data/head-arm64-default/p522076_s356364/logs/errors/pkg-1.12.0.log pkgs are not building anymore since February 28th. https://pkg-status.freebsd.org/builds?jailname=head-arm64 pkgs for 12.X also stopped building for some time now. http://thunderx1.nyi.freebsd.org/ I made a bugreport about it, but I see no confirmation or anything. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244577 Am I the only one without pkg updates on arm64? Or do I look wrong at the wrong places? Who could I contact about this? Regards, Ronald. From owner-freebsd-arm@freebsd.org Tue Mar 24 15:57:55 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 999C0260934 for ; Tue, 24 Mar 2020 15:57:55 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48mwr73nbTz4NMX for ; Tue, 24 Mar 2020 15:57:41 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id 02OFvr5m092021 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Mar 2020 08:57:54 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id 02OFvrab092020; Tue, 24 Mar 2020 08:57:53 -0700 (PDT) (envelope-from fbsd) Date: Tue, 24 Mar 2020 08:57:53 -0700 From: bob prohaska To: freebsd-arm@freebsd.org Subject: Belated out of swap kill on rpi3 at r359216 Message-ID: <20200324155753.GA91922@www.zefox.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Queue-Id: 48mwr73nbTz4NMX X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [2.85 / 15.00]; ARC_NA(0.00)[]; WWW_DOT_DOMAIN(0.50)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(0.06)[ip: (0.24), ipnet: 50.1.16.0/20(0.12), asn: 7065(-0.03), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.24)[0.244,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.65)[0.652,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MID_RHS_WWW(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 15:57:55 -0000 An attempt to buildworld on an rpi3 running r359216 stopped with an OOMA kill. Sources were at 359264, loader.conf contained vm.pageout_oom_seq="4096" . A snipped of gstat log suggests the worst congestion in the storage I/O happened at Tue Mar 24 04:52:26 PDT 2020 with an L(q) of 37, but the OOMA kill happened at Tue Mar 24 04:53:04 PDT 2020, by which time the L(q) had dropped to one, half a minute later. Is the delay in OOMA action to be expected? Here's the relevant part of the log, I hope the columns display readably: 0/2/2/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 0 0 29 1897320 47312 12851 9 4 5 13330 1669 0 0 14172 8020 3034 65 29 6 dT: 1.056s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 37 367 323 5463 6.4 45 1511 76.8 0 0 0.0 91.7 mmcsd0 37 367 323 5463 6.5 45 1511 76.9 0 0 0.0 93.3 mmcsd0s2 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.0 mmcsd0s2a 3 235 212 2254 5.9 23 814 21.5 0 0 0.0 70.0 mmcsd0s2b 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.2 ufs/rootfs Tue Mar 24 04:52:26 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 224484 4179768 5% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 0/0/0/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 3 0 29 1936132 19232 12849 9 4 5 13328 1671 0 0 14173 8020 3036 65 29 6 dT: 1.010s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 9 323 148 2531 20.7 175 3327 14.6 0 0 0.0 97.2 mmcsd0 9 323 148 2531 20.8 175 3327 14.6 0 0 0.0 98.4 mmcsd0s2 2 96 96 1869 16.9 0 0 0.0 0 0 0.0 82.9 mmcsd0s2a 7 227 51 661 28.3 175 3327 14.6 0 0 0.0 84.0 mmcsd0s2b 2 96 96 1869 16.9 0 0 0.0 0 0 0.0 83.0 ufs/rootfs Tue Mar 24 04:52:41 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 272928 4131324 6% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 0/0/0/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 0 0 29 1939940 48016 12849 9 4 5 13328 1676 948 0 14174 8019 3037 65 29 6 dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 3 371 151 1989 91.0 221 3847 14.0 0 0 0.0 100.1 mmcsd0 3 371 151 1989 91.1 221 3847 14.0 0 0 0.0 100.1 mmcsd0s2 1 41 41 415 74.5 0 0 0.0 0 0 0.0 304.8 mmcsd0s2a 2 331 110 1574 97.3 221 3847 14.0 0 0 0.0 99.3 mmcsd0s2b 1 41 41 415 74.5 0 0 0.0 0 0 0.0 304.9 ufs/rootfs Tue Mar 24 04:52:45 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 282480 4121772 6% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 0/2/2/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 2 0 29 1944652 42768 12849 9 4 5 13328 1678 0 0 14174 8019 3037 65 29 6 dT: 1.010s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 2 498 498 6345 3.5 0 0 0.0 0 0 0.0 88.8 mmcsd0 2 498 498 6345 3.9 0 0 0.0 0 0 0.0 98.3 mmcsd0s2 1 152 152 1501 4.1 0 0 0.0 0 0 0.0 62.1 mmcsd0s2a 1 346 346 4844 3.9 0 0 0.0 0 0 0.0 88.8 mmcsd0s2b 1 152 152 1501 4.2 0 0 0.0 0 0 0.0 62.5 ufs/rootfs Tue Mar 24 04:52:47 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 282396 4121856 6% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 0/0/0/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 5 0 28 1980200 19676 12848 9 4 5 13327 1684 714 0 14175 8019 3039 65 29 6 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 9 487 174 1449 26.2 313 2427 9.1 0 0 0.0 98.4 mmcsd0 9 487 174 1449 26.4 313 2427 9.1 0 0 0.0 100.0 mmcsd0s2 1 45 45 267 22.9 0 0 0.0 0 0 0.0 80.3 mmcsd0s2a 9 442 129 1182 27.7 313 2427 9.1 0 0 0.0 97.2 mmcsd0s2b 1 45 45 267 22.9 0 0 0.0 0 0 0.0 80.4 ufs/rootfs Tue Mar 24 04:53:01 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 309996 4094256 7% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 0/2/2/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 0 0 29 1983636 44400 12848 9 4 5 13326 1689 0 0 14175 8018 3040 65 29 6 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 12 528 0 0 0.0 528 3797 7.5 0 0 0.0 100.0 mmcsd0 12 528 0 0 0.0 528 3797 7.5 0 0 0.0 100.0 mmcsd0s2 11 528 0 0 0.0 528 3797 7.5 0 0 0.0 100.0 mmcsd0s2b Tue Mar 24 04:53:05 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 322360 4081892 7% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 3/769/772/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 1 0 29 1826272 165204 12848 9 4 5 13326 1689 971 0 14176 8018 3040 65 29 6 dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 1 614 604 4661 4.9 10 168 40.6 0 0 0.0 93.8 mmcsd0 1 614 604 4661 4.9 10 168 40.6 0 0 0.0 94.5 mmcsd0s2 0 210 209 1243 4.1 1 32 181.0 0 0 0.0 83.1 mmcsd0s2a 1 404 395 3418 5.4 9 136 25.1 0 0 0.0 81.0 mmcsd0s2b 0 210 209 1243 4.1 1 32 181.0 0 0 0.0 83.6 ufs/rootfs Tue Mar 24 04:53:08 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 230064 4174188 5% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 Mar 24 04:53:04 www kernel: pid 96708 (c++), jid 0, uid 0, was killed: out of swap space 0/772/772/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 2 0 28 1705884 221144 12848 9 4 5 13326 1689 1109 0 14176 8018 3041 65 29 6 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 545 545 3580 2.1 0 0 0.0 0 0 0.0 74.0 mmcsd0 0 545 545 3580 2.1 0 0 0.0 0 0 0.0 75.7 mmcsd0s2 0 288 288 1517 2.0 0 0 0.0 0 0 0.0 56.4 mmcsd0s2a 0 256 256 2064 2.4 0 0 0.0 0 0 0.0 49.0 mmcsd0s2b 0 288 288 1517 2.0 0 0 0.0 0 0 0.0 57.1 ufs/rootfs Tue Mar 24 04:53:10 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 230064 4174188 5% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 Mar 24 04:53:04 www kernel: pid 96708 (c++), jid 0, uid 0, was killed: out of swap space [end of excerpt] Thanks for looking! bob prohaska From owner-freebsd-arm@freebsd.org Tue Mar 24 17:09:58 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8F143262278 for ; Tue, 24 Mar 2020 17:09:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-24.consmr.mail.gq1.yahoo.com (sonic311-24.consmr.mail.gq1.yahoo.com [98.137.65.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48myRH1GY7z3Lmt for ; Tue, 24 Mar 2020 17:09:45 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 3NcAURgVM1n4QNtRJ5jV4GwUha.eqDqTIbkHe3yZrhSru9ryUm2hYVMAYEEOhKz X9z8tbQQa53qMY8ZMkOEEZuN7Bj_a1FQE6IV80UsYpfbHHjqVNkXsQudtLiQyyOJQYRBJ2OOo32n R2ThG.fzc3vt4kGDLkEsEQy4cJ4hRaWMwT.MZy0R3fJ78NQHL.oUlcTTuADJpXDcQ0Fnt5u0LkP6 d8rVOEHqVPa2E75nvVF8u6Ok4jrKy0E_5jW_CGn1ZZaSuILzS6B9QaAHcOU7.j2v3NqDv3tovs4W faocyTx0A5rF5e6q_HY_EMzLCnjqWnnGzJJt8QQuNoGFL8lcxqRBFOit1FskzKHVeH_LQ8Z1xbjX v6HSsKPmrub59ZC9S9LCAvE3gVw5o1RxXo1e.DMTGafiIdPpiU6qblaUq9WQvXu5IalNO1.SSRf9 fZSvEd5ofaZ0ovEvCkjufaE4Fec4xAlP64MWvxef1mpS6n50ObCqjvC1G8AO1ARLw0mxqgh6hZ5O Pn3ikBdd8pDc.C.yB.jdfUOu.ZZx3Ke7Ki9xw_R1lyKdqa2Rgo2.9MsLyYFcqfpfVW6mgyBnau1o Log.fUWFcxu7t86vlBSv8BlMwK4oxQ9OaO.xBEkCVDuTFbi_1FcZ3lQNNU_z53DgQ_CyknmqxkwB _PbAdjGtb79_tQ3JNhhP6aGHAKOFAhzOHHLq3yq2FI0lEaYCbyXZwKqfuhWT7IJ_mvCICssgfizQ ZkzQ2M4daTHHyTn8fTXnHVBkoTHOIVnJ4D6avBEgU3E9bW7PwN29MxWnycRjl0m4qOgvyl6rXzIY RVMf12nctLIZlWqjXU7faOZBUAtytBnfPXDD18aHTSjxgWUogxwh8IX73vXGKmRq.HyF6xl5MG3g toDiJd2720ez8gl3Iu5jpfnxjnYR3JPQcchHOVIMno_gF1Fxm.8CpgbHlm2GxHnzKXPwh1ls4PGV bI_ryhomQXR1zHTNQMUE0pLJX9IjZdG5q9LV12y3IjFPlrjj2bUjyidi78.34mZTfeWP5.pNmU5b M6CPKMo.s6CgLX4vrlYwrkUsvlO_HOgO3a0IiEIrFMR5t8tNmvkTxw76QFK7_pRGnNdxoldp4uQV 9uvuAm1fQBNCmTaBEKV8E2SjXmAaNI6QTXBsgdirhJcdK8Tm8DuRgNQglpHCIrJ3FIV9jGGO3r99 1EEchc8qIRzC0TNNndZ3AZK1qQtSBYGTqQ0uEdrVSFKdp2HF40g6hro_b2V1dEqNlIQdaFRaJuxG N3SUYB2Ajpx3F.N7qYYFckA.7fVUWTL9b2Mg5qrFDQUFVRaFObzyl7Hg9dBSy_AOfg5QH.539nqt VhACfHi.QHSubyjOjGkQ5RNARxu8VZHh3wULw2vnKzCvjVFbd8Hbq48hCjE1rrt_1Fro9.ZRIQRE UQL4LvAFJizwrpEWm3Z3SeVXoP0GTPaLC9QTQ452ELFCt9tKNj.x14vad Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Tue, 24 Mar 2020 17:09:35 +0000 Received: by smtp414.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID b71740cd43012fc2b0f0e710945367f4; Tue, 24 Mar 2020 17:09:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <20200324155753.GA91922@www.zefox.net> Date: Tue, 24 Mar 2020 10:09:28 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> References: <20200324155753.GA91922@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48myRH1GY7z3Lmt X-Spamd-Bar: - X-Spamd-Result: default: False [-1.53 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.45)[-0.451,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; URIBL_BLOCKED(0.00)[dsl-only.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.58)[-0.583,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (4.47), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[205.65.137.98.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 17:09:58 -0000 On 2020-Mar-24, at 08:57, bob prohaska wrote: > An attempt to buildworld on an rpi3 running r359216 stopped with an > OOMA kill. Sources were at 359264, loader.conf contained > vm.pageout_oom_seq=3D"4096" .=20 What of the value of vm.pfault_oom_attempts ? If vm.pfault_oom_attempts was not -1, what of the value of vm.pfault_oom_wait ? > A snipped of gstat log suggests the worst congestion in the storage = I/O > happened at Tue Mar 24 04:52:26 PDT 2020 with an L(q) of 37, but the > OOMA kill happened at Tue Mar 24 04:53:04 PDT 2020, by which time the > L(q) had dropped to one, half a minute later. >=20 > Is the delay in OOMA action to be expected?=20 >=20 > Here's the relevant part of the log, I hope the columns display = readably: >=20 > 0/2/2/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 0 0 29 1897320 47312 12851 9 4 5 13330 1669 0 0 14172 = 8020 3034 65 29 6 > dT: 1.056s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 37 367 323 5463 6.4 45 1511 76.8 0 0 = 0.0 91.7 mmcsd0 > 37 367 323 5463 6.5 45 1511 76.9 0 0 = 0.0 93.3 mmcsd0s2 > 34 133 111 3209 7.6 22 697 134.7 0 0 = 0.0 74.0 mmcsd0s2a > 3 235 212 2254 5.9 23 814 21.5 0 0 = 0.0 70.0 mmcsd0s2b > 34 133 111 3209 7.6 22 697 134.7 0 0 = 0.0 74.2 ufs/rootfs > Tue Mar 24 04:52:26 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 224484 4179768 5% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 How bad were things back when the "indefinate wait buffer" notices were generated (Mar 24 04:20:50 time frame)? > 0/0/0/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 3 0 29 1936132 19232 12849 9 4 5 13328 1671 0 0 14173 = 8020 3036 65 29 6 > dT: 1.010s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 9 323 148 2531 20.7 175 3327 14.6 0 0 = 0.0 97.2 mmcsd0 > 9 323 148 2531 20.8 175 3327 14.6 0 0 = 0.0 98.4 mmcsd0s2 > 2 96 96 1869 16.9 0 0 0.0 0 0 = 0.0 82.9 mmcsd0s2a > 7 227 51 661 28.3 175 3327 14.6 0 0 = 0.0 84.0 mmcsd0s2b > 2 96 96 1869 16.9 0 0 0.0 0 0 = 0.0 83.0 ufs/rootfs > Tue Mar 24 04:52:41 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 272928 4131324 6% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 > 0/0/0/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 0 0 29 1939940 48016 12849 9 4 5 13328 1676 948 0 14174 = 8019 3037 65 29 6 > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 3 371 151 1989 91.0 221 3847 14.0 0 0 = 0.0 100.1 mmcsd0 > 3 371 151 1989 91.1 221 3847 14.0 0 0 = 0.0 100.1 mmcsd0s2 > 1 41 41 415 74.5 0 0 0.0 0 0 = 0.0 304.8 mmcsd0s2a > 2 331 110 1574 97.3 221 3847 14.0 0 0 = 0.0 99.3 mmcsd0s2b > 1 41 41 415 74.5 0 0 0.0 0 0 = 0.0 304.9 ufs/rootfs > Tue Mar 24 04:52:45 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 282480 4121772 6% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 > 0/2/2/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 2 0 29 1944652 42768 12849 9 4 5 13328 1678 0 0 14174 = 8019 3037 65 29 6 > dT: 1.010s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 2 498 498 6345 3.5 0 0 0.0 0 0 = 0.0 88.8 mmcsd0 > 2 498 498 6345 3.9 0 0 0.0 0 0 = 0.0 98.3 mmcsd0s2 > 1 152 152 1501 4.1 0 0 0.0 0 0 = 0.0 62.1 mmcsd0s2a > 1 346 346 4844 3.9 0 0 0.0 0 0 = 0.0 88.8 mmcsd0s2b > 1 152 152 1501 4.2 0 0 0.0 0 0 = 0.0 62.5 ufs/rootfs > Tue Mar 24 04:52:47 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 282396 4121856 6% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 > 0/0/0/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 5 0 28 1980200 19676 12848 9 4 5 13327 1684 714 0 14175 = 8019 3039 65 29 6 > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 9 487 174 1449 26.2 313 2427 9.1 0 0 = 0.0 98.4 mmcsd0 > 9 487 174 1449 26.4 313 2427 9.1 0 0 = 0.0 100.0 mmcsd0s2 > 1 45 45 267 22.9 0 0 0.0 0 0 = 0.0 80.3 mmcsd0s2a > 9 442 129 1182 27.7 313 2427 9.1 0 0 = 0.0 97.2 mmcsd0s2b > 1 45 45 267 22.9 0 0 0.0 0 0 = 0.0 80.4 ufs/rootfs > Tue Mar 24 04:53:01 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 309996 4094256 7% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 > 0/2/2/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 0 0 29 1983636 44400 12848 9 4 5 13326 1689 0 0 14175 = 8018 3040 65 29 6 > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 12 528 0 0 0.0 528 3797 7.5 0 0 = 0.0 100.0 mmcsd0 > 12 528 0 0 0.0 528 3797 7.5 0 0 = 0.0 100.0 mmcsd0s2 > 11 528 0 0 0.0 528 3797 7.5 0 0 = 0.0 100.0 mmcsd0s2b > Tue Mar 24 04:53:05 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 322360 4081892 7% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 > 3/769/772/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 1 0 29 1826272 165204 12848 9 4 5 13326 1689 971 0 14176 = 8018 3040 65 29 6 > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 1 614 604 4661 4.9 10 168 40.6 0 0 = 0.0 93.8 mmcsd0 > 1 614 604 4661 4.9 10 168 40.6 0 0 = 0.0 94.5 mmcsd0s2 > 0 210 209 1243 4.1 1 32 181.0 0 0 = 0.0 83.1 mmcsd0s2a > 1 404 395 3418 5.4 9 136 25.1 0 0 = 0.0 81.0 mmcsd0s2b > 0 210 209 1243 4.1 1 32 181.0 0 0 = 0.0 83.6 ufs/rootfs > Tue Mar 24 04:53:08 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 230064 4174188 5% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 > Mar 24 04:53:04 www kernel: pid 96708 (c++), jid 0, uid 0, was killed: = out of swap space > 0/772/772/19177 mbuf clusters in use (current/cache/total/max) > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 2 0 28 1705884 221144 12848 9 4 5 13326 1689 1109 0 14176 = 8018 3041 65 29 6 > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 0 545 545 3580 2.1 0 0 0.0 0 0 = 0.0 74.0 mmcsd0 > 0 545 545 3580 2.1 0 0 0.0 0 0 = 0.0 75.7 mmcsd0s2 > 0 288 288 1517 2.0 0 0 0.0 0 0 = 0.0 56.4 mmcsd0s2a > 0 256 256 2064 2.4 0 0 0.0 0 0 = 0.0 49.0 mmcsd0s2b > 0 288 288 1517 2.0 0 0 0.0 0 0 = 0.0 57.1 ufs/rootfs > Tue Mar 24 04:53:10 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 230064 4174188 5% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 > Mar 24 04:53:04 www kernel: pid 96708 (c++), jid 0, uid 0, was killed: = out of swap space >=20 Below I show code changes to be more explicit in the output messaging about what contributes to initiating OOM kills, without needing boot verbose or the like. There are also some messages from Mark J.'s old code for reporting on things related to initiating OOM kills, or some minor variations of them. You may want to try such changes to provide more context for your OOM kills when they happen. Below the 4 reasons reported are (showing the most detailed of the related messages from different stages): swp_pager_meta_build: swap blk uma zone exhausted swp_pager_meta_build: swap pctrie uma zone exhausted vm_fault_allocate: proc %d (%s) failed to alloc page on fault, starting = OOM m_pageout_mightbe_oom: kill context: v_free_count: %u, v_inactive_count: = %u The below is based on my head -r358966 context for the detailed diff's. # svnlite diff /usr/src/sys/vm/ | more Index: /usr/src/sys/vm/swap_pager.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/swap_pager.c (revision 358966) +++ /usr/src/sys/vm/swap_pager.c (working copy) @@ -1998,6 +1998,7 @@ 0, 1)) printf("swap blk zone exhausted, = " "increase = kern.maxswzone\n"); + printf("swp_pager_meta_build: swap blk = uma zone exhausted\n"); vm_pageout_oom(VM_OOM_SWAPZ); pause("swzonxb", 10); } else @@ -2028,6 +2029,7 @@ 0, 1)) printf("swap pctrie zone = exhausted, " "increase = kern.maxswzone\n"); + printf("swp_pager_meta_build: swap = pctrie uma zone exhausted\n"); vm_pageout_oom(VM_OOM_SWAPZ); pause("swzonxp", 10); } else Index: /usr/src/sys/vm/vm_fault.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/vm_fault.c (revision 358966) +++ /usr/src/sys/vm/vm_fault.c (working copy) @@ -1074,9 +1074,8 @@ fs->oom++; vm_waitpfault(dset, vm_pfault_oom_wait * hz); } else { - if (bootverbose) - printf( - "proc %d (%s) failed to alloc page on fault, starting = OOM\n", + printf( + "vm_fault_allocate: proc %d (%s) failed to alloc page on = fault, starting OOM\n", curproc->p_pid, curproc->p_comm); vm_pageout_oom(VM_OOM_MEM_PF); fs->oom =3D 0; Index: /usr/src/sys/vm/vm_page.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/vm_page.c (revision 358966) +++ /usr/src/sys/vm/vm_page.c (working copy) @@ -3134,6 +3134,7 @@ * race-free vm_wait_domain(). */ if (curproc =3D=3D pageproc) { + printf("thread %d waiting for memory\n", = curthread->td_tid); mtx_lock(&vm_domainset_lock); vm_pageproc_waiters++; msleep(&vm_pageproc_waiters, &vm_domainset_lock, PVM | = PDROP, Index: /usr/src/sys/vm/vm_pageout.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/vm_pageout.c (revision 358966) +++ /usr/src/sys/vm/vm_pageout.c (working copy) @@ -1741,6 +1741,8 @@ * start OOM. Initiate the selection and signaling of the * victim. */ + printf("vm_pageout_mightbe_oom: kill context: v_free_count: %u, = v_inactive_count: %u\n", + vmd->vmd_free_count, = vmd->vmd_pagequeues[PQ_INACTIVE].pq_cnt); vm_pageout_oom(VM_OOM_MEM); =20 /* @@ -1933,10 +1935,24 @@ } sx_sunlock(&allproc_lock); if (bigproc !=3D NULL) { + char *reason_text; + switch (shortage) { + case VM_OOM_MEM_PF: + reason_text=3D "fault's page allocation failed"; + break; + case VM_OOM_MEM: + reason_text=3D "free RAM stayed below = threshold"; + break; + case VM_OOM_SWAPZ: + reason_text=3D "swblk or swpctrie zone = exhausted"; + break; + default: + reason_text=3D "Unknown Reason"; + } if (vm_panic_on_oom !=3D 0 && --vm_panic_on_oom =3D=3D = 0) - panic("out of swap space"); + panic("%s",reason_text); PROC_LOCK(bigproc); - killproc(bigproc, "out of swap space"); + killproc(bigproc, reason_text); sched_nice(bigproc, PRIO_MIN); _PRELE(bigproc); PROC_UNLOCK(bigproc); =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Tue Mar 24 18:55:10 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 696BF2673F7 for ; Tue, 24 Mar 2020 18:55:10 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48n0mn2DPgz4WZw for ; Tue, 24 Mar 2020 18:55:04 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id 02OItJ0Y092406 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Mar 2020 11:55:20 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id 02OItJKb092405; Tue, 24 Mar 2020 11:55:19 -0700 (PDT) (envelope-from fbsd) Date: Tue, 24 Mar 2020 11:55:18 -0700 From: bob prohaska To: Mark Millard Cc: freebsd-arm@freebsd.org, bob prohaska Subject: Re: Belated out of swap kill on rpi3 at r359216 Message-ID: <20200324185518.GA92311@www.zefox.net> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Queue-Id: 48n0mn2DPgz4WZw X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [2.42 / 15.00]; ARC_NA(0.00)[]; WWW_DOT_DOMAIN(0.50)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; IP_SCORE(0.06)[ip: (0.24), ipnet: 50.1.16.0/20(0.12), asn: 7065(-0.03), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.03)[0.029,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.44)[0.437,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MID_RHS_WWW(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 18:55:11 -0000 On Tue, Mar 24, 2020 at 10:09:28AM -0700, Mark Millard wrote: > > > On 2020-Mar-24, at 08:57, bob prohaska wrote: > > > An attempt to buildworld on an rpi3 running r359216 stopped with an > > OOMA kill. Sources were at 359264, loader.conf contained > > vm.pageout_oom_seq="4096" . > > What of the value of vm.pfault_oom_attempts ? > > If vm.pfault_oom_attempts was not -1, > what of the value of vm.pfault_oom_wait ? > bob@www:/usr/src % sysctl vm.pfault_oom_wait vm.pfault_oom_wait: 10 I didn't change it, this must be the default. Would it be useful to add something like vm.pfault_oom_wait="20" to /boot/loader.conf? > > A snipped of gstat log suggests the worst congestion in the storage I/O > > happened at Tue Mar 24 04:52:26 PDT 2020 with an L(q) of 37, but the > > OOMA kill happened at Tue Mar 24 04:53:04 PDT 2020, by which time the > > L(q) had dropped to one, half a minute later. > > > > Is the delay in OOMA action to be expected? > > > > Here's the relevant part of the log, I hope the columns display readably: > > > > 0/2/2/19177 mbuf clusters in use (current/cache/total/max) > > procs memory page disks faults cpu > > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > > 0 0 29 1897320 47312 12851 9 4 5 13330 1669 0 0 14172 8020 3034 65 29 6 > > dT: 1.056s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 37 367 323 5463 6.4 45 1511 76.8 0 0 0.0 91.7 mmcsd0 > > 37 367 323 5463 6.5 45 1511 76.9 0 0 0.0 93.3 mmcsd0s2 > > 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.0 mmcsd0s2a > > 3 235 212 2254 5.9 23 814 21.5 0 0 0.0 70.0 mmcsd0s2b > > 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.2 ufs/rootfs > > Tue Mar 24 04:52:26 PDT 2020 > > Device 1K-blocks Used Avail Capacity > > /dev/mmcsd0s2b 4404252 224484 4179768 5% > > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 > > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 > > How bad were things back when the "indefinate wait buffer" notices > were generated (Mar 24 04:20:50 time frame)? > It looks like _new_ indefinite wait messages started coming at around Mon Mar 23 23:00:28 PDT 2020: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id 2 0 18 1588824 197676 14108 2 1 0 14759 238 0 0 14055 9263 2668 62 32 5 dT: 1.027s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 9 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0 9 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0s2 6 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0s2a 6 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 ufs/rootfs Mon Mar 23 23:00:28 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 245288 4158964 6% Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 280324, size: 8192 Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 285903, size: 16384 Admittedly, 300% busy looks pretty bad. Still the machine didn't quit..... On the next sample %busy went down to 18% for swap, less for all else. > > Below I show code changes to be more explicit in the > output messaging about what contributes to initiating > OOM kills, without needing boot verbose or the like. > There are also some messages from Mark J.'s old code > for reporting on things related to initiating OOM > kills, or some minor variations of them. > > You may want to try such changes to provide more > context for your OOM kills when they happen. Below > the 4 reasons reported are (showing the most > detailed of the related messages from different > stages): > > swp_pager_meta_build: swap blk uma zone exhausted > > swp_pager_meta_build: swap pctrie uma zone exhausted > > vm_fault_allocate: proc %d (%s) failed to alloc page on fault, starting OOM > > m_pageout_mightbe_oom: kill context: v_free_count: %u, v_inactive_count: %u > > Could inquiries instead be added to the logging script? Right now it's #!/bin/sh while true do vmstat ; gstat -abd -I 1s ; date ; swapinfo ; tail -n 2 /var/log/messages ; netstat -m | grep "mbuf clusters" done I'd much rather tamper with the logging script than the kernel 8-) Many thanks for all your help! bob prohaska From owner-freebsd-arm@freebsd.org Tue Mar 24 20:22:08 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1335D26970A for ; Tue, 24 Mar 2020 20:22:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-20.consmr.mail.gq1.yahoo.com (sonic313-20.consmr.mail.gq1.yahoo.com [98.137.65.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n2j16K17z45Z9 for ; Tue, 24 Mar 2020 20:21:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: utDf450VM1ldyPv6l1g5SivZzGpDLEouEk9lYK_sxXH94kQ9ARQVnSCdLJXxgH0 PLQ9FhP0_QfjlaNVorZwRLVgdZ2IzRiOPZPy18KyJKRQzUQ9XVOcqWbMyZUV67doEtrkNThNZKo8 rAMdVrArCQAXn6VaR5LT5nvCo54jDHAFi3mImRrlld1OTw7zIrh9tAZtahMm8rZORaq2VwEIRkdx aQCfAcLHtBrJejDlR1NOrWqlPy.6W2LYqA5dH9wye7.1ucr_H_0EPpoE2wilLFXTF_Aq3AAPB4nu fZ0KSYPYWIGvPo3KYUYF1gbj8sb9GrqDtrFb3iflyheTW6Xw58TE5i9JCeGTLyl.hb4tdttomBpt 8onZj4G2TAVHHtoaqcqBjdSu9VedKSPcWUQjmFYfJ.5s8YnWL3JC2cHczWOwFa.o6CMgw54wfrwU wKcYcx4LnRWd1J9P_F52Z3HT_vSGoXRlEqCpummy_Dmj8_tEi_7pwR7gW5GhL_7Y8vSeHBHqCVJX grY1mD4lEHEDi53rN.af7Ndqq7Dx7O2gI0i6K3ZamDKFo9O6wUOUsw23zLYHQ62h2TnKPB7IhIow SJD1SN5OQu6DI3QLNHOBir3VVBQm2U4yJd5l_KzCTwfJpxxqRVZnhjcgolgowZHCs7hbzzZIS9AL 4sakLXSK7hy3WP1vg3rPt9YcIVUtRK6uaPSM6mz8Emr5Z7uhMGOwzv8KJ16OLwDuH73hgBE3qc6s INRWO2pbxTLL.0KUZD0PDZ5XmSTxuioeQBuw0o9GRyvowYCMFGeL2uSKhEpskDhpntXladtiS.rN TdJaSXbMfHBr1lMwRKtxZhGIJz1QPZ2ZZpYHV_woHNTPl5cg8.SEEA02_Nbh2gbq1JHVDCUII5I2 1h1noQv3CIZtMcpHSXapYCp6zB1uQv8eo5_FJMu34z6PTVZMshTDz7z7EJ7XDkcZsWspG9Ii4MTg EckPe5V2ljMAGtaPV0BM2x4uHW.ysTumRetrOq66WUXol3WN5E3GoBRqeyH5uK45pG4lRQfpNbPn KRN2dyc5FI8N4Xpn46ZKAN7DA7NqMPxqQCdW7OMDiXorZEBwFYdRdd0ZlKgbtS6W.tvZh68aatMa yV38JQr9A2gIMTFNzJLZ5buDZ.dnK5zL10MeQfZx1Zq5V8LyJY5Xm1YXYtvYObWKz4a7vuL.cPUm zakJ2BEbs0AmtplTW90kCEG8sPAbjwDU34603wQH0wglPCwUs2q.L0lM1ZUsDOypaaAf8955hOv1 5pPDrOs_gV3PkewCstF.OtxcKicrJbpzGegnmvpIWxVTqlawiDVe1qXA9zTuJ0Xurox_FFGxQM2w SF2e34uCXSiVEIWSB3hOBnFpk5_Z31jUDOIHqunhb7wkfHnL_VwgzplAf35p40SI4hnhHNxgcCpc Ipyn3nY6qLtktQlWgZST3pa34GQ0l4Dxeh4BAY.u2jYTAQ4ZSJFL2Kw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Tue, 24 Mar 2020 20:21:49 +0000 Received: by smtp426.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 9b577911a420bb135562a20b1228b863; Tue, 24 Mar 2020 20:21:43 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <20200324185518.GA92311@www.zefox.net> Date: Tue, 24 Mar 2020 13:21:42 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48n2j16K17z45Z9 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.09 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.73)[-0.730,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; URIBL_BLOCKED(0.00)[dsl-only.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.86)[-0.864,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (2.66), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[83.65.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[83.65.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 20:22:08 -0000 On 2020-Mar-24, at 11:55, bob prohaska wrote: > On Tue, Mar 24, 2020 at 10:09:28AM -0700, Mark Millard wrote: >>=20 >>=20 >> On 2020-Mar-24, at 08:57, bob prohaska wrote: >>=20 >>> An attempt to buildworld on an rpi3 running r359216 stopped with an >>> OOMA kill. Sources were at 359264, loader.conf contained >>> vm.pageout_oom_seq=3D"4096" .=20 >>=20 >> What of the value of vm.pfault_oom_attempts ? >>=20 >> If vm.pfault_oom_attempts was not -1, >> what of the value of vm.pfault_oom_wait ? >>=20 > bob@www:/usr/src % sysctl vm.pfault_oom_wait > vm.pfault_oom_wait: 10 > I didn't change it, this must be the default.=20 > Would it be useful to add something like > vm.pfault_oom_wait=3D"20"=20 > to /boot/loader.conf? I assume that this answer means vm.pfault_oom_attempts was not -1. But you did not list what it was. vm.pfault_oom_attempts=3D-1 is the means of avoiding the the vm.pfault_oom_attempts,vm.pfault_oom_wait pair from causing OOM at all. (vm.pfault_oom_wait becomes irrelevant.) When vm.pfault_oom_attempts !=3D -1 , then there are potential time-outs that overall involve: vm.pfault_oom_attempts * vm.pfault_oom_wait but has some tradeoffs in the partitioning between the 2 factors: # sysctl -d vm.pfault_oom_attempts vm.pfault_oom_attempts: Number of page allocation attempts in page fault = handler before it triggers OOM handling # sysctl -d vm.pfault_oom_wait vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler As I understand, the following cases could each have very different results depending on what the latencies are like and such: vm.pfault_oom_attempts=3D=3D20 && vm.pfault_oom_wait=3D=3D1 vs. vm.pfault_oom_attempts=3D=3D1 && vm.pfault_oom_wait=3D=3D20 vs. vm.pfault_oom_attempts=3D=3D4 && vm.pfault_oom_wait=3D=3D5 As I remember, Konstantin Belousov once asked someone that was having a repeatable problem to try some alternatives that explored this but, as I remember, he got no reply to the request. The person might have just used: vm.pfault_oom_attempts=3D-1 and continued with their primary activity, for all I know. vm.pfault_oom_attempts=3D-1 is only recommended when one is sure that they will not run out of swap/paging space, if I understand right. For reference, for other settings: # sysctl -d vm.pageout_oom_seq vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM There is also: # sysctl -d vm.oom_pf_secs vm.oom_pf_secs:=20 but it seems to have an empty description. May be it reports vm.pfault_oom_attempts * vm.pfault_oom_wait when vm.pfault_oom_attempts !=3D -1 . (I've not looked.) >>> A snipped of gstat log suggests the worst congestion in the storage = I/O >>> happened at Tue Mar 24 04:52:26 PDT 2020 with an L(q) of 37, but the >>> OOMA kill happened at Tue Mar 24 04:53:04 PDT 2020, by which time = the >>> L(q) had dropped to one, half a minute later. >>>=20 >>> Is the delay in OOMA action to be expected?=20 >>>=20 >>> Here's the relevant part of the log, I hope the columns display = readably: >>>=20 >>> 0/2/2/19177 mbuf clusters in use (current/cache/total/max) >>> procs memory page disks faults = cpu >>> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >>> 0 0 29 1897320 47312 12851 9 4 5 13330 1669 0 0 14172 = 8020 3034 65 29 6 >>> dT: 1.056s w: 1.000s >>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >>> 37 367 323 5463 6.4 45 1511 76.8 0 0 = 0.0 91.7 mmcsd0 >>> 37 367 323 5463 6.5 45 1511 76.9 0 0 = 0.0 93.3 mmcsd0s2 >>> 34 133 111 3209 7.6 22 697 134.7 0 0 = 0.0 74.0 mmcsd0s2a >>> 3 235 212 2254 5.9 23 814 21.5 0 0 = 0.0 70.0 mmcsd0s2b >>> 34 133 111 3209 7.6 22 697 134.7 0 0 = 0.0 74.2 ufs/rootfs >>> Tue Mar 24 04:52:26 PDT 2020 >>> Device 1K-blocks Used Avail Capacity >>> /dev/mmcsd0s2b 4404252 224484 4179768 5% >>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 >>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 >>=20 >> How bad were things back when the "indefinate wait buffer" notices >> were generated (Mar 24 04:20:50 time frame)? >>=20 > It looks like _new_ indefinite wait messages started coming at around = Mon Mar 23 23:00:28 PDT 2020: > procs memory page disks faults = cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id > 2 0 18 1588824 197676 14108 2 1 0 14759 238 0 0 14055 = 9263 2668 62 32 5 > dT: 1.027s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 9 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 mmcsd0 > 9 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 mmcsd0s2 > 6 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 mmcsd0s2a > 6 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 ufs/rootfs > Mon Mar 23 23:00:28 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 245288 4158964 6% > Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 280324, size: 8192 > Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 285903, size: 16384 >=20 > Admittedly, 300% busy looks pretty bad. Still the machine didn't = quit..... > On the next sample %busy went down to 18% for swap, less for all else. As I understand, the ms/w value 13451 means that there was for a time (a mean of ?) somewhat under 14 seconds from a write being queued to it being completed. If a read could be blocked for such time frames by such writes, it suggests that vm.pfault_oom_wait might need to be larger than 13 if vm.pfault_oom_attempts=3D-1 is not in use. Or vm.pfault_oom_attempts=3D-1 could be used to avoid large time frames from directly leading to OOM activity. >>=20 >> Below I show code changes to be more explicit in the >> output messaging about what contributes to initiating >> OOM kills, without needing boot verbose or the like. >> There are also some messages from Mark J.'s old code >> for reporting on things related to initiating OOM >> kills, or some minor variations of them. >>=20 >=20 >> You may want to try such changes to provide more >> context for your OOM kills when they happen. Below >> the 4 reasons reported are (showing the most >> detailed of the related messages from different >> stages): >>=20 >> swp_pager_meta_build: swap blk uma zone exhausted >>=20 >> swp_pager_meta_build: swap pctrie uma zone exhausted >>=20 >> vm_fault_allocate: proc %d (%s) failed to alloc page on fault, = starting OOM >>=20 >> m_pageout_mightbe_oom: kill context: v_free_count: %u, = v_inactive_count: %u >>=20 >>=20 >=20 > Could inquiries instead be added to the logging script? > Right now it's > #!/bin/sh > while true > do vmstat ; gstat -abd -I 1s ; date ; swapinfo ; tail -n 2 = /var/log/messages ; netstat -m | grep "mbuf clusters" > done >=20 > I'd much rather tamper with the logging script than the kernel 8-) Unfortunately, the reason I made the kernel messaging changes is that, as far as I know, the normal kernel simply does not publish the information anywhere when the internal criteria leads to OOM activity: No inquiry available without kernel changes. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Tue Mar 24 22:46:57 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 06B5826CFDC for ; Tue, 24 Mar 2020 22:46:57 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48n5w63Gtwz42Sq for ; Tue, 24 Mar 2020 22:46:45 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id 02OMkxEp092965 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Mar 2020 15:47:00 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id 02OMkwgm092964; Tue, 24 Mar 2020 15:46:58 -0700 (PDT) (envelope-from fbsd) Date: Tue, 24 Mar 2020 15:46:58 -0700 From: bob prohaska To: Mark Millard Cc: freebsd-arm@freebsd.org, bob prohaska Subject: Re: Belated out of swap kill on rpi3 at r359216 Message-ID: <20200324224658.GA92726@www.zefox.net> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Queue-Id: 48n5w63Gtwz42Sq X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [3.15 / 15.00]; ARC_NA(0.00)[]; WWW_DOT_DOMAIN(0.50)[]; URIBL_BLOCKED(0.00)[zefox.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.37)[0.369,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(0.06)[ip: (0.24), ipnet: 50.1.16.0/20(0.12), asn: 7065(-0.03), country: US(-0.05)]; NEURAL_SPAM_LONG(0.83)[0.827,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MID_RHS_WWW(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 22:46:57 -0000 The logfile being discussed is at http://www.zefox.net/~fbsd/rpi3/swaptests/r359216/swapscript.osupdate.log for convenient reference. More replies below. On Tue, Mar 24, 2020 at 01:21:42PM -0700, Mark Millard wrote: > > > On 2020-Mar-24, at 11:55, bob prohaska wrote: > > > On Tue, Mar 24, 2020 at 10:09:28AM -0700, Mark Millard wrote: > >> > >> > >> On 2020-Mar-24, at 08:57, bob prohaska wrote: > >> > >>> An attempt to buildworld on an rpi3 running r359216 stopped with an > >>> OOMA kill. Sources were at 359264, loader.conf contained > >>> vm.pageout_oom_seq="4096" . > >> > >> What of the value of vm.pfault_oom_attempts ? > >> > >> If vm.pfault_oom_attempts was not -1, > >> what of the value of vm.pfault_oom_wait ? > >> > > bob@www:/usr/src % sysctl vm.pfault_oom_wait > > vm.pfault_oom_wait: 10 > > I didn't change it, this must be the default. > > Would it be useful to add something like > > vm.pfault_oom_wait="20" > > to /boot/loader.conf? > > I assume that this answer means vm.pfault_oom_attempts > was not -1. But you did not list what it was. > Sorry, the variable names are running together in my head. bob@www:/usr/src % sysctl vm.pfault_oom_attempts vm.pfault_oom_attempts: 3 It's now reset to -1, hopefully it'll work better than last time 8-) > vm.pfault_oom_attempts=-1 is the means of avoiding > the the vm.pfault_oom_attempts,vm.pfault_oom_wait > pair from causing OOM at all. (vm.pfault_oom_wait > becomes irrelevant.) > > When vm.pfault_oom_attempts != -1 , then there are > potential time-outs that overall involve: > > vm.pfault_oom_attempts * vm.pfault_oom_wait > > but has some tradeoffs in the partitioning between > the 2 factors: > > # sysctl -d vm.pfault_oom_attempts > vm.pfault_oom_attempts: Number of page allocation attempts in page fault handler before it triggers OOM handling > > # sysctl -d vm.pfault_oom_wait > vm.pfault_oom_wait: Number of seconds to wait for free pages before retrying the page fault handler > > As I understand, the following cases could each have > very different results depending on what the latencies > are like and such: > > vm.pfault_oom_attempts==20 && vm.pfault_oom_wait==1 > vs. > vm.pfault_oom_attempts==1 && vm.pfault_oom_wait==20 > vs. > vm.pfault_oom_attempts==4 && vm.pfault_oom_wait==5 > > As I remember, Konstantin Belousov once asked someone > that was having a repeatable problem to try some > alternatives that explored this but, as I remember, > he got no reply to the request. > Konstantin wrote to both me and the list in a thread on Re: panic: non-current pmap. If it's relevant please indicate. He also wrote to Don Lewis, in a thread on Re: spurious out of swap kills but that was on a " 16-thread Ryzen machine, with 64 GB of RAM and 40 GB of swap" which is surely unlike my predicament 8-) > The person might have just used: vm.pfault_oom_attempts=-1 > and continued with their primary activity, for all I know. > > vm.pfault_oom_attempts=-1 is only recommended when one > is sure that they will not run out of swap/paging space, > if I understand right. > > For reference, for other settings: > > # sysctl -d vm.pageout_oom_seq > vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM > > There is also: > > # sysctl -d vm.oom_pf_secs > vm.oom_pf_secs: > > but it seems to have an empty description. May be > it reports vm.pfault_oom_attempts * vm.pfault_oom_wait > when vm.pfault_oom_attempts != -1 . (I've not looked.) > > >>> A snipped of gstat log suggests the worst congestion in the storage I/O > >>> happened at Tue Mar 24 04:52:26 PDT 2020 with an L(q) of 37, but the > >>> OOMA kill happened at Tue Mar 24 04:53:04 PDT 2020, by which time the > >>> L(q) had dropped to one, half a minute later. > >>> > >>> Is the delay in OOMA action to be expected? > >>> > >>> Here's the relevant part of the log, I hope the columns display readably: > >>> > >>> 0/2/2/19177 mbuf clusters in use (current/cache/total/max) > >>> procs memory page disks faults cpu > >>> r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > >>> 0 0 29 1897320 47312 12851 9 4 5 13330 1669 0 0 14172 8020 3034 65 29 6 > >>> dT: 1.056s w: 1.000s > >>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > >>> 37 367 323 5463 6.4 45 1511 76.8 0 0 0.0 91.7 mmcsd0 > >>> 37 367 323 5463 6.5 45 1511 76.9 0 0 0.0 93.3 mmcsd0s2 > >>> 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.0 mmcsd0s2a > >>> 3 235 212 2254 5.9 23 814 21.5 0 0 0.0 70.0 mmcsd0s2b > >>> 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.2 ufs/rootfs > >>> Tue Mar 24 04:52:26 PDT 2020 > >>> Device 1K-blocks Used Avail Capacity > >>> /dev/mmcsd0s2b 4404252 224484 4179768 5% > >>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 > >>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 > >> > >> How bad were things back when the "indefinate wait buffer" notices > >> were generated (Mar 24 04:20:50 time frame)? > >> > > It looks like _new_ indefinite wait messages started coming at around Mon Mar 23 23:00:28 PDT 2020: > > procs memory page disks faults cpu > > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > > 2 0 18 1588824 197676 14108 2 1 0 14759 238 0 0 14055 9263 2668 62 32 5 > > dT: 1.027s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 9 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0 > > 9 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0s2 > > 6 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0s2a > > 6 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 ufs/rootfs > > Mon Mar 23 23:00:28 PDT 2020 > > Device 1K-blocks Used Avail Capacity > > /dev/mmcsd0s2b 4404252 245288 4158964 6% > > Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 280324, size: 8192 > > Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 285903, size: 16384 > > > > Admittedly, 300% busy looks pretty bad. Still the machine didn't quit..... > > On the next sample %busy went down to 18% for swap, less for all else. > > As I understand, the ms/w value 13451 means that there was > for a time (a mean of ?) somewhat under 14 seconds from a > write being queued to it being completed. > > If a read could be blocked for such time frames by such > writes, it suggests that vm.pfault_oom_wait might need to > be larger than 13 if vm.pfault_oom_attempts=-1 is not in > use. > Ok, I'm starting to get it. On this machine it's 10. But the 13 second delay appeared at Mon Mar 23 23:00:28 PDT 2020. The kill occurred around Tue Mar 24 04:53:08 PDT 2020 under what look like much more benign circumstances. > Or vm.pfault_oom_attempts=-1 could be used to avoid large > time frames from directly leading to OOM activity. > Understood. I simply forgot to restore the setting after the initial troubles with it. > >> > >> Below I show code changes to be more explicit in the > >> output messaging about what contributes to initiating > >> OOM kills, without needing boot verbose or the like. > >> There are also some messages from Mark J.'s old code > >> for reporting on things related to initiating OOM > >> kills, or some minor variations of them. > >> > > > >> You may want to try such changes to provide more > >> context for your OOM kills when they happen. Below > >> the 4 reasons reported are (showing the most > >> detailed of the related messages from different > >> stages): > >> > >> swp_pager_meta_build: swap blk uma zone exhausted > >> > >> swp_pager_meta_build: swap pctrie uma zone exhausted > >> > >> vm_fault_allocate: proc %d (%s) failed to alloc page on fault, starting OOM > >> > >> m_pageout_mightbe_oom: kill context: v_free_count: %u, v_inactive_count: %u > >> > >> > > > > Could inquiries instead be added to the logging script? > > Right now it's > > #!/bin/sh > > while true > > do vmstat ; gstat -abd -I 1s ; date ; swapinfo ; tail -n 2 /var/log/messages ; netstat -m | grep "mbuf clusters" > > done > > > > I'd much rather tamper with the logging script than the kernel 8-) > > Unfortunately, the reason I made the kernel messaging > changes is that, as far as I know, the normal kernel > simply does not publish the information anywhere when > the internal criteria leads to OOM activity: No > inquiry available without kernel changes. If all else fails I'll try to apply your patches to the kernel and recompile. With my thanks, bob prohaska From owner-freebsd-arm@freebsd.org Wed Mar 25 00:22:09 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 71D9D26F571 for ; Wed, 25 Mar 2020 00:22:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n81y6bBZz4cTp for ; Wed, 25 Mar 2020 00:21:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: PVU8nvIVM1k55OesVvilFuuB.xpEH6_3Nrw9pdw2xC0HwqSMbVkS7EI2QVPO1eJ _928NzBgZ_RUFM9hGUAy.VlhwImNEY6AD8wWD2FNE5dLvTr4PDuKACDAO1BRC9s9cTCbmXffb0lJ 5Ir9Zqk2j7UdxvRPQpz_qWlYqhPj7aD4JniwyOUftaWuO369rm_rp5Zeq8m6EDjlhSg7zqKRrZTZ xlVXI37qkCrNhyLIsGcEBdQeI43WmlnE7T2Vb3bPbndrs3gjbDU0tqGSjEywrThCs5pz1xnhKpLL DLeF6eu3OItbk9vgv1sI9Us1hDylv1hAiJZ6BUCcKe2ef1_kwhcwSJer6AVS5El1PtGCRaS2QGe1 m1ewKwLW9_pa6tLpuJDNjOIme7clDbtAL2sUnVs.nljyTgG2dTLqkwyD_2oUs_iiIwSN1B8XDn9. k7T35kXsJ3Ub.txJd.QYrQPVI6n8byCQn1sS99ALsWI44R302IX9L2l_E5F2gMohyee8DpJ2dUSU MwuUIZ0SBuC3oukl3c_biiB.aw_gDtZ1uNL0tPj9A0jDmhdZiZPBW7Hxab15jC8kE7prwtTgpMOM 0iFPmOisuOTBpOQlY9flUV9efl6C6Qp1ZcgtgT6Tox3JD3z_qn9lESZaNVQNuXXpt8huyzWtqUaq EP5xajr29AJg0Bfh3obNETeQe_AEJ_FzlumzpNEFBf9szns4h_1ANpc7BE86Jdf7N55FBFdM3sud ihNW8cGJPd8JEkCtphZXNSnHN_GaK9za_OS8KfjL2nTfoa5YW7WljRwsDqVtLo6FJAnFAWwJ5O2l ePbUgwIFMT.9sgkZIfa9SAoginLU3uJzlS_VeuH5CGdiPG0g_b6ov78H4lJDsuqy5ul4lt_s47zU kKPvHeJCnjPm8YAjlsKVift.sOcDwAfTlCnc_d38I7783LjUIi6TtGK1zpJRUwGKVJRO3rkyXSMN wR5O3uzpAtVgQkGz_DvhiXuzZV8EUBF3nmzrFxyRwJnrvjEz2DdHDtz3O5.Ds4KSnxQngqMfEHwA J31E6NFrA3JuCb2jrVV.5RFTU7IlpI6FE7YJV6bXd4NBDexqbvHKYMvUydO7_jXmI4EokUEEB8Yf Pav25lSg2GrF4ONJhFnlniIJrI0Sh5CYN7ImaIYl5qvlOsfNUaTqtgAfnZgo_36rq220aHZgeqkO 88wUeFgTnjLwyKKTaD76P8bFqVKBxfsU8ZemvBL1MkxynPAXght0Dk6coX0wtmDScTVkk9ec0DI3 wqFoGJwwkFE00DkEO1J1qJ2itaUaKDlIv83UpleEuLd1pnJyJIzkRcKPIbUJpKf1aZ3dZ8Nkzn9g MmQr7mKI0EP0UcaSKw23h7Vlc7jssquMGJVR.YQFQg8EGu_SnZ3RpInnt2J5u3KEYJoXznRh1EFr s4V924h26Pt44e7RDFG1S Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Wed, 25 Mar 2020 00:21:50 +0000 Received: by smtp415.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 054ab904faaea82c9cff83d6a8dd3bc8; Wed, 25 Mar 2020 00:21:48 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <20200324224658.GA92726@www.zefox.net> Date: Tue, 24 Mar 2020 17:21:47 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48n81y6bBZz4cTp X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.48 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.984,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; URIBL_BLOCKED(0.00)[zefox.net.multi.uribl.com,dsl-only.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (-0.46), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[148.69.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[148.69.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 00:22:10 -0000 On 2020-Mar-24, at 15:46, bob prohaska wrote: > The logfile being discussed is at > = http://www.zefox.net/~fbsd/rpi3/swaptests/r359216/swapscript.osupdate.log > for convenient reference. More replies below. >=20 >=20 > On Tue, Mar 24, 2020 at 01:21:42PM -0700, Mark Millard wrote: >>=20 >>=20 >> On 2020-Mar-24, at 11:55, bob prohaska wrote: >>=20 >>> On Tue, Mar 24, 2020 at 10:09:28AM -0700, Mark Millard wrote: >>>>=20 >>>>=20 >>>> On 2020-Mar-24, at 08:57, bob prohaska = wrote: >>>>=20 >>>>> An attempt to buildworld on an rpi3 running r359216 stopped with = an >>>>> OOMA kill. Sources were at 359264, loader.conf contained >>>>> vm.pageout_oom_seq=3D"4096" .=20 >>>>=20 >>>> What of the value of vm.pfault_oom_attempts ? >>>>=20 >>>> If vm.pfault_oom_attempts was not -1, >>>> what of the value of vm.pfault_oom_wait ? >>>>=20 >>> bob@www:/usr/src % sysctl vm.pfault_oom_wait >>> vm.pfault_oom_wait: 10 >>> I didn't change it, this must be the default.=20 >>> Would it be useful to add something like >>> vm.pfault_oom_wait=3D"20"=20 >>> to /boot/loader.conf? >>=20 >> I assume that this answer means vm.pfault_oom_attempts >> was not -1. But you did not list what it was. >>=20 >=20 > Sorry, the variable names are running together in my head. >=20 > bob@www:/usr/src % sysctl vm.pfault_oom_attempts > vm.pfault_oom_attempts: 3 > It's now reset to -1, hopefully it'll work better than last time 8-) >=20 >> vm.pfault_oom_attempts=3D-1 is the means of avoiding >> the the vm.pfault_oom_attempts,vm.pfault_oom_wait >> pair from causing OOM at all. (vm.pfault_oom_wait >> becomes irrelevant.) >>=20 >=20 >> When vm.pfault_oom_attempts !=3D -1 , then there are >> potential time-outs that overall involve: >>=20 >> vm.pfault_oom_attempts * vm.pfault_oom_wait >>=20 >> but has some tradeoffs in the partitioning between >> the 2 factors: >>=20 >> # sysctl -d vm.pfault_oom_attempts >> vm.pfault_oom_attempts: Number of page allocation attempts in page = fault handler before it triggers OOM handling >>=20 >> # sysctl -d vm.pfault_oom_wait >> vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler >>=20 >> As I understand, the following cases could each have >> very different results depending on what the latencies >> are like and such: >>=20 >> vm.pfault_oom_attempts=3D=3D20 && vm.pfault_oom_wait=3D=3D1 >> vs. >> vm.pfault_oom_attempts=3D=3D1 && vm.pfault_oom_wait=3D=3D20 >> vs. >> vm.pfault_oom_attempts=3D=3D4 && vm.pfault_oom_wait=3D=3D5 >>=20 >> As I remember, Konstantin Belousov once asked someone >> that was having a repeatable problem to try some >> alternatives that explored this but, as I remember, >> he got no reply to the request. >>=20 > Konstantin wrote to both me and the list in a thread on=20 > Re: panic: non-current pmap. If it's relevant please indicate. >=20 > He also wrote to Don Lewis, in a thread on Re: spurious out of swap = kills > but that was on a=20 > " 16-thread Ryzen machine, with 64 GB of RAM and 40 GB of swap" > which is surely unlike my predicament 8-) >=20 >> The person might have just used: vm.pfault_oom_attempts=3D-1 >> and continued with their primary activity, for all I know. >>=20 >> vm.pfault_oom_attempts=3D-1 is only recommended when one >> is sure that they will not run out of swap/paging space, >> if I understand right. >>=20 >> For reference, for other settings: >>=20 >> # sysctl -d vm.pageout_oom_seq >> vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM >>=20 >> There is also: >>=20 >> # sysctl -d vm.oom_pf_secs >> vm.oom_pf_secs:=20 >>=20 >> but it seems to have an empty description. May be >> it reports vm.pfault_oom_attempts * vm.pfault_oom_wait >> when vm.pfault_oom_attempts !=3D -1 . (I've not looked.) >>=20 >>>>> A snipped of gstat log suggests the worst congestion in the = storage I/O >>>>> happened at Tue Mar 24 04:52:26 PDT 2020 with an L(q) of 37, but = the >>>>> OOMA kill happened at Tue Mar 24 04:53:04 PDT 2020, by which time = the >>>>> L(q) had dropped to one, half a minute later. >>>>>=20 >>>>> Is the delay in OOMA action to be expected?=20 >>>>>=20 >>>>> Here's the relevant part of the log, I hope the columns display = readably: >>>>>=20 >>>>> 0/2/2/19177 mbuf clusters in use (current/cache/total/max) >>>>> procs memory page disks faults = cpu >>>>> r b w avm fre flt re pi po fr sr mm0 da0 in = sy cs us sy id >>>>> 0 0 29 1897320 47312 12851 9 4 5 13330 1669 0 0 14172 = 8020 3034 65 29 6 >>>>> dT: 1.056s w: 1.000s >>>>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s = kBps ms/d %busy Name >>>>> 37 367 323 5463 6.4 45 1511 76.8 0 0 = 0.0 91.7 mmcsd0 >>>>> 37 367 323 5463 6.5 45 1511 76.9 0 0 = 0.0 93.3 mmcsd0s2 >>>>> 34 133 111 3209 7.6 22 697 134.7 0 0 = 0.0 74.0 mmcsd0s2a >>>>> 3 235 212 2254 5.9 23 814 21.5 0 0 = 0.0 70.0 mmcsd0s2b >>>>> 34 133 111 3209 7.6 22 697 134.7 0 0 = 0.0 74.2 ufs/rootfs >>>>> Tue Mar 24 04:52:26 PDT 2020 >>>>> Device 1K-blocks Used Avail Capacity >>>>> /dev/mmcsd0s2b 4404252 224484 4179768 5% >>>>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 >>>>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 >>>>=20 >>>> How bad were things back when the "indefinate wait buffer" notices >>>> were generated (Mar 24 04:20:50 time frame)? >>>>=20 >>> It looks like _new_ indefinite wait messages started coming at = around Mon Mar 23 23:00:28 PDT 2020: >>> procs memory page disks faults = cpu >>> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >>> 2 0 18 1588824 197676 14108 2 1 0 14759 238 0 0 14055 = 9263 2668 62 32 5 >>> dT: 1.027s w: 1.000s >>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >>> 9 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 mmcsd0 >>> 9 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 mmcsd0s2 >>> 6 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 mmcsd0s2a >>> 6 1 0 0 0.0 1 31 13451 0 0 = 0.0 326.9 ufs/rootfs >>> Mon Mar 23 23:00:28 PDT 2020 >>> Device 1K-blocks Used Avail Capacity >>> /dev/mmcsd0s2b 4404252 245288 4158964 6% >>> Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 280324, size: 8192 >>> Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 285903, size: 16384 >>>=20 >>> Admittedly, 300% busy looks pretty bad. Still the machine didn't = quit..... >>> On the next sample %busy went down to 18% for swap, less for all = else. >>=20 >> As I understand, the ms/w value 13451 means that there was >> for a time (a mean of ?) somewhat under 14 seconds from a >> write being queued to it being completed. >>=20 >> If a read could be blocked for such time frames by such >> writes, it suggests that vm.pfault_oom_wait might need to >> be larger than 13 if vm.pfault_oom_attempts=3D-1 is not in >> use. >>=20 > Ok, I'm starting to get it. On this machine it's 10. >=20 > But the 13 second delay appeared at Mon Mar 23 23:00:28 PDT 2020.=20 > The kill occurred around Tue Mar 24 04:53:08 PDT 2020 > under what look like much more benign circumstances.=20 I had intended to ask about the text next-to/just-above, not much earlier [thus, the "(Mar 24 04:20:50 time frame)"]. So I was asking about: Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 25477, size: 4096 and did not originally notice that you replied with information for "Mon Mar 23 23:00:28 PDT 2020". Other than showing examples of what the delays can be, the "Mar 24 04:20:50" is of more direct relevance for the OOM kill that happened. I'll show the text around/before 04:20:50 later in this note. It proves somewhat interesting. >> Or vm.pfault_oom_attempts=3D-1 could be used to avoid large >> time frames from directly leading to OOM activity. >>=20 > Understood. I simply forgot to restore the setting after > the initial troubles with it.=20 >=20 >>>>=20 >>>> Below I show code changes to be more explicit in the >>>> output messaging about what contributes to initiating >>>> OOM kills, without needing boot verbose or the like. >>>> There are also some messages from Mark J.'s old code >>>> for reporting on things related to initiating OOM >>>> kills, or some minor variations of them. >>>>=20 >>>=20 >>>> You may want to try such changes to provide more >>>> context for your OOM kills when they happen. Below >>>> the 4 reasons reported are (showing the most >>>> detailed of the related messages from different >>>> stages): >>>>=20 >>>> swp_pager_meta_build: swap blk uma zone exhausted >>>>=20 >>>> swp_pager_meta_build: swap pctrie uma zone exhausted >>>>=20 >>>> vm_fault_allocate: proc %d (%s) failed to alloc page on fault, = starting OOM >>>>=20 >>>> m_pageout_mightbe_oom: kill context: v_free_count: %u, = v_inactive_count: %u >>>>=20 >>>>=20 >>>=20 >>> Could inquiries instead be added to the logging script? >>> Right now it's >>> #!/bin/sh >>> while true >>> do vmstat ; gstat -abd -I 1s ; date ; swapinfo ; tail -n 2 = /var/log/messages ; netstat -m | grep "mbuf clusters" >>> done >>>=20 >>> I'd much rather tamper with the logging script than the kernel 8-) >>=20 >> Unfortunately, the reason I made the kernel messaging >> changes is that, as far as I know, the normal kernel >> simply does not publish the information anywhere when >> the internal criteria leads to OOM activity: No >> inquiry available without kernel changes. >=20 > If all else fails I'll try to apply your patches to the kernel > and recompile.=20 Up to you, of course. The text before "Mar 24 04:20:50" messages, with notes and blank lines mixed in to group things: . . . Tue Mar 24 04:20:38 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 3/773/776/19177 mbuf clusters in use (current/cache/total/max) procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 0 0 30 2034696 54180 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.031s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name 18 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 mmcsd0 18 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 mmcsd0s2 10 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 mmcsd0s2a 10 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 ufs/rootfs Tue Mar 24 04:20:40 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 6/770/776/19177 mbuf clusters in use (current/cache/total/max) Note that the above has 26sec+ ms/w figures. No reads. No mmcsd0s2b. procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 0 0 30 2034696 54252 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.064s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name Tue Mar 24 04:20:42 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 9/1023/1032/19177 mbuf clusters in use (current/cache/total/max) For the above, note the lack of any lines with ms/r or ms/w figures, just the title rows show. procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 0 0 30 2034696 53668 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.046s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name 17 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 mmcsd0 17 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 mmcsd0s2 9 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 mmcsd0s2a 9 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 ufs/rootfs Tue Mar 24 04:20:43 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 9/1023/1032/19177 mbuf clusters in use (current/cache/total/max) Note that the above has 30sec+ ms/w figures. No reads. Also note that the L(q) figures dropped by only 1 over the about 04:20:40 to 04:20:43 interval (18->17 and 10->9). (I only take these deltas as suggestive. They might be accidental near matches.) No mmcsd0s2b. procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 0 0 30 2034696 53624 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.003s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name Tue Mar 24 04:20:45 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 12/1020/1032/19177 mbuf clusters in use (current/cache/total/max) For the above, note the lack of any lines with ms/r or ms/w figures, just the title rows show. procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 0 0 30 2034696 53600 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.063s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name Tue Mar 24 04:20:47 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 12/1020/1032/19177 mbuf clusters in use (current/cache/total/max) For the above, note the lack of any lines with ms/r or ms/w figures, just the title rows show. procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 0 0 30 2034696 53592 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.006s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name Tue Mar 24 04:20:48 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 15/1017/1032/19177 mbuf clusters in use (current/cache/total/max) For the above, note the lack of any lines with ms/r or ms/w figures, just the title rows show. procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 0 0 30 2034696 53672 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name Tue Mar 24 04:20:50 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 403784 4000468 9% Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 623056, size: 16384 Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 1033854, size: 4096 15/1017/1032/19177 mbuf clusters in use (current/cache/total/max) For the above, note the list of any lines with ms/r or ms/w figures, just the title rows. procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mm0 da0 in sy cs = us sy id 2 0 30 2045028 54160 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name 2 329 236 3839 13.0 93 1262 18.6 0 0 = 0.0 99.4 mmcsd0 2 329 236 3839 13.2 93 1262 18.7 0 0 = 0.0 101.1 mmcsd0s2 1 153 152 2816 12.5 1 32 366.5 0 0 = 0.0 95.2 mmcsd0s2a 1 176 84 1023 14.5 92 1230 14.9 0 0 = 0.0 66.8 mmcsd0s2b 1 153 152 2816 12.5 1 32 366.6 0 0 = 0.0 95.3 ufs/rootfs Tue Mar 24 04:20:51 PDT 2020 Device 1K-blocks Used Avail Capacity /dev/mmcsd0s2b 4404252 296976 4107276 7% Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 21842, size: 12288 Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: = 0, blkno: 25477, size: 4096 0/1032/1032/19177 mbuf clusters in use (current/cache/total/max) For the above, the lines with ms/r and ms/w figures are back but the indefinite wait buffer for "Mar 24 04:20:50" happened too, before the "Tue Mar 24 04:20:51 PDT 2020" above. I'd guess they are from before the data that the gstat output is based on. Also there are lots of reads and some writes in the above gstat output. mmcsd0s2b is again showing as well. Note: It looks like ufs/rootfs might be a label identifying mmcsd0s2a, so they track up to slight time-of-information differences. I'm guessing that mmcsd0s2b is just a swap/paging partition. So it looks like activity for ufs/rootfs may have blocked/delayed activity for paging/swaping (mmcsd0s2b), at least for a time, even if it is not directly the cause of the specific OOM activity. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Wed Mar 25 01:56:35 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AA14A271FF8 for ; Wed, 25 Mar 2020 01:56:35 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48nB6v4vb5z4DrH for ; Wed, 25 Mar 2020 01:56:23 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id 02P1uYmU093429 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Mar 2020 18:56:35 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id 02P1uXkj093428; Tue, 24 Mar 2020 18:56:33 -0700 (PDT) (envelope-from fbsd) Date: Tue, 24 Mar 2020 18:56:33 -0700 From: bob prohaska To: Mark Millard Cc: freebsd-arm@freebsd.org, bob prohaska Subject: Re: Belated out of swap kill on rpi3 at r359216 Message-ID: <20200325015633.GA93057@www.zefox.net> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Queue-Id: 48nB6v4vb5z4DrH X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [3.18 / 15.00]; ARC_NA(0.00)[]; WWW_DOT_DOMAIN(0.50)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; IP_SCORE(0.06)[ip: (0.24), ipnet: 50.1.16.0/20(0.12), asn: 7065(-0.03), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; URIBL_BLOCKED(0.00)[zefox.net.multi.uribl.com]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.37)[0.371,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.85)[0.849,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MID_RHS_WWW(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 01:56:35 -0000 [this thread needs a haircut, new comments start near line 288] On Tue, Mar 24, 2020 at 05:21:47PM -0700, Mark Millard wrote: > On 2020-Mar-24, at 15:46, bob prohaska wrote: > > > The logfile being discussed is at > > http://www.zefox.net/~fbsd/rpi3/swaptests/r359216/swapscript.osupdate.log > > for convenient reference. More replies below. > > > > > > On Tue, Mar 24, 2020 at 01:21:42PM -0700, Mark Millard wrote: > >> > >> > >> On 2020-Mar-24, at 11:55, bob prohaska wrote: > >> > >>> On Tue, Mar 24, 2020 at 10:09:28AM -0700, Mark Millard wrote: > >>>> > >>>> > >>>> On 2020-Mar-24, at 08:57, bob prohaska wrote: > >>>> > >>>>> An attempt to buildworld on an rpi3 running r359216 stopped with an > >>>>> OOMA kill. Sources were at 359264, loader.conf contained > >>>>> vm.pageout_oom_seq="4096" . > >>>> > >>>> What of the value of vm.pfault_oom_attempts ? > >>>> > >>>> If vm.pfault_oom_attempts was not -1, > >>>> what of the value of vm.pfault_oom_wait ? > >>>> > >>> bob@www:/usr/src % sysctl vm.pfault_oom_wait > >>> vm.pfault_oom_wait: 10 > >>> I didn't change it, this must be the default. > >>> Would it be useful to add something like > >>> vm.pfault_oom_wait="20" > >>> to /boot/loader.conf? > >> > >> I assume that this answer means vm.pfault_oom_attempts > >> was not -1. But you did not list what it was. > >> > > > > Sorry, the variable names are running together in my head. > > > > bob@www:/usr/src % sysctl vm.pfault_oom_attempts > > vm.pfault_oom_attempts: 3 > > It's now reset to -1, hopefully it'll work better than last time 8-) > > > >> vm.pfault_oom_attempts=-1 is the means of avoiding > >> the the vm.pfault_oom_attempts,vm.pfault_oom_wait > >> pair from causing OOM at all. (vm.pfault_oom_wait > >> becomes irrelevant.) > >> > > > >> When vm.pfault_oom_attempts != -1 , then there are > >> potential time-outs that overall involve: > >> > >> vm.pfault_oom_attempts * vm.pfault_oom_wait > >> > >> but has some tradeoffs in the partitioning between > >> the 2 factors: > >> > >> # sysctl -d vm.pfault_oom_attempts > >> vm.pfault_oom_attempts: Number of page allocation attempts in page fault handler before it triggers OOM handling > >> > >> # sysctl -d vm.pfault_oom_wait > >> vm.pfault_oom_wait: Number of seconds to wait for free pages before retrying the page fault handler > >> > >> As I understand, the following cases could each have > >> very different results depending on what the latencies > >> are like and such: > >> > >> vm.pfault_oom_attempts==20 && vm.pfault_oom_wait==1 > >> vs. > >> vm.pfault_oom_attempts==1 && vm.pfault_oom_wait==20 > >> vs. > >> vm.pfault_oom_attempts==4 && vm.pfault_oom_wait==5 > >> > >> As I remember, Konstantin Belousov once asked someone > >> that was having a repeatable problem to try some > >> alternatives that explored this but, as I remember, > >> he got no reply to the request. > >> > > Konstantin wrote to both me and the list in a thread on > > Re: panic: non-current pmap. If it's relevant please indicate. > > > > He also wrote to Don Lewis, in a thread on Re: spurious out of swap kills > > but that was on a > > " 16-thread Ryzen machine, with 64 GB of RAM and 40 GB of swap" > > which is surely unlike my predicament 8-) > > > >> The person might have just used: vm.pfault_oom_attempts=-1 > >> and continued with their primary activity, for all I know. > >> > >> vm.pfault_oom_attempts=-1 is only recommended when one > >> is sure that they will not run out of swap/paging space, > >> if I understand right. > >> > >> For reference, for other settings: > >> > >> # sysctl -d vm.pageout_oom_seq > >> vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM > >> > >> There is also: > >> > >> # sysctl -d vm.oom_pf_secs > >> vm.oom_pf_secs: > >> > >> but it seems to have an empty description. May be > >> it reports vm.pfault_oom_attempts * vm.pfault_oom_wait > >> when vm.pfault_oom_attempts != -1 . (I've not looked.) > >> > >>>>> A snipped of gstat log suggests the worst congestion in the storage I/O > >>>>> happened at Tue Mar 24 04:52:26 PDT 2020 with an L(q) of 37, but the > >>>>> OOMA kill happened at Tue Mar 24 04:53:04 PDT 2020, by which time the > >>>>> L(q) had dropped to one, half a minute later. > >>>>> > >>>>> Is the delay in OOMA action to be expected? > >>>>> > >>>>> Here's the relevant part of the log, I hope the columns display readably: > >>>>> > >>>>> 0/2/2/19177 mbuf clusters in use (current/cache/total/max) > >>>>> procs memory page disks faults cpu > >>>>> r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > >>>>> 0 0 29 1897320 47312 12851 9 4 5 13330 1669 0 0 14172 8020 3034 65 29 6 > >>>>> dT: 1.056s w: 1.000s > >>>>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > >>>>> 37 367 323 5463 6.4 45 1511 76.8 0 0 0.0 91.7 mmcsd0 > >>>>> 37 367 323 5463 6.5 45 1511 76.9 0 0 0.0 93.3 mmcsd0s2 > >>>>> 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.0 mmcsd0s2a > >>>>> 3 235 212 2254 5.9 23 814 21.5 0 0 0.0 70.0 mmcsd0s2b > >>>>> 34 133 111 3209 7.6 22 697 134.7 0 0 0.0 74.2 ufs/rootfs > >>>>> Tue Mar 24 04:52:26 PDT 2020 > >>>>> Device 1K-blocks Used Avail Capacity > >>>>> /dev/mmcsd0s2b 4404252 224484 4179768 5% > >>>>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 > >>>>> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 > >>>> > >>>> How bad were things back when the "indefinate wait buffer" notices > >>>> were generated (Mar 24 04:20:50 time frame)? > >>>> > >>> It looks like _new_ indefinite wait messages started coming at around Mon Mar 23 23:00:28 PDT 2020: > >>> procs memory page disks faults cpu > >>> r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > >>> 2 0 18 1588824 197676 14108 2 1 0 14759 238 0 0 14055 9263 2668 62 32 5 > >>> dT: 1.027s w: 1.000s > >>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > >>> 9 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0 > >>> 9 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0s2 > >>> 6 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 mmcsd0s2a > >>> 6 1 0 0 0.0 1 31 13451 0 0 0.0 326.9 ufs/rootfs > >>> Mon Mar 23 23:00:28 PDT 2020 > >>> Device 1K-blocks Used Avail Capacity > >>> /dev/mmcsd0s2b 4404252 245288 4158964 6% > >>> Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 280324, size: 8192 > >>> Mar 23 23:00:28 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 285903, size: 16384 > >>> > >>> Admittedly, 300% busy looks pretty bad. Still the machine didn't quit..... > >>> On the next sample %busy went down to 18% for swap, less for all else. > >> > >> As I understand, the ms/w value 13451 means that there was > >> for a time (a mean of ?) somewhat under 14 seconds from a > >> write being queued to it being completed. > >> > >> If a read could be blocked for such time frames by such > >> writes, it suggests that vm.pfault_oom_wait might need to > >> be larger than 13 if vm.pfault_oom_attempts=-1 is not in > >> use. > >> > > Ok, I'm starting to get it. On this machine it's 10. > > > > But the 13 second delay appeared at Mon Mar 23 23:00:28 PDT 2020. > > The kill occurred around Tue Mar 24 04:53:08 PDT 2020 > > under what look like much more benign circumstances. > > I had intended to ask about the text next-to/just-above, > not much earlier [thus, the "(Mar 24 04:20:50 time frame)"]. > So I was asking about: > > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 > > and did not originally notice that you replied with > information for "Mon Mar 23 23:00:28 PDT 2020". > > Other than showing examples of what the delays can be, > the "Mar 24 04:20:50" is of more direct relevance for > the OOM kill that happened. > > I'll show the text around/before 04:20:50 later in this > note. It proves somewhat interesting. > > >> Or vm.pfault_oom_attempts=-1 could be used to avoid large > >> time frames from directly leading to OOM activity. > >> > > Understood. I simply forgot to restore the setting after > > the initial troubles with it. > > > >>>> > >>>> Below I show code changes to be more explicit in the > >>>> output messaging about what contributes to initiating > >>>> OOM kills, without needing boot verbose or the like. > >>>> There are also some messages from Mark J.'s old code > >>>> for reporting on things related to initiating OOM > >>>> kills, or some minor variations of them. > >>>> > >>> > >>>> You may want to try such changes to provide more > >>>> context for your OOM kills when they happen. Below > >>>> the 4 reasons reported are (showing the most > >>>> detailed of the related messages from different > >>>> stages): > >>>> > >>>> swp_pager_meta_build: swap blk uma zone exhausted > >>>> > >>>> swp_pager_meta_build: swap pctrie uma zone exhausted > >>>> > >>>> vm_fault_allocate: proc %d (%s) failed to alloc page on fault, starting OOM > >>>> > >>>> m_pageout_mightbe_oom: kill context: v_free_count: %u, v_inactive_count: %u > >>>> > >>>> > >>> > >>> Could inquiries instead be added to the logging script? > >>> Right now it's > >>> #!/bin/sh > >>> while true > >>> do vmstat ; gstat -abd -I 1s ; date ; swapinfo ; tail -n 2 /var/log/messages ; netstat -m | grep "mbuf clusters" > >>> done > >>> > >>> I'd much rather tamper with the logging script than the kernel 8-) > >> > >> Unfortunately, the reason I made the kernel messaging > >> changes is that, as far as I know, the normal kernel > >> simply does not publish the information anywhere when > >> the internal criteria leads to OOM activity: No > >> inquiry available without kernel changes. > > > > If all else fails I'll try to apply your patches to the kernel > > and recompile. > > Up to you, of course. > > > The text before "Mar 24 04:20:50" messages, with notes > and blank lines mixed in to group things: > > . . . > Tue Mar 24 04:20:38 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 3/773/776/19177 mbuf clusters in use (current/cache/total/max) > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 0 0 30 2034696 54180 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.031s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 18 1 0 0 0.0 1 31 26686 0 0 0.0 287.1 mmcsd0 > 18 1 0 0 0.0 1 31 26686 0 0 0.0 287.1 mmcsd0s2 > 10 1 0 0 0.0 1 31 26686 0 0 0.0 287.1 mmcsd0s2a > 10 1 0 0 0.0 1 31 26686 0 0 0.0 287.1 ufs/rootfs > Tue Mar 24 04:20:40 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 6/770/776/19177 mbuf clusters in use (current/cache/total/max) > > Note that the above has 26sec+ ms/w figures. No reads. > No mmcsd0s2b. > > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 0 0 30 2034696 54252 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.064s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > Tue Mar 24 04:20:42 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 9/1023/1032/19177 mbuf clusters in use (current/cache/total/max) > > For the above, note the lack of any lines with ms/r or ms/w figures, > just the title rows show. > Same thing happens when gstat -a is run on an idle machine. I thought it normal, the CPU had all the data it needed. > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 0 0 30 2034696 53668 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.046s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 17 1 0 0 0.0 1 31 30042 0 0 0.0 320.7 mmcsd0 > 17 1 0 0 0.0 1 31 30042 0 0 0.0 320.7 mmcsd0s2 > 9 1 0 0 0.0 1 31 30042 0 0 0.0 320.7 mmcsd0s2a > 9 1 0 0 0.0 1 31 30042 0 0 0.0 320.7 ufs/rootfs > Tue Mar 24 04:20:43 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 9/1023/1032/19177 mbuf clusters in use (current/cache/total/max) > > Note that the above has 30sec+ ms/w figures. No reads. > > Also note that the L(q) figures dropped by only 1 over > the about 04:20:40 to 04:20:43 interval (18->17 and > 10->9). (I only take these deltas as suggestive. They > might be accidental near matches.) No mmcsd0s2b. > > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 0 0 30 2034696 53624 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.003s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > Tue Mar 24 04:20:45 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 12/1020/1032/19177 mbuf clusters in use (current/cache/total/max) > > For the above, note the lack of any lines with ms/r or ms/w figures, > just the title rows show. > It's tempting, but maybe not correct, the think the I/O caught up. > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 0 0 30 2034696 53600 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.063s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > Tue Mar 24 04:20:47 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 12/1020/1032/19177 mbuf clusters in use (current/cache/total/max) > > For the above, note the lack of any lines with ms/r or ms/w figures, > just the title rows show. > Can't that be interpreted as gstat -a having nothing to report? Just _why_ it might have nothing to report never crossed my mind.... > > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 0 0 30 2034696 53592 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.006s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > Tue Mar 24 04:20:48 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 15/1017/1032/19177 mbuf clusters in use (current/cache/total/max) > > For the above, note the lack of any lines with ms/r or ms/w figures, > just the title rows show. > > > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 0 0 30 2034696 53672 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > Tue Mar 24 04:20:50 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 403784 4000468 9% > Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 623056, size: 16384 > Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1033854, size: 4096 > 15/1017/1032/19177 mbuf clusters in use (current/cache/total/max) > > For the above, note the list of any lines with ms/r or ms/w figures, > just the title rows. > > > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id > 2 0 30 2045028 54160 12942 7 2 4 13432 1374 0 0 14074 8102 2849 65 29 5 > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 2 329 236 3839 13.0 93 1262 18.6 0 0 0.0 99.4 mmcsd0 > 2 329 236 3839 13.2 93 1262 18.7 0 0 0.0 101.1 mmcsd0s2 > 1 153 152 2816 12.5 1 32 366.5 0 0 0.0 95.2 mmcsd0s2a > 1 176 84 1023 14.5 92 1230 14.9 0 0 0.0 66.8 mmcsd0s2b > 1 153 152 2816 12.5 1 32 366.6 0 0 0.0 95.3 ufs/rootfs > Tue Mar 24 04:20:51 PDT 2020 > Device 1K-blocks Used Avail Capacity > /dev/mmcsd0s2b 4404252 296976 4107276 7% > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 21842, size: 12288 > Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 25477, size: 4096 > 0/1032/1032/19177 mbuf clusters in use (current/cache/total/max) > > For the above, the lines with ms/r and ms/w figures are back > but the indefinite wait buffer for "Mar 24 04:20:50" happened > too, before the "Tue Mar 24 04:20:51 PDT 2020" above. I'd guess > they are from before the data that the gstat output is based on. > > Also there are lots of reads and some writes in the above > gstat output. mmcsd0s2b is again showing as well. > > Note: It looks like ufs/rootfs might be a label > identifying mmcsd0s2a, so they track up to slight > time-of-information differences. I'm guessing that > mmcsd0s2b is just a swap/paging partition. > That's correct. > So it looks like activity for ufs/rootfs may have > blocked/delayed activity for paging/swaping > (mmcsd0s2b), at least for a time, even if it is > not directly the cause of the specific OOM > activity. > How would one distinguish slow swap i/o from slow filesystem i/o ? Would moving swap to a USB device avoid blocking by writes to microSD? That's not hard to try, but it puts a whole new kettle of fish on the fire. It should be added that the original buildworld has been restarted with vm.pfault_oom_attempts: -1 and so far, though it still generates the "indefinite wait" warnings, the ms/w times look better. It's not done yet, though. How much overhead comes with OOMA? Many thanks for your insights, bob prohaska From owner-freebsd-arm@freebsd.org Wed Mar 25 08:19:05 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B3EDE27A293 for ; Wed, 25 Mar 2020 08:19:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-54.consmr.mail.gq1.yahoo.com (sonic308-54.consmr.mail.gq1.yahoo.com [98.137.68.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48nLcC2Kq5z4NT1 for ; Wed, 25 Mar 2020 08:18:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: HiN1tZ0VM1mteVGUkvE9Zh77bkmSgyZ2td4wJSop7SiqrauhxBKdJqgaouKRByw lpmeRmZajwLZ5CBv3zEjECh6jST09MBv6LALSIJeLIASjQOcRwMVTa.tl62rME8Lk_PBBZoeOPnW 5NnU7oNW3rkxEbUEYRmNUISgXWGkDHMROdKqUhRYdl_65PDDyUTRINSFOZFsRGm0qNGnTYEpzpNn y3R6YrOFSpwPDe288v_9yhGpOdK7Kz9VMf9yjHuBNX8n9x.PPz6N_maafXzaJNq1C6CyVmOsbKat EvsDBs5kszwfne5bG2iZm6RNXYSOkv.J7HTQDOLBNPk4htnphDwrWBZdfkKt9Pvb0AvafpDwa1.i qqQQHzVumBRc6vG9Fz_KWqPFXBDrG6CVTsOysSJ.FeSxEesnXPHhkZGuPZwYk2kSfyYTCvcgRBId 5ulq0GNq.6PLFmvplKFVPjpdUqzL88tBA1CgdpIUa8qUm9KMJerjWwjXGDfaVCKZ2kmu0k5_8jSW SDBFLlT.rA01.ENSOVN5GlFoJf0lcki1eP4pvZvUM8VFJbAgaf0WKc5Ntydm7Wn3nWRUT0SN83je oy1_S6aU9Kdiig8rN5GaJegj_VbD09YXIU8H9YgkxBkrXMqCYC.Ai0fjliXuY_qTRr1AwkJJkK3M ywCFluo08uAP.fBIoBcSjelVgMgU4YcB9S9Grlvp3DxTD3SPORZkaIECMrKyOEdMMXYxDGNQPvv6 9q8AHZOV20MHHATvHrA7CmMeDJrOIghMLykC1.HT8vCvgUFklVtfrZt.EJkxFq4w1PY2lwT9QlMn Z5KPWL83TdXf5JMKBNT0qqQfpKVvnzDlzkyQBdd1K.vO6KRTWIQtxoC.8GcIHmupRWRBlR4uyRru v83Eue0.WeBL_8vInUMwkFEA3NvnGey2CmXVyqY2oRyfNFpZTgEZ7UF3YOpcpqIFHg4W_MVCC5.X TgJX03VDewG4agABUb1o0REwSqRDKojlyKj5R1d2ERZFfmvp5XoOOhbVMQ6h5Ow953xynpLAPMGK pfpDxGTWwSv_gJkAu4coNJNe.bqA1mrNuGCEG1rqEjxEcxd09teH5yjnu0lLBs0VvSLJxv2K4QWf ketPnSYiZ8XT9RJ.k_ISJ6GhbtaS8X_2uTKej9uiGOen1melUtiKxVlO89.tSaHJtOquxWlDM0N3 t54uHMwYHqsR2PdkKerM6QFC1FJ6fWwKBC1IlUOArm4Fni4pkO5q5LatFLZmnOQfmiQqiL1qFKnj 6_32pkac9nG6AreCnuaD_9pfv0j9y8maBylML41ij0WgTFtlOVqa8Q.TXlyYOMk8fgfJ3haj76FU yBRPcUROzsG9Tto.hbVxEbASuImM4.GGcP.TuFU6PUo4wCoJzBTqZtLolZRp92j.1Q4e43lgVo2u yAkcCc8tYth9G1OUDVeBrl6v7FUZuT1ohEBSpiw2czrbTH3AOWJzY18M- Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Wed, 25 Mar 2020 08:18:42 +0000 Received: by smtp416.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID d326b47f009cf90864aa4ccb9235bbcb; Wed, 25 Mar 2020 08:18:38 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <20200325015633.GA93057@www.zefox.net> Date: Wed, 25 Mar 2020 01:18:36 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> <20200325015633.GA93057@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48nLcC2Kq5z4NT1 X-Spamd-Bar: - X-Spamd-Result: default: False [-1.84 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.88)[-0.877,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; URIBL_BLOCKED(0.00)[dsl-only.net.multi.uribl.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.46)[-0.460,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (4.68), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[30.68.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[30.68.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 08:19:05 -0000 On 2020-Mar-24, at 18:56, bob prohaska wrote: > [this thread needs a haircut, new comments start near line 288] > On Tue, Mar 24, 2020 at 05:21:47PM -0700, Mark Millard wrote: >> On 2020-Mar-24, at 15:46, bob prohaska wrote: >>=20 >>> . . . >>=20 >> . . . >>=20 >>=20 >> The text before "Mar 24 04:20:50" messages, with notes >> and blank lines mixed in to group things: >>=20 >> . . . >> Tue Mar 24 04:20:38 PDT 2020 Note the time above. >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 3/773/776/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 0 0 30 2034696 54180 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.031s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> 18 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 mmcsd0 >> 18 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 mmcsd0s2 >> 10 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 mmcsd0s2a >> 10 1 0 0 0.0 1 31 26686 0 0 = 0.0 287.1 ufs/rootfs >> Tue Mar 24 04:20:40 PDT 2020 >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 6/770/776/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> Note that the above has 26sec+ ms/w figures. No reads. >> No mmcsd0s2b. >>=20 >>=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 0 0 30 2034696 54252 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.064s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> Tue Mar 24 04:20:42 PDT 2020 >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 9/1023/1032/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> For the above, note the lack of any lines with ms/r or ms/w figures, >> just the title rows show. >>=20 >=20 > Same thing happens when gstat -a is run on an idle machine. > I thought it normal, the CPU had all the data it needed.=20 >=20 >=20 >=20 >>=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 0 0 30 2034696 53668 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.046s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> 17 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 mmcsd0 >> 17 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 mmcsd0s2 >> 9 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 mmcsd0s2a >> 9 1 0 0 0.0 1 31 30042 0 0 = 0.0 320.7 ufs/rootfs >> Tue Mar 24 04:20:43 PDT 2020 Note the time above compared to the earlier one that I referenced: around 5sec later (=3D=3D 43sec-38sec), despite the ms/w figures involved (over 26sec/w and over 30sec/w with a nothing-listed in the middle). >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 9/1023/1032/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> Note that the above has 30sec+ ms/w figures. No reads. >>=20 >> Also note that the L(q) figures dropped by only 1 over >> the about 04:20:40 to 04:20:43 interval (18->17 and >> 10->9). (I only take these deltas as suggestive. They >> might be accidental near matches.) No mmcsd0s2b. >>=20 >>=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 0 0 30 2034696 53624 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.003s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> Tue Mar 24 04:20:45 PDT 2020 >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 12/1020/1032/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> For the above, note the lack of any lines with ms/r or ms/w figures, >> just the title rows show. >>=20 >=20 > It's tempting, but maybe not correct, the think the I/O caught up. On this side of the 30sec+ example, things are less clear than the above note, so maybe. (Same below.) >>=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 0 0 30 2034696 53600 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.063s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> Tue Mar 24 04:20:47 PDT 2020 >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 12/1020/1032/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> For the above, note the lack of any lines with ms/r or ms/w figures, >> just the title rows show. >>=20 >=20 > Can't that be interpreted as gstat -a having nothing to report?=20 > Just _why_ it might have nothing to report never crossed my mind.... >=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 0 0 30 2034696 53592 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.006s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> Tue Mar 24 04:20:48 PDT 2020 >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 15/1017/1032/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> For the above, note the lack of any lines with ms/r or ms/w figures, >> just the title rows show. >>=20 >>=20 >>=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 0 0 30 2034696 53672 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.002s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> Tue Mar 24 04:20:50 PDT 2020 >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 403784 4000468 9% >> Mar 24 02:50:01 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 623056, size: 16384 >> Mar 24 04:17:17 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 1033854, size: 4096 >> 15/1017/1032/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> For the above, note the list of any lines with ms/r or ms/w figures, >> just the title rows. >>=20 >>=20 >>=20 >> procs memory page disks faults = cpu >> r b w avm fre flt re pi po fr sr mm0 da0 in sy = cs us sy id >> 2 0 30 2045028 54160 12942 7 2 4 13432 1374 0 0 14074 = 8102 2849 65 29 5 >> dT: 1.001s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name >> 2 329 236 3839 13.0 93 1262 18.6 0 0 = 0.0 99.4 mmcsd0 >> 2 329 236 3839 13.2 93 1262 18.7 0 0 = 0.0 101.1 mmcsd0s2 >> 1 153 152 2816 12.5 1 32 366.5 0 0 = 0.0 95.2 mmcsd0s2a >> 1 176 84 1023 14.5 92 1230 14.9 0 0 = 0.0 66.8 mmcsd0s2b >> 1 153 152 2816 12.5 1 32 366.6 0 0 = 0.0 95.3 ufs/rootfs >> Tue Mar 24 04:20:51 PDT 2020 >> Device 1K-blocks Used Avail Capacity >> /dev/mmcsd0s2b 4404252 296976 4107276 7% >> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 21842, size: 12288 >> Mar 24 04:20:50 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 25477, size: 4096 >> 0/1032/1032/19177 mbuf clusters in use (current/cache/total/max) >>=20 >> For the above, the lines with ms/r and ms/w figures are back >> but the indefinite wait buffer for "Mar 24 04:20:50" happened >> too, before the "Tue Mar 24 04:20:51 PDT 2020" above. I'd guess >> they are from before the data that the gstat output is based on. >>=20 >> Also there are lots of reads and some writes in the above >> gstat output. mmcsd0s2b is again showing as well. >>=20 >> Note: It looks like ufs/rootfs might be a label >> identifying mmcsd0s2a, so they track up to slight >> time-of-information differences. I'm guessing that >> mmcsd0s2b is just a swap/paging partition. >>=20 >=20 > That's correct. =20 >=20 >> So it looks like activity for ufs/rootfs may have >> blocked/delayed activity for paging/swaping >> (mmcsd0s2b), at least for a time, even if it is >> not directly the cause of the specific OOM >> activity. >>=20 >=20 > How would one distinguish slow swap i/o from slow > filesystem i/o ?=20 The 26sec+ and 30sec+ examples do not list mmcsd0s2b but do list mmcsd0s2a (and ufs/rootfs). Presumably that means no reas/writes to mmcsd0s2b in the time frame spanned. That is what I was referring to. I was not comparing against mmcsd0s2b I/O activity. > Would moving swap to a USB device avoid blocking=20 > by writes to microSD? That's not hard to try, but > it puts a whole new kettle of fish on the fire. Presuming the USB device is well performing, splitting the load across buses might well help. A good USB SSD might handle both activities better than involving the microsd card at all(?). It is possible to have only /boot/ materials on ufs on the microsd card and so to have the (ufs) root file system on the USB. For example, in the Pine64+2GB's /boot/loader.conf on the microsd card I have: vfs.root.mountfrom=3D"ufs:/dev/gpt/PINE642Groot" That label identifies the USB SSD's ufs partition holding the root file system. On that file system is an empty directory /microsd_ufs used as a mount point to get to the microsd card's ufs-based /boot/ . # gpart show -p =3D> 63 249737153 mmcsd0 MBR (119G) 63 16380 - free - (8.0M) 16443 131040 mmcsd0s1 fat32lba [active] (64M) 147483 997 - free - (499K) 148480 241172480 mmcsd0s2 freebsd (115G) 241320960 8416256 - free - (4.0G) =3D> 0 241172480 mmcsd0s2 BSD (115G) 0 230686720 mmcsd0s2a freebsd-ufs (110G) 230686720 10485760 - free - (5.0G) =3D> 40 468862048 da0 GPT (224G) 40 2008 - free - (1.0M) 2048 413138944 da0p1 freebsd-ufs (197G) 413140992 6291456 da0p2 freebsd-swap (3.0G) 419432448 6291456 da0p4 freebsd-swap (3.0G) 425723904 43138184 - free - (21G) # gpart show -pl =3D> 63 249737153 mmcsd0 MBR (119G) 63 16380 - free - (8.0M) 16443 131040 mmcsd0s1 (null) [active] (64M) 147483 997 - free - (499K) 148480 241172480 mmcsd0s2 (null) (115G) 241320960 8416256 - free - (4.0G) =3D> 0 241172480 mmcsd0s2 BSD (115G) 0 230686720 mmcsd0s2a (null) (110G) 230686720 10485760 - free - (5.0G) =3D> 40 468862048 da0 GPT (224G) 40 2008 - free - (1.0M) 2048 413138944 da0p1 PINE642Groot (197G) 413140992 6291456 da0p2 PINE642Gswap (3.0G) 419432448 6291456 da0p4 PINE642Gswp2 (3.0G) 425723904 43138184 - free - (21G) After booting it looks like: # df -m Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/gpt/PINE642Groot 195378 34613 145135 19% / devfs 0 0 0 100% /dev /dev/label/PINE64P2Groot 109101 75 100297 0% /microsd_ufs /dev/label/PINE642GAboot 63 43 20 69% /boot/efi (The Pine64+2GB uses a /boot/efi/ instead of a /boot/msdos/ .) # swapinfo Device 1K-blocks Used Avail Capacity /dev/gpt/PINE642Gswap 3145728 0 3145728 0% /dev/gpt/PINE642Gswp2 3145728 0 3145728 0% Total 6291456 0 6291456 0% (I'm not suggesting that having 2 swap partitions is important, it just is what I happen to have.) # dumpon -l gpt/PINE642Gswp2,gpt/PINE642Gswp2 (The duplication is a result of something I report later.) I also have in the USB SSD's ufs file system's /etc/fstab : /dev/label/PINE64P2Groot /microsd_ufs ufs rw,noatime = 1 1 (Again, I used labeling, in this case glabel based on the microsd card. Having a /boot/ is a subset of having a full root file system and I still used "root" terminology in the label.) I keep a /boot/ on the USB SSD and update the microsd card's copy from the USB copy: # more /boot/copy_boot_to_microsd.sh=20 rsync -axHh --info=3Dprogress2 --exclude=3D/boot/entropy --delete -r = /boot /microsd_ufs/ (I do not have /boot/entropy on the USB SSD, just on the microsd card at its /boot/entropy . The exclude prevents the --delete from touching the /boot/entropy file on the microsd card.) I also have in /etc/rc.conf (so: on the USB SSD): entropy_boot_file=3D"/microsd_ufs/boot/entropy" For reference: # ls -ldT /microsd_ufs/* -r--r--r-- 1 root wheel 6170 Feb 1 04:48:34 2020 = /microsd_ufs/COPYRIGHT drwxr-xr-x 11 root wheel 1024 Mar 24 19:06:27 2020 /microsd_ufs/boot (The COPYRIGHT file is, by content, the normal one that is on a root filesystem.) # ls -ldT /microsd_ufs/*/* -r--r--r-- 1 root wheel 3547 Mar 14 15:05:54 2020 = /microsd_ufs/boot/beastie.4th . . . -rw------- 1 root wheel 4096 Dec 31 16:00:25 2009 = /microsd_ufs/boot/entropy . . . drwxr-xr-x 2 root wheel 512 Dec 23 22:17:51 2019 = /microsd_ufs/boot/zfs (The date/time for /microsd_ufs/boot/entropy and /entropy are from before time is adjusted via ntp, so the dates can be odd.) I also use: dumpdev=3D"/dev/gpt/PINE642Gswp2" in /boot/loader.conf (both media) and in /etc/rc.conf . (This leads to the duplication noted earlier.) The same media-pair (microsd card plus USB SSD) can boot the RPi3. For the Pine64+2GB and RPi3, these days I use a small powered USB hub instead of directly plugging in the USB SSD to the arm-based board. I had power problems otherwise, at least on the Pine64+2GB in recent times. Technically, I use a USB3 hub and a USB3 SSD, both of which happen to allow USB2 use as well. The OPi+2e (armv7) and Rock64 (aarch64) are set up similarly. In all cases the USB SSDs used happen to perform better than the microsd cards used. (On the Rock64, the USB SSD also out performs the e.MMC that is in use --and the e.MMC outperforms the microsd card.) (The Rock64 USB3 is not operational yet. As stands, it too is a USB2 based context in my use.) I do not have other external USB devices involved. Mulitple external USB devices may share an internal hub on some arm boards, leading to a lack of multi-channel performance gains for multiple drives. > It should be added that the original buildworld has been restarted = with > vm.pfault_oom_attempts: -1 and so far, though it still generates the > "indefinite wait" warnings, the ms/w times look better. It's not done = yet, > though. How much overhead comes with OOMA? I doubt the ms/w figures span execution of the code that deals with OOM kill decisions or retries logic. But vm.pfault_oom_attempts=3D=3D-1 would avoid any retries, if I understand right. That in turn might put less of a load on the microsd card? (Guess work on my part.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Wed Mar 25 17:50:36 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AF1FE2A7285 for ; Wed, 25 Mar 2020 17:50:36 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.49.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gromit.dlib.vt.edu", Issuer "Chumby Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48nbHl2YM8z443j for ; Wed, 25 Mar 2020 17:50:26 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from mather.gromit23.net (c-98-244-101-97.hsd1.va.comcast.net [98.244.101.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id 4A06D108; Wed, 25 Mar 2020 13:40:47 -0400 (EDT) Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: arm64 pkg build broken since January From: Paul Mather In-Reply-To: Date: Wed, 25 Mar 2020 13:40:45 -0400 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <7755160D-36AB-447F-8711-4E5D1C130B7A@gromit.dlib.vt.edu> References: To: Ronald Klop X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 48nbHl2YM8z443j X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=vt.edu (policy=none); spf=none (mx1.freebsd.org: domain of paul@gromit.dlib.vt.edu has no SPF policy when checking 128.173.49.70) smtp.mailfrom=paul@gromit.dlib.vt.edu X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[vt.edu : No valid SPF, No valid DKIM,none]; RECEIVED_SPAMHAUS_PBL(0.00)[97.101.244.98.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.51)[ip: (-1.28), ipnet: 128.173.0.0/16(-0.64), asn: 1312(-0.57), country: US(-0.05)]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1312, ipnet:128.173.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 17:50:37 -0000 On Mar 23, 2020, at 5:48 PM, Ronald Klop wrote: > Hi, > > AFAIK the pkg build of arm64/aarch64 is broken for quite some time. > > ports-mgmt/pkg does not build on the build cluster since January 13th > http://thunderx1.nyi.freebsd.org/data/head-arm64-default/p522076_s356364/logs/errors/pkg-1.12.0.log > > pkgs are not building anymore since February 28th. > https://pkg-status.freebsd.org/builds?jailname=head-arm64 > > pkgs for 12.X also stopped building for some time now. > http://thunderx1.nyi.freebsd.org/ > > > I made a bugreport about it, but I see no confirmation or anything. > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244577 > > > Am I the only one without pkg updates on arm64? Or do I look wrong at the > wrong places? > Who could I contact about this? I cross-build my arm64/aarch64 12-STABLE packages using Poudriere running on a FreeBSD/amd64 system. Quite a few packages are still building for me but since at least 2020-02-01 I've had these three packages consistently fail to build: - net/p5-Socket6 - devel/p5-Locale-gettext - devel/p5-Locale-libintl The failure of those to build currently causes 64 packages to be skipped of the ones I normally build. According to my Poudriere logs, I last successfully built ports-mgmt/pkg on 2020-02-18. From owner-freebsd-arm@freebsd.org Thu Mar 26 22:06:50 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 80508262E10 for ; Thu, 26 Mar 2020 22:06:50 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48pJww54Fxz3DLy for ; Thu, 26 Mar 2020 22:06:39 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id 02QM6ouv099873 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 26 Mar 2020 15:06:51 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id 02QM6oWH099872; Thu, 26 Mar 2020 15:06:50 -0700 (PDT) (envelope-from fbsd) Date: Thu, 26 Mar 2020 15:06:49 -0700 From: bob prohaska To: Mark Millard Cc: freebsd-arm@freebsd.org, bob prohaska Subject: Re: Belated out of swap kill on rpi3 at r359216 Message-ID: <20200326220649.GA99824@www.zefox.net> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> <20200325015633.GA93057@www.zefox.net> <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Queue-Id: 48pJww54Fxz3DLy X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [1.39 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.44)[-0.438,0]; WWW_DOT_DOMAIN(0.50)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; IP_SCORE(0.06)[ip: (0.24), ipnet: 50.1.16.0/20(0.12), asn: 7065(-0.03), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-0.13)[-0.129,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MID_RHS_WWW(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 22:06:50 -0000 Just to wrap up, I tend to agree that delays writing to the microSD filesystem were blocking swap traffic and causing OOM kills. Turning off OOM allowed the OS build/install to complete successfully. Why this behavior started recently is less clear. The card was placed in service in October of 2018 and has been in strenuous use since then. The first hints of trouble occurred in late 2019. Perhaps this phenomenon is a useful warning of impending wearout. The gstat logs are at http://www.zefox.net/~fbsd/rpi3/swaptests/r359216/ in case anybody's curious. Thanks for all your attention, bob prohaska From owner-freebsd-arm@freebsd.org Thu Mar 26 23:25:28 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 35C6F264775 for ; Thu, 26 Mar 2020 23:25:28 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48pLgX6JQYz4CvB for ; Thu, 26 Mar 2020 23:25:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: bf4m7QcVM1nsSwiCiAXmII897i9fhaTM5XNG6RmG3zbKvw1QVR9i7t3BnQEb6op Qf5aVOleBoh5k_I9xVYYKpU6wAwPg0na8j9efst97bgITNX95nf5mVrqLFGgjVz6cYcP6mMLu6D_ jZR_NSlfQ2.2PWFA6k8LlZOaLA4WQ1AT_EB_rpNAdJB9gMJ3_55xVLZdV8ee5Ie2ld2qSieow4u8 NRgwoQScajTQHrSadPJAYCd4ZsS85q1RtwNRAbv2arvBMIK_Wq8hKuj0iGCQN1cdi1x3gxxbnHKT 99BfgdjHpdaFkhBW0kru5dCgbsrfQErdqiC0uphC40FX_FsDVMa8LO6_d8TpedjAnL3ywWCJImg1 0ZVnJDY5h4eixfSDq4qfhaoDTDuNYqKL1eaC4R8LoupHg9NRNAl5hbMn0tmgJQiuaf9U7a7oZ0P_ 3CadpbJvdo7EteuHUSWr2Lbz_TUqFMZesVGhwW5ADskya6JXLbTou_ad1.5uKO44PDNBpYjtzT0R 3juh43gf9WyoyZqTSp0_GNICidWWw4wm15SSrcrnrbpm.tVoK.IluMlE3zEMj8flZ9TfRoPkAuM7 HAhWBmmRzOPj3sgzEUMhnO24KMNfPqn25UlmxWMFza6HM9FeU.mlom_o7fzqkMU4314Xb50SgIxs 5sru0Tova8inFvDvvplJgB2p4xVHrEZz08ubhXbPzUnIBcQeo75dKWv4YJZiq46fq7tgy1Unx42V 36k0FFx4q3h_pDwoDh0V9lxmirwL__5_w2ChuJmgBeH.7ImFEUoYPJ4ayNZZpLQtPNM_FWCPtaCO aEGkBI15XpmR1KZ5zNSc7VuGio3uijAIsVs9lX5F8FZo1ZAZxa32RzZth8SswAY_3pQUN.EPfdm7 P25BfJ.4HX8cr2pXIO4ngveXCOWJz8VEw7LuAYp7gIr3JBtv95_hYmC3xIQ3ZbgfclWlmdzbDCaT NJ7AeR9tuQjCGh8Cc.164eMSpm.O6UDrMPKYjobr11FTDPcwzwgTuhlLSmKAWA6UAejsHeQBN5lF QC1WNEiE9LdjkA6bbxa4AKLAMT8CJhJsvLx48dgcLRtSgVMSYjz3J9X6J538kwHr.iARyAvgFu2f .IdttVMA69s.s6.Sk_1BUPI.3_wq3TEzScULhtHnSvdexxoknGgGJl1LdQzqhMr_by2K8P..TGr1 qzVqxUAwga0cFQth6EuA5t8rBJYj_fgquBAe.u1EJBOXI3kL7hB7_hNrJguT.0RkEqUIz4tjzqmG oouSje0NiwbQaSlI3RsCfVcQcBmr6yqBUj5ODEJH7GPIG3yGNzb4Sf6M_wVvpSYhy8XFIca38KBp 3MCUbXb0RM44lqgt4ezt5aAJhiVDCGkBPP1.4xuqcWr.fjxV5S6bRDpdwAKjp0Elqc7TUI3RO3Oa Ec1Lxcxf5DpzXAH.aAhU- Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Thu, 26 Mar 2020 23:25:02 +0000 Received: by smtp412.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID c6d89609fe3116e518e2388efb074be5; Thu, 26 Mar 2020 23:24:59 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <20200326220649.GA99824@www.zefox.net> Date: Thu, 26 Mar 2020 16:24:57 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <0A8CF8D1-8D0F-40E2-A10D-EB44BEEAB557@yahoo.com> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> <20200325015633.GA93057@www.zefox.net> <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> <20200326220649.GA99824@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48pLgX6JQYz4CvB X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.46 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.961,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (0.87), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[206.68.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[206.68.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 23:25:28 -0000 On 2020-Mar-26, at 15:06, bob prohaska wrote: > Just to wrap up, I tend to agree that > delays writing to the microSD filesystem were > blocking swap traffic and causing OOM kills. > Turning off OOM allowed the OS build/install > to complete successfully. There are still 2 types of "uma zone exhausted" issues that can lead to OOM activity, as well as vm.pageout_oom_seq being more of a "needs to be sufficiently large" than a direct-disable of the OOM handling of sustained-low-free-RAM periods. (And, only trying examples establishes what figures are sufficient for specific activities.) > Why this behavior started recently is less clear. > The card was placed in service in October of 2018 and > has been in strenuous use since then. The first hints > of trouble occurred in late 2019. Perhaps this phenomenon > is a useful warning of impending wearout. I'm not sure of the relative timing, but vm.pfault_oom_attempts and vm.pfault_oom_wait are fairly new and were (are still?) specific to head at 13. I expect they were in place for a while before we learned of them and what to do with them ( and so started using vm.pfault_oom_attempts=-1 ). This is sort of like vm.pageout_oom_seq being around for a long time before I learned of it and the background to understand using it: Mark J.'s earlier material from an earlier round of finding why OOM activity was happening on small arm boards. > The gstat logs are at > http://www.zefox.net/~fbsd/rpi3/swaptests/r359216/ > in case anybody's curious. > One of the unfortunate things is that having the logging of gstat and such also go to the same media (or even over a common channel but different media) also adds to the competing I/O load for paging/swapping. Also, if I remember right, all USB ports on the RPi3 share a common channel in the path (internal USB hub), limiting the utility of using multiple USB drives for helping manage these issues. It might be that one USB drive and one microsd card are as far as one can go for independent channels (ignoring WiFi and such). Microsd cards have issues of their own when involved, however. Side note: I've got access to the old RPi3 because the Pine64+2GB is having problems with I/O failures to longterm media (microsd cards, USB media directly attached, USB media via a powered hub). It might go a week without failure but when it does it tends to be a large sequence of failures. Various separate media all got such a problem eventually. I'm trying to see if the RPi3 also gets such issues with some of the same media the Pine64+2GB did. Anyway, I may, for a time, have one context that is more like yours than is normal for me. As stands, the RPi3 is doing a from-scratch buildworld buildkernel . (Reconstructing the head -r358966 that it is already running.) It is not splitting the I/O load but is using a USB SSD (via a powered hub), not the microsd card. No extra logging. vm.pfault_oom_attempts=-1 and vm.pageout_oom_seq=120 for this attempt. 3072 MiBytes of page/swap space. It is a -j4 build attempt. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Sat Mar 28 02:26:12 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BB73526C974 for ; Sat, 28 Mar 2020 02:26:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-21.consmr.mail.gq1.yahoo.com (sonic313-21.consmr.mail.gq1.yahoo.com [98.137.65.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48q2dg4TW6z4ZJM for ; Sat, 28 Mar 2020 02:25:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: iGtJN5kVM1kJzcPGqJ7VjsNwx0Eqb_tlBH5o6hWHsVy2wZ4.mbUZtkf5icD.Wvi C8c_LpV.IMKcW0Gi5nxcF52k6FGAwdjPIEPhi2PilOxt__z55bxDr09SURoG3gNPRIuPS3oqEcWa qMTqTaovzA5p56iuGhNvFR6r4mPaFipfQpD8iMd5YSotJYDqB2popPUwB19TwrCKUAfjrSf1OLqa LJQCosvbSXntLYXCfP1hFWfXqBU1S.eqx52x6mdLl0S5vlCVEPS3m_dK_C6Bxuwxv8XPdSdV0SQm dahZMF4dNlQZ87Yz.hnTspoaagqIKqzDMimsHZSNUpaRhUhSnXpc.AjSZ_iwiHxLS7_OY6iQzB76 TjuUaXJde2SvkScWgI_vlhQatEvDK25cRWprToNW_7Qv0.8RUzlVo42tjuRFVhzu8hbDwYlNIAqf V2I.Q4LArI0qKd4bhB7gdYEKDghNILlWd4mcOPvZ7I.RbAb4LgGIbEj8IrGfX47dlvCsBpKY0jgT nqktSNUtmyrsfI3p5IneKN_Ng6vYyTRxYyAXJH3Wyw2wmPF1LM9f18d6VVUw.L6KhZ_HKZfjYhtZ YDZeIe9Pt_6REsAcK_pGVRDyroHkLS8yC9JfpfhZKdq.NKc3asR_SwNdt3PrNN8GTbNBBaoE_onE 0RDIklRrq0k8YIU0aFFNGaoWLPVS1pzDdzqg27quyfkT_DIrF_EgwHIf_K9N18LbwGqJw58NuLts fNPoxwRQx3hV3_T23lp2hwTkkH9Go5d0Rw1u1QPKdNraT_qwRjfzxUoLK_4JX.bBiLMEv3XgE.LG T_rKbmXwMKumatFpwywbRurulzdmfVXuo6mio.lUa8ojoFPwiHcRm7N69HNbZqXNdQ58kIwYbTh9 vdvUM_.NGMwFWAUxCxEX_x9a3h6aAvrjUq31JoZzK.PdmyBDzwFcAmsiOupgxqrd0IbS9Q6rgpSR qy3L1hD3dyNx3cbaAzql.Rujh2j1E6bnWjmDJC0r7TbvMnMBZwIfgnQjI0uAE4OyWXepUs8pkbs. 1KCPiCQNnskPaB2hOh6kIjVgRkYfHWJ6V20_yCOF.II0ajhLWGS0OuvkweGrQXibpvTd.xLJEmpA bLlrZyfK9zzGRx0Nzj2MQiFClDBsPWC.29fX8aaLJbel3DwM3L2YqdrFKtBTSJnjStDVs_sPI_Hr h8M3k66rlYz2mNu4yBg9SiaYbgbkRGUy0oGlunKQUPlWReR91tUyjzsDmnecGBEDkNiFF179wi7f bS4V7E67586dtBAkHqNjdAKO3S4tKbiFnHKxa2qwT.wJYdsLGA.UMy.JdoKh6mkLu3TJtByRHsQc xeiHNF1rGZGlz.rFljnpyfcRdmt8TFgRsa7gdlZlIWYr4rpq313TVTyIeAp7rLNU5PyfSNPKpmn1 ftgPgaTGl0ySKFi4KKB1j Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sat, 28 Mar 2020 02:25:48 +0000 Received: by smtp426.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 8c4bccfc6b76987d39d15c26065aa22e; Sat, 28 Mar 2020 02:25:46 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <0A8CF8D1-8D0F-40E2-A10D-EB44BEEAB557@yahoo.com> Date: Fri, 27 Mar 2020 19:25:45 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5549E63B-0784-4B58-AD36-2A2EDC518308@yahoo.com> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> <20200325015633.GA93057@www.zefox.net> <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> <20200326220649.GA99824@www.zefox.net> <0A8CF8D1-8D0F-40E2-A10D-EB44BEEAB557@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48q2dg4TW6z4ZJM X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.40 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.91)[-0.907,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.991,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (1.61), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[84.65.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[84.65.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 02:26:12 -0000 On 2020-Mar-26, at 16:24, Mark Millard wrote: > On 2020-Mar-26, at 15:06, bob prohaska wrote: >=20 >> . . . >=20 > Anyway, I may, for a time, have one context that is > more like yours than is normal for me. As stands, the > RPi3 is doing a from-scratch buildworld buildkernel . > (Reconstructing the head -r358966 that it is already > running.) It is not splitting the I/O load but is > using a USB SSD (via a powered hub), not the microsd > card. No extra logging. vm.pfault_oom_attempts=3D-1 > and vm.pageout_oom_seq=3D120 for this attempt. 3072 > MiBytes of page/swap space. It is a -j4 build attempt. >=20 ("No extra logging" meant: beyond my normal typescript recording of the build output. That file ended up at 7741518 Bytes for size.) The build completed without any /var/log/message or console output during the build. My modified version of top reported (details copied from a ssh window) . . . For Mem: 738512Ki MaxObsActive, 190608Ki MaxObsWired, 906372Ki = MaxObs(Act+Wir) For Swap: 1927Mi MaxObsUsed (top was started before the build. "MaxObs" is short for "Maximum Observed".) The build took a few minutes under 31 hrs. (Ending: 2010-03-27:18:54:03 Starting: 2020-03-26:12:02:47). Because it was rebuilding -r358966 that it was already running, no bootstrap compiler or linker was built, despite it being a from-scratch build: The system compiler and linker were sufficient. For reference: the details of what was specified for building (contributes to "how long it took"). . . # more ~/src.configs/src.conf.cortexA53-clang-bootstrap.aarch64-host=20 TO_TYPE=3Daarch64 # KERNCONF=3DGENERIC-NODBG TARGET=3Darm64 .if ${.MAKE.LEVEL} =3D=3D 0 TARGET_ARCH=3D${TO_TYPE} .export TARGET_ARCH .endif # WITH_SYSTEM_COMPILER=3D WITH_SYSTEM_LINKER=3D # WITH_LIBCPLUSPLUS=3D WITHOUT_BINUTILS_BOOTSTRAP=3D WITH_ELFTOOLCHAIN_BOOTSTRAP=3D #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=3D WITH_LLVM_TARGET_AARCH64=3D WITH_LLVM_TARGET_ARM=3D WITHOUT_LLVM_TARGET_MIPS=3D WITHOUT_LLVM_TARGET_POWERPC=3D WITHOUT_LLVM_TARGET_RISCV=3D WITHOUT_LLVM_TARGET_X86=3D WITH_CLANG=3D WITH_CLANG_IS_CC=3D WITH_CLANG_FULL=3D WITH_CLANG_EXTRAS=3D WITH_LLD=3D WITH_LLD_IS_LD=3D WITHOUT_BINUTILS=3D WITH_LLDB=3D # WITH_BOOT=3D WITHOUT_LIB32=3D # # NO_WERROR=3D #WERROR=3D MALLOC_PRODUCTION=3D # # Avoid stripping but do not control host -g status as well: DEBUG_FLAGS+=3D # WITH_DEBUG_FILES=3D # # Use of the .clang 's here avoids # interfering with other CFLAGS # usage, such as ?=3D usage. CFLAGS.clang+=3D -mcpu=3Dcortex-a53 CXXFLAGS.clang+=3D -mcpu=3Dcortex-a53 CPPFLAGS.clang+=3D -mcpu=3Dcortex-a53 ACFLAGS.arm64cpuid.S+=3D -mcpu=3Dcortex-a53+crypto ACFLAGS.aesv8-armx.S+=3D -mcpu=3Dcortex-a53+crypto ACFLAGS.ghashv8-armx.S+=3D -mcpu=3Dcortex-a53+crypto # more /usr/src/sys/arm64/conf/GENERIC-NODBG=20 # # GENERIC -- Custom configuration for the arm64/aarch64 # include "GENERIC" ident GENERIC-NODBG makeoptions DEBUG=3D-g # Build kernel with gdb(1) = debug symbols options ALT_BREAK_TO_DEBUGGER options KDB # Enable kernel debugger support # For minimum debugger support (stable branch) use: #options KDB_TRACE # Print a stack trace for a = panic options DDB # Enable the kernel debugger # Extra stuff: #options VERBOSE_SYSINIT=3D0 # Enable verbose sysinit = messages #options BOOTVERBOSE=3D1 #options BOOTHOWTO=3DRB_VERBOSE #options KTR #options KTR_MASK=3DKTR_TRAP ##options KTR_CPUMASK=3D0xF #options KTR_VERBOSE # Disable any extra checking for. . . nooptions DEADLKRES # Enable the deadlock resolver nooptions INVARIANTS # Enable calls of extra sanity = checking nooptions INVARIANT_SUPPORT # Extra sanity checks of = internal structures, required by INVARIANTS nooptions WITNESS # Enable checks to detect = deadlocks and cycles nooptions WITNESS_SKIPSPIN # Don't run witness on spinlocks = for speed nooptions DIAGNOSTIC nooptions MALLOC_DEBUG_MAXZONES # Separate malloc(9) zones nooptions BUF_TRACKING nooptions FULL_BUF_TRACKING =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Sat Mar 28 16:17:42 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C81012A69E3 for ; Sat, 28 Mar 2020 16:17:42 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qP566QJ1z43MW for ; Sat, 28 Mar 2020 16:17:30 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id 02SGHgTN007665 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 28 Mar 2020 09:17:43 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id 02SGHgmc007664; Sat, 28 Mar 2020 09:17:42 -0700 (PDT) (envelope-from fbsd) Date: Sat, 28 Mar 2020 09:17:42 -0700 From: bob prohaska To: Mark Millard Cc: freebsd-arm@freebsd.org, bob prohaska Subject: Re: Belated out of swap kill on rpi3 at r359216 Message-ID: <20200328161742.GA7571@www.zefox.net> References: <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> <20200325015633.GA93057@www.zefox.net> <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> <20200326220649.GA99824@www.zefox.net> <0A8CF8D1-8D0F-40E2-A10D-EB44BEEAB557@yahoo.com> <5549E63B-0784-4B58-AD36-2A2EDC518308@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5549E63B-0784-4B58-AD36-2A2EDC518308@yahoo.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Queue-Id: 48qP566QJ1z43MW X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [2.23 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.11)[-0.108,0]; WWW_DOT_DOMAIN(0.50)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; IP_SCORE(0.05)[ip: (0.24), ipnet: 50.1.16.0/20(0.12), asn: 7065(-0.03), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.38)[0.383,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MID_RHS_WWW(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 16:17:42 -0000 On Fri, Mar 27, 2020 at 07:25:45PM -0700, Mark Millard wrote: > > > On 2020-Mar-26, at 16:24, Mark Millard wrote: > > > > > Anyway, I may, for a time, have one context that is > > more like yours than is normal for me. As stands, the > > RPi3 is doing a from-scratch buildworld buildkernel . > > (Reconstructing the head -r358966 that it is already > > running.) It is not splitting the I/O load but is > > using a USB SSD (via a powered hub), not the microsd > > card. No extra logging. vm.pfault_oom_attempts=-1 > > and vm.pageout_oom_seq=120 for this attempt. 3072 > > MiBytes of page/swap space. It is a -j4 build attempt. > > > > ("No extra logging" meant: beyond my normal typescript > recording of the build output. That file ended up at > 7741518 Bytes for size.) Does the process capture all the output from make buildworld? On my machines (pi2 and pi3) that's usually ~30 MB. > > > The build completed without any /var/log/message or > console output during the build. My modified version > of top reported (details copied from a ssh window) . . . > That seems to settle matters. My problems are with the old microSD card. New, it was marginally ok. Old, it's not. That crudely quantifies lifespan at around a year of active use, with trouble appearing roughly when the card was 75% full, at least a hint of required overprovisioning. Out of curiosity, have you tried leaving vm.pfault_oom_attempts at its default value? An OOM kill would be unexpected, but interesting if observed. > For Mem: 738512Ki MaxObsActive, 190608Ki MaxObsWired, 906372Ki MaxObs(Act+Wir) > For Swap: 1927Mi MaxObsUsed > Thanks for posting! bob prohaska > (top was started before the build. "MaxObs" is short > for "Maximum Observed".) > > The build took a few minutes under 31 hrs. > (Ending: 2010-03-27:18:54:03 > Starting: 2020-03-26:12:02:47). > > Because it was rebuilding -r358966 that it was already > running, no bootstrap compiler or linker was built, > despite it being a from-scratch build: The system > compiler and linker were sufficient. > > > > For reference: the details of what was specified for > building (contributes to "how long it took"). . . > > # more ~/src.configs/src.conf.cortexA53-clang-bootstrap.aarch64-host > TO_TYPE=aarch64 > # > KERNCONF=GENERIC-NODBG > TARGET=arm64 > .if ${.MAKE.LEVEL} == 0 > TARGET_ARCH=${TO_TYPE} > .export TARGET_ARCH > .endif > # > WITH_SYSTEM_COMPILER= > WITH_SYSTEM_LINKER= > # > WITH_LIBCPLUSPLUS= > WITHOUT_BINUTILS_BOOTSTRAP= > WITH_ELFTOOLCHAIN_BOOTSTRAP= > #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL= > WITH_LLVM_TARGET_AARCH64= > WITH_LLVM_TARGET_ARM= > WITHOUT_LLVM_TARGET_MIPS= > WITHOUT_LLVM_TARGET_POWERPC= > WITHOUT_LLVM_TARGET_RISCV= > WITHOUT_LLVM_TARGET_X86= > WITH_CLANG= > WITH_CLANG_IS_CC= > WITH_CLANG_FULL= > WITH_CLANG_EXTRAS= > WITH_LLD= > WITH_LLD_IS_LD= > WITHOUT_BINUTILS= > WITH_LLDB= > # > WITH_BOOT= > WITHOUT_LIB32= > # > # > NO_WERROR= > #WERROR= > MALLOC_PRODUCTION= > # > # Avoid stripping but do not control host -g status as well: > DEBUG_FLAGS+= > # > WITH_DEBUG_FILES= > # > # Use of the .clang 's here avoids > # interfering with other CFLAGS > # usage, such as ?= usage. > CFLAGS.clang+= -mcpu=cortex-a53 > CXXFLAGS.clang+= -mcpu=cortex-a53 > CPPFLAGS.clang+= -mcpu=cortex-a53 > ACFLAGS.arm64cpuid.S+= -mcpu=cortex-a53+crypto > ACFLAGS.aesv8-armx.S+= -mcpu=cortex-a53+crypto > ACFLAGS.ghashv8-armx.S+= -mcpu=cortex-a53+crypto > > # more /usr/src/sys/arm64/conf/GENERIC-NODBG > # > # GENERIC -- Custom configuration for the arm64/aarch64 > # > > include "GENERIC" > > ident GENERIC-NODBG > > makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols > > options ALT_BREAK_TO_DEBUGGER > > options KDB # Enable kernel debugger support > > # For minimum debugger support (stable branch) use: > #options KDB_TRACE # Print a stack trace for a panic > options DDB # Enable the kernel debugger > > # Extra stuff: > #options VERBOSE_SYSINIT=0 # Enable verbose sysinit messages > #options BOOTVERBOSE=1 > #options BOOTHOWTO=RB_VERBOSE > #options KTR > #options KTR_MASK=KTR_TRAP > ##options KTR_CPUMASK=0xF > #options KTR_VERBOSE > > # Disable any extra checking for. . . > nooptions DEADLKRES # Enable the deadlock resolver > nooptions INVARIANTS # Enable calls of extra sanity checking > nooptions INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS > nooptions WITNESS # Enable checks to detect deadlocks and cycles > nooptions WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed > nooptions DIAGNOSTIC > nooptions MALLOC_DEBUG_MAXZONES # Separate malloc(9) zones > nooptions BUF_TRACKING > nooptions FULL_BUF_TRACKING > > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > From owner-freebsd-arm@freebsd.org Sat Mar 28 16:23:05 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 072E12A6CFD for ; Sat, 28 Mar 2020 16:23:05 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail.karels.net (mail.karels.net [216.160.39.52]) by mx1.freebsd.org (Postfix) with ESMTP id 48qPCH2pqWz45Bd for ; Sat, 28 Mar 2020 16:22:50 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail.karels.net (localhost [127.0.0.1]) by mail.karels.net (8.15.2/8.15.2) with ESMTP id 02SGMcmi027728 for ; Sat, 28 Mar 2020 11:22:38 -0500 (CDT) (envelope-from mike@karels.net) Message-Id: <202003281622.02SGMcmi027728@mail.karels.net> To: freebsd-arm@freebsd.org From: Mike Karels Reply-to: mike@karels.net Subject: CFT: alpha test of Ethernet driver for RPi4 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <27726.1585412558.1@mail.karels.net> Date: Sat, 28 Mar 2020 11:22:38 -0500 X-Rspamd-Queue-Id: 48qPCH2pqWz45Bd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@karels.net designates 216.160.39.52 as permitted sender) smtp.mailfrom=mike@karels.net X-Spamd-Result: default: False [-4.47 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[mike@karels.net]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:216.160.39.52]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[karels.net]; REPLYTO_ADDR_EQ_FROM(0.00)[]; IP_SCORE(-2.27)[ip: (-7.40), ipnet: 216.160.36.0/22(-3.78), asn: 209(-0.10), country: US(-0.05)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:209, ipnet:216.160.36.0/22, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 16:23:05 -0000 I have a driver for the Ethernet on the RPi4 that is nearly ready for initial testing. If you have a RPi4 and are willing to test, let me know. Also, if anyone is able to review the busdma hooks, that would be appreciated. There are a lot of loose ends. The main missing feature is checksum offload. There are also some question marks in the initialization code that make configuration fairly noisy at the moment. This driver is derived in part from the NetBSD bcmgenet.c driver by Jared McNeill. I also used framework from the Allwinner if_awg.c driver, also by Jared McNeill. Mike From owner-freebsd-arm@freebsd.org Sat Mar 28 18:38:57 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F3868261E9B for ; Sat, 28 Mar 2020 18:38:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48qSCz53qHz3y5D for ; Sat, 28 Mar 2020 18:38:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: MO6UVRAVM1nF.Wgk0k_f2IavZrcM1lhjAz_oo7l2P5xd1SKt6YpybVdAb47LieJ XLkgikIv2PSnxpkLft7KDZeX4x32ITLXT70ymumDgzaNhhdn051r1mLx9mgxs4ozKvZa8iXXIzCP OlZ2fgveQw.TyhIOavCLPvEOMAmWxwqbnZ_o_8M_oa0Mwy7Dl6GC2BGTNxk19YSpjxcztkGGv4ig llrLETjnUfaWxxLlQVmsjppowntTfgmJpz78PrLxhHA9lL6tRUZ87zY.NnAObPJMoAEGH5x743tC 0jNXEjqsxWPW.JvrYM7ODdj7qlBylvU.F0idLYWF8XWWuxkDbZj88hXj6_ZqxPhFAZU.jN.hy1dD .PLV0C.a4jhUZoM0MIF6tHu_OYov83oKnfrcSyyAwLfztiYSZSYk1o6cvx696tghE7CSqFizT6TH 2bsm75Qq41BvVjoTc64B7y_nb.t9DNWksoEiT.SqkdEztdorjXLv50bHUBzuczd7AXqNsy21dtL2 anOmNVrWmCjy53b4mKR6qHV1UaLW6S0vH7wKjNIhqGwz7pWdLRjAoKc36DJqRhSc50XYdmpahi6g 4rEfK33HeOTchK3rXLu9ZdIM8Mvmb973zNhk3R.qJZpn32Ec0xaU3oTxot9LBPJ1OTcYTHKVJ3KN vFNbiEstwUQEptMf9LhxDLuN4do3wiAJ1L7Rf2ModGGvRii.iT3vqqUlimxECma887.hXmiLEJjM 7tfLbPfUP6l1pdCRqOsvliCtKGJC6H5m.HIunWkZI8bJZKX1coZ.pe.Og8MkkAD25cUMt9LkHS82 bz8lahpnIEbhaG1cpeBINdSeugi0kLyIfgGwuimS9fS26YLu.6dgCAaND.jap0QyU4AV8oMEKjx6 rsPRQ0zf8.I_ekp8d_yZMCNZ70BvEmuYwVLcdvG6RFkWQKybhwpTS9iEHVEKqbiBF8k01x0VFmsk M0wGgeWbWo2KEQ8LNq.9SC_HhexN_oQOegBb.r0kfZXcWTivSbU7hRgkpL368K3769VqOUbJ4vjC nH_CJzgdYVZ94ksXDHOd31CeEjow8rj5DEPhH7Ph2WM2_PiQs8Cs3W5ZhAQqaEAEwXNywV44.lF7 t4nQZia4AeDHc.pxHfh3IF1EGPA3MfQmfnnWYLfyudCUq.u9QoZ_uJeKgk.XGpHfn20KQsuW3QJb lTRaTByfqrpGenwtsmveQf4t.TycsCLBA44MLL7fiarih81ScexIFoBFQoD74xQJnnqcaotOg794 mbpS0QArvnvlH5XxZg7_8bdst_Y7EEnA8FxNcXyxtwjeQ3yp9rguF942k7bklmz5H5GWL9HFGT5t SHmUrKQiwBeDoooj6JmJXqpp4kPhssBEXBq_yPNmpBIPQ1bIeGGqWvycgX1YU708M_5zuhn5HR5d 6FL_V2gphGBaxBA3.Scg- Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Sat, 28 Mar 2020 18:38:27 +0000 Received: by smtp409.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID e6f364188e4d4d03a1ed14c05ce53ac0; Sat, 28 Mar 2020 18:38:21 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <20200328161742.GA7571@www.zefox.net> Date: Sat, 28 Mar 2020 11:38:19 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5CBFD168-D533-4BF4-AB9C-64B8B98F4B84@yahoo.com> References: <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> <20200325015633.GA93057@www.zefox.net> <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> <20200326220649.GA99824@www.zefox.net> <0A8CF8D1-8D0F-40E2-A10D-EB44BEEAB557@yahoo.com> <5549E63B-0784-4B58-AD36-2A2EDC518308@yahoo.com> <20200328161742.GA7571@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48qSCz53qHz3y5D X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.44 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.982,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-0.96)[-0.960,0]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[205.69.137.98.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (2.54), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.65), country: US(-0.05)]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 18:38:57 -0000 On 2020-Mar-28, at 09:17, bob prohaska wrote: > On Fri, Mar 27, 2020 at 07:25:45PM -0700, Mark Millard wrote: >>=20 >>=20 >> On 2020-Mar-26, at 16:24, Mark Millard wrote: >>=20 >>>=20 >>> Anyway, I may, for a time, have one context that is >>> more like yours than is normal for me. As stands, the >>> RPi3 is doing a from-scratch buildworld buildkernel . >>> (Reconstructing the head -r358966 that it is already >>> running.) It is not splitting the I/O load but is >>> using a USB SSD (via a powered hub), not the microsd >>> card. No extra logging. vm.pfault_oom_attempts=3D-1 >>> and vm.pageout_oom_seq=3D120 for this attempt. 3072 >>> MiBytes of page/swap space. It is a -j4 build attempt. >>>=20 >>=20 >> ("No extra logging" meant: beyond my normal typescript >> recording of the build output. That file ended up at >> 7741518 Bytes for size.) >=20 > Does the process capture all the output from make buildworld? > On my machines (pi2 and pi3) that's usually ~30 MB.=20 A likely explanation is that I use WITH_META_MODE and you might not: WITH_META_MODE . . . The build hides commands that are executed unless NO_SILENT = is defined. Errors cause make(1) to show some of its = environment for further debugging. . . . (I do not use NO_SILENT, so I get the hiding.) Over 1/2 of the lines recorded looked like sequences similar to: . . . Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/obj-tools= /tools/build/dummy.o Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/obj-tools= /tools/build/libegacy.a . . . In this case: # grep "^Building " = /root/sys_typescripts/typescript_make_cortexA53_nodebug_clang_bootstrap-aa= rch64-host-2020-03-26:12:02:47 | wc 52487 104974 5767152 vs. the file overall: # wc = /root/sys_typescripts/typescript_make_cortexA53_nodebug_clang_bootstrap-aa= rch64-host-2020-03-26:12:02:47 94908 256377 7741518 = /root/sys_typescripts/typescript_make_cortexA53_nodebug_clang_bootstrap-aa= rch64-host-2020-03-26:12:02:47 WITH_META_MODE does record details for each "Building" line in a .meta file specific to that line. A .meta file even includes a list of what files were involved (opened) for that step. So their is still file I/O for such logging, likely more in total than when not using WITH_META_MODE. (Not that I'd thought about that before.) >>=20 >> The build completed without any /var/log/message or >> console output during the build. My modified version >> of top reported (details copied from a ssh window) . . . >>=20 >=20 > That seems to settle matters. My problems are with the old > microSD card. New, it was marginally ok. Old, it's not. That > crudely quantifies lifespan at around a year of active use, > with trouble appearing roughly when the card was 75% full, > at least a hint of required overprovisioning. Since FreeBSD provides no means of having the SATA drive in the USB enclosure trimmed(?), I do not know how long before it would have issues from that. It is a small form factor 240 GByte SSD [user space, not GiByte, likely from internal over-provisioning of a 240 GiByte media]. I left a 21 GiByte area at the end free as well. The 197 GiByte ufs file system is only about 19% used. smartctl reports for the USB SSD internals: ATA Version is: ATA8-ACS, ACS-2 T13/2015-D revision 3 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) The Firmware version 609ABBF0 listed suggests a Seagate SATA controller is involved, if I understand right. The USB SSD drive is far from new. It now gets the report: Device Statistics (GP Log 0x04) Page Offset Size Value Flags Description . . . 0x01 0x018 6 5499176259 --- Logical Sectors Written 0x01 0x028 6 2406890437 --- Logical Sectors Read . . . where earlier smartctl reported: Sector Size: 512 bytes logical/physical > Out of curiosity, have you tried leaving vm.pfault_oom_attempts at=20 > its default value? An OOM kill would be unexpected, but interesting=20 > if observed.=20 Nope. I've thought of locally updating gstat to do something similar to what I did with top: record and report the maximum observed figures for ms/r, ms/w, ms/d, but for each line of data in this case. I'd not be surprised if the heavier paging times had some large figures compared to what I saw when watching the display. (Rarely more than 20ms.) But my observations are not much of a sample. I'd be more likely to try picking vm.pfault_oom_wait after seeing what is reported, then picking a positive vm.pfault_oom_attempts value to go with it. I'm not sure if I'll ever do this sort of experiment. The resulting figures used would be rather context-specific as well. >> For Mem: 738512Ki MaxObsActive, 190608Ki MaxObsWired, 906372Ki = MaxObs(Act+Wir) >> For Swap: 1927Mi MaxObsUsed >>=20 >=20 > . . . =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-arm@freebsd.org Sat Mar 28 19:38:20 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D3536263A98 for ; Sat, 28 Mar 2020 19:38:20 +0000 (UTC) (envelope-from db@db.net) Received: from tfm.com (mtbaker.tfm.com [192.231.224.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.tfm.com", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qTXc2pnpz4Lr6 for ; Sat, 28 Mar 2020 19:38:07 +0000 (UTC) (envelope-from db@db.net) Received: from night.db.net (DB-DSL.ServerNorth.com [98.124.61.131]) (authenticated bits=0) by tfm.com (8.14.4/8.14.4) with ESMTP id 02SJblMB003790 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 28 Mar 2020 12:37:50 -0700 (PDT) Date: Sat, 28 Mar 2020 15:37:45 -0400 From: Diane Bruce To: Mike Karels Cc: freebsd-arm@freebsd.org Subject: Re: CFT: alpha test of Ethernet driver for RPi4 Message-ID: <20200328193745.GA97797@night.db.net> References: <202003281622.02SGMcmi027728@mail.karels.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <202003281622.02SGMcmi027728@mail.karels.net> X-Rspamd-Queue-Id: 48qTXc2pnpz4Lr6 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of db@db.net has no SPF policy when checking 192.231.224.2) smtp.mailfrom=db@db.net X-Spamd-Result: default: False [-1.79 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.967,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.985,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[db.net]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:10488, ipnet:192.231.224.0/22, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-0.74)[ip: (-1.92), ipnet: 192.231.224.0/22(-0.96), asn: 10488(-0.77), country: US(-0.05)] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 19:38:21 -0000 On Sat, Mar 28, 2020 at 11:22:38AM -0500, Mike Karels wrote: > I have a driver for the Ethernet on the RPi4 that is nearly ready for > initial testing. If you have a RPi4 and are willing to test, let me > know. Also, if anyone is able to review the busdma hooks, that would > be appreciated. Yes please. > > There are a lot of loose ends. The main missing feature is checksum > offload. There are also some question marks in the initialization code > that make configuration fairly noisy at the moment. > > This driver is derived in part from the NetBSD bcmgenet.c driver by > Jared McNeill. I also used framework from the Allwinner if_awg.c driver, > also by Jared McNeill. > > Mike > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" -- - db@FreeBSD.org db@db.net http://www.db.net/~db From owner-freebsd-arm@freebsd.org Sat Mar 28 21:23:34 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 89C60266A84 for ; Sat, 28 Mar 2020 21:23:34 +0000 (UTC) (envelope-from maciphone2@googlemail.com) Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qWt55YH5z42Jq for ; Sat, 28 Mar 2020 21:23:25 +0000 (UTC) (envelope-from maciphone2@googlemail.com) Received: by mail-wr1-x442.google.com with SMTP id j17so16133440wru.13 for ; Sat, 28 Mar 2020 14:23:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=jiNXSjCgR2B1xeILhNuxSmMTWc3EWHekfnZxieNQtdo=; b=DBbCO8jqP0tTME4Ps5Cye+DhAhxysvzCuFO+wSk53BrNThpxdJDDFbPX6xxr/XO2vJ DHSk0ChkKd4onsbHW9BsjHXrTmcUFGDF7m/ygoBWwgGpDfNhzmaaiDV2+dmqX3cL1sHX YR+dve49D4kcP27BbthFoFZcesCu6IB3KQ+z4nG+bwTy4EutEg4b+3WW4B7mvczoztjS YsAEUXBQxc2i5T/sY1dIM4ZRd/STen0emMiAXlElClKZjDTLjkboYPG0GdS/TbjeedfD EcegYo+ofeeEvo5SWwdkf4A9NWVXkYLstir3w2iLtqTgvDYk0q0U1kZ02trlbmP/gXpB T+Hw== X-Gm-Message-State: ANhLgQ1+w7i3Evx8pqwJwOUZIJWL1PeVOPt9yQxSsGZ314TNmzz2xoss 70fw/ZQs0HXKnJQJCB2xpMQ7D1f2 X-Google-Smtp-Source: ADFU+vs8TuCTa1lsAvtOvJSJ1Tc5f67xQJAS3xD+es4CSbySKHazcnnEgvVWrv4Y9A1M8zbryWt8KQ== X-Received: by 2002:adf:aade:: with SMTP id i30mr6512771wrc.336.1585430594874; Sat, 28 Mar 2020 14:23:14 -0700 (PDT) Received: from [192.168.1.168] (x59cc9a4f.dyn.telefonica.de. [89.204.154.79]) by smtp.googlemail.com with ESMTPSA id f12sm13711785wmh.4.2020.03.28.14.23.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Mar 2020 14:23:13 -0700 (PDT) From: =?utf-8?Q?Klaus_K=C3=BCchemann?= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Switch to UEFI Re: CFT: alpha test of Ethernet driver for RPi4 Date: Sat, 28 Mar 2020 22:23:12 +0100 References: <202003281622.02SGMcmi027728@mail.karels.net> To: Mike Karels , freebsd-arm@freebsd.org In-Reply-To: <202003281622.02SGMcmi027728@mail.karels.net> Message-Id: X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48qWt55YH5z42Jq X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[googlemail.com]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[googlemail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[googlemail.com,quarantine]; RECEIVED_SPAMHAUS_PBL(0.00)[79.154.204.89.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[googlemail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (2.48), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-0.46), country: US(-0.05)]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 21:23:34 -0000 > Am 28.03.2020 um 17:22 schrieb Mike Karels : >=20 > I have a driver for the Ethernet on the RPi4 that is nearly ready for > initial testing. If you have a RPi4 and are willing to test, let me > know. Also, if anyone is able to review the busdma hooks, that would > be appreciated. >=20 > There are a lot of loose ends. The main missing feature is checksum > offload. There are also some question marks in the initialization = code > that make configuration fairly noisy at the moment. >=20 > This driver is derived in part from the NetBSD bcmgenet.c driver by > Jared McNeill. I also used framework from the Allwinner if_awg.c = driver, > also by Jared McNeill. >=20 > Mike > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to = "freebsd-arm-unsubscribe@freebsd.org=E2=80=9C Hi, I`m also working on pi4 a little while not ready=E2=80=A6 I recommend to switch the whole RPI-thing to UEFI/ ACPI, J.McNeill is = working on that , afaik=E2=80=A6 So next step I see is to implement ACPI-xhci-driver since FreeBSD-boot = PANICS on xhci- The good news is that I first saw USB working in UEFI-boot...=E2=80=A6the = ethernet driver should then later correspond or to be integrated into = UEFI =E2=80=A6 nice to boot from a =E2=80=9EBios=E2=80=9C -style menu = with a plugged USB-keyboard... Regards Klaus=20 =20 From owner-freebsd-arm@freebsd.org Sat Mar 28 21:34:00 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 95A46266E3F for ; Sat, 28 Mar 2020 21:34:00 +0000 (UTC) (envelope-from maciphone2@googlemail.com) Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qX6051Gvz45dL for ; Sat, 28 Mar 2020 21:33:44 +0000 (UTC) (envelope-from maciphone2@googlemail.com) Received: by mail-wr1-x42f.google.com with SMTP id p10so16217756wrt.6 for ; Sat, 28 Mar 2020 14:33:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=P+HkubRCEYXjCcwndKdoYmQvFMngTFuThCnjP8HRv2A=; b=eKxZmZBrafcTZf+mdsGqZqMeI/EWPIE8+xleJYkExPoFDoapowzMDDzzUTrUFKFE8X 6eyoXAIeG2hKxsN4+sxwkzZwLItp3kWN2wbiJYzVjyUFzKBpuwD29d0aMxC1B0WKvjmG /COLoyFwu+pD6OIccUWOGxbeir4UCn2QHnIHazD5ZSISBHIrlZohajx/Q+Hv3xUoJKwm 85f7CO8hhAQl5synSCBD92RbsJ0vbZeszRUeK1wc1oJWdhZQrncNTlee/09ehGyLp9HX 0jyMUeq96+ISF2v+XyAKJQ3eT0oPFxAp1h4cT5EXAPnUK0FyTWvjsoTpFjhyapepwrCo R34w== X-Gm-Message-State: ANhLgQ1unfm69Y73PXnVPDsgmh1jOxQa5/4TIxAZ2AdAiwbJRYr8Wkb/ VTdzYb5VsMlmnbqrR2CKlIU= X-Google-Smtp-Source: ADFU+vsryGcjkRK7wYmfRtAGsfv5VT0GTUDlQFFBNJP9N90DUQ5R28/Lu8dG6bokbnmLWTxu4j+DgQ== X-Received: by 2002:adf:f68a:: with SMTP id v10mr6575395wrp.80.1585431215361; Sat, 28 Mar 2020 14:33:35 -0700 (PDT) Received: from [192.168.1.168] (x59cc9a4f.dyn.telefonica.de. [89.204.154.79]) by smtp.googlemail.com with ESMTPSA id o145sm2275824wme.42.2020.03.28.14.33.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Mar 2020 14:33:34 -0700 (PDT) From: =?utf-8?Q?Klaus_K=C3=BCchemann?= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: git? Re: CFT: alpha test of Ethernet driver for RPi4 Date: Sat, 28 Mar 2020 22:33:33 +0100 References: <202003281622.02SGMcmi027728@mail.karels.net> To: Mike Karels , freebsd-arm@freebsd.org In-Reply-To: <202003281622.02SGMcmi027728@mail.karels.net> Message-Id: <82938D52-F986-4D83-8389-2BBA958651CE@googlemail.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48qX6051Gvz45dL X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[googlemail.com]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[googlemail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[googlemail.com,quarantine]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[79.154.204.89.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[googlemail.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[f.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (-9.07), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-0.46), country: US(-0.05)]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 21:34:00 -0000 > Am 28.03.2020 um 17:22 schrieb Mike Karels : >=20 > ... > This driver is derived in part from the NetBSD bcmgenet.c driver by > Jared McNeill. I also used framework from the Allwinner if_awg.c = driver, > also by Jared McNeill. >=20 > Mike > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to = "freebsd-arm-unsubscribe@freebsd.org=E2=80=9C Forgot to ask:=20 Do you have git repo where we can track your ethernet-implementation ? By the way =E2=80=A6 there are discussing channels for the UEFI .. if = someone=E2=80=99s interested in we could step=20 In there or open an own discussing channel there =E2=80=A6 Regards Klaus= From owner-freebsd-arm@freebsd.org Sat Mar 28 22:15:53 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7907C2686C7 for ; Sat, 28 Mar 2020 22:15:53 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: from out.migadu.com (out.migadu.com [91.121.223.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.migadu.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qY2M5pdkz4MP9 for ; Sat, 28 Mar 2020 22:15:38 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: (Migadu outbound); Sat, 28 Mar 2020 22:15:27 +0000 Received: from wms0-eu-central.migadu.com (wms0-eu-central.migadu.com [139.162.159.86]) by out.migadu.com (Haraka/2.8.16) with ESMTPSA id B05238C0-B4DC-4AD1-A813-CC63379A5502.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 verify=FAIL); Sat, 28 Mar 2020 22:15:27 +0000 MIME-Version: 1.0 Date: Sat, 28 Mar 2020 22:15:20 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: RainLoop/1.12.1 From: greg@unrelenting.technology Message-ID: <57d4ba4ef95eeaf382d2c0b2407e9dab@unrelenting.technology> Subject: Re: Switch to UEFI Re: CFT: alpha test of Ethernet driver for RPi4 To: "=?utf-8?B?S2xhdXMgS8O8Y2hlbWFubg==?=" , "Mike Karels" , freebsd-arm@freebsd.org In-Reply-To: References: <202003281622.02SGMcmi027728@mail.karels.net> DKIM-Signature: v=1; a=rsa-sha256; bh=7nTofmwOAYysRVsJqHwRLR7igSIDQNSif1ZEUIaqLGY=; c=relaxed/simple; d=unrelenting.technology; h=from:subject:date:to; s=default; b=Knx0D237JHMrRUPIkeE0lP63q+n6tlATx1iOISI+8aSZTVPbrnNwF77yBi/4PAvwSD+zxkjsKSrXn0ulEavNudXzMMqBDY+BjFsFhZb9HOKIH0RYjWG4jExiWrsm0UDNwIAt3DxD2W/NtCpm2DckMcnw9sXCVxJ0kwZ59LYRO8w= X-Rspamd-Queue-Id: 48qY2M5pdkz4MP9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=unrelenting.technology header.s=default header.b=Knx0D237; dmarc=pass (policy=none) header.from=unrelenting.technology; spf=pass (mx1.freebsd.org: domain of greg@unrelenting.technology designates 91.121.223.63 as permitted sender) smtp.mailfrom=greg@unrelenting.technology X-Spamd-Result: default: False [-4.85 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[unrelenting.technology:s=default]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:91.121.223.63]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[unrelenting.technology:+]; DMARC_POLICY_ALLOW(-0.50)[unrelenting.technology,none]; RCVD_IN_DNSWL_NONE(0.00)[63.223.121.91.list.dnswl.org : 127.0.10.0]; FROM_NO_DN(0.00)[]; FREEMAIL_TO(0.00)[googlemail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-1.85)[ip: (-9.79), ipnet: 91.121.0.0/16(-1.51), asn: 16276(2.05), country: FR(0.00)]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 22:15:53 -0000 March 29, 2020 12:23 AM, "Klaus K=C3=BCchemann via freebsd-arm" wrote:=0A=0A> I`m also working on pi4 a little while not= ready=E2=80=A6=0A> I recommend to switch the whole RPI-thing to UEFI/ AC= PI, J.McNeill is working on that , afaik=E2=80=A6=0A=0AHe's working on Ne= tBSD a lot, but not on FreeBSD.=0A=0A> So next step I see is to implement= ACPI-xhci-driver since FreeBSD-boot PANICS on xhci-=0A=0AThat's not "imp= lement a driver", that's "fix a bug" :)=0AI was looking into this a month= ago..=0AFor some reason reading from the XHCI's memory space returns jun= k like 0xdead.=0AAnd seems like nobody has seen that problem on NetBSD or= Linux.=0AI'll test this a bit more now.=0A=0AOnce we get that working, a= nd the Ethernet driver port is published, I could add ACPI attachment to = the Ethernet driver.=0A=0A> The good news is that I first saw USB working= in UEFI-boot...=E2=80=A6the ethernet driver should then later=0A> corres= pond or to be integrated into UEFI =E2=80=A6 nice to boot from a =E2=80= =9EBios=E2=80=9C -style menu with a plugged=0A> USB-keyboard...=0A=0AYeah= , eventually there's going to be a NetBSD-derived Ethernet driver in UEFI= for network booting. From owner-freebsd-arm@freebsd.org Sat Mar 28 22:28:16 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A369C268DAB for ; Sat, 28 Mar 2020 22:28:16 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail.karels.net (mail.karels.net [216.160.39.52]) by mx1.freebsd.org (Postfix) with ESMTP id 48qYJd47QGz4RMm for ; Sat, 28 Mar 2020 22:28:00 +0000 (UTC) (envelope-from mike@karels.net) Received: from [10.0.2.10] (mjk-mac.karels.net [10.0.2.10]) by mail.karels.net (8.15.2/8.15.2) with ESMTP id 02SMRrTm029085; Sat, 28 Mar 2020 17:27:53 -0500 (CDT) (envelope-from mike@karels.net) From: "Mike Karels" To: "Klaus =?utf-8?q?K=C3=BCchemann?=" Cc: freebsd-arm@freebsd.org Subject: Re: git? Re: CFT: alpha test of Ethernet driver for RPi4 Date: Sat, 28 Mar 2020 17:27:24 -0500 X-Mailer: MailMate (1.13.1r5671) Message-ID: In-Reply-To: <82938D52-F986-4D83-8389-2BBA958651CE@googlemail.com> References: <202003281622.02SGMcmi027728@mail.karels.net> <82938D52-F986-4D83-8389-2BBA958651CE@googlemail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mail.karels.net id 02SMRrTm029085 X-Rspamd-Queue-Id: 48qYJd47QGz4RMm X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@karels.net designates 216.160.39.52 as permitted sender) smtp.mailfrom=mike@karels.net X-Spamd-Result: default: False [-4.48 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:216.160.39.52]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[karels.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-2.28)[ip: (-7.42), ipnet: 216.160.36.0/22(-3.83), asn: 209(-0.11), country: US(-0.05)]; FREEMAIL_TO(0.00)[googlemail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:209, ipnet:216.160.36.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 22:28:17 -0000 On 28 Mar 2020, at 16:33, Klaus K=C3=BCchemann wrote: >> Am 28.03.2020 um 17:22 schrieb Mike Karels : >> >> ... >> This driver is derived in part from the NetBSD bcmgenet.c driver by >> Jared McNeill. I also used framework from the Allwinner if_awg.c=20 >> driver, >> also by Jared McNeill. >> >> Mike >> _______________________________________________ >> freebsd-arm@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-arm >> To unsubscribe, send any mail to=20 >> "freebsd-arm-unsubscribe@freebsd.org=E2=80=9C > Forgot to ask: > Do you have git repo where we can track your ethernet-implementation ? > By the way =E2=80=A6 there are discussing channels for the UEFI .. if=20 > someone=E2=80=99s interested in we could step > In there or open an own discussing channel there =E2=80=A6 > > Regards > Klaus I=E2=80=99m not ready to put the source out in public, as the copyright n= otice=20 is not yet approved. I=E2=80=99ll put it in phabricator when it=E2=80=99= s ready. About UEFI: I don=E2=80=99t know what=E2=80=99s involved, but half of the= driver=20 interfaces with the rest of the network stack, and I don=E2=80=99t see ho= w=20 using UEFI simplifies any of that. Mike From owner-freebsd-arm@freebsd.org Sat Mar 28 22:36:19 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1ADD72691A9 for ; Sat, 28 Mar 2020 22:36:19 +0000 (UTC) (envelope-from maciphone2@googlemail.com) Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qYTz1q91z4V2q for ; Sat, 28 Mar 2020 22:36:06 +0000 (UTC) (envelope-from maciphone2@googlemail.com) Received: by mail-wr1-x434.google.com with SMTP id w10so16421678wrm.4 for ; Sat, 28 Mar 2020 15:36:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=eF/1S7F6hqXNaOwgNtg78PiUQ39oByDJ2gNDyCe2joc=; b=ZU8VDg2I40SdsFHny4UgOCwvkwY2SfcVhFJwWlKzoURFX9Gj9p25t4mLD/0WNjTikZ oyJ3P5WslmAXwPvOP9SKHEOK9Rb8cJmGzHhl8P+5ItgbmVKkLIGtIeH/3hW8xFQvx4Sn ploxbuLVLJCBO6STgntYruiX5hz7C4XoZcoV778lY0Mbrhtapj0oC2VLiqXmtORKEM0+ hhLnvB6YcOoJm0ESjqReltjqE05GhQV9/oCsOCD85o0h3YRWX7JchPgcq9Xet88KL9RZ iPTlRcykM3jFSocvn57+ZqGPCCaI2lSFBrk/1iMh5PS2Q5GdhjuBJfKyyaUGkavMK8Li SEFQ== X-Gm-Message-State: ANhLgQ0bnLcXJTaeggXnWsfFluOQseSsOSJJ7BiMh30BMF+041s/BS5e wMy2+h3tKmKtbYGIYLYBR8Hznu+h X-Google-Smtp-Source: ADFU+vvEjTvY4lURv2AtjOLoL/OI3Q4FRbsfaftJjhejt7GANs2fOt8saAf8Qqf5W7twq/0WK9r8EQ== X-Received: by 2002:adf:e98b:: with SMTP id h11mr6842346wrm.409.1585434487534; Sat, 28 Mar 2020 15:28:07 -0700 (PDT) Received: from [192.168.1.168] (x59cc9a4f.dyn.telefonica.de. [89.204.154.79]) by smtp.googlemail.com with ESMTPSA id k3sm13844676wmf.16.2020.03.28.15.28.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Mar 2020 15:28:06 -0700 (PDT) From: =?utf-8?Q?Klaus_K=C3=BCchemann?= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Switch to UEFI Re: CFT: alpha test of Ethernet driver for RPi4 Date: Sat, 28 Mar 2020 23:28:05 +0100 References: <202003281622.02SGMcmi027728@mail.karels.net> <57d4ba4ef95eeaf382d2c0b2407e9dab@unrelenting.technology> To: Greg V , freebsd-arm@freebsd.org In-Reply-To: <57d4ba4ef95eeaf382d2c0b2407e9dab@unrelenting.technology> Message-Id: <964BBCA2-5EE1-4673-966E-63D37FEDB4EA@googlemail.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48qYTz1q91z4V2q X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[googlemail.com]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[googlemail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[googlemail.com,quarantine]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(0.00)[ip: (-9.41), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-0.46), country: US(-0.05)]; RECEIVED_SPAMHAUS_PBL(0.00)[79.154.204.89.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[googlemail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 22:36:19 -0000 > Am 28.03.2020 um 23:15 schrieb greg@unrelenting.technology: >=20 > March 29, 2020 12:23 AM, "Klaus K=C3=BCchemann via freebsd-arm" = wrote: >=20 >> I`m also working on pi4 a little while not ready=E2=80=A6 >> I recommend to switch the whole RPI-thing to UEFI/ ACPI, J.McNeill is = working on that , afaik=E2=80=A6 >=20 > He's working on NetBSD a lot, but not on FreeBSD. He is working on RPiUefi/TianoCore and NetBSD is the only OS which has = already integrated=20 the Ethernet driver into UEFI with a published Mac-Adress >=20 >> So next step I see is to implement ACPI-xhci-driver since = FreeBSD-boot PANICS on xhci- >=20 > That's not "implement a driver", that's "fix a bug" :) > I was looking into this a month ago.. > For some reason reading from the XHCI's memory space returns junk like = 0xdead. > And seems like nobody has seen that problem on NetBSD or Linux. > I'll test this a bit more now. I know the exact place where it panics(IRQ-message) and=20 at first glance I would say: the driver is incomplete and not fully = implemented(only for allwinner) , but let`s take a closer look there and = work on it =E2=80=A6 >=20 > Once we get that working, and the Ethernet driver port is published, I = could add ACPI attachment to the Ethernet driver. >=20 >> The good news is that I first saw USB working in UEFI-boot...=E2=80=A6t= he ethernet driver should then later >> correspond or to be integrated into UEFI =E2=80=A6 nice to boot from = a =E2=80=9EBios=E2=80=9C -style menu with a plugged >> USB-keyboard... >=20 > Yeah, eventually there's going to be a NetBSD-derived Ethernet driver = in UEFI for network booting. Yes, afaik they are planning to do that Klaus= From owner-freebsd-arm@freebsd.org Sat Mar 28 22:47:17 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B14B3269578 for ; Sat, 28 Mar 2020 22:47:17 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: from out.migadu.com (out.migadu.com [91.121.223.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.migadu.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qYkj4vPwz4YZ9 for ; Sat, 28 Mar 2020 22:47:06 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: (Migadu outbound); Sat, 28 Mar 2020 22:39:21 +0000 Received: from wms0-eu-central.migadu.com (wms0-eu-central.migadu.com [139.162.159.86]) by out.migadu.com (Haraka/2.8.16) with ESMTPSA id B96FD185-D504-475C-879D-35B68195080A.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 verify=FAIL); Sat, 28 Mar 2020 22:39:21 +0000 MIME-Version: 1.0 Date: Sat, 28 Mar 2020 22:39:17 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: RainLoop/1.12.1 From: greg@unrelenting.technology Message-ID: <3d84dbd6acea80b04bee712b59661a86@unrelenting.technology> Subject: Re: Switch to UEFI Re: CFT: alpha test of Ethernet driver for RPi4 To: "=?utf-8?B?S2xhdXMgS8O8Y2hlbWFubg==?=" , freebsd-arm@freebsd.org In-Reply-To: <964BBCA2-5EE1-4673-966E-63D37FEDB4EA@googlemail.com> References: <964BBCA2-5EE1-4673-966E-63D37FEDB4EA@googlemail.com> <202003281622.02SGMcmi027728@mail.karels.net> <57d4ba4ef95eeaf382d2c0b2407e9dab@unrelenting.technology> DKIM-Signature: v=1; a=rsa-sha256; bh=er+JjJ4CUKriqzsMFsVRnILyg6uiVYT15X4lST8Jja4=; c=relaxed/simple; d=unrelenting.technology; h=from:subject:date:to; s=default; b=WRbwbVxXQ7bVq0bSxGxpjNpeQNeP/ZNTc3NFw8xMlsQnen/V/zAL3bQi21FuuqabExrm72iwREz/yKBw466fTTtZtP/DOonrYEN9wLQm2ELa2BTZ43sPxRuQ3pNCn2oZbTUJh8DaCpzLTy/aN0joP56jVLFpLjtauqNDbsHW6u4= X-Rspamd-Queue-Id: 48qYkj4vPwz4YZ9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=unrelenting.technology header.s=default header.b=WRbwbVxX; dmarc=pass (policy=none) header.from=unrelenting.technology; spf=pass (mx1.freebsd.org: domain of greg@unrelenting.technology designates 91.121.223.63 as permitted sender) smtp.mailfrom=greg@unrelenting.technology X-Spamd-Result: default: False [-4.86 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[unrelenting.technology:s=default]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:91.121.223.63]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[unrelenting.technology:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[63.223.121.91.list.dnswl.org : 127.0.10.0]; FROM_NO_DN(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[unrelenting.technology,none]; FREEMAIL_TO(0.00)[googlemail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-1.86)[ip: (-9.79), ipnet: 91.121.0.0/16(-1.55), asn: 16276(2.05), country: FR(0.00)]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 22:47:17 -0000 March 29, 2020 1:28 AM, "Klaus K=C3=BCchemann" wrote:=0A=0A> I know the exact place where it panics(IRQ-message) and= =0A> at first glance I would say: the driver is incomplete and not fully = implemented(only for allwinner)=0A> , but let`s take a closer look there = and work on it =E2=80=A6=0A=0AWhat allwinner?=0AHow did you find anything= allwinner-related in ACPI code, when AFAIK no Allwinner SoC ever had ACP= I tables written for it?=0A=0AI'm talking about generic XHCI.