From owner-freebsd-arm@freebsd.org Thu Sep 6 02:05:25 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 016B4FE3240 for ; Thu, 6 Sep 2018 02:05:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-12.consmr.mail.ne1.yahoo.com (sonic307-12.consmr.mail.ne1.yahoo.com [66.163.190.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7995580797 for ; Thu, 6 Sep 2018 02:05:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 1j4irzEVM1n5m490J0zjqTOM6iRWZrGYDdL3bxmfD_g99ys14NxAPyHDp5aDbmx c3MelA.78QaOOYEqakqLxULBfXvw1KEWxNJGzUFljvueFPo3TW_SwWd.bcTgKISG2cFHlfvXMO8h 6ZErgmefbF3ywe71kW.2SIS_pp_QJ_oUq3R0pTbJkrU0J.H8w0VQX8SCfd9ZWeZwefwSD4zRZepi hYI_ZFAVSYtAYIDYZ96zscq2fQFMkgArlHlOiSOzkM0f_wCuksApVhcelTbFTKRKbYxUI0HrndYY aGkAkA3c5eRfjfriJQ6_K2nG7SuYyZV2Skjf1bng8_N.INWvBjGeOWT0plPAfSEg8oz6im87NlhV i1T9ou6WE5uH0KHS5EabAjC9Uabkku3LO5xsy3C4YfA8P0HAFT2ILQYdsi9V4MSj3qsRcLWcKpv0 HKm564OYsCbqdvz2a5ox.VNTTQtHhBCInhvRPfSWg2EGRbF7MEnUwPdDog5VhgYGBJ90eApPSHkp Q4cfef.CrdZ7O76YCsxkgAXquOUra3pvorCUsD.WmmQj3eEjgEvuzWdEmV7hUt6plnb4oLZFz__4 8Q_IVo.bKyACyUxS_n0xZhx58R5PgC8h4pyrDEprvH3TsoCEEE2lyiH.d2UvrTlkolVhIP.EJYq_ fy4rV8SU80ixY0eeUpiNbbJYQ7UeUjsdX75HheApHHEKEQa6_jpLeYoR.QLqmvMkQhvs7QK43K1h vCpxOufp2cFPOacyVdYTCbbSXdPWasoQaV4.Xc9aoF85KJxwcTmcD7tbIQpIYc25lDezxc6z3dnx NGRgAmuLrZt6StedycqETCkD0PdCWi.r4mA9pH5AMiPpVDPR4CZkDDdd.s2b9ecWtqAXV.qwtyrH L7TgaugCyT2ka8ni2f99ZfcPMx2DNSbilq.T.nFVKMyZ_vw6ZFEk93mmYXfzWw_YwzKGKbzVbRDr h9VH8umDq1HIm9OP72p0H1q0EiBoys3ze.VkqRyw46tcrDZvHLuaJ2hR9fWu9PDLEVfrKVb95wEJ Ofxej.TE- Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.ne1.yahoo.com with HTTP; Thu, 6 Sep 2018 02:05:17 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp427.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 108d42328177c67bf89bbe60bf210b12; Thu, 06 Sep 2018 02:05:13 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: RPI3 swap experiments (r338342 with vm.pageout_oom_seq="1024" and 6 GB swap) From: Mark Millard In-Reply-To: <20180906003829.GC818@www.zefox.net> Date: Wed, 5 Sep 2018 19:05:11 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <0D8B9A29-DD95-4FA3-8F7D-4B85A3BB54D7@yahoo.com> <20180813185350.GA47132@www.zefox.net> <20180814014226.GA50013@www.zefox.net> <20180815013612.GB51051@www.zefox.net> <20180815225504.GB59074@www.zefox.net> <20180901230233.GA42895@www.zefox.net> <20180906003829.GC818@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Sep 2018 02:05:25 -0000 [I've omitted Kirk McKusick as my notes are largely off subject for what he asked about for testing specific to his changes.] On 2018-Sep-5, at 5:38 PM, bob prohaska wrote: > On Sat, Sep 01, 2018 at 04:02:33PM -0700, bob prohaska wrote: >>=20 >> With r338342 and >> vm.pageout_oom_seq=3D"1024" >> in /boot/loader.conf the RPI3 is a bit closer to a Mars Rover. >> No panics, crashes or USB errors, -j4 buildworld runs to completion. >> When swap usage goes over about 50% the system slows, but doesn't = give up. >> There are six 1 GB swap partitions available, 3 on USB and 3 on = microSD. >>=20 >> Log files are at >> http://www.zefox.net/~fbsd/rpi3/swaptests/r338342/ >> for the combinations tried so far. >>=20 >=20 > It looks as if using all six GB of swap doesn't cause any immediate = problem, > at least so long as swap usage stays relatively low, say 1.5 GB. In a = final > test, TRIM was turned on without catastrophe, though it had little to = do > given that all the busy filesystems were on USB. The penalty was about = one > hour extra (25 vs 24 hours) to run -j4 buildworld from a clean start. What UFS file systems with TRIM enabled were on some /dev/mmcsd0* ? Did you 1st use "fsck_ffs -E" on any of the file systems where trim would work? If I gather right, the "clean start" was on USB where TRIM during the clean would not be available. The extra swap space may have contributed to the extra time? Having more swap uses more kernel memory for keeping track of the swap if I understand right. That leaves less for other things. That could have consequences other than outright failure. Quoting "man 8 loader" related to kern.maxswzone : Note that swap metadata can be fragmented, which means = that the system can run out of space before it reaches the theoretical limit. Therefore, care should be taken to = not configure more swap than approximately half of the theoretical maximum. Running out of space for swap metadata can leave the = system in an unrecoverable state. This wording suggests not allocating 6 GiBytes of swap when 3.5 GiBytes is approximately half the theoretical maximum --even if the system does still operate with 6 GiBytes. (Note: The man page's reference to "eight times the amount of physical = memory" and such does not seem to apply to all platforms. And rpi2 V1.1 and an = rpi3 with the same amount of RAM get rather difference recommended figures according to the messages generated.) > One chance observation caught my attention, however. I'd always = thought > the VM system would favor fast swap devices over slow, but the gstat = log > recorded this, visible at > = http://www.zefox.net/~fbsd/rpi3/swaptests/r338342/3gbsd_3gbusb/trim_on/swa= pscript.log >=20 >=20 >=20 > dT: 10.004s w: 10.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > 3 175 91 673 4.0 84 701 4.0 0 0 = 0.0 24.4 mmcsd0 > 4 173 88 693 106.6 86 723 176.5 0 0 = 0.0 103.4 da0 > 1 58 30 224 4.5 28 220 4.1 0 0 = 0.0 14.5 mmcsd0s2b > 3 175 91 673 4.0 84 701 4.0 0 0 = 0.0 24.7 mmcsd0s2 > 1 58 30 223 4.0 28 244 3.8 0 0 = 0.0 14.0 mmcsd0s2d > 1 59 31 227 3.7 28 237 4.3 0 0 = 0.0 14.9 mmcsd0s2e > 2 57 28 235 140.2 28 236 103.8 0 0 = 0.0 186.1 da0a > 0 56 28 224 178.4 28 222 35.9 0 0 = 0.0 131.5 da0b > 2 59 31 234 9.4 28 240 59.1 0 0 = 0.0 99.5 da0d > 0 0 0 0 0.0 0 3 15011 0 0 = 0.0 150.1 da0e > 0 1 0 0 0.0 1 22 13376 0 0 = 0.0 147.8 da0g Are there any examples of "d/s kBps ms/d" being non-zero? If they are always zero then no TRIMing likely happened. That in turn would make TRIM an unlikely use of an extra hour. > Tue Sep 4 15:07:39 PDT 2018 > Device 1K-blocks Used Avail Capacity > /dev/da0b 1048576 236872 811704 23% > /dev/mmcsd0s2b 1048576 221568 827008 21% > /dev/da0d 1048576 218636 829940 21% > /dev/da0a 1048576 222028 826548 21% > /dev/mmcsd0s2d 1048576 221660 826916 21% > /dev/mmcsd0s2e 1048576 221392 827184 21% > Total 6291456 1342156 4949300 21% As I understand the normal use of multiple swap partitions is to split the load across channels that can operate independently in parallel. Having 3 such partitions on the same channel/device may only add overhead vs. one full-size partition per channel/device. I also do not know if mmcsd0 and da0 can have independent, parallel I/O activity in the rpi3 context. > Sep 4 14:57:52 www sshd[41673]: error: Received disconnect from = 103.207.39.197 port 64499:3: com.jcraft.jsch.JSchException: Auth cancel = [preauth] > Sep 4 15:04:19 www kernel: swap_pager: indefinite wait buffer: = bufobj: 0, blkno: 2217840, size: 12288 Note: my context is very different from yours and I get no console messages about I/O or waits during buildworld buildkernel or other such build/install tests. > The system has lots of fast swap available on microSD, but is = seemingly choking=20 > trying to use the slow swap on da0 _and_ run traffic to /usr and /var. = Buildworld > doesn't run any faster with less swap, so I don't think the oversupply = is the problem. If I understand right, your only 6 GiByte swap experiment was slower but you attributed all time variations to an (inactive? ever used?) TRIM enabled status. You might want to manipulate the two separately. For all I know something else may also have contributed. I've no clue if having so many swap partitions on the same = channel/device has consequences that having only one per channel/device would avoid. > Is this expected behavior? =20 As I understand the approximately even split across the in-use swap partitions is the normal way things are split. It is the placement of the partitions themselves that contributes to how effective that split is at improving the swap/paging I/O if I understand right. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)