From owner-freebsd-current@freebsd.org Sat Nov 24 09:46:49 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A771111490D9 for ; Sat, 24 Nov 2018 09:46:49 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 219E47132B for ; Sat, 24 Nov 2018 09:46:49 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: by mailman.ysv.freebsd.org (Postfix) id D8A4011490D5; Sat, 24 Nov 2018 09:46:48 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5BD911490CD; Sat, 24 Nov 2018 09:46:48 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1309A71329; Sat, 24 Nov 2018 09:46:47 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id wAO94T0h040918 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 24 Nov 2018 01:04:30 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id wAO94TLk040917; Sat, 24 Nov 2018 01:04:29 -0800 (PST) (envelope-from jmg) Date: Sat, 24 Nov 2018 01:04:29 -0800 From: John-Mark Gurney To: current@FreeBSD.org Cc: freebsd-arm@FreeBSD.org Subject: maxswzone NOT used correctly and defaults incorrect? Message-ID: <20181124090429.GI10067@funkthat.com> Mail-Followup-To: current@FreeBSD.org, freebsd-arm@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Sat, 24 Nov 2018 01:04:30 -0800 (PST) X-Rspamd-Queue-Id: 219E47132B X-Spamd-Result: default: False [-4.56 / 15.00]; ARC_NA(0.00)[]; R_SPF_FAIL(1.00)[-all]; RCVD_COUNT_FIVE(0.00)[6]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.986,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[funkthat.com]; IP_SCORE(-3.66)[ip: (-9.87), ipnet: 2001:1900:2254::/48(-4.74), asn: 10310(-3.61), country: US(-0.09)]; RCVD_IN_DNSWL_MED(-0.20)[5.0.0.0.0.5.0.0.0.0.0.0.0.0.0.0.a.6.0.2.4.5.2.2.0.0.9.1.1.0.0.2.list.dnswl.org : 127.0.9.2]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[cached: gold.funkthat.com]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MID_RHS_MATCH_FROM(0.00)[]; FORGED_RECIPIENTS(0.00)[current@FreeBSD.org ..,freebsd-current@freebsd.org] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 09:46:49 -0000 I have an BeagleBoard Black. I'm running a recent snapshot: FreeBSD generic 13.0-CURRENT FreeBSD 13.0-CURRENT r340239 GENERIC arm aka: FreeBSD-13.0-CURRENT-arm-armv7-BEAGLEBONE-20181107-r340239.img.xz It has 512MB of memory on board. I created a 4GB swap file. According to loader(8), this should be the default capable: in bytes of KVA space. If no value is provided, the system allocates enough memory to handle an amount of swap that corresponds to eight times the amount of physical memory present in the system. avail memory = 505909248 (482 MB) but I get this: warning: total configured swap (1048576 pages) exceeds maximum recommended amount (248160 pages). warning: increase kern.maxswzone or reduce amount of swap. So, this appears that it's only 2x amount of memory, NOT 8x like the documentation says. When running make in sbin/ggate/ggated, make consumes a large amount of memory. Before the OOM killer just kicked in, top showed: Mem: 224M Active, 4096 Inact, 141M Laundry, 121M Wired, 57M Buf, 2688K Free Swap: 1939M Total, 249M Used, 1689M Free, 12% Inuse, 1196K Out PID UID THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 1029 1001 1 44 0 594M 3848K RUN 2:03 38.12% make swapinfo -k showed: /dev/md99 4194304 254392 3939912 6% sysctl: vm.swzone: 4466880 vm.swap_maxpages: 496320 kern.maxswzone: 0 dmesg when OOM strikes: swap blk zone exhausted, increase kern.maxswzone pid 1029 (make), uid 1001, was killed: out of swap space pid 984 (bash), uid 1001, was killed: out of swap space pid 956 (bash), uid 1001, was killed: out of swap space pid 952 (sshd), uid 0, was killed: out of swap space pid 1043 (bash), uid 1001, was killed: out of swap space pid 626 (dhclient), uid 65, was killed: out of swap space pid 955 (sshd), uid 1001, was killed: out of swap space pid 1025 (bash), uid 1001, was killed: out of swap space swblk zone ok lock order reversal: 1st 0xd374d028 filedesc structure (filedesc structure) @ /usr/src/sys/kern/sys_generic.c:1451 2nd 0xd41a5bc4 devfs (devfs) @ /usr/src/sys/kern/vfs_vnops.c:1513 stack backtrace: swap blk zone exhausted, increase kern.maxswzone pid 981 (tmux), uid 1001, was killed: out of swap space pid 983 (tmux), uid 1001, was killed: out of swap space pid 1031 (bash), uid 1001, was killed: out of swap space pid 580 (dhclient), uid 0, was killed: out of swap space swblk zone ok swap blk zone exhausted, increase kern.maxswzone pid 577 (dhclient), uid 0, was killed: out of swap space pid 627 (devd), uid 0, was killed: out of swap space swblk zone ok swap blk zone exhausted, increase kern.maxswzone pid 942 (getty), uid 0, was killed: out of swap space swblk zone ok swap blk zone exhausted, increase kern.maxswzone pid 1205 (init), uid 0, was killed: out of swap space swblk zone ok swap blk zone exhausted, increase kern.maxswzone pid 1206 (init), uid 0, was killed: out of swap space swblk zone ok swap blk zone exhausted, increase kern.maxswzone swblk zone ok swap blk zone exhausted, increase kern.maxswzone swblk zone ok So, as you can see, despite having plenty of swap, and swap usage being well below any of the maximums, the OOM killer kicked in, and killed off a bunch of processes. It also looks like the algorithm for calculating kern.maxswzone is not correct. I just tried to run the system w/: kern.maxswzone: 21474836 and it again died w/ plenty of swap free: /dev/md99 4194304 238148 3956156 6% This time I had vmstat -z | grep sw running, and saw: swpctrie: 48, 62084, 145, 270, 203, 0, 0 swblk: 72, 62040, 56357, 18, 56587, 0, 0 after the system died, I logged back in as see: swpctrie: 48, 62084, 28, 387, 240, 0, 0 swblk: 72, 62040, 175, 61865, 62957, 16, 0 so, it clearly ran out of swblk space VERY early, when only consuming around 232MB of swap... Hmm... it looks like swblk and swpctrie are not affected by the setting of kern.maxswzone... I just set it to: kern.maxswzone: 85899344 and the limits for the zones did not increase at ALL: swpctrie: 48, 62084, 0, 0, 0, 0, 0 swblk: 72, 62040, 0, 0, 0, 0, 0 Thoughts? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."