Date: Mon, 27 Jan 2020 11:07:09 -0800 From: bob prohaska <fbsd@www.zefox.net> To: freebsd-arm@freebsd.org Subject: OOMA kill with vm.pfault_oom_attempts="-1" on RPi3 at r357147 Message-ID: <20200127190709.GA11328@www.zefox.net>
next in thread | raw e-mail | index | archive | help
The latest attempt at buildworld on a Pi3 with kernel and sources at
r357147 stopped with an "out of swap" kill. The activity log reported,
in the one second samples before, during and after the kill recorded:
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id
4 0 0 1670108 58012 1332 5 2 1 1414 597 0 0 10572 1251 2340 80 18 2
dT: 1.015s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
Mon Jan 27 10:32:06 PST 2020
Device 1K-blocks Used Avail Capacity
/dev/mmcsd0s2b 4404252 135540 4268712 3%
/dev/da0p6 5242880 132148 5110732 3%
Total 9647132 267688 9379444 3%
Jan 26 19:44:00 www sshd[1289]: error: maximum authentication attempts exceeded for invalid user from 45.136.108.85 port 16543 ssh2 [preauth]
Jan 26 19:44:12 www sshd[1298]: error: maximum authentication attempts exceeded for invalid user from 45.136.108.85 port 4581 ssh2 [preauth]
0/254/254/19180 mbuf clusters in use (current/cache/total/max)
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id
3 0 0 1683560 54412 1332 5 2 1 1414 603 0 0 10572 1251 2340 80 18 2
dT: 1.051s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
0 79 3 11 917.7 76 1557 212.6 0 0 0.0 318.6 mmcsd0
0 79 3 11 918.0 76 1557 212.8 0 0 0.0 318.6 mmcsd0s2
0 52 1 4 1.4 51 780 1.4 0 0 0.0 6.9 da0
0 22 0 0 0.0 22 700 177.2 0 0 0.0 32.5 mmcsd0s2a
0 57 3 11 918.0 54 856 227.2 0 0 0.0 318.6 mmcsd0s2b
0 22 0 0 0.0 22 700 177.3 0 0 0.0 32.5 ufs/rootfs
0 51 1 4 1.4 50 780 1.4 0 0 0.0 7.1 da0p6
Mon Jan 27 10:32:12 PST 2020
Device 1K-blocks Used Avail Capacity
/dev/mmcsd0s2b 4404252 137184 4267068 3%
/dev/da0p6 5242880 134284 5108596 3%
Total 9647132 271468 9375664 3%
Jan 26 19:44:00 www sshd[1289]: error: maximum authentication attempts exceeded for invalid user from 45.136.108.85 port 16543 ssh2 [preauth]
Jan 26 19:44:12 www sshd[1298]: error: maximum authentication attempts exceeded for invalid user from 45.136.108.85 port 4581 ssh2 [preauth]
0/256/256/19180 mbuf clusters in use (current/cache/total/max)
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr mm0 da0 in sy cs us sy id
3 0 0 1394192 142844 1332 5 2 1 1415 607 0 0 10573 1251 2341 80 18 2
dT: 1.006s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
0 25 25 557 1.9 0 0 0.0 0 0 0.0 4.6 mmcsd0
0 25 25 557 1.9 0 0 0.0 0 0 0.0 4.8 mmcsd0s2
0 17 17 159 1.3 0 0 0.0 0 0 0.0 2.2 da0
0 8 8 231 2.4 0 0 0.0 0 0 0.0 1.9 mmcsd0s2a
0 17 17 326 1.7 0 0 0.0 0 0 0.0 2.9 mmcsd0s2b
0 8 8 231 2.4 0 0 0.0 0 0 0.0 1.9 ufs/rootfs
0 17 17 159 1.3 0 0 0.0 0 0 0.0 2.3 da0p6
Mon Jan 27 10:32:21 PST 2020
Device 1K-blocks Used Avail Capacity
/dev/mmcsd0s2b 4404252 43020 4361232 1%
/dev/da0p6 5242880 42128 5200752 1%
Total 9647132 85148 9561984 1%
Jan 26 19:44:12 www sshd[1298]: error: maximum authentication attempts exceeded for invalid user from 45.136.108.85 port 4581 ssh2 [preauth]
Jan 27 10:32:18 www kernel: pid 97756 (c++), jid 0, uid 0, was killed: out of swap space
Here's the command used to collect the activity log:
#!/bin/sh
while true
do vmstat ; gstat -abd -I 1s ; date ; swapinfo ; tail -n 2 /var/log/messages ; netstat -m | grep "mbuf clusters"
done
It looks as if the vm.pfault_oom_attempts="-1" no longer shuts OOMA off.
Is there another way to deal with the problem?
As an aside, it appears the activity percentages in top have changed:
Formerly the per-cpu numbers totalled about four times the total %busy.
Now the per-cpu numbers roughly add up to total %busy. Not sure it matters,
but it's certainly different from previous behavior.
Perhaps most surprisingly, after buildworld and the activity logger had
stopped (while I was writing this little missive) the machine again panic'd,
reporting:
panic: deadlres_td_sleep_q: possible deadlock detected for 0xfffffd0000eff000, blocked for 1800269 ticks
Thanks for reading and any ideas.
bob prohaska
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200127190709.GA11328>