Date: Fri, 27 Oct 2023 11:15:25 +0100 From: void <void@f-m.fm> To: freebsd-stable@freebsd.org Subject: periodic daily takes a very long time to run (14-stable) Message-ID: <ZTuNvVMW_XG3mZKU@int21h>
next in thread | raw e-mail | index | archive | help
Hello list,
I'm asking this in order to determine whether I've got something misconfigured or
maybe my expectations need recalibrating, or maybe both.
context is rpi4b with 8GB and usb3-connected 1tb zfs hard disk.
/etc/daily.local has been moved out of the way to try to fix the issue.
When periodic daily runs, it seems to take a long time. It's running now and has
been running for overan hour so far.
top shows
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
21649 root 1 20 0 18M 6668K zio->i 1 0:40 0.53% find
There's plenty of resources
93 processes: 1 running, 92 sleeping
CPU: 0.1% user, 0.0% nice, 0.5% system, 0.0% interrupt, 99.4% idle
Mem: 202M Active, 688M Inact, 69M Laundry, 3079M Wired, 216K Buf, 3780M Free
ARC: 975M Total, 552M MFU, 244M MRU, 808K Anon, 21M Header, 158M Other
229M Compressed, 905M Uncompressed, 3.95:1 Ratio
Swap: 12G Total, 12G Free
but the 'find' thing does heavy disk i/o and everything else reading from or writing
to disk slows to a crawl.
# cat /etc/periodic.conf
daily_status_smart_devices="da0"
daily_status_include_submit_mailq="NO"
daily_clean_hoststat_enable="NO"
daily_queuerun_enable="NO"
daily_submit_queuerun="NO"
daily_scrub_zfs_enable="YES"
daily_scrub_zfs_pools=""
daily_scrub_zfs_default_threshold="7"
daily_status_ntpd_enable="YES"
daily_clean_disks_enable="YES"
zpool last scrubbed on the 21st; it's not happening right now
there's the following in /etc/sysctl.conf relevant to the context:
vfs.zfs.min_auto_ashift=12
#
# filesystem
vm.pageout_oom_seq=120
vm.pfault_oom_attempts=-1
vm.pageout_update_period=0
# vfs.zfs.resilver_min_time_ms=5000
vfs.zfs.arc.min=536870912
vfs.zfs.arc_max=1073741824
The OOM settings are there for poudriere.
# zdb | grep ashift
ashift: 12
# ps xx | grep periodic
68824 11 S+ 0:00.00 grep periodic
16115 14 I+ 0:00.00 /bin/sh - /usr/sbin/periodic daily
17425 14 I+ 0:00.00 lockf -s -t 0 /var/run/periodic.daily.lock /bin/sh /usr/sbin/periodic LOCKED daily
17747 14 I+ 0:00.01 /bin/sh /usr/sbin/periodic LOCKED daily <=== why are there two of these?
19460 14 I+ 0:00.00 /bin/sh /usr/sbin/periodic LOCKED daily <===
19921 14 I+ 0:00.00 /bin/sh /etc/periodic/daily/100.clean-disks <=== also here
21312 14 I+ 0:00.00 /bin/sh /etc/periodic/daily/100.clean-disks <===
I'm not sure if it's relevant, but the disk, when the zroot/zfs was initialised, was set
to enforce 4k blocks, geli-encrypted swap, geli-encrypted fs. Because it's an rpi4, there's
no aes-ni equivalent hardware. But the slowdown doesn't show much in other areas, only a couple,
like this problem and when git updates src or ports. smartctl shows no remapped or pending blocks.
What else can I do to debug the issue?
--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ZTuNvVMW_XG3mZKU>