Date: Fri, 27 Oct 2023 13:45:24 +0200 (CEST) From: Ronald Klop <ronald-lists@klop.ws> To: void <void@f-m.fm> Cc: freebsd-stable@freebsd.org Subject: Re: periodic daily takes a very long time to run (14-stable) Message-ID: <1122335317.4913.1698407124469@localhost> In-Reply-To: <ZTuNvVMW_XG3mZKU@int21h> References: <ZTuNvVMW_XG3mZKU@int21h>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_4912_1359293768.1698407124396 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Van: void <void@f-m.fm> Datum: vrijdag, 27 oktober 2023 12:15 Aan: freebsd-stable@freebsd.org Onderwerp: periodic daily takes a very long time to run (14-stable) > > Hello list, > > I'm asking this in order to determine whether I've got something misconfigured or > maybe my expectations need recalibrating, or maybe both. > > context is rpi4b with 8GB and usb3-connected 1tb zfs hard disk. > /etc/daily.local has been moved out of the way to try to fix the issue. > > When periodic daily runs, it seems to take a long time. It's running now and has > been running for overan hour so far. > > top shows > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 21649 root 1 20 0 18M 6668K zio->i 1 0:40 0.53% find > > There's plenty of resources > > 93 processes: 1 running, 92 sleeping > CPU: 0.1% user, 0.0% nice, 0.5% system, 0.0% interrupt, 99.4% idle > Mem: 202M Active, 688M Inact, 69M Laundry, 3079M Wired, 216K Buf, 3780M Free > ARC: 975M Total, 552M MFU, 244M MRU, 808K Anon, 21M Header, 158M Other > 229M Compressed, 905M Uncompressed, 3.95:1 Ratio > Swap: 12G Total, 12G Free > > but the 'find' thing does heavy disk i/o and everything else reading from or writing to disk slows to a crawl. > > # cat /etc/periodic.conf > daily_status_smart_devices="da0" > daily_status_include_submit_mailq="NO" > daily_clean_hoststat_enable="NO" > daily_queuerun_enable="NO" > daily_submit_queuerun="NO" > daily_scrub_zfs_enable="YES" > daily_scrub_zfs_pools="" > daily_scrub_zfs_default_threshold="7" > daily_status_ntpd_enable="YES" > daily_clean_disks_enable="YES" > > zpool last scrubbed on the 21st; it's not happening right now > > there's the following in /etc/sysctl.conf relevant to the context: > > vfs.zfs.min_auto_ashift=12 > # > # filesystem > vm.pageout_oom_seq=120 > vm.pfault_oom_attempts=-1 > vm.pageout_update_period=0 > # vfs.zfs.resilver_min_time_ms=5000 > vfs.zfs.arc.min=536870912 > vfs.zfs.arc_max=1073741824 > > The OOM settings are there for poudriere. > > # zdb | grep ashift > ashift: 12 > > # ps xx | grep periodic > 68824 11 S+ 0:00.00 grep periodic > 16115 14 I+ 0:00.00 /bin/sh - /usr/sbin/periodic daily > 17425 14 I+ 0:00.00 lockf -s -t 0 /var/run/periodic.daily.lock /bin/sh /usr/sbin/periodic LOCKED daily > 17747 14 I+ 0:00.01 /bin/sh /usr/sbin/periodic LOCKED daily <=== why are there two of these? > 19460 14 I+ 0:00.00 /bin/sh /usr/sbin/periodic LOCKED daily <=== > 19921 14 I+ 0:00.00 /bin/sh /etc/periodic/daily/100.clean-disks <=== also here > 21312 14 I+ 0:00.00 /bin/sh /etc/periodic/daily/100.clean-disks <=== > > I'm not sure if it's relevant, but the disk, when the zroot/zfs was initialised, was set > to enforce 4k blocks, geli-encrypted swap, geli-encrypted fs. Because it's an rpi4, there's > no aes-ni equivalent hardware. But the slowdown doesn't show much in other areas, only a couple, > like this problem and when git updates src or ports. smartctl shows no remapped or pending blocks. > > What else can I do to debug the issue? > -- > > > > Hi, Can you run "gstat" or "iostat -x -d 1" to see how busy your disk is? And how much bandwidth is uses. The output of "zpool status", "zpool list" and "zfs list" can also be interesting. ZFS is known to become slow when the zpool is full almost full. Regards, Ronald. ------=_Part_4912_1359293768.1698407124396 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit <html><head></head><body><br> <p><strong>Van:</strong> void <void@f-m.fm><br> <strong>Datum:</strong> vrijdag, 27 oktober 2023 12:15<br> <strong>Aan:</strong> freebsd-stable@freebsd.org<br> <strong>Onderwerp:</strong> periodic daily takes a very long time to run (14-stable)</p> <blockquote style="padding-right: 0px; padding-left: 5px; margin-left: 5px; border-left: #000000 2px solid; margin-right: 0px"> <div class="MessageRFC822Viewer" id="P"> <div class="TextPlainViewer" id="P.P">Hello list,<br> <br> I'm asking this in order to determine whether I've got something misconfigured or<br> maybe my expectations need recalibrating, or maybe both.<br> <br> context is rpi4b with 8GB and usb3-connected 1tb zfs hard disk.<br> /etc/daily.local has been moved out of the way to try to fix the issue.<br> <br> When periodic daily runs, it seems to take a long time. It's running now and has<br> been running for overan hour so far.<br> <br> top shows<br> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND<br> 21649 root 1 20 0 18M 6668K zio->i 1 0:40 0.53% find<br> <br> There's plenty of resources<br> <br> 93 processes: 1 running, 92 sleeping<br> CPU: 0.1% user, 0.0% nice, 0.5% system, 0.0% interrupt, 99.4% idle<br> Mem: 202M Active, 688M Inact, 69M Laundry, 3079M Wired, 216K Buf, 3780M Free<br> ARC: 975M Total, 552M MFU, 244M MRU, 808K Anon, 21M Header, 158M Other<br> 229M Compressed, 905M Uncompressed, 3.95:1 Ratio<br> Swap: 12G Total, 12G Free<br> <br> but the 'find' thing does heavy disk i/o and everything else reading from or writing to disk slows to a crawl.<br> <br> # cat /etc/periodic.conf<br> daily_status_smart_devices="da0"<br> daily_status_include_submit_mailq="NO"<br> daily_clean_hoststat_enable="NO"<br> daily_queuerun_enable="NO"<br> daily_submit_queuerun="NO"<br> daily_scrub_zfs_enable="YES"<br> daily_scrub_zfs_pools=""<br> daily_scrub_zfs_default_threshold="7"<br> daily_status_ntpd_enable="YES"<br> daily_clean_disks_enable="YES"<br> <br> zpool last scrubbed on the 21st; it's not happening right now<br> <br> there's the following in /etc/sysctl.conf relevant to the context:<br> <br> vfs.zfs.min_auto_ashift=12<br> #<br> # filesystem<br> vm.pageout_oom_seq=120<br> vm.pfault_oom_attempts=-1<br> vm.pageout_update_period=0<br> # vfs.zfs.resilver_min_time_ms=5000<br> vfs.zfs.arc.min=536870912<br> vfs.zfs.arc_max=1073741824<br> <br> The OOM settings are there for poudriere.<br> <br> # zdb | grep ashift<br> ashift: 12 <br> <br> # ps xx | grep periodic<br> 68824 11 S+ 0:00.00 grep periodic<br> 16115 14 I+ 0:00.00 /bin/sh - /usr/sbin/periodic daily<br> 17425 14 I+ 0:00.00 lockf -s -t 0 /var/run/periodic.daily.lock /bin/sh /usr/sbin/periodic LOCKED daily<br> 17747 14 I+ 0:00.01 /bin/sh /usr/sbin/periodic LOCKED daily <=== why are there two of these?<br> 19460 14 I+ 0:00.00 /bin/sh /usr/sbin/periodic LOCKED daily <===<br> 19921 14 I+ 0:00.00 /bin/sh /etc/periodic/daily/100.clean-disks <=== also here<br> 21312 14 I+ 0:00.00 /bin/sh /etc/periodic/daily/100.clean-disks <===<br> <br> I'm not sure if it's relevant, but the disk, when the zroot/zfs was initialised, was set<br> to enforce 4k blocks, geli-encrypted swap, geli-encrypted fs. Because it's an rpi4, there's<br> no aes-ni equivalent hardware. But the slowdown doesn't show much in other areas, only a couple,<br> like this problem and when git updates src or ports. smartctl shows no remapped or pending blocks.<br> <br> What else can I do to debug the issue?<br> -- <br> </div> <hr></div> </blockquote> <br> <br> Hi,<br> <br> Can you run "gstat" or "iostat -x -d 1" to see how busy your disk is? And how much bandwidth is uses.<br> <br> The output of "zpool status", "zpool list" and "zfs list" can also be interesting.<br> <br> ZFS is known to become slow when the zpool is full almost full.<br> <br> Regards,<br> Ronald.<br> </body></html> ------=_Part_4912_1359293768.1698407124396--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1122335317.4913.1698407124469>