From owner-freebsd-fs@freebsd.org Sun Jun 7 17:09:27 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19053331972 for ; Sun, 7 Jun 2020 17:09:27 +0000 (UTC) (envelope-from SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49g2tF5SGTz4Gpx for ; Sun, 7 Jun 2020 17:09:25 +0000 (UTC) (envelope-from SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 62F6B28416; Sun, 7 Jun 2020 19:09:17 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 3440028422; Sun, 7 Jun 2020 19:09:15 +0200 (CEST) Subject: Re: ZFS on FreeBSD 11.3 slower than 10.4 To: Ronald Klop , "freebsd-fs@freebsd.org" References: <1ff455a5-d111-86fa-ceb1-1021b6d9a5b6@quip.cz> <8c64cc48-7d79-7591-8bb5-67f3127463b7@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <1c8983f7-8aee-61ca-b8ee-fbaa24506841@quip.cz> Date: Sun, 7 Jun 2020 19:09:14 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49g2tF5SGTz4Gpx X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [-0.52 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-0.57)[-0.571]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.54)[-0.538]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.61)[-0.615]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz, SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Jun 2020 17:09:27 -0000 On 2020-06-06 00:57, Ronald Klop wrote: > On Fri, 05 Jun 2020 22:57:15 +0200, Ronald Klop > wrote: > >> On Sat, 30 May 2020 23:29:48 +0200, Miroslav Lachman >> <000.fbsd@quip.cz> wrote: >> >>> On 2020-05-30 22:10, Ronald Klop wrote: >>>> On Sat, 23 May 2020 21:44:03 +0200, Miroslav Lachman >>>> <000.fbsd@quip.cz> wrote: >>>> >>>>> I upgraded my old desktop computer few month ago from old 10.4 >>>>> based PC-BSD to stock FreeBSD 11.3. It uses single 2TB HDD 7200rpm. >>>>> My problem is that upgraded version is really slow and some desktop >>>>> applications are very lagging (playing multimedia is interrupted >>>>> for a fraction of seconds) when there is heavy filesystem activity. >>>>> >>>>> I am using zfsnap2 for taking snapshots periodically and when there >>>>> is enough snapshots zfs destroy is called. In this time the user >>>>> experience is terrible. Starting new application like browser or >>>>> even something much smaller takes minutes. The old version based on >>>>> FreeBSD 10.4 behaves much better. I used the old version for years >>>>> and never have problems with interrupted multimedia playback. >>>>> >>>>> Are there some sysctls to tune to get better desktop interactivity >>>>> in heavy filesystem operations like zfs destroy, pkg check or other >>>>> "find" periodic scripts? >>> >>> >>>> How full is the disk? ZFS has poor performance if the disk becomes >>>> full. >>>> What is in /etc/sysctl.conf and /boot/loader.conf? >>>> And did you try to boot 12.1 and did it have the same behavious? >>> >>> It is currently 77% full. But it is the same pool with the same >>> capacity as with 10.4. >>> >>> I didn't try 12.1, I need to stay on 11.3 for now. >>> >>> ## loader.conf >>> >>> nvidia_load="YES" >>> drm_load="YES" >>> drm2_load="YES" >>> iicbus_load="YES" >>> vboxdrv_load="YES" >>> crypto_load="YES" >>> aesni_load="YES" >>> geom_eli_load="YES" >>> vfs.zfs.arc_max="1024M" >>> zfs_load="YES" >>> iicbus_load="YES" >>> >>> ## sysctl.conf >>> >>> kern.coredump=0 >>> kern.maxfiles=49312 >>> vfs.usermount=1 >>> security.jail.allow_raw_sockets=1 >>> security.jail.sysvipc_allowed=1 >>> security.jail.mount_allowed=1 >>> security.jail.chflags_allowed=1 >>> hw.syscons.bell=0 >>> kern.sched.preempt_thresh=224 >>> kern.ipc.shm_allow_removed=1 >>> kern.shutdown.poweroff_delay=500 >>> kern.bootfile=/boot/kernel/kernel >>> hw.usb.no_shutdown_wait=1 >>> hw.snd.default_unit=3 >>> kern.sched.interact=10 >>> vfs.aio.max_aio_per_proc=256 >>> vfs.aio.max_aio_queue=8192 >>> vfs.aio.max_aio_queue_per_proc=1024 >>> vfs.aio.max_buf_aio=64 >>> net.local.stream.recvspace=65536 >>> net.local.stream.sendspace=65536 >>> >>> >>> loader.conf and sysctl.conf are the same for 10.4 and 11.3 but 11.3 >>> is much much slower when it comes to heavy IO like "find" daily >>> periodic scripts, zfs destroy, starting new applications etc. >>> >>> >>> Kind regards >>> Miroslav Lachman >> >> >> I don't have anything I see which I'm sure will fix things, but you >> could try to remove/comment some of these sysctls to see if 11.3 has >> better defaults now. >> kern.sched.preempt_thresh, kern.maxfiles, kern.sched.interact, vfs.aio.* >> >> What kind of machine is it? CPU, MEM? >> What does gstat say about the saturation of the disk? >> >> Regards, >> Ronald. > > > What might also be interesting is which timer is selected. Sometimes for > some reason another time-source is chosen which can influence a lot of > things (like sound). > Please post the output of "sysctl kern.eventtimer" and "sysctl > kern.timecounter" if possible of 10.4 and 11.3. > > Or compare the /var/run/dmesg.boot files of 10.4 and 11.3 to see if some > hardware is recognized differently. sysctls from 11.3 instance # sysctl kern.eventtimer kern.eventtimer.periodic: 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) HPET3(440) HPET4(440) i8254(100) RTC(0) kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.HPET4.quality: 440 kern.eventtimer.et.HPET4.frequency: 14318180 kern.eventtimer.et.HPET4.flags: 3 kern.eventtimer.et.HPET3.quality: 440 kern.eventtimer.et.HPET3.frequency: 14318180 kern.eventtimer.et.HPET3.flags: 3 kern.eventtimer.et.HPET2.quality: 440 kern.eventtimer.et.HPET2.frequency: 14318180 kern.eventtimer.et.HPET2.flags: 3 kern.eventtimer.et.HPET1.quality: 440 kern.eventtimer.et.HPET1.frequency: 14318180 kern.eventtimer.et.HPET1.flags: 3 kern.eventtimer.et.HPET.quality: 550 kern.eventtimer.et.HPET.frequency: 14318180 kern.eventtimer.et.HPET.flags: 7 kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 3192676730 kern.eventtimer.et.LAPIC.flags: 7 # sysctl kern.timecounter kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 1 kern.timecounter.invariant_tsc: 1 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000) dummy(-1000000) kern.timecounter.hardware: TSC-low kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 16156069 kern.timecounter.tc.ACPI-fast.mask: 16777215 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 49991 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 14318180 kern.timecounter.tc.HPET.counter: 3887656701 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.TSC-low.quality: 1000 kern.timecounter.tc.TSC-low.frequency: 1596338365 kern.timecounter.tc.TSC-low.counter: 1764358755 kern.timecounter.tc.TSC-low.mask: 4294967295 CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.68-MHz K8-class CPU) real memory = 13388218368 (12768 MB) avail memory = 12337872896 (11766 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs last pid: 15694; load averages: 0.42, 0.46, 0.38 up 0+00:30:40 19:01:30 138 processes: 1 running, 136 sleeping, 1 zombie CPU: 3.2% user, 0.0% nice, 0.4% system, 0.5% interrupt, 95.9% idle Mem: 1531M Active, 977M Inact, 39M Laundry, 3047M Wired, 6251M Free ARC: 1275M Total, 311M MFU, 286M MRU, 2542K Anon, 23M Header, 652M Other 124M Compressed, 476M Uncompressed, 3.85:1 Ratio Swap: 4096M Total, 4096M Free As you can see there is a lot of free memory. I will try to catch gstat output next time and also post some info from 10.4 instance. Kind regards Miroslav Lachman