From owner-freebsd-fs@freebsd.org Sun Jun 7 17:09:27 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19053331972 for ; Sun, 7 Jun 2020 17:09:27 +0000 (UTC) (envelope-from SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49g2tF5SGTz4Gpx for ; Sun, 7 Jun 2020 17:09:25 +0000 (UTC) (envelope-from SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 62F6B28416; Sun, 7 Jun 2020 19:09:17 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 3440028422; Sun, 7 Jun 2020 19:09:15 +0200 (CEST) Subject: Re: ZFS on FreeBSD 11.3 slower than 10.4 To: Ronald Klop , "freebsd-fs@freebsd.org" References: <1ff455a5-d111-86fa-ceb1-1021b6d9a5b6@quip.cz> <8c64cc48-7d79-7591-8bb5-67f3127463b7@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <1c8983f7-8aee-61ca-b8ee-fbaa24506841@quip.cz> Date: Sun, 7 Jun 2020 19:09:14 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49g2tF5SGTz4Gpx X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [-0.52 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-0.57)[-0.571]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.54)[-0.538]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.61)[-0.615]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz, SRS0=1dUP=7U=quip.cz=000.fbsd@elsa.codelab.cz] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Jun 2020 17:09:27 -0000 On 2020-06-06 00:57, Ronald Klop wrote: > On Fri, 05 Jun 2020 22:57:15 +0200, Ronald Klop > wrote: > >> On Sat, 30 May 2020 23:29:48 +0200, Miroslav Lachman >> <000.fbsd@quip.cz> wrote: >> >>> On 2020-05-30 22:10, Ronald Klop wrote: >>>> On Sat, 23 May 2020 21:44:03 +0200, Miroslav Lachman >>>> <000.fbsd@quip.cz> wrote: >>>> >>>>> I upgraded my old desktop computer few month ago from old 10.4 >>>>> based PC-BSD to stock FreeBSD 11.3. It uses single 2TB HDD 7200rpm. >>>>> My problem is that upgraded version is really slow and some desktop >>>>> applications are very lagging (playing multimedia is interrupted >>>>> for a fraction of seconds) when there is heavy filesystem activity. >>>>> >>>>> I am using zfsnap2 for taking snapshots periodically and when there >>>>> is enough snapshots zfs destroy is called. In this time the user >>>>> experience is terrible. Starting new application like browser or >>>>> even something much smaller takes minutes. The old version based on >>>>> FreeBSD 10.4 behaves much better. I used the old version for years >>>>> and never have problems with interrupted multimedia playback. >>>>> >>>>> Are there some sysctls to tune to get better desktop interactivity >>>>> in heavy filesystem operations like zfs destroy, pkg check or other >>>>> "find" periodic scripts? >>> >>> >>>> How full is the disk? ZFS has poor performance if the disk becomes >>>> full. >>>> What is in /etc/sysctl.conf and /boot/loader.conf? >>>> And did you try to boot 12.1 and did it have the same behavious? >>> >>> It is currently 77% full. But it is the same pool with the same >>> capacity as with 10.4. >>> >>> I didn't try 12.1, I need to stay on 11.3 for now. >>> >>> ## loader.conf >>> >>> nvidia_load="YES" >>> drm_load="YES" >>> drm2_load="YES" >>> iicbus_load="YES" >>> vboxdrv_load="YES" >>> crypto_load="YES" >>> aesni_load="YES" >>> geom_eli_load="YES" >>> vfs.zfs.arc_max="1024M" >>> zfs_load="YES" >>> iicbus_load="YES" >>> >>> ## sysctl.conf >>> >>> kern.coredump=0 >>> kern.maxfiles=49312 >>> vfs.usermount=1 >>> security.jail.allow_raw_sockets=1 >>> security.jail.sysvipc_allowed=1 >>> security.jail.mount_allowed=1 >>> security.jail.chflags_allowed=1 >>> hw.syscons.bell=0 >>> kern.sched.preempt_thresh=224 >>> kern.ipc.shm_allow_removed=1 >>> kern.shutdown.poweroff_delay=500 >>> kern.bootfile=/boot/kernel/kernel >>> hw.usb.no_shutdown_wait=1 >>> hw.snd.default_unit=3 >>> kern.sched.interact=10 >>> vfs.aio.max_aio_per_proc=256 >>> vfs.aio.max_aio_queue=8192 >>> vfs.aio.max_aio_queue_per_proc=1024 >>> vfs.aio.max_buf_aio=64 >>> net.local.stream.recvspace=65536 >>> net.local.stream.sendspace=65536 >>> >>> >>> loader.conf and sysctl.conf are the same for 10.4 and 11.3 but 11.3 >>> is much much slower when it comes to heavy IO like "find" daily >>> periodic scripts, zfs destroy, starting new applications etc. >>> >>> >>> Kind regards >>> Miroslav Lachman >> >> >> I don't have anything I see which I'm sure will fix things, but you >> could try to remove/comment some of these sysctls to see if 11.3 has >> better defaults now. >> kern.sched.preempt_thresh, kern.maxfiles, kern.sched.interact, vfs.aio.* >> >> What kind of machine is it? CPU, MEM? >> What does gstat say about the saturation of the disk? >> >> Regards, >> Ronald. > > > What might also be interesting is which timer is selected. Sometimes for > some reason another time-source is chosen which can influence a lot of > things (like sound). > Please post the output of "sysctl kern.eventtimer" and "sysctl > kern.timecounter" if possible of 10.4 and 11.3. > > Or compare the /var/run/dmesg.boot files of 10.4 and 11.3 to see if some > hardware is recognized differently. sysctls from 11.3 instance # sysctl kern.eventtimer kern.eventtimer.periodic: 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) HPET3(440) HPET4(440) i8254(100) RTC(0) kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.HPET4.quality: 440 kern.eventtimer.et.HPET4.frequency: 14318180 kern.eventtimer.et.HPET4.flags: 3 kern.eventtimer.et.HPET3.quality: 440 kern.eventtimer.et.HPET3.frequency: 14318180 kern.eventtimer.et.HPET3.flags: 3 kern.eventtimer.et.HPET2.quality: 440 kern.eventtimer.et.HPET2.frequency: 14318180 kern.eventtimer.et.HPET2.flags: 3 kern.eventtimer.et.HPET1.quality: 440 kern.eventtimer.et.HPET1.frequency: 14318180 kern.eventtimer.et.HPET1.flags: 3 kern.eventtimer.et.HPET.quality: 550 kern.eventtimer.et.HPET.frequency: 14318180 kern.eventtimer.et.HPET.flags: 7 kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 3192676730 kern.eventtimer.et.LAPIC.flags: 7 # sysctl kern.timecounter kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 1 kern.timecounter.invariant_tsc: 1 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000) dummy(-1000000) kern.timecounter.hardware: TSC-low kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 16156069 kern.timecounter.tc.ACPI-fast.mask: 16777215 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 49991 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 14318180 kern.timecounter.tc.HPET.counter: 3887656701 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.TSC-low.quality: 1000 kern.timecounter.tc.TSC-low.frequency: 1596338365 kern.timecounter.tc.TSC-low.counter: 1764358755 kern.timecounter.tc.TSC-low.mask: 4294967295 CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.68-MHz K8-class CPU) real memory = 13388218368 (12768 MB) avail memory = 12337872896 (11766 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs last pid: 15694; load averages: 0.42, 0.46, 0.38 up 0+00:30:40 19:01:30 138 processes: 1 running, 136 sleeping, 1 zombie CPU: 3.2% user, 0.0% nice, 0.4% system, 0.5% interrupt, 95.9% idle Mem: 1531M Active, 977M Inact, 39M Laundry, 3047M Wired, 6251M Free ARC: 1275M Total, 311M MFU, 286M MRU, 2542K Anon, 23M Header, 652M Other 124M Compressed, 476M Uncompressed, 3.85:1 Ratio Swap: 4096M Total, 4096M Free As you can see there is a lot of free memory. I will try to catch gstat output next time and also post some info from 10.4 instance. Kind regards Miroslav Lachman From owner-freebsd-fs@freebsd.org Sun Jun 7 17:34:02 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BA3213323CC for ; Sun, 7 Jun 2020 17:34:02 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49g3Qd3s7Zz4KkF for ; Sun, 7 Jun 2020 17:34:01 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.110.112]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1jhzBG-0000Da-3E; Sun, 07 Jun 2020 19:33:58 +0200 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "freebsd-fs@freebsd.org" , "Miroslav Lachman" <000.fbsd@quip.cz> Subject: Re: ZFS on FreeBSD 11.3 slower than 10.4 References: <1ff455a5-d111-86fa-ceb1-1021b6d9a5b6@quip.cz> <8c64cc48-7d79-7591-8bb5-67f3127463b7@quip.cz> <1c8983f7-8aee-61ca-b8ee-fbaa24506841@quip.cz> Date: Sun, 07 Jun 2020 19:33:56 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <1c8983f7-8aee-61ca-b8ee-fbaa24506841@quip.cz> User-Agent: Opera Mail/12.16 (FreeBSD) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50 autolearn=disabled version=3.4.2 X-Scan-Signature: 0b90d2ac815c9ef578df8ccc1b50cfb0 X-Rspamd-Queue-Id: 49g3Qd3s7Zz4KkF X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 195.190.28.88 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-2.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[195.190.28.88:from]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.190.28.64/27]; NEURAL_HAM_LONG(-0.96)[-0.959]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_NA(0.00)[klop.ws]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.72)[-0.715]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[195.190.28.88:from]; NEURAL_HAM_MEDIUM(-1.03)[-1.030]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:47172, ipnet:195.190.28.0/24, country:NL]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Jun 2020 17:34:02 -0000 On Sun, 07 Jun 2020 19:09:14 +0200, Miroslav Lachman <000.fbsd@quip.cz> wrote: > On 2020-06-06 00:57, Ronald Klop wrote: >> On Fri, 05 Jun 2020 22:57:15 +0200, Ronald Klop >> wrote: >> >>> On Sat, 30 May 2020 23:29:48 +0200, Miroslav Lachman >>> <000.fbsd@quip.cz> wrote: >>> >>>> On 2020-05-30 22:10, Ronald Klop wrote: >>>>> On Sat, 23 May 2020 21:44:03 +0200, Miroslav Lachman >>>>> <000.fbsd@quip.cz> wrote: >>>>> >>>>>> I upgraded my old desktop computer few month ago from old 10.4 >>>>>> based PC-BSD to stock FreeBSD 11.3. It uses single 2TB HDD 7200rpm. >>>>>> My problem is that upgraded version is really slow and some desktop >>>>>> applications are very lagging (playing multimedia is interrupted >>>>>> for a fraction of seconds) when there is heavy filesystem activity. >>>>>> >>>>>> I am using zfsnap2 for taking snapshots periodically and when there >>>>>> is enough snapshots zfs destroy is called. In this time the user >>>>>> experience is terrible. Starting new application like browser or >>>>>> even something much smaller takes minutes. The old version based on >>>>>> FreeBSD 10.4 behaves much better. I used the old version for years >>>>>> and never have problems with interrupted multimedia playback. >>>>>> >>>>>> Are there some sysctls to tune to get better desktop interactivity >>>>>> in heavy filesystem operations like zfs destroy, pkg check or other >>>>>> "find" periodic scripts? >>>> >>>> >>>>> How full is the disk? ZFS has poor performance if the disk becomes >>>>> full. >>>>> What is in /etc/sysctl.conf and /boot/loader.conf? >>>>> And did you try to boot 12.1 and did it have the same behavious? >>>> >>>> It is currently 77% full. But it is the same pool with the same >>>> capacity as with 10.4. >>>> >>>> I didn't try 12.1, I need to stay on 11.3 for now. >>>> >>>> ## loader.conf >>>> >>>> nvidia_load="YES" >>>> drm_load="YES" >>>> drm2_load="YES" >>>> iicbus_load="YES" >>>> vboxdrv_load="YES" >>>> crypto_load="YES" >>>> aesni_load="YES" >>>> geom_eli_load="YES" >>>> vfs.zfs.arc_max="1024M" >>>> zfs_load="YES" >>>> iicbus_load="YES" >>>> >>>> ## sysctl.conf >>>> >>>> kern.coredump=0 >>>> kern.maxfiles=49312 >>>> vfs.usermount=1 >>>> security.jail.allow_raw_sockets=1 >>>> security.jail.sysvipc_allowed=1 >>>> security.jail.mount_allowed=1 >>>> security.jail.chflags_allowed=1 >>>> hw.syscons.bell=0 >>>> kern.sched.preempt_thresh=224 >>>> kern.ipc.shm_allow_removed=1 >>>> kern.shutdown.poweroff_delay=500 >>>> kern.bootfile=/boot/kernel/kernel >>>> hw.usb.no_shutdown_wait=1 >>>> hw.snd.default_unit=3 >>>> kern.sched.interact=10 >>>> vfs.aio.max_aio_per_proc=256 >>>> vfs.aio.max_aio_queue=8192 >>>> vfs.aio.max_aio_queue_per_proc=1024 >>>> vfs.aio.max_buf_aio=64 >>>> net.local.stream.recvspace=65536 >>>> net.local.stream.sendspace=65536 >>>> >>>> >>>> loader.conf and sysctl.conf are the same for 10.4 and 11.3 but 11.3 >>>> is much much slower when it comes to heavy IO like "find" daily >>>> periodic scripts, zfs destroy, starting new applications etc. >>>> >>>> >>>> Kind regards >>>> Miroslav Lachman >>> >>> >>> I don't have anything I see which I'm sure will fix things, but you >>> could try to remove/comment some of these sysctls to see if 11.3 has >>> better defaults now. >>> kern.sched.preempt_thresh, kern.maxfiles, kern.sched.interact, >>> vfs.aio.* >>> >>> What kind of machine is it? CPU, MEM? >>> What does gstat say about the saturation of the disk? >>> >>> Regards, >>> Ronald. >> What might also be interesting is which timer is selected. Sometimes >> for some reason another time-source is chosen which can influence a lot >> of things (like sound). >> Please post the output of "sysctl kern.eventtimer" and "sysctl >> kern.timecounter" if possible of 10.4 and 11.3. >> Or compare the /var/run/dmesg.boot files of 10.4 and 11.3 to see if >> some hardware is recognized differently. > > sysctls from 11.3 instance > > # sysctl kern.eventtimer > kern.eventtimer.periodic: 0 > kern.eventtimer.timer: LAPIC > kern.eventtimer.idletick: 0 > kern.eventtimer.singlemul: 2 > kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) > HPET3(440) HPET4(440) i8254(100) RTC(0) > kern.eventtimer.et.i8254.quality: 100 > kern.eventtimer.et.i8254.frequency: 1193182 > kern.eventtimer.et.i8254.flags: 1 > kern.eventtimer.et.RTC.quality: 0 > kern.eventtimer.et.RTC.frequency: 32768 > kern.eventtimer.et.RTC.flags: 17 > kern.eventtimer.et.HPET4.quality: 440 > kern.eventtimer.et.HPET4.frequency: 14318180 > kern.eventtimer.et.HPET4.flags: 3 > kern.eventtimer.et.HPET3.quality: 440 > kern.eventtimer.et.HPET3.frequency: 14318180 > kern.eventtimer.et.HPET3.flags: 3 > kern.eventtimer.et.HPET2.quality: 440 > kern.eventtimer.et.HPET2.frequency: 14318180 > kern.eventtimer.et.HPET2.flags: 3 > kern.eventtimer.et.HPET1.quality: 440 > kern.eventtimer.et.HPET1.frequency: 14318180 > kern.eventtimer.et.HPET1.flags: 3 > kern.eventtimer.et.HPET.quality: 550 > kern.eventtimer.et.HPET.frequency: 14318180 > kern.eventtimer.et.HPET.flags: 7 > kern.eventtimer.et.LAPIC.quality: 600 > kern.eventtimer.et.LAPIC.frequency: 3192676730 > kern.eventtimer.et.LAPIC.flags: 7 > > > # sysctl kern.timecounter > kern.timecounter.tsc_shift: 1 > kern.timecounter.smp_tsc_adjust: 0 > kern.timecounter.smp_tsc: 1 > kern.timecounter.invariant_tsc: 1 > kern.timecounter.fast_gettime: 1 > kern.timecounter.tick: 1 > kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000) > dummy(-1000000) > kern.timecounter.hardware: TSC-low Does setting "sysctl kern.timecounter.hardware=HPET" or "sysctl kern.timecounter.hardware=ACPI-fast" help? Ronald. > kern.timecounter.alloweddeviation: 5 > kern.timecounter.stepwarnings: 0 > kern.timecounter.tc.ACPI-fast.quality: 900 > kern.timecounter.tc.ACPI-fast.frequency: 3579545 > kern.timecounter.tc.ACPI-fast.counter: 16156069 > kern.timecounter.tc.ACPI-fast.mask: 16777215 > kern.timecounter.tc.i8254.quality: 0 > kern.timecounter.tc.i8254.frequency: 1193182 > kern.timecounter.tc.i8254.counter: 49991 > kern.timecounter.tc.i8254.mask: 65535 > kern.timecounter.tc.HPET.quality: 950 > kern.timecounter.tc.HPET.frequency: 14318180 > kern.timecounter.tc.HPET.counter: 3887656701 > kern.timecounter.tc.HPET.mask: 4294967295 > kern.timecounter.tc.TSC-low.quality: 1000 > kern.timecounter.tc.TSC-low.frequency: 1596338365 > kern.timecounter.tc.TSC-low.counter: 1764358755 > kern.timecounter.tc.TSC-low.mask: 4294967295 > > > CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.68-MHz K8-class > CPU) > real memory = 13388218368 (12768 MB) > avail memory = 12337872896 (11766 MB) > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > > > last pid: 15694; load averages: 0.42, 0.46, 0.38 > up 0+00:30:40 > 19:01:30 > 138 processes: 1 running, 136 sleeping, 1 zombie > CPU: 3.2% user, 0.0% nice, 0.4% system, 0.5% interrupt, 95.9% idle > Mem: 1531M Active, 977M Inact, 39M Laundry, 3047M Wired, 6251M Free > ARC: 1275M Total, 311M MFU, 286M MRU, 2542K Anon, 23M Header, 652M Other > 124M Compressed, 476M Uncompressed, 3.85:1 Ratio > Swap: 4096M Total, 4096M Free > > As you can see there is a lot of free memory. > > I will try to catch gstat output next time and also post some info from > 10.4 instance. > > Kind regards > Miroslav Lachman From owner-freebsd-fs@freebsd.org Mon Jun 8 18:20:43 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C34FD337846 for ; Mon, 8 Jun 2020 18:20:43 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49ghQ24vZqz4CLp for ; Mon, 8 Jun 2020 18:20:42 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 49ghPv19wqz6dQk for ; Mon, 8 Jun 2020 20:20:35 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10026) with ESMTP id bu7AB82XbSti for ; Mon, 8 Jun 2020 20:20:33 +0200 (CEST) To: freebsd-fs@freebsd.org From: Guido Falsi Subject: ZFS panic on boot (head) Autocrypt: addr=mad@madpilot.net; keydata= mQENBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAG0Hkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PokBOQQTAQgAIwIbAwIeAQIXgAUL CQgHAwUVCgkICwQWAgMBBQJS79AgAhkBAAoJEBrmhg5Wy9KTc0kH/RO64ORBlTbTHaUaOj8F Je5O5NU2Pt9Cyt5ZWBRvxntr1zPTJGKRPS9ihlIfqT4ZvEngQGp57EUyFbCpI0UWasTerImM tt5WACnGmCzUTB39UXx8Oy4b1EgWeTJQ747e/F1mQLXTNa6ijRBE9fYlTb4gAkPN88/wVV9v 3PZozKLTg16ghBzHM/P7Lk8L7clPEZChX1FTa/6eSt3nvzfCuTMZbBPJF/ph+q1KyPqRgVfh tyhu5dvgMoPz/ni41IfeSrkJTD5RXzdyGR9q4Z1NYeBsLkRjC4LxKAP5KqUsvlOUjKvO1byj ApYdMarol+IGkaSk9e3zVYAJkWKjn/ni8Xa5Ag0EUxB7QQEQAKFhrDceoPdK/IHDSmoj6SQY isvM7VdhcleS7E9DoEAVt7yMbf6HbbMVTTY6ckvwTWQssywLBXNVqxgc4WLJjzfUhgef+WE7 5M3+WFYlOVQLGZY/zEVgma1raYnOHNAOzeHLDmEXjbZP6vGAeDyBbGfQPpE7qGYZ7ubeT3Xw QO+PklcCrvOPj2ZPcAxGNS2xVU/LzONqCrJqLMJSIcCdsbiSP4G5PnDFHtMokaTY6OEr8OEQ fOAerhcHUa/z7Uu8YtmaqKH+QGkE/WEgaRqSiTnv0JOTD+DxehaqvoKPPZ++2NpCZMHB2i6A /xifmQwEiIjEXtcueBRzkNUQkxhqZyS13SrhocL9ydtaVPBzZatAEjUDDEJmAMLVFs45qfyh MiNapHJo2n3MW/E5omqCvEkDdWX/en3P7CK2TemeaDghMsgkNKax/z0wNo5UZCkOPOz0xpNi UilOVbkuezZZNg65741qee2lfXhQIaZ66yT7hphc/N/z3PIAtLeze4u1VR2EXAuZ2sWAdlKC NTlJMsaU/x70BV11Wd/ypnVzM68dfdQIIAj1iMFAD/lXGlEUmKXg5Ov2VQDlTntQoanCYrAg +8CttPzjrydgLZFq3hrtQmfc0se5yv1WHS69+BsUOG09RvvawUDZxUjW19kyeN9THaNRgow3 kSuArUp6zSmJABEBAAGJAR8EGAEIAAkFAlMQe0ECGwwACgkQGuaGDlbL0pMN5wgA4bCkX/qw EVC06ToeR6C2putmSWQMgpDaqrv65Hubo+QGmg2P4ewTYQQ4g6oYWS03qHxqVVWhKz7FjfrV +dH8qbCLfSgIcvdBha7ayGZVrsiuMLKGbw36fcmkZPpSDOfHcP0XH8Z+u9CWj0xUkTxAlZ/7 i6gYSUpG2JWNtdmE/X8VVEyXusCLwy0K0BI60A/4dRTIX3C4QKrJ3ZbUXegz70ynjHf+lQMZ 9IZKASoRMuS5FozPQh6abvmwZEPdf5I9riUElzvHrqJ8Bx0t3Pujdoth+yNHpnBxrtO8LkQd rQ58P0SwcaIX33T2U9pG8bhu5YVR88FQ8OQ0cEsPBpDncrkBDQRPhvpdAQgAsd6mrOq1GSZw lzRscNQa9W2WB/3Tj4ON4PL2e9B+hc9lT/ny2zB3agXu5wbsXTzwxgJpQT7hNHkCSckW98h3 HRjFfhZPNCgInuUGsjcNyVguQh+/47ckhph0s7U+6B4yNuIiqQZk4mo8WgCNj1YIihVmGWEs gDOwMaajbDYZ0r1/3GkKlYjOXeUuT/WgourrSR5oZJVNA/k4X2H7M3JUr1BSc32L7BJt8M7A ntul6k17J0L8GmkvLvTUtQTO+p+DYQMna2ngD3PbAvQRcbEGnkg9ABrdEF0Wp4Gx+gGGWsyF KlHvPdMtgWAy3JsS+rQapG6LoW3yUJpwpEpA86KdBwARAQABiQEfBCgBCAAJBQJTEH0NAh0B AAoJEBrmhg5Wy9KTMZcIAMSsidGF4KpjGcKzhkNK0sEpevcelQ6DzgT7kcXuq6LQ6YOrbof2 /KPgGie9/ToFZfJXH8zE5GefqkKvHZbYssWilFvkI90F9n138kG205NB/2zlaQb74/v9ZMXJ XcipnIx+T2tOMCBgHJU41IMJmB+NfRt5A6CDytJdhWxqppsEo5jjy/7tJM1Nn47G87tAV8qV NUtzbS6zdnbHB4W2BJwCObbVv8epL3hu/L5efV2j2tSbVTmyvK/ClYMBqdtUo3uPX75GF/Ku YDCOP1BTA5zzmzp4PMVd+gmHcMgCZKY6lvcEtdi5FLI0we2kcY8ffPvM2d6MNhFsGLaVI95J 0oqJAR8EGAECAAkFAk+G+l0CGwwACgkQGuaGDlbL0pM18Qf9HTNNhu8N0ISKtmR8lgPhJuu8 9rOEa8KKEatr4fQ7gL+hmYOEqZ/yHLcPQvGxbAlLR7F0SheKvAEk4B1aFwGULPo0SzuO0d/W tVMEbGa95JTm/6mfiymWMlWf8UifD1MDKzzPR7Om0ybeoPM8S/RQTboUU1WLpwd4mg9pVJlK 0xr55GOSHNf4m7S+P1kvl3xgmEj14zVMq9yJBNWFlsQK5ciifh7sFpfuxWdEVbtgIdxpzImK LXSLA0vOroKAvxFTGBrBq3vxV6eUmaKyd5HbbWejmafY1ua5dcnew9lxpWKLdqkC27Vt0Cku +LtTY3325V+BChncwNcJJS7IMmBz6w== Message-ID: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> Date: Mon, 8 Jun 2020 20:20:32 +0200 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49ghQ24vZqz4CLp X-Spamd-Bar: - X-Spamd-Result: default: False [-1.44 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.01)[-1.015]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=bjowvop61wgh]; FROM_HAS_DN(0.00)[]; MISSING_MIME_VERSION(2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.998]; RCVD_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; DKIM_TRACE(0.00)[madpilot.net:+]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; NEURAL_HAM_SHORT(-0.43)[-0.430]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2020 18:20:43 -0000 Hi, On a laptop running head r361728, after a normal reboot I'm experiencing an had panic every boot. Booting a kernel with kdb I get: panic: Solaris(panic): zfs: allocating allocated segment(offset=433818959872 size=4096) of (offset=433818021200 size=32768) I can provide a screenshot of the stack trace if required. Is there a way to recover? Or should I simply reinstall the machine? What is actually happened? Maybe a disk data error? (it's a laptop with only one disk) Or something misbehaved in ZFS? Thanks in advance for any information/help. -- Guido Falsi From owner-freebsd-fs@freebsd.org Mon Jun 8 19:44:05 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A38C339627 for ; Mon, 8 Jun 2020 19:44:05 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49gkGD0RP3z4MhS for ; Mon, 8 Jun 2020 19:44:03 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 49gkGB3sR4z6dQk for ; Mon, 8 Jun 2020 21:44:02 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10026) with ESMTP id AnnpywEwCDpl for ; Mon, 8 Jun 2020 21:44:00 +0200 (CEST) Subject: Re: ZFS panic on boot (head) To: freebsd-fs@freebsd.org References: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> From: Guido Falsi Autocrypt: addr=mad@madpilot.net; prefer-encrypt=mutual; keydata= mQENBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAG0Hkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PokBOQQTAQgAIwIbAwIeAQIXgAUL CQgHAwUVCgkICwQWAgMBBQJS79AgAhkBAAoJEBrmhg5Wy9KTc0kH/RO64ORBlTbTHaUaOj8F Je5O5NU2Pt9Cyt5ZWBRvxntr1zPTJGKRPS9ihlIfqT4ZvEngQGp57EUyFbCpI0UWasTerImM tt5WACnGmCzUTB39UXx8Oy4b1EgWeTJQ747e/F1mQLXTNa6ijRBE9fYlTb4gAkPN88/wVV9v 3PZozKLTg16ghBzHM/P7Lk8L7clPEZChX1FTa/6eSt3nvzfCuTMZbBPJF/ph+q1KyPqRgVfh tyhu5dvgMoPz/ni41IfeSrkJTD5RXzdyGR9q4Z1NYeBsLkRjC4LxKAP5KqUsvlOUjKvO1byj ApYdMarol+IGkaSk9e3zVYAJkWKjn/ni8Xa5Ag0EUxB7QQEQAKFhrDceoPdK/IHDSmoj6SQY isvM7VdhcleS7E9DoEAVt7yMbf6HbbMVTTY6ckvwTWQssywLBXNVqxgc4WLJjzfUhgef+WE7 5M3+WFYlOVQLGZY/zEVgma1raYnOHNAOzeHLDmEXjbZP6vGAeDyBbGfQPpE7qGYZ7ubeT3Xw QO+PklcCrvOPj2ZPcAxGNS2xVU/LzONqCrJqLMJSIcCdsbiSP4G5PnDFHtMokaTY6OEr8OEQ fOAerhcHUa/z7Uu8YtmaqKH+QGkE/WEgaRqSiTnv0JOTD+DxehaqvoKPPZ++2NpCZMHB2i6A /xifmQwEiIjEXtcueBRzkNUQkxhqZyS13SrhocL9ydtaVPBzZatAEjUDDEJmAMLVFs45qfyh MiNapHJo2n3MW/E5omqCvEkDdWX/en3P7CK2TemeaDghMsgkNKax/z0wNo5UZCkOPOz0xpNi UilOVbkuezZZNg65741qee2lfXhQIaZ66yT7hphc/N/z3PIAtLeze4u1VR2EXAuZ2sWAdlKC NTlJMsaU/x70BV11Wd/ypnVzM68dfdQIIAj1iMFAD/lXGlEUmKXg5Ov2VQDlTntQoanCYrAg +8CttPzjrydgLZFq3hrtQmfc0se5yv1WHS69+BsUOG09RvvawUDZxUjW19kyeN9THaNRgow3 kSuArUp6zSmJABEBAAGJAR8EGAEIAAkFAlMQe0ECGwwACgkQGuaGDlbL0pMN5wgA4bCkX/qw EVC06ToeR6C2putmSWQMgpDaqrv65Hubo+QGmg2P4ewTYQQ4g6oYWS03qHxqVVWhKz7FjfrV +dH8qbCLfSgIcvdBha7ayGZVrsiuMLKGbw36fcmkZPpSDOfHcP0XH8Z+u9CWj0xUkTxAlZ/7 i6gYSUpG2JWNtdmE/X8VVEyXusCLwy0K0BI60A/4dRTIX3C4QKrJ3ZbUXegz70ynjHf+lQMZ 9IZKASoRMuS5FozPQh6abvmwZEPdf5I9riUElzvHrqJ8Bx0t3Pujdoth+yNHpnBxrtO8LkQd rQ58P0SwcaIX33T2U9pG8bhu5YVR88FQ8OQ0cEsPBpDncrkBDQRPhvpdAQgAsd6mrOq1GSZw lzRscNQa9W2WB/3Tj4ON4PL2e9B+hc9lT/ny2zB3agXu5wbsXTzwxgJpQT7hNHkCSckW98h3 HRjFfhZPNCgInuUGsjcNyVguQh+/47ckhph0s7U+6B4yNuIiqQZk4mo8WgCNj1YIihVmGWEs gDOwMaajbDYZ0r1/3GkKlYjOXeUuT/WgourrSR5oZJVNA/k4X2H7M3JUr1BSc32L7BJt8M7A ntul6k17J0L8GmkvLvTUtQTO+p+DYQMna2ngD3PbAvQRcbEGnkg9ABrdEF0Wp4Gx+gGGWsyF KlHvPdMtgWAy3JsS+rQapG6LoW3yUJpwpEpA86KdBwARAQABiQEfBCgBCAAJBQJTEH0NAh0B AAoJEBrmhg5Wy9KTMZcIAMSsidGF4KpjGcKzhkNK0sEpevcelQ6DzgT7kcXuq6LQ6YOrbof2 /KPgGie9/ToFZfJXH8zE5GefqkKvHZbYssWilFvkI90F9n138kG205NB/2zlaQb74/v9ZMXJ XcipnIx+T2tOMCBgHJU41IMJmB+NfRt5A6CDytJdhWxqppsEo5jjy/7tJM1Nn47G87tAV8qV NUtzbS6zdnbHB4W2BJwCObbVv8epL3hu/L5efV2j2tSbVTmyvK/ClYMBqdtUo3uPX75GF/Ku YDCOP1BTA5zzmzp4PMVd+gmHcMgCZKY6lvcEtdi5FLI0we2kcY8ffPvM2d6MNhFsGLaVI95J 0oqJAR8EGAECAAkFAk+G+l0CGwwACgkQGuaGDlbL0pM18Qf9HTNNhu8N0ISKtmR8lgPhJuu8 9rOEa8KKEatr4fQ7gL+hmYOEqZ/yHLcPQvGxbAlLR7F0SheKvAEk4B1aFwGULPo0SzuO0d/W tVMEbGa95JTm/6mfiymWMlWf8UifD1MDKzzPR7Om0ybeoPM8S/RQTboUU1WLpwd4mg9pVJlK 0xr55GOSHNf4m7S+P1kvl3xgmEj14zVMq9yJBNWFlsQK5ciifh7sFpfuxWdEVbtgIdxpzImK LXSLA0vOroKAvxFTGBrBq3vxV6eUmaKyd5HbbWejmafY1ua5dcnew9lxpWKLdqkC27Vt0Cku +LtTY3325V+BChncwNcJJS7IMmBz6w== Message-ID: Date: Mon, 8 Jun 2020 21:43:59 +0200 In-Reply-To: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49gkGD0RP3z4MhS X-Spamd-Bar: / X-Spamd-Result: default: False [-0.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.02)[-1.017]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=bjowvop61wgh]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; MISSING_MIME_VERSION(2.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.996]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[madpilot.net:+]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; NEURAL_SPAM_SHORT(0.19)[0.193]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2020 19:44:05 -0000 On 08/06/20 20:20, Guido Falsi via freebsd-fs wrote: > Hi, > > On a laptop running head r361728, after a normal reboot I'm > experiencing an had panic every boot. I meant ZFS panic here. BTW I don't mean any accusation or demand here. I am aware this could be an hardware problem or whatever. I'm just curious, and would like to know if I can spare reinstalling the OS. I'd like to understand what is going on. I also tried too import the pool using a live CD on a usb key. Recent snapshot downloaded from freebsd.org. Unluckily "zfs import" does not list the pool as expected, so it's not even identifying it as such. -- Guido Falsi From owner-freebsd-fs@freebsd.org Mon Jun 8 20:22:51 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3F56A33A1F7 for ; Mon, 8 Jun 2020 20:22:51 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49gl6y2zb2z4S26 for ; Mon, 8 Jun 2020 20:22:50 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 49gl6x0qjzz6dR2 for ; Mon, 8 Jun 2020 22:22:49 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10026) with ESMTP id EZpPnfpysQQi for ; Mon, 8 Jun 2020 22:22:47 +0200 (CEST) Subject: Re: ZFS panic on boot (head) To: freebsd-fs@freebsd.org References: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> From: Guido Falsi Autocrypt: addr=mad@madpilot.net; prefer-encrypt=mutual; keydata= mQENBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAG0Hkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PokBOQQTAQgAIwIbAwIeAQIXgAUL CQgHAwUVCgkICwQWAgMBBQJS79AgAhkBAAoJEBrmhg5Wy9KTc0kH/RO64ORBlTbTHaUaOj8F Je5O5NU2Pt9Cyt5ZWBRvxntr1zPTJGKRPS9ihlIfqT4ZvEngQGp57EUyFbCpI0UWasTerImM tt5WACnGmCzUTB39UXx8Oy4b1EgWeTJQ747e/F1mQLXTNa6ijRBE9fYlTb4gAkPN88/wVV9v 3PZozKLTg16ghBzHM/P7Lk8L7clPEZChX1FTa/6eSt3nvzfCuTMZbBPJF/ph+q1KyPqRgVfh tyhu5dvgMoPz/ni41IfeSrkJTD5RXzdyGR9q4Z1NYeBsLkRjC4LxKAP5KqUsvlOUjKvO1byj ApYdMarol+IGkaSk9e3zVYAJkWKjn/ni8Xa5Ag0EUxB7QQEQAKFhrDceoPdK/IHDSmoj6SQY isvM7VdhcleS7E9DoEAVt7yMbf6HbbMVTTY6ckvwTWQssywLBXNVqxgc4WLJjzfUhgef+WE7 5M3+WFYlOVQLGZY/zEVgma1raYnOHNAOzeHLDmEXjbZP6vGAeDyBbGfQPpE7qGYZ7ubeT3Xw QO+PklcCrvOPj2ZPcAxGNS2xVU/LzONqCrJqLMJSIcCdsbiSP4G5PnDFHtMokaTY6OEr8OEQ fOAerhcHUa/z7Uu8YtmaqKH+QGkE/WEgaRqSiTnv0JOTD+DxehaqvoKPPZ++2NpCZMHB2i6A /xifmQwEiIjEXtcueBRzkNUQkxhqZyS13SrhocL9ydtaVPBzZatAEjUDDEJmAMLVFs45qfyh MiNapHJo2n3MW/E5omqCvEkDdWX/en3P7CK2TemeaDghMsgkNKax/z0wNo5UZCkOPOz0xpNi UilOVbkuezZZNg65741qee2lfXhQIaZ66yT7hphc/N/z3PIAtLeze4u1VR2EXAuZ2sWAdlKC NTlJMsaU/x70BV11Wd/ypnVzM68dfdQIIAj1iMFAD/lXGlEUmKXg5Ov2VQDlTntQoanCYrAg +8CttPzjrydgLZFq3hrtQmfc0se5yv1WHS69+BsUOG09RvvawUDZxUjW19kyeN9THaNRgow3 kSuArUp6zSmJABEBAAGJAR8EGAEIAAkFAlMQe0ECGwwACgkQGuaGDlbL0pMN5wgA4bCkX/qw EVC06ToeR6C2putmSWQMgpDaqrv65Hubo+QGmg2P4ewTYQQ4g6oYWS03qHxqVVWhKz7FjfrV +dH8qbCLfSgIcvdBha7ayGZVrsiuMLKGbw36fcmkZPpSDOfHcP0XH8Z+u9CWj0xUkTxAlZ/7 i6gYSUpG2JWNtdmE/X8VVEyXusCLwy0K0BI60A/4dRTIX3C4QKrJ3ZbUXegz70ynjHf+lQMZ 9IZKASoRMuS5FozPQh6abvmwZEPdf5I9riUElzvHrqJ8Bx0t3Pujdoth+yNHpnBxrtO8LkQd rQ58P0SwcaIX33T2U9pG8bhu5YVR88FQ8OQ0cEsPBpDncrkBDQRPhvpdAQgAsd6mrOq1GSZw lzRscNQa9W2WB/3Tj4ON4PL2e9B+hc9lT/ny2zB3agXu5wbsXTzwxgJpQT7hNHkCSckW98h3 HRjFfhZPNCgInuUGsjcNyVguQh+/47ckhph0s7U+6B4yNuIiqQZk4mo8WgCNj1YIihVmGWEs gDOwMaajbDYZ0r1/3GkKlYjOXeUuT/WgourrSR5oZJVNA/k4X2H7M3JUr1BSc32L7BJt8M7A ntul6k17J0L8GmkvLvTUtQTO+p+DYQMna2ngD3PbAvQRcbEGnkg9ABrdEF0Wp4Gx+gGGWsyF KlHvPdMtgWAy3JsS+rQapG6LoW3yUJpwpEpA86KdBwARAQABiQEfBCgBCAAJBQJTEH0NAh0B AAoJEBrmhg5Wy9KTMZcIAMSsidGF4KpjGcKzhkNK0sEpevcelQ6DzgT7kcXuq6LQ6YOrbof2 /KPgGie9/ToFZfJXH8zE5GefqkKvHZbYssWilFvkI90F9n138kG205NB/2zlaQb74/v9ZMXJ XcipnIx+T2tOMCBgHJU41IMJmB+NfRt5A6CDytJdhWxqppsEo5jjy/7tJM1Nn47G87tAV8qV NUtzbS6zdnbHB4W2BJwCObbVv8epL3hu/L5efV2j2tSbVTmyvK/ClYMBqdtUo3uPX75GF/Ku YDCOP1BTA5zzmzp4PMVd+gmHcMgCZKY6lvcEtdi5FLI0we2kcY8ffPvM2d6MNhFsGLaVI95J 0oqJAR8EGAECAAkFAk+G+l0CGwwACgkQGuaGDlbL0pM18Qf9HTNNhu8N0ISKtmR8lgPhJuu8 9rOEa8KKEatr4fQ7gL+hmYOEqZ/yHLcPQvGxbAlLR7F0SheKvAEk4B1aFwGULPo0SzuO0d/W tVMEbGa95JTm/6mfiymWMlWf8UifD1MDKzzPR7Om0ybeoPM8S/RQTboUU1WLpwd4mg9pVJlK 0xr55GOSHNf4m7S+P1kvl3xgmEj14zVMq9yJBNWFlsQK5ciifh7sFpfuxWdEVbtgIdxpzImK LXSLA0vOroKAvxFTGBrBq3vxV6eUmaKyd5HbbWejmafY1ua5dcnew9lxpWKLdqkC27Vt0Cku +LtTY3325V+BChncwNcJJS7IMmBz6w== Message-ID: <88e1c4e4-f328-e3be-5b3b-4e8992d388d9@madpilot.net> Date: Mon, 8 Jun 2020 22:22:46 +0200 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49gl6y2zb2z4S26 X-Spamd-Bar: - X-Spamd-Result: default: False [-1.19 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.02)[-1.017]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=bjowvop61wgh]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; MISSING_MIME_VERSION(2.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.996]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[madpilot.net:+]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; NEURAL_HAM_SHORT(-0.18)[-0.176]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2020 20:22:51 -0000 On 08/06/20 21:43, Guido Falsi via freebsd-fs wrote: > On 08/06/20 20:20, Guido Falsi via freebsd-fs wrote: >> Hi, >> >> On a laptop running head r361728, after a normal reboot I'm >> experiencing an had panic every boot. > > I meant ZFS panic here. > > BTW I don't mean any accusation or demand here. I am aware this could be > an hardware problem or whatever. I'm just curious, and would like to > know if I can spare reinstalling the OS. I'd like to understand what is > going on. > > I also tried too import the pool using a live CD on a usb key. Recent > snapshot downloaded from freebsd.org. Unluckily "zfs import" does not > list the pool as expected, so it's not even identifying it as such. > Interesting discover: setting vfs.zfs.recover=1 in loader allows the machine to mount the FS rw and boot successfully. It still gives two errors: Solaris: WARNING: zfs: allocating allocated segment(offset=433818959872 size=4096) of (offset=433818931200 size=32768) Solaris: WARNING: zfs: allocating allocated segment(offset=433815740416 size=4096) of (offset=433815736320 size=8192) Unluckily trying to reboot without vfs.zfs.recover=1 leads to the same panic, so the underlying corruption is not fixed. a zpool scrub shows no errors. rebooting without the recover option still panics. Is there a way to repair this now that I'm able to boot multiuser? Apart from scrub is there some more diagnostic to force rebuilding the broken metadata? Since now this laptop is not reliable anymore IMHO, I'd reinstall it at some pint, but if someone is interested in debug data that can be extracted from it, and is willing to hep me extract it, I can avoid reinstalling it for some days. So, really, if anyone is interested let me know! -- Guido Falsi From owner-freebsd-fs@freebsd.org Mon Jun 8 20:44:16 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA4FE33AEC6 for ; Mon, 8 Jun 2020 20:44:16 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49glbg5BBZz4W9K for ; Mon, 8 Jun 2020 20:44:15 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 49glbf39wkz6dR2 for ; Mon, 8 Jun 2020 22:44:14 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10026) with ESMTP id Kdvybyv0_9cw for ; Mon, 8 Jun 2020 22:44:12 +0200 (CEST) Subject: Re: ZFS panic on boot (head) To: freebsd-fs@freebsd.org References: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> <88e1c4e4-f328-e3be-5b3b-4e8992d388d9@madpilot.net> From: Guido Falsi Autocrypt: addr=mad@madpilot.net; prefer-encrypt=mutual; keydata= mQENBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAG0Hkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PokBOQQTAQgAIwIbAwIeAQIXgAUL CQgHAwUVCgkICwQWAgMBBQJS79AgAhkBAAoJEBrmhg5Wy9KTc0kH/RO64ORBlTbTHaUaOj8F Je5O5NU2Pt9Cyt5ZWBRvxntr1zPTJGKRPS9ihlIfqT4ZvEngQGp57EUyFbCpI0UWasTerImM tt5WACnGmCzUTB39UXx8Oy4b1EgWeTJQ747e/F1mQLXTNa6ijRBE9fYlTb4gAkPN88/wVV9v 3PZozKLTg16ghBzHM/P7Lk8L7clPEZChX1FTa/6eSt3nvzfCuTMZbBPJF/ph+q1KyPqRgVfh tyhu5dvgMoPz/ni41IfeSrkJTD5RXzdyGR9q4Z1NYeBsLkRjC4LxKAP5KqUsvlOUjKvO1byj ApYdMarol+IGkaSk9e3zVYAJkWKjn/ni8Xa5Ag0EUxB7QQEQAKFhrDceoPdK/IHDSmoj6SQY isvM7VdhcleS7E9DoEAVt7yMbf6HbbMVTTY6ckvwTWQssywLBXNVqxgc4WLJjzfUhgef+WE7 5M3+WFYlOVQLGZY/zEVgma1raYnOHNAOzeHLDmEXjbZP6vGAeDyBbGfQPpE7qGYZ7ubeT3Xw QO+PklcCrvOPj2ZPcAxGNS2xVU/LzONqCrJqLMJSIcCdsbiSP4G5PnDFHtMokaTY6OEr8OEQ fOAerhcHUa/z7Uu8YtmaqKH+QGkE/WEgaRqSiTnv0JOTD+DxehaqvoKPPZ++2NpCZMHB2i6A /xifmQwEiIjEXtcueBRzkNUQkxhqZyS13SrhocL9ydtaVPBzZatAEjUDDEJmAMLVFs45qfyh MiNapHJo2n3MW/E5omqCvEkDdWX/en3P7CK2TemeaDghMsgkNKax/z0wNo5UZCkOPOz0xpNi UilOVbkuezZZNg65741qee2lfXhQIaZ66yT7hphc/N/z3PIAtLeze4u1VR2EXAuZ2sWAdlKC NTlJMsaU/x70BV11Wd/ypnVzM68dfdQIIAj1iMFAD/lXGlEUmKXg5Ov2VQDlTntQoanCYrAg +8CttPzjrydgLZFq3hrtQmfc0se5yv1WHS69+BsUOG09RvvawUDZxUjW19kyeN9THaNRgow3 kSuArUp6zSmJABEBAAGJAR8EGAEIAAkFAlMQe0ECGwwACgkQGuaGDlbL0pMN5wgA4bCkX/qw EVC06ToeR6C2putmSWQMgpDaqrv65Hubo+QGmg2P4ewTYQQ4g6oYWS03qHxqVVWhKz7FjfrV +dH8qbCLfSgIcvdBha7ayGZVrsiuMLKGbw36fcmkZPpSDOfHcP0XH8Z+u9CWj0xUkTxAlZ/7 i6gYSUpG2JWNtdmE/X8VVEyXusCLwy0K0BI60A/4dRTIX3C4QKrJ3ZbUXegz70ynjHf+lQMZ 9IZKASoRMuS5FozPQh6abvmwZEPdf5I9riUElzvHrqJ8Bx0t3Pujdoth+yNHpnBxrtO8LkQd rQ58P0SwcaIX33T2U9pG8bhu5YVR88FQ8OQ0cEsPBpDncrkBDQRPhvpdAQgAsd6mrOq1GSZw lzRscNQa9W2WB/3Tj4ON4PL2e9B+hc9lT/ny2zB3agXu5wbsXTzwxgJpQT7hNHkCSckW98h3 HRjFfhZPNCgInuUGsjcNyVguQh+/47ckhph0s7U+6B4yNuIiqQZk4mo8WgCNj1YIihVmGWEs gDOwMaajbDYZ0r1/3GkKlYjOXeUuT/WgourrSR5oZJVNA/k4X2H7M3JUr1BSc32L7BJt8M7A ntul6k17J0L8GmkvLvTUtQTO+p+DYQMna2ngD3PbAvQRcbEGnkg9ABrdEF0Wp4Gx+gGGWsyF KlHvPdMtgWAy3JsS+rQapG6LoW3yUJpwpEpA86KdBwARAQABiQEfBCgBCAAJBQJTEH0NAh0B AAoJEBrmhg5Wy9KTMZcIAMSsidGF4KpjGcKzhkNK0sEpevcelQ6DzgT7kcXuq6LQ6YOrbof2 /KPgGie9/ToFZfJXH8zE5GefqkKvHZbYssWilFvkI90F9n138kG205NB/2zlaQb74/v9ZMXJ XcipnIx+T2tOMCBgHJU41IMJmB+NfRt5A6CDytJdhWxqppsEo5jjy/7tJM1Nn47G87tAV8qV NUtzbS6zdnbHB4W2BJwCObbVv8epL3hu/L5efV2j2tSbVTmyvK/ClYMBqdtUo3uPX75GF/Ku YDCOP1BTA5zzmzp4PMVd+gmHcMgCZKY6lvcEtdi5FLI0we2kcY8ffPvM2d6MNhFsGLaVI95J 0oqJAR8EGAECAAkFAk+G+l0CGwwACgkQGuaGDlbL0pM18Qf9HTNNhu8N0ISKtmR8lgPhJuu8 9rOEa8KKEatr4fQ7gL+hmYOEqZ/yHLcPQvGxbAlLR7F0SheKvAEk4B1aFwGULPo0SzuO0d/W tVMEbGa95JTm/6mfiymWMlWf8UifD1MDKzzPR7Om0ybeoPM8S/RQTboUU1WLpwd4mg9pVJlK 0xr55GOSHNf4m7S+P1kvl3xgmEj14zVMq9yJBNWFlsQK5ciifh7sFpfuxWdEVbtgIdxpzImK LXSLA0vOroKAvxFTGBrBq3vxV6eUmaKyd5HbbWejmafY1ua5dcnew9lxpWKLdqkC27Vt0Cku +LtTY3325V+BChncwNcJJS7IMmBz6w== Message-ID: Date: Mon, 8 Jun 2020 22:44:11 +0200 In-Reply-To: <88e1c4e4-f328-e3be-5b3b-4e8992d388d9@madpilot.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49glbg5BBZz4W9K X-Spamd-Bar: - X-Spamd-Result: default: False [-1.19 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.02)[-1.018]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=bjowvop61wgh]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; MISSING_MIME_VERSION(2.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.997]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[madpilot.net:+]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; NEURAL_HAM_SHORT(-0.18)[-0.176]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2020 20:44:17 -0000 On 08/06/20 22:22, Guido Falsi via freebsd-fs wrote: > So, really, if anyone is interested let me know! > I've found bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=180001 which references this same error. I'm writing any further information I get there. -- Guido Falsi From owner-freebsd-fs@freebsd.org Tue Jun 9 05:53:14 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C5A5D349CC0 for ; Tue, 9 Jun 2020 05:53:14 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f67.google.com (mail-lf1-f67.google.com [209.85.167.67]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49gzn608J3z3RRg for ; Tue, 9 Jun 2020 05:53:13 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f67.google.com with SMTP id w15so11663018lfe.11 for ; Mon, 08 Jun 2020 22:53:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=QC2+nTIzAQydTwQodb6GEFJXnWyGWx9ob9nbEY1FZ3Q=; b=D+lTIpcouD3qFPGJUS9cPHeNieQY2kWBoFW0W3L4Z6dVarEUVS8+HVp3xPgaFViyBg kyuLu7GCWrAi7GHtFxHmqRPv5g+KScr8+gxOFGEEvaiCaNjrBhNIyvu7iWBaQEKYOgAC ouW/IT3zP27jL582VKpJLGEcOCqS+Yxjjy4wElUX2yG8qCGHcJSpK9JiCUH8ajQCyxCB vopuIIV8UsMWevuzeiHKR1OWAZKoNOwlGyPPlH6uKEUx6bbyCpCKyKWmZymLp/HYk+Ks Iq+WwAxvXVT4sdiJY+7JJHamKpx+mtmOP+WMZtiVr5WoMPP4pI0AD8uwDSy7dftOVCrw Zt7A== X-Gm-Message-State: AOAM530NTJWwFnmabGf2b0sipKqr9Ip6TeI0UGJGipQ72um2lyvhEIt4 LcV73x9SFLtOpsWmPlQG7cQS1esbqk8= X-Google-Smtp-Source: ABdhPJyWkX3UIQ34cx53Sv0FU9ky97bpadPJ18Dn90IHCBsgzm8lzHBeay13DmcEQbYshYTts2Vq8Q== X-Received: by 2002:ac2:53a2:: with SMTP id j2mr14123545lfh.139.1591681991683; Mon, 08 Jun 2020 22:53:11 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 13sm4129397ljj.83.2020.06.08.22.53.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 08 Jun 2020 22:53:10 -0700 (PDT) Subject: Re: ZFS panic on boot (head) To: Guido Falsi , freebsd-fs@freebsd.org References: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <5c6933c3-7996-694e-91e2-3dda125926e7@FreeBSD.org> Date: Tue, 9 Jun 2020 08:53:09 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49gzn608J3z3RRg X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.67 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.22 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.04)[-1.041]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; DMARC_NA(0.00)[FreeBSD.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.16)[-0.158]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.67:from]; NEURAL_HAM_MEDIUM(-1.02)[-1.023]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.67:from]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[93.72.151.96:received] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jun 2020 05:53:14 -0000 On 08/06/2020 21:20, Guido Falsi via freebsd-fs wrote: > Hi, > > On a laptop running head r361728, after a normal reboot I'm > experiencing an had panic every boot. > > Booting a kernel with kdb I get: > > panic: Solaris(panic): zfs: allocating allocated > segment(offset=433818959872 size=4096) of (offset=433818021200 size=32768) > > I can provide a screenshot of the stack trace if required. > > Is there a way to recover? Or should I simply reinstall the machine? > > What is actually happened? Maybe a disk data error? (it's a laptop with > only one disk) > > Or something misbehaved in ZFS? > > Thanks in advance for any information/help. What you see is inconsistent information in a space map. There are two elements that describe overlapping ranges of disk space. That is something that must never happen. I do not know how that can be repaired, the safest solution is to copy / restore the data to a fresh pool. As to why, this could be a result of a bug in ZFS. If you google for the panic messages (sans concrete numbers) you can find some bugs that were fixed in the past. Also, and it once happened to me, it could be a hardware problem. I used non-ECC memory, (at least) one bit got flipped and it was a bit describing a space map entry. After the corruption the good entry turned into a conflicting entry. A disk error can probably be ruled out. First, ZFS stores 3 copies of all important metadata including space maps, so all three would have to be corrupted. Second, even if all copies were corrupted then you would get a checksum error (this is ZFS). -- Andriy Gapon From owner-freebsd-fs@freebsd.org Tue Jun 9 07:54:44 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4740B34CC11 for ; Tue, 9 Jun 2020 07:54:44 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49h2TH5vz2z3ghn; Tue, 9 Jun 2020 07:54:43 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 49h2TG2HrTz6dQk; Tue, 9 Jun 2020 09:54:42 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10026) with ESMTP id pRs2KWYS3lPr; Tue, 9 Jun 2020 09:54:39 +0200 (CEST) Subject: Re: ZFS panic on boot (head) To: Andriy Gapon , freebsd-fs@freebsd.org References: <8a092b2f-ae13-ef3a-0086-9d4aaaf8466b@madpilot.net> <5c6933c3-7996-694e-91e2-3dda125926e7@FreeBSD.org> From: Guido Falsi Autocrypt: addr=mad@madpilot.net; keydata= mQENBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAG0Hkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PokBOQQTAQgAIwIbAwIeAQIXgAUL CQgHAwUVCgkICwQWAgMBBQJS79AgAhkBAAoJEBrmhg5Wy9KTc0kH/RO64ORBlTbTHaUaOj8F Je5O5NU2Pt9Cyt5ZWBRvxntr1zPTJGKRPS9ihlIfqT4ZvEngQGp57EUyFbCpI0UWasTerImM tt5WACnGmCzUTB39UXx8Oy4b1EgWeTJQ747e/F1mQLXTNa6ijRBE9fYlTb4gAkPN88/wVV9v 3PZozKLTg16ghBzHM/P7Lk8L7clPEZChX1FTa/6eSt3nvzfCuTMZbBPJF/ph+q1KyPqRgVfh tyhu5dvgMoPz/ni41IfeSrkJTD5RXzdyGR9q4Z1NYeBsLkRjC4LxKAP5KqUsvlOUjKvO1byj ApYdMarol+IGkaSk9e3zVYAJkWKjn/ni8Xa5Ag0EUxB7QQEQAKFhrDceoPdK/IHDSmoj6SQY isvM7VdhcleS7E9DoEAVt7yMbf6HbbMVTTY6ckvwTWQssywLBXNVqxgc4WLJjzfUhgef+WE7 5M3+WFYlOVQLGZY/zEVgma1raYnOHNAOzeHLDmEXjbZP6vGAeDyBbGfQPpE7qGYZ7ubeT3Xw QO+PklcCrvOPj2ZPcAxGNS2xVU/LzONqCrJqLMJSIcCdsbiSP4G5PnDFHtMokaTY6OEr8OEQ fOAerhcHUa/z7Uu8YtmaqKH+QGkE/WEgaRqSiTnv0JOTD+DxehaqvoKPPZ++2NpCZMHB2i6A /xifmQwEiIjEXtcueBRzkNUQkxhqZyS13SrhocL9ydtaVPBzZatAEjUDDEJmAMLVFs45qfyh MiNapHJo2n3MW/E5omqCvEkDdWX/en3P7CK2TemeaDghMsgkNKax/z0wNo5UZCkOPOz0xpNi UilOVbkuezZZNg65741qee2lfXhQIaZ66yT7hphc/N/z3PIAtLeze4u1VR2EXAuZ2sWAdlKC NTlJMsaU/x70BV11Wd/ypnVzM68dfdQIIAj1iMFAD/lXGlEUmKXg5Ov2VQDlTntQoanCYrAg +8CttPzjrydgLZFq3hrtQmfc0se5yv1WHS69+BsUOG09RvvawUDZxUjW19kyeN9THaNRgow3 kSuArUp6zSmJABEBAAGJAR8EGAEIAAkFAlMQe0ECGwwACgkQGuaGDlbL0pMN5wgA4bCkX/qw EVC06ToeR6C2putmSWQMgpDaqrv65Hubo+QGmg2P4ewTYQQ4g6oYWS03qHxqVVWhKz7FjfrV +dH8qbCLfSgIcvdBha7ayGZVrsiuMLKGbw36fcmkZPpSDOfHcP0XH8Z+u9CWj0xUkTxAlZ/7 i6gYSUpG2JWNtdmE/X8VVEyXusCLwy0K0BI60A/4dRTIX3C4QKrJ3ZbUXegz70ynjHf+lQMZ 9IZKASoRMuS5FozPQh6abvmwZEPdf5I9riUElzvHrqJ8Bx0t3Pujdoth+yNHpnBxrtO8LkQd rQ58P0SwcaIX33T2U9pG8bhu5YVR88FQ8OQ0cEsPBpDncrkBDQRPhvpdAQgAsd6mrOq1GSZw lzRscNQa9W2WB/3Tj4ON4PL2e9B+hc9lT/ny2zB3agXu5wbsXTzwxgJpQT7hNHkCSckW98h3 HRjFfhZPNCgInuUGsjcNyVguQh+/47ckhph0s7U+6B4yNuIiqQZk4mo8WgCNj1YIihVmGWEs gDOwMaajbDYZ0r1/3GkKlYjOXeUuT/WgourrSR5oZJVNA/k4X2H7M3JUr1BSc32L7BJt8M7A ntul6k17J0L8GmkvLvTUtQTO+p+DYQMna2ngD3PbAvQRcbEGnkg9ABrdEF0Wp4Gx+gGGWsyF KlHvPdMtgWAy3JsS+rQapG6LoW3yUJpwpEpA86KdBwARAQABiQEfBCgBCAAJBQJTEH0NAh0B AAoJEBrmhg5Wy9KTMZcIAMSsidGF4KpjGcKzhkNK0sEpevcelQ6DzgT7kcXuq6LQ6YOrbof2 /KPgGie9/ToFZfJXH8zE5GefqkKvHZbYssWilFvkI90F9n138kG205NB/2zlaQb74/v9ZMXJ XcipnIx+T2tOMCBgHJU41IMJmB+NfRt5A6CDytJdhWxqppsEo5jjy/7tJM1Nn47G87tAV8qV NUtzbS6zdnbHB4W2BJwCObbVv8epL3hu/L5efV2j2tSbVTmyvK/ClYMBqdtUo3uPX75GF/Ku YDCOP1BTA5zzmzp4PMVd+gmHcMgCZKY6lvcEtdi5FLI0we2kcY8ffPvM2d6MNhFsGLaVI95J 0oqJAR8EGAECAAkFAk+G+l0CGwwACgkQGuaGDlbL0pM18Qf9HTNNhu8N0ISKtmR8lgPhJuu8 9rOEa8KKEatr4fQ7gL+hmYOEqZ/yHLcPQvGxbAlLR7F0SheKvAEk4B1aFwGULPo0SzuO0d/W tVMEbGa95JTm/6mfiymWMlWf8UifD1MDKzzPR7Om0ybeoPM8S/RQTboUU1WLpwd4mg9pVJlK 0xr55GOSHNf4m7S+P1kvl3xgmEj14zVMq9yJBNWFlsQK5ciifh7sFpfuxWdEVbtgIdxpzImK LXSLA0vOroKAvxFTGBrBq3vxV6eUmaKyd5HbbWejmafY1ua5dcnew9lxpWKLdqkC27Vt0Cku +LtTY3325V+BChncwNcJJS7IMmBz6w== Message-ID: <6547bdf1-2332-ba2f-b7f6-51dbeed92d63@madpilot.net> Date: Tue, 9 Jun 2020 09:54:39 +0200 In-Reply-To: <5c6933c3-7996-694e-91e2-3dda125926e7@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49h2TH5vz2z3ghn X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; REPLY(-4.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jun 2020 07:54:44 -0000 On 09/06/20 07:53, Andriy Gapon wrote: > On 08/06/2020 21:20, Guido Falsi via freebsd-fs wrote: >> Hi, >> >> On a laptop running head r361728, after a normal reboot I'm >> experiencing an had panic every boot. >> >> Booting a kernel with kdb I get: >> >> panic: Solaris(panic): zfs: allocating allocated >> segment(offset=433818959872 size=4096) of (offset=433818021200 size=32768) >> >> I can provide a screenshot of the stack trace if required. >> >> Is there a way to recover? Or should I simply reinstall the machine? >> >> What is actually happened? Maybe a disk data error? (it's a laptop with >> only one disk) >> >> Or something misbehaved in ZFS? >> >> Thanks in advance for any information/help. > > What you see is inconsistent information in a space map. > There are two elements that describe overlapping ranges of disk space. > That is something that must never happen. > I do not know how that can be repaired, the safest solution is to copy / restore > the data to a fresh pool. I see. Being a laptop with no important data on it, I think I'll just reinstall/reconfigure the system. > > As to why, this could be a result of a bug in ZFS. If you google for the panic > messages (sans concrete numbers) you can find some bugs that were fixed in the > past. Also, and it once happened to me, it could be a hardware problem. I used > non-ECC memory, (at least) one bit got flipped and it was a bit describing a > space map entry. After the corruption the good entry turned into a conflicting > entry. Again since the hardware does not have ECC memory (and I suspect no support for it), this could be the explanation for what happened. I bet I've been very unlucky! > > A disk error can probably be ruled out. First, ZFS stores 3 copies of all > important metadata including space maps, so all three would have to be > corrupted. Second, even if all copies were corrupted then you would get a > checksum error (this is ZFS). > At least I can have faith in the disk hardware :) Thanks for taking your time to give me this information! -- Guido Falsi From owner-freebsd-fs@freebsd.org Thu Jun 11 18:03:16 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5C5D833C9CA for ; Thu, 11 Jun 2020 18:03:16 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49jWtX1mBrz4BWB for ; Thu, 11 Jun 2020 18:03:16 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from localhost (p4fd3af72.dip0.t-ipconnect.de [79.211.175.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: gbe) by smtp.freebsd.org (Postfix) with ESMTPSA id D69931E614 for ; Thu, 11 Jun 2020 18:03:15 +0000 (UTC) (envelope-from gbe@freebsd.org) Date: Thu, 11 Jun 2020 20:03:12 +0200 From: Gordon Bergling To: freebsd-fs@freebsd.org Subject: ZFS failure (vdev probe failure) Message-ID: <20200611180312.GA42266@lion.0xfce3.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="RnlQjJ0d97Da+TV1" Content-Disposition: inline X-Url: X-Operating-System: FreeBSD 12.1-STABLE amd64 X-Host-Uptime: 8:02PM up 40 mins, 2 users, load averages: 1.36, 1.27, 1.17 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2020 18:03:16 -0000 --RnlQjJ0d97Da+TV1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, has anyone seen this kind of failure on a vdev / zfs pool? Jun 11 19:17:23 lion ZFS[16825]: vdev probe failure, zpool=$boiler path=$/dev/da2 Jun 11 19:17:23 lion ZFS[17154]: pool I/O failure, zpool=$boiler error=$6 Jun 11 19:17:23 lion ZFS[17181]: catastrophic pool I/O failure, zpool=$boiler Jun 11 19:17:26 lion ZFS[17394]: pool I/O failure, zpool=$boiler error=$28 Jun 11 19:17:26 lion ZFS[17702]: pool I/O failure, zpool=$boiler error=$28 Jun 11 19:17:26 lion ZFS[17840]: pool I/O failure, zpool=$boiler error=$28 Jun 11 19:17:26 lion ZFS[18189]: pool I/O failure, zpool=$boiler error=$28 Jun 11 19:17:26 lion ZFS[18382]: pool I/O failure, zpool=$boiler error=$28 Jun 11 19:17:26 lion ZFS[18733]: pool I/O failure, zpool=$boiler error=$28 Jun 11 19:17:26 lion ZFS[19051]: pool I/O failure, zpool=$boiler error=$28 Jun 11 19:17:26 lion ZFS[19346]: catastrophic pool I/O failure, zpool=$boiler After an reboot everything seems to be okay and I do daily scrubs and none of them as shown an error before. --Gordon --RnlQjJ0d97Da+TV1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAEBCgB9FiEEYbWI0KY5X7yH/Fy4OQX2V8rP09wFAl7icd9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDYx QjU4OEQwQTYzOTVGQkM4N0ZDNUNCODM5MDVGNjU3Q0FDRkQzREMACgkQOQX2V8rP 09y+lgf+JvTgZ/7A+LfqmGmctAvtK+wvBxbL7w63ca5UmJ/PwoLAA5kx+QICpPgN FaeN/TVNNP0AdG3IGimpnGgtb09GsSWpKSrcWN6rjQRmTLiVvHgO4zoSyBX3dAfV hjrDsSWcGoSI6D2dHBGxhJmt7V3lgxD97nz0DEFoT3vYJphVd3+4knIwlAqBSGWD CxuqDdJ9s2uUpK7OMWYnp7uLFLf2iqTYGWqF/LZK39u7+GwTfPrT9TUnxugvfMYJ yEGt1GyxbKtpp3lt0QBTulVDP+r1KnF2GB5F70rc+LWp7r2zKtbHthcR7OujXoL0 FllIm92kVRg2v0NtBIjwoP/kE9rjyg== =OmPX -----END PGP SIGNATURE----- --RnlQjJ0d97Da+TV1-- From owner-freebsd-fs@freebsd.org Fri Jun 12 10:35:48 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1C90B3363B5 for ; Fri, 12 Jun 2020 10:35:48 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from mail.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 49jxvk657Kz4Js6; Fri, 12 Jun 2020 10:35:46 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id 05CAZYo4000595; Fri, 12 Jun 2020 11:35:34 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id 05CAZY5o012554; Fri, 12 Jun 2020 11:35:34 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id 05CAZYmv012550; Fri, 12 Jun 2020 11:35:34 +0100 Date: Fri, 12 Jun 2020 11:35:34 +0100 Message-Id: <202006121035.05CAZYmv012550@higson.cam.lispworks.com> From: Martin Simmons To: Gordon Bergling CC: freebsd-fs@freebsd.org In-reply-to: <20200611180312.GA42266@lion.0xfce3.net> (message from Gordon Bergling on Thu, 11 Jun 2020 20:03:12 +0200) Subject: Re: ZFS failure (vdev probe failure) References: <20200611180312.GA42266@lion.0xfce3.net> X-Rspamd-Queue-Id: 49jxvk657Kz4Js6 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [2.63 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.73)[0.728]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.01)[-0.010]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[46.17.166.21:from]; NEURAL_SPAM_LONG(0.92)[0.916]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; RWL_MAILSPIKE_VERYGOOD(0.00)[46.17.166.21:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2020 10:35:48 -0000 >>>>> On Thu, 11 Jun 2020 20:03:12 +0200, Gordon Bergling said: > > Hi, > > has anyone seen this kind of failure on a vdev / zfs pool? > > Jun 11 19:17:23 lion ZFS[16825]: vdev probe failure, zpool=$boiler path=$/dev/da2 > Jun 11 19:17:23 lion ZFS[17154]: pool I/O failure, zpool=$boiler error=$6 > Jun 11 19:17:23 lion ZFS[17181]: catastrophic pool I/O failure, zpool=$boiler > Jun 11 19:17:26 lion ZFS[17394]: pool I/O failure, zpool=$boiler error=$28 > Jun 11 19:17:26 lion ZFS[17702]: pool I/O failure, zpool=$boiler error=$28 > Jun 11 19:17:26 lion ZFS[17840]: pool I/O failure, zpool=$boiler error=$28 > Jun 11 19:17:26 lion ZFS[18189]: pool I/O failure, zpool=$boiler error=$28 > Jun 11 19:17:26 lion ZFS[18382]: pool I/O failure, zpool=$boiler error=$28 > Jun 11 19:17:26 lion ZFS[18733]: pool I/O failure, zpool=$boiler error=$28 > Jun 11 19:17:26 lion ZFS[19051]: pool I/O failure, zpool=$boiler error=$28 > Jun 11 19:17:26 lion ZFS[19346]: catastrophic pool I/O failure, zpool=$boiler > > After an reboot everything seems to be okay and I do daily scrubs and none > of them as shown an error before. > > --Gordon Were there any other syslog messages about /dev/da2? It looks like it stopped working temporarily. __Martin From owner-freebsd-fs@freebsd.org Fri Jun 12 13:30:58 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3258C33A984 for ; Fri, 12 Jun 2020 13:30:58 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49k1ns37fCz4Vyp; Fri, 12 Jun 2020 13:30:57 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:615c:68ac:a14a:abad] ([IPv6:2607:f3e0:0:4:615c:68ac:a14a:abad]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 05CDUuHK039769 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 12 Jun 2020 09:30:56 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: ZFS failure (vdev probe failure) To: Martin Simmons , Gordon Bergling Cc: freebsd-fs@freebsd.org References: <20200611180312.GA42266@lion.0xfce3.net> <202006121035.05CAZYmv012550@higson.cam.lispworks.com> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: <6da378af-8ca5-ffc4-16f1-8da5960d1c18@sentex.net> Date: Fri, 12 Jun 2020 09:30:56 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <202006121035.05CAZYmv012550@higson.cam.lispworks.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 49k1ns37fCz4Vyp X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.85)[-0.849]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; NEURAL_HAM_LONG(-0.98)[-0.983]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.08)[-0.079]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2020 13:30:58 -0000 On 6/12/2020 6:35 AM, Martin Simmons wrote: >>>>>> On Thu, 11 Jun 2020 20:03:12 +0200, Gordon Bergling said: >> Hi, >> >> has anyone seen this kind of failure on a vdev / zfs pool? >> >> Jun 11 19:17:23 lion ZFS[16825]: vdev probe failure, zpool=$boiler path=$/dev/da2 >> Jun 11 19:17:23 lion ZFS[17154]: pool I/O failure, zpool=$boiler error=$6 >> Jun 11 19:17:23 lion ZFS[17181]: catastrophic pool I/O failure, zpool=$boiler >> Jun 11 19:17:26 lion ZFS[17394]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[17702]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[17840]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[18189]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[18382]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[18733]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[19051]: pool I/O failure, zpool=$boiler error=$28 >> Jun 11 19:17:26 lion ZFS[19346]: catastrophic pool I/O failure, zpool=$boiler >> >> After an reboot everything seems to be okay and I do daily scrubs and none >> of them as shown an error before. >> >> --Gordon > Were there any other syslog messages about /dev/da2? It looks like it stopped > working temporarily. See if there are any incrementing  errors via smartctl  on the drive(s) in that vdev     ---Mike