From nobody Fri Sep 8 04:32:09 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RhjrX5v7Tz4sNJY for ; Fri, 8 Sep 2023 04:32:28 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.gq1.yahoo.com (sonic314-21.consmr.mail.gq1.yahoo.com [98.137.69.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RhjrW5cQXz3RLK for ; Fri, 8 Sep 2023 04:32:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Rly7Vloo; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1694147545; bh=T/859hwkDHR2VGagWWbzN+fjv13WjUFSPw4ddwNJIIY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Rly7VloohIcKt8EesI4ZSWhplqv+bXmnKAoOiGVGtSqX/fETBtihJlBnEb3rL/AVC5jqRmM/78/Rz3ss6g7j6jHC+QH+ltrz8JCnoHs9hEVh/sk4j7ldmN2VzRDswtpaPBFxBmmLLiTtxvoV+juy8a7cHtRzZuQLl+2kWE3KCrUbQYjRg/lHogHuWG6YBHjRv2wdkjpxh96jb+OW6Jx6YS+fpznQl7os9G+SeElHOLhyP5s924HHxLnZMVDk/y20Ca9zRimEQbYPvpyJe66CPl9cvm6BOSseUefPkama/ziOiZwggQXcMghHLSQKYXzhhsHI/s/xK/W/kgOPq/naCQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1694147545; bh=thI0DAxiZxrCFQSMHaBOatWwW7+D5yBRTnfQPiBHHeP=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=qMm2kEZfQTLCGrYKcW5sDYg2KpU8sOJVwUCOy7HJUJM+cgSU6Qq5B68bIiQBbfNEQgwB2/a1DeVe6LpCJTh5Pz7Tq/Z6na+1I4J6yaEKwm2YBgA9VcM/yA6m5DEZK3fQZO9D1PjiFdNlaE53JktCDjqEGNmHN/sYPMtITYHIgpplvD8d6RH97tJ1a4J21rLN+RH9SXNdJSu388u3zSnQKITg18sKUr/uXPZAw9ktb0sb5BNx8dFlj9OojHTMxXfdEnHfSc0oJcHfwULcfctGACzOxzl9UpXFbhkGMDGMolq2AS1ZTD8ai8pi8OPig1gPWvPvoeUtygel2KAekhvhnA== X-YMail-OSG: PWQ3i68VM1k8XJfD7_RzRicVaJrYYmjnPK7T6jqv3xvza.vEOhtw1fDT73b8_qp rUq29CLnzOok013gqBGpMq4EyctE4LcPVPjisLvDLSAfj0Hl1.yZdS4Q16ZmO9RRd5Cr1VwZ7MLP OGaECnqaQF1vH5fvMhy5EEMg5f1FJjluDi8KHRc6se3lYMHWw.1Bih2CXRphj30bxE5aQNTLkEaE xMHprZ9rulMstDQSXMXFMHPE3scr2Wqd3_XAKEAy5AW9Z1MF8R02lqbJ.DyESyjiyyThQStYE0an J_sdnUinjPjLyupxxMkpUPqHvYex0CY6oslxl6DpADI2C1uc7GV6GlN4It260jP3E03YUeuKY8.o HRXQfiYDl5miSf9h7l6TheFtYVMvCWy1PVSlfnJZatCXFNsM3HQMuSsTv3Frh_g5XJYQvbuTxmFc Jaml7seJSw_pXxfAUoTPmRIElvtYb3wZ7lSeuBKwWW6FCoCHkqs1SlStn_pgcGF85WqLHxColy_t qVvko9H9TA1uNAEQXVAhxJI37ZyIjWW4n92vR1rIubuTKp72nKrqlkBhFiX2aYjVnElmbQRuoJzW cJbrVY15LH02.BMbOHTvo9fZhYIwIaRHdhgJb3vlXuH6TE2lSIew8nyFKRObLgy_MtTXaZSv4KP9 gpSK6fLlSHYWxqRA14Q4nSgDmJCSTwwOSqyahUUsIf78x8AE36JKWtbOKpbJD.y8RcizC7Vb9tN4 rbpNxX3a3ZKdvjv_4QOFHKa5dH3dCVwJOYBoAGpi6gcs2JP6FuC9XkXcV2u6g8E2brmWizcCAH7p Pz39rjINGfR9z6sTAHuUxu4OGaHc0HNvg66R20zlMosLirN51CMYvEUr8oe_d.rx_20DAh7_nff7 ypNCd5EaSDBpdDBDnJwPaMx19HuiPAiSN9BrZ6Js6gqXu2Ps815JZk58Y8Y_V1qyJx59uBGPM30S VwBcf7PdikQr4KYwERKDrnSs..H.NAVtD0wDu3TUscgjsZR0IudHMjFsjosws3jH8CD2QhZE_Ush bpOgkjZkp3RScKT7E3fPa8KNRWd7SpXqQEvujwYxKRBb_wtbC3jWuWADh0yYmaNBBG__PPszpJa8 XE1rRZmwknoTn8FjWuNED0JqSiGKRmbeQdTdcZd5kTdOcKadZ4Y.s4u2f_gqz2fyHknYlxAilQk5 EK76SOW7LWuE5CQ8QyPzkmVWTMJOW46av0NK0iS4mkD6O9x9WHnU8hkC9pJQXH6bob1LuXYjz5Ol g3tXc4ivCJ4E5rOIMCKyKpLJdAP77a9yqHeA51PxxFkrujPR4jqxPCak2MQongoV8FYcXiTKetwP lTbq42kJOoMQywwE.sm74g77dZ4caGwPCoGU3hZgZ_4j57RSGBfsDTsezAWUj_gLGyGSYVYYBA6y ZunszwCPZcFmMrIzDoGLeH.bUt0a4xHc33bTwT0NbXZCiK_qBJK6uj1efQxPwRRk4l4cGTjACdHK 2h2FOeIAqdcJ_P0KbA5864eYIskpMNjhM6KrRyGrkqiPUASSpM8EP0xeGADRtLQdDpUFD3Cw6O7t TNTpitNRlQ3EoKS33LFBc5AM8OJCLn5i_wxwYmdL.r5OowM05ErxjfzTOkiumjUDP6ytFRdp2Q3V 5cwDkAPRfjpwIp66R3BI_KMYVF7tr72089Fn9uAe9yPEkPcGefWFo2Ysq0XYLX0c4oClUyPLW7Uf 8stDNxq13X2uNDSaVmP2dyAu2.agXWS1ynOD8q9OXC8ODI43y1lZ3.qrkz630VTfJ1rXXCj7hThV PaMBwLHFEGMvIsKLKV3HHpdEzQ3RPVt9xEHFuuZanBaRMqCzJBWH0Ph00zksrG9Kw2cx8A7EDpfr 2Yy9O9xwJoIvYNTGNV1uta.AyPQHD.RuhFBO3Qze2vdN9SZMBlcilpJZ2FHdulAvED24qDFk7Mrx ztwijNAY8zQXTrAlzdV3tR7SUxZ5cogoLgTd7yqpqa8163M1Ft1daXkGmIIZgrfGrrFBaWBxsjaI 6WtrxqcsiowCKAvvpityMF0PwL4NTIisJqyBa_u8Yx.Lixv1YV2deB75xpeYnPoy.2C7wsk_DkLZ iWuWZajR29lvlpbuUi5drCYTAYZ9O3yko84KiTh.rmp5HvrSXoe.LVvtXvrzslUWfUBB6.gHvgO. dJ3DAHQ94qlCqgjO3tgZMw1TnEB4N3STKGVX6m.EY_WeRggZWTXJaMRobT7AFVcpGxhBcotJBb2i WYXwalPXtKgJYydEfqTHkb1CmbWpsUAIYCFokrg1VXh.POvdmMSpOIeZtqkueYhiXOD5LMG.gwqy Q6DLjB3A6Eef71rXy X-Sonic-MF: X-Sonic-ID: e4541169-7798-47f3-b7a8-60622728fefc Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Fri, 8 Sep 2023 04:32:25 +0000 Received: by hermes--production-ne1-7b767b77cc-27nt8 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 4475f12bd510bf710db871c929ca1675; Fri, 08 Sep 2023 04:32:22 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: main [and, likely, stable/14]: do not set vfs.zfs.bclone_enabled=1 with that zpool feature enabled because it still leads to panics From: Mark Millard In-Reply-To: <354C5B8C-4216-4171-B8C2-8E827817F8E5@yahoo.com> Date: Thu, 7 Sep 2023 21:32:09 -0700 Cc: Martin Matuska , Current FreeBSD , FreeBSD-STABLE Mailing List , Pawel Jakub Dawidek Content-Transfer-Encoding: quoted-printable Message-Id: <8B8B3707-4B37-4621-8124-D6A77CAF6879@yahoo.com> References: <7CE2CAAF-8BB0-4422-B194-4A6B0A4BC12C@yahoo.com> <08B7E72B-78F1-4ACA-B09D-E8C34BCE2335@yahoo.com> <20230907184823.GC4090@FreeBSD.org> <4f4e2b68-57e0-a475-e2bd-1f2b8844ebfe@FreeBSD.org> <354C5B8C-4216-4171-B8C2-8E827817F8E5@yahoo.com> To: Alexander Motin , Glen Barber X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.84:from]; BLOCKLISTDE_FAIL(0.00)[98.137.69.84:server fail]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.84:from]; TO_DN_ALL(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RhjrW5cQXz3RLK [Today's main-snapshot kernel panics as well.] On Sep 7, 2023, at 16:32, Mark Millard wrote: > On Sep 7, 2023, at 13:07, Alexander Motin wrote: >=20 >> Thanks, Mark. >>=20 >> On 07.09.2023 15:40, Mark Millard wrote: >>> On Sep 7, 2023, at 11:48, Glen Barber wrote: >>>> On Thu, Sep 07, 2023 at 11:17:22AM -0700, Mark Millard wrote: >>>>> When I next have time, should I retry based on a more recent >>>>> vintage of main that includes 969071be938c ? >>>>=20 >>>> Yes, please, if you can. >>> As stands, I rebooted that machine into my normal >>> enviroment, so the after-crash-with-dump-info >>> context is preserved. I'll presume lack of a need >>> to preserve that context unless I hear otherwise. >>> (But I'll work on this until later today.) >>> Even my normal environment predates the commit in >>> question by a few commits. So I'll end up doing a >>> more general round of updates overall. >>> Someone can let me know if there is a preference >>> for debug over non-debug for the next test run. >>=20 >> It is not unknown when some bugs disappear once debugging is enabled = due to different execution timings, but generally debug may to detect = the problem closer to its origin instead of looking on random = consequences. I am only starting to look on this report (unless Pawel or = somebody beat me on it), and don't have additional requests yet, but if = you can repeat the same with debug kernel (in-base ZFS's ZFS_DEBUG = setting follows kernel's INVARIANTS), it may give us some additional = information. >=20 > So I did a zpool import, rewinding to the checkpoint. > (This depends on the questionable zfs doing fully as > desired for this. Notably the normal environment has > vfs.zfs.bclone_enabled=3D0 , including when it was > doing this activity.) My normal environment reported > no problems. >=20 > Note: the earlier snapshot from my first setup was > still in place since it was made just before the > original checkpoint used above. >=20 > However, the rewind did remove the /var/crash/ > material that had been added. >=20 > I did the appropriate zfs mount. >=20 > I installed a debug kernel and world to the import. Again, > no problems reported. >=20 > I did the appropriate zfs umount. >=20 > I did the appropriate zpool export. >=20 > I rebooted with the test media. >=20 > # sysctl vfs.zfs.bclone_enabled > vfs.zfs.bclone_enabled: 1 >=20 > # zpool trim -w zamd64 >=20 > # zpool checkpoint zamd64 >=20 > # uname -apKU > FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #74 = main-n265188-117c54a78ccd-dirty: Tue Sep 5 21:29:53 PDT 2023 = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-dbg-clang/usr/main-src/amd64.amd= 64/sys/GENERIC-DBG amd64 amd64 1500000 1500000 >=20 > (So, before the 969071be938c vintage, same sources as for > my last run but a debug build.) >=20 > # poudriere bulk -jmain-amd64-bulk_a -a > . . . > [00:03:23] Building 34214 packages using up to 32 builders > [00:03:23] Hit CTRL+t at any time to see build progress and stats > [00:03:23] [01] [00:00:00] Builder starting > [00:04:19] [01] [00:00:56] Builder started > [00:04:20] [01] [00:00:01] Building ports-mgmt/pkg | pkg-1.20.6 > [00:05:33] [01] [00:01:14] Finished ports-mgmt/pkg | pkg-1.20.6: = Success > [00:05:53] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 > [00:05:53] [02] [00:00:00] Builder starting > . . . > [00:05:54] [32] [00:00:00] Builder starting > [00:06:11] [01] [00:00:18] Finished print/indexinfo | indexinfo-0.3.1: = Success > [00:06:12] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22_1 > [00:08:24] [01] [00:02:12] Finished devel/gettext-runtime | = gettext-runtime-0.22_1: Success > [00:08:31] [01] [00:00:00] Building devel/libtextstyle | = libtextstyle-0.22 > [00:10:06] [05] [00:04:13] Builder started > [00:10:06] [05] [00:00:00] Building devel/autoconf-switch | = autoconf-switch-20220527 > [00:10:06] [31] [00:04:12] Builder started > [00:10:06] [31] [00:00:00] Building devel/libatomic_ops | = libatomic_ops-7.8.0 > . . . >=20 > Crashed again, with 158 *.pkg files in .building/All/ after > rebooting. >=20 > The crash is similar to the non-debug one. No extra output > from the debug build. >=20 > For reference: >=20 > Unread portion of the kernel message buffer: > panic: Solaris(panic): zfs: accessing past end of object 422/10b1c02 = (size=3D2560 access=3D2560+2560) > . . . Same world with newer snapshot main kernel that should be compatible with the world: # uname -apKU FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #0 = main-n265205-03a7c36ddbc0: Thu Sep 7 03:10:34 UTC 2023 = root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC = amd64 amd64 1500000 1500000 Steps: #NOTE: (re)boot to normal environment #NOTE: login cd ~/artifacts/ #NOTE: as needed . . . fetch = http://ftp3.freebsd.org/pub/FreeBSD/snapshots/amd64/15.0-CURRENT/kernel.tx= z fetch = http://ftp3.freebsd.org/pub/FreeBSD/snapshots/amd64/15.0-CURRENT/kernel-db= g.txz fetch = http://ftp3.freebsd.org/pub/FreeBSD/snapshots/amd64/15.0-CURRENT/src.txz #NOTE: /zamd64-mnt is a pre-existing empty directory for such use: zpool import --rewind-to-checkpoint -f -N -R /zamd64-mnt -t zamd64 = zpamd64 zpool checkpoint zpamd64 #NOTE: My BE's for bectl use have mountpoint=3Dnone normally. zfs set mountpoint=3D/ zpamd64/ROOT/main-amd64 zfs mount zpamd64/ROOT/main-amd64 mv /zamd64-mnt/boot/kernel /zamd64-mnt/boot/kernel.mm tar -xpf kernel.txz -C /zamd64-mnt/ mv /zamd64-mnt/usr/lib/debug/boot/kernel = /zamd64-mnt/usr/lib/debug/boot/kernel.mm tar -xpf kernel-dbg.txz -C /zamd64-mnt/ zfs umount zpamd64/ROOT/main-amd64 zfs set mountpoint=3Dnone zpamd64/ROOT/main-amd64 zfs mount zpamd64/usr/src #NOTE: /zamd64-mnt/usr/src/ being empty initially #NOTE: My normal environment's source is in a worktree /usr/main-src/ tar -xpf src.txz -C /zamd64-mnt/ zfs umount zpamd64/usr/src zpool export zpamd64 #NOTE: reboot to zamd64 #(my test environment) #NOTE: login uname -apKU sysctl vfs.zfs.bclone_enabled zpool trim -w zamd64 poudriere bulk -jmain-amd64-bulk_a -a Paniced. #NOTE: dump #NOTE: reboot to zamd64 #(my test environment) #NOTE: login ls -Tlod = /usr/local/poudriere/data/packages/main-amd64-bulk_a-default/.building/All= /*.pkg | wc -l 122 less /var/crash/core.txt.3 . . . Unread portion of the kernel message buffer: panic: Solaris(panic): zfs: accessing past end of object 422/10d670b = (size=3D2560 access=3D2560+2560) cpuid =3D 18 time =3D 1694146387 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe033f721590 vpanic() at vpanic+0x132/frame 0xfffffe033f7216c0 panic() at panic+0x43/frame 0xfffffe033f721720 vcmn_err() at vcmn_err+0xeb/frame 0xfffffe033f721850 zfs_panic_recover() at zfs_panic_recover+0x59/frame 0xfffffe033f7218b0 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0xb8/frame = 0xfffffe033f721960 dmu_brt_clone() at dmu_brt_clone+0x61/frame 0xfffffe033f7219e0 zfs_clone_range() at zfs_clone_range+0xaa3/frame 0xfffffe033f721bb0 zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0x18a/frame = 0xfffffe033f721c30 vn_copy_file_range() at vn_copy_file_range+0x163/frame = 0xfffffe033f721ce0 kern_copy_file_range() at kern_copy_file_range+0x36c/frame = 0xfffffe033f721db0 sys_copy_file_range() at sys_copy_file_range+0x78/frame = 0xfffffe033f721e00 amd64_syscall() at amd64_syscall+0x138/frame 0xfffffe033f721f30 fast_syscall_common() at fast_syscall_common+0xf8/frame = 0xfffffe033f721f30 --- syscall (569, FreeBSD ELF64, copy_file_range), rip =3D = 0x2c05a58a355a, rsp =3D 0x2c05a4312138, rbp =3D 0x2c05a43125d0 --- KDB: enter: panic __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 57 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" = (offsetof(struct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=3Dtextdump@entry=3D0) at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xffffffff804a401a in db_dump (dummy=3D, = dummy2=3D, dummy3=3D, dummy4=3D) at /usr/src/sys/ddb/db_command.c:591 #3 0xffffffff804a3e1d in db_command (last_cmdp=3D, = cmd_table=3D, dopager=3Dtrue) at /usr/src/sys/ddb/db_command.c:504 #4 0xffffffff804a3add in db_command_loop () at /usr/src/sys/ddb/db_command.c:551 #5 0xffffffff804a71b6 in db_trap (type=3D, = code=3D) at /usr/src/sys/ddb/db_main.c:268 #6 0xffffffff80b9dfb3 in kdb_trap (type=3Dtype@entry=3D3, = code=3Dcode@entry=3D0, tf=3Dtf@entry=3D0xfffffe033f7214d0) at = /usr/src/sys/kern/subr_kdb.c:790 #7 0xffffffff8104d579 in trap (frame=3D0xfffffe033f7214d0) at /usr/src/sys/amd64/amd64/trap.c:608 #8 #9 kdb_enter (why=3D, msg=3D) at /usr/src/sys/kern/subr_kdb.c:556 #10 0xffffffff80b4f683 in vpanic (fmt=3D0xffffffff8223d54e "%s%s", = ap=3Dap@entry=3D0xfffffe033f721700) at = /usr/src/sys/kern/kern_shutdown.c:958 #11 0xffffffff80b4f463 in panic ( fmt=3D0xffffffff8196c800 = "$\222\024\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:894 #12 0xffffffff81f9964b in vcmn_err (ce=3D, = fmt=3D0xffffffff82274462 "zfs: accessing past end of object %llx/%llx = (size=3D%u access=3D%llu+%llu)", adx=3D0xfffffe033f721890) at = /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_cmn_err.c:60 #13 0xffffffff820b6a59 in zfs_panic_recover ( fmt=3D0x12 ) at /usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c:1594 #14 0xffffffff820150d8 in dmu_buf_hold_array_by_dnode = (dn=3D0xfffff802c3c41be8, offset=3Doffset@entry=3D2560, = length=3Dlength@entry=3D2560, read=3Dread@entry=3D0, = tag=3D0xffffffff8222139b, numbufsp=3Dnumbufsp@entry=3D0xfffffe033f7219ac, = dbpp=3D0xfffffe033f7219b0, flags=3D0) at /usr/src/sys/contrib/openzfs/module/zfs/dmu.c:543 #15 0xffffffff82018c31 in dmu_buf_hold_array (os=3D, = object=3D, read=3D0, numbufsp=3D0xfffffe033f7219ac, = dbpp=3D0xfffffe033f7219b0, offset=3D, length=3D, tag=3D) at = /usr/src/sys/contrib/openzfs/module/zfs/dmu.c:654 #16 dmu_brt_clone (os=3Dos@entry=3D0xfffff8019696b800, object=3D, offset=3Doffset@entry=3D2560, length=3Dlength@entry=3D2560, = tx=3Dtx@entry=3D0xfffff81433e03d00, = bps=3Dbps@entry=3D0xfffffe0442ce5000, nbps=3D1, replay=3D0) at = /usr/src/sys/contrib/openzfs/module/zfs/dmu.c:2301 #17 0xffffffff82183333 in zfs_clone_range (inzp=3D0xfffff80e37f64000, = inoffp=3D0xfffff8019ab32b38, outzp=3D0xfffff80d931a5910, = outoffp=3D0xfffff80662d0a548, lenp=3Dlenp@entry=3D0xfffffe033f721bf0, = cr=3D0xfffff8051d1c4800) at /usr/src/sys/contrib/openzfs/module/zfs/zfs_vnops.c:1302 #18 0xffffffff81fbc76a in zfs_freebsd_copy_file_range = (ap=3D0xfffffe033f721c48) at = /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:6294 #19 0xffffffff80c612a3 in VOP_COPY_FILE_RANGE (invp=3D0xfffff80eb09c08c0, = inoffp=3D0xfffff8019ab32b38, outvp=3D0xfffff80e7f998000, = outoffp=3D0xfffff80662d0a548, lenp=3D0xfffffe033f721d40, = incred=3D0xfffff8051d1c4800, flags=3D, = outcred=3D, fsizetd=3D) at = ./vnode_if.h:2385 #20 vn_copy_file_range (invp=3Dinvp@entry=3D0xfffff80eb09c08c0, = inoffp=3Dinoffp@entry=3D0xfffff8019ab32b38, = outvp=3Doutvp@entry=3D0xfffff80e7f998000, outoffp=3D0xfffff80662d0a548, = lenp=3Dlenp@entry=3D0xfffffe033f721d40, flags=3Dflags@entry=3D0, = incred=3D0xfffff8051d1c4800, outcred=3D0xfffff8051d1c4800, = fsize_td=3D0xfffffe033fdf1c80) at /usr/src/sys/kern/vfs_vnops.c:3087 #21 0xffffffff80c5be9c in kern_copy_file_range ( td=3Dtd@entry=3D0xfffffe033fdf1c80, infd=3D, = inoffp=3D0xfffff8019ab32b38, inoffp@entry=3D0x0, outfd=3D, outoffp=3D0xffffffff8120f47f, outoffp@entry=3D0x0, = len=3D9223372036854775807, flags=3D0) at = /usr/src/sys/kern/vfs_syscalls.c:4971 #22 0xffffffff80c5bfa8 in sys_copy_file_range (td=3D0xfffffe033fdf1c80, = uap=3D0xfffffe033fdf2080) at /usr/src/sys/kern/vfs_syscalls.c:5009 #23 0xffffffff8104e3d8 in syscallenter (td=3D) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:187 #24 amd64_syscall (td=3D0xfffffe033fdf1c80, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1197 #25 #26 0x00002c05a58a355a in ?? () Backtrace stopped: Cannot access memory at address 0x2c05a4312138 (kgdb)=20 . . . So the problem is not special to my personal kernel builds. >>> Looking at "git: 969071be938c - main", the relevant >>> part seems to be just (white space possibly not >>> preserved accurately): >>> diff --git a/sys/kern/vfs_vnops.c b/sys/kern/vfs_vnops.c >>> index 9fb5aee6a023..4e4161ef1a7f 100644 >>> --- a/sys/kern/vfs_vnops.c >>> +++ b/sys/kern/vfs_vnops.c >>> @@ -3076,12 +3076,14 @@ vn_copy_file_range(struct vnode *invp, off_t = *inoffp, struct vnode *outvp, >>> goto out; >>> /* >>> - * If the two vnode are for the same file system, call >>> + * If the two vnodes are for the same file system type, call >>> * VOP_COPY_FILE_RANGE(), otherwise call = vn_generic_copy_file_range() >>> - * which can handle copies across multiple file systems. >>> + * which can handle copies across multiple file system types. >>> */ >>> *lenp =3D len; >>> - if (invp->v_mount =3D=3D outvp->v_mount) >>> + if (invp->v_mount =3D=3D outvp->v_mount || >>> + strcmp(invp->v_mount->mnt_vfc->vfc_name, >>> + outvp->v_mount->mnt_vfc->vfc_name) =3D=3D 0) >>> error =3D VOP_COPY_FILE_RANGE(invp, inoffp, outvp, outoffp, >>> lenp, flags, incred, outcred, fsize_td); >>> else >>> That looks to call VOP_COPY_FILE_RANGE in more contexts and >>> vn_generic_copy_file_range in fewer. >>> The backtrace I reported involves: VOP_COPY_FILE_RANGE >>> So it appears this change is unlikely to invalidate my >>> test result, although failure might happen sooner if >>> more VOP_COPY_FILE_RANGE calls happen with the newer code. >>=20 >> Your logic is likely right, but if you have block cloning requests = both within and between datasets, this patch may change the pattern. = Though it is obviously not a fix for the issue. I responded to the = commit email only because it makes no difference while = vfs.zfs.bclone_enabled is 0. >>=20 >>> That in turns means that someone may come up with some >>> other change for me to test by the time I get around to >>> setting up another test. Let me know if so. >> . . . =3D=3D=3D Mark Millard marklmi at yahoo.com