Date: Thu, 10 Aug 2023 20:32:51 -0700 From: Kevin Bowling <kevin.bowling@kev009.com> To: Cy Schubert <Cy.Schubert@cschubert.com> Cc: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= <des@freebsd.org>, current@freebsd.org Subject: Re: ZFS deadlock in 14 Message-ID: <CAK7dMtDZP9gZcxvo2QPxHSVXOoLyaDsUTmTA-yeTTQuNfmm-kA@mail.gmail.com> In-Reply-To: <20230810233312.0E10AF4@slippy.cwsent.com> References: <86leeltqcb.fsf@ltc.des.no> <CAK7dMtDJeuf8rjWbsNEZABUfeqpjUyCHzuOL9AAhKk93sy%2BPKg@mail.gmail.com> <20230810133745.D0EC0178@slippy.cwsent.com> <CAK7dMtDJQtaai3_6VjEkwVwW5JN6e8v=kKTOPffp371xb=ORUg@mail.gmail.com> <20230810233312.0E10AF4@slippy.cwsent.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000b7db4b06029d5ea5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Spoke too soon still seeing zfs lockups under heavy poudriere workload after the MFVs. Regression time matches what has been reported here. On Thu, Aug 10, 2023 at 4:33 PM Cy Schubert <Cy.Schubert@cschubert.com> wrote: > I haven't experienced any problems (yet) either. > > > -- > Cheers, > Cy Schubert <Cy.Schubert@cschubert.com> > FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org > NTP: <cy@nwtime.org> Web: https://nwtime.org > > e^(i*pi)+1=3D0 > > > In message > <CAK7dMtDJQtaai3_6VjEkwVwW5JN6e8v=3DkKTOPffp371xb=3DORUg@mail.gmail.c > om> > , Kevin Bowling writes: > > The two MFVs on head have improved/fixed stability with poudriere for > > me 48 core bare metal. > > > > On Thu, Aug 10, 2023 at 6:37=3DE2=3D80=3DAFAM Cy Schubert > <Cy.Schubert@cschubert.=3D > > com> wrote: > > > > > > In message > <CAK7dMtDJeuf8rjWbsNEZABUfeqpjUyCHzuOL9AAhKk93sy+PKg@mail.gmai=3D > > l.c > > > om> > > > , Kevin Bowling writes: > > > > Possibly > https://github.com/openzfs/zfs/commit/2cb992a99ccadb78d97049b4=3D > > 0bd4=3D3D > > > > 42eb4fdc549d > > > > > > > > On Tue, Aug 8, 2023 at 10:08=3D3DE2=3D3D80=3D3DAFAM Dag-Erling > Sm=3D3DC3=3D3DB8rg=3D > > rav <des@freeb=3D3D > > > > sd.org> wrote: > > > > > > > > > > At some point between 42d088299c (4 May) and f0c9703301 (26 June)= , > a > > > > > deadlock was introduced in ZFS. It is still present as of > 9c2823bae9=3D > > (4 > > > > > August) and is 100% reproducable just by starting poudriere bulk > in a > > > > > 16-core VM and waiting a few hours until deadlkres kicks in. In > the > > > > > latest instance, deadlkres complained about a bash process: > > > > > > > > > > #0 sched_switch (td=3D3D3Dtd@entry=3D3D3D0xfffffe02fb1d8000, > flags=3D > > =3D3D3Dflags@e=3D3D > > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3D3Dflags@entry > =3D3D3D25=3D > > 9) at /u=3D3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D3D0xfffff818543a=3D > > 9e70, =3D3D > > > > pri=3D3D3D64) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D3D<unavailable>, p=3D > > ri=3D3D3D<=3D3D > > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80b1c1b0 in sleeplk (lk=3D3D3Dlk@entry > =3D3D3D0xfffff818=3D > > 543a9e70=3D3D > > > > , flags=3D3D3Dflags@entry=3D3D3D2121728, ilk=3D3D3Dilk@entry=3D3D3D= 0x0, > wmesg=3D > > =3D3D3Dwmesg@entry=3D3D > > > > =3D3D3D0xffffffff8222a054 "zfs", pri=3D3D3D<optimized out>, pri@ent= ry > =3D3D3D6=3D > > 4, timo=3D3D3D=3D3D > > > > timo@entry=3D3D3D6, queue=3D3D3D1) at /usr/src/sys/kern/kern_lock.c= :310 > > > > > #5 0xffffffff80b1a23f in lockmgr_slock_hard > (lk=3D3D3D0xfffff81854=3D > > 3a9e70=3D3D > > > > , flags=3D3D3D2121728, ilk=3D3D3D<optimized out>, > file=3D3D3D0xffffffff812544=3D > > fb "/usr/s=3D3D > > > > rc/sys/kern/vfs_subr.c", line=3D3D3D3057, lwa=3D3D3D0x0) at > /usr/src/sys/ke=3D > > rn/kern_=3D3D > > > > lock.c:705 > > > > > #6 0xffffffff80c59ec3 in VOP_LOCK1 > (vp=3D3D3D0xfffff818543a9e00, f=3D > > lags=3D3D > > > > =3D3D3D2105344, file=3D3D3D0xffffffff812544fb > "/usr/src/sys/kern/vfs_subr.c=3D > > ", line=3D3D > > > > =3D3D3D3057) at ./vnode_if.h:1120 > > > > > #7 _vn_lock (vp=3D3D3Dvp@entry=3D3D3D0xfffff818543a9e00, > flags=3D3D3D2=3D > > 105344, fi=3D3D > > > > le=3D3D3D<unavailable>, line=3D3D3D<unavailable>, line@entry=3D3D3D= 3057) > at /=3D > > usr/src/sy=3D3D > > > > s/kern/vfs_vnops.c:1815 > > > > > #8 0xffffffff80c4173d in vget_finish > (vp=3D3D3D0xfffff818543a9e00,=3D > > flags=3D3D > > > > =3D3D3D<unavailable>, vs=3D3D3Dvs@entry=3D3D3DVGET_USECOUNT) at > /usr/src/sys/=3D > > kern/vfs_s=3D3D > > > > ubr.c:3057 > > > > > #9 0xffffffff80c1c9b7 in cache_lookup (dvp=3D3D3Ddvp@entry > =3D3D3D0xf=3D > > ffff802c=3D3D > > > > d02ac40, vpp=3D3D3Dvpp@entry=3D3D3D0xfffffe046b20ac30, cnp=3D3D3Dcn= p@entry > =3D3D=3D > > 3D0xfffffe04=3D3D > > > > 6b20ac58, tsp=3D3D3Dtsp@entry=3D3D3D0x0, ticksp=3D3D3Dticksp@entry= =3D3D3D0x0) > a=3D > > t /usr/src/s=3D3D > > > > ys/kern/vfs_cache.c:2086 > > > > > #10 0xffffffff80c2150c in vfs_cache_lookup (ap=3D3D3D<optimiz= ed > out=3D > > >) at =3D3D > > > > /usr/src/sys/kern/vfs_cache.c:3068 > > > > > #11 0xffffffff80c32c37 in VOP_LOOKUP > (dvp=3D3D3D0xfffff802cd02ac40,=3D > > vpp=3D3D > > > > =3D3D3D0xfffffe046b20ac30, cnp=3D3D3D0xfffffe046b20ac58) at > ./vnode_if.h:69 > > > > > #12 vfs_lookup (ndp=3D3D3Dndp@entry=3D3D3D0xfffffe046b20abd8)= at > /usr=3D > > /src/sys=3D3D > > > > /kern/vfs_lookup.c:1266 > > > > > #13 0xffffffff80c31ce1 in namei (ndp=3D3D3Dndp@entry > =3D3D3D0xfffffe04=3D > > 6b20abd8=3D3D > > > > ) at /usr/src/sys/kern/vfs_lookup.c:689 > > > > > #14 0xffffffff80c52090 in kern_statat > (td=3D3D3D0xfffffe02fb1d8000,=3D > > flag=3D3D > > > > =3D3D3D<optimized out>, fd=3D3D3D-100, path=3D3D3D0xa75b480e070 <er= ror: > Canno=3D > > t access m=3D3D > > > > emory at address 0xa75b480e070>, pathseg=3D3D3Dpathseg@entry > =3D3D3DUIO_USER=3D > > SPACE, s=3D3D > > > > bp=3D3D3Dsbp@entry=3D3D3D0xfffffe046b20ad18) > > > > > at /usr/src/sys/kern/vfs_syscalls.c:2441 > > > > > #15 0xffffffff80c52797 in sys_fstatat (td=3D3D3D<unavailable>= , > uap=3D > > =3D3D3D0xff=3D3D > > > > fffe02fb1d8400) at /usr/src/sys/kern/vfs_syscalls.c:2419 > > > > > #16 0xffffffff81049398 in syscallenter (td=3D3D3D<optimized > out>) a=3D > > t /usr=3D3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #17 amd64_syscall (td=3D3D3D0xfffffe02fb1d8000, traced=3D3D3D= 0) at > /u=3D > > sr/src/s=3D3D > > > > ys/amd64/amd64/trap.c:1199 > > > > > #18 <signal handler called> > > > > > > > > > > The lock it is trying to acquire in frame 5 belongs to another ba= sh > > > > > process which is in the process of creating a fifo: > > > > > > > > > > #0 sched_switch (td=3D3D3Dtd@entry=3D3D3D0xfffffe046acd8e40, > flags=3D > > =3D3D3Dflags@e=3D3D > > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3D3Dflags@entry > =3D3D3D25=3D > > 9) at /u=3D3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D3D0xfffff8018acb=3D > > f154, =3D3D > > > > pri=3D3D3D87) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D3D<unavailable>, p=3D > > ri=3D3D3D<=3D3D > > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80b59606 in _sleep (ident=3D3D3Dident@entry > =3D3D3D0xfff=3D > > ff8018ac=3D3D > > > > bf154, lock=3D3D3Dlock@entry=3D3D3D0xfffff8018acbf120, > priority=3D3D3Dpriorit=3D > > y@entry=3D3D3D=3D3D > > > > 87, wmesg=3D3D3D0xffffffff8223af0e "zfs teardown inactive", > sbt=3D3D3Dsbt@e=3D > > ntry=3D3D3D0=3D3D > > > > , pr=3D3D3Dpr@entry=3D3D3D0, flags=3D3D3D256) > > > > > at /usr/src/sys/kern/kern_synch.c:225 > > > > > #5 0xffffffff80b45dc0 in rms_rlock_fallback > (rms=3D3D3D0xfffff8018=3D > > acbf12=3D3D > > > > 0) at /usr/src/sys/kern/kern_rmlock.c:1015 > > > > > #6 0xffffffff80b45c93 in rms_rlock (rms=3D3D3D<unavailable>, > rms@e=3D > > ntry=3D3D > > > > =3D3D3D0xfffff8018acbf120) at /usr/src/sys/kern/kern_rmlock.c:1036 > > > > > #7 0xffffffff81fb147b in zfs_freebsd_reclaim > (ap=3D3D3D<optimized =3D > > out>) =3D3D > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:51=3D > > 64 > > > > > #8 0xffffffff8111d245 in VOP_RECLAIM_APV > (vop=3D3D3D0xffffffff822e=3D > > 71a0 <=3D3D > > > > zfs_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1c9c8) at > vnode_if.c:2180 > > > > > #9 0xffffffff80c43569 in VOP_RECLAIM > (vp=3D3D3D0xfffff802cdbaca80)=3D > > at ./=3D3D > > > > vnode_if.h:1084 > > > > > #10 vgonel (vp=3D3D3Dvp@entry=3D3D3D0xfffff802cdbaca80) at > /usr/src/s=3D > > ys/kern/=3D3D > > > > vfs_subr.c:4143 > > > > > #11 0xffffffff80c3ef61 in vtryrecycle > (vp=3D3D3D0xfffff802cdbaca80)=3D > > at /u=3D3D > > > > sr/src/sys/kern/vfs_subr.c:1693 > > > > > #12 vnlru_free_impl (count=3D3D3Dcount@entry=3D3D3D1, > mnt_op=3D3D3Dmnt_=3D > > op@entry=3D3D > > > > =3D3D3D0x0, mvp=3D3D3D0xfffff8010864da00) at > /usr/src/sys/kern/vfs_subr.c:1=3D > > 344 > > > > > #13 0xfffff =C3=A3=C2=B5=C2=B6=E2=80=93, fff80c49553 in vn= lru_free_locked > (count=3D3D3D1) at /usr=3D > > /src/s=3D3D > > > > ys/kern/vfs_subr.c:1357 > > > > > #14 vn_alloc_hard (mp=3D3D3Dmp@entry=3D3D3D0x0) at > /usr/src/sys/kern/=3D > > vfs_subr=3D3D > > > > .c:1744 > > > > > #15 0xffffffff80c3f6f0 in vn_alloc (mp=3D3D3D0x0) at > /usr/src/sys/a=3D > > md64/i=3D3D > > > > nclude/atomic.h:375 > > > > > #16 getnewvnode_reserve () at /usr/src/sys/kern/vfs_subr.c:18= 88 > > > > > #17 0xffffffff81faa072 in zfs_create > (dzp=3D3D3D0xfffff812200261d0,=3D > > name=3D3D > > > > =3D3D3D0xfffff8011b8ac805 "sh-np.yPbxoo", vap=3D3D3D0xfffffe0410f1c= c20, > exc=3D > > l=3D3D3D<opt=3D3D > > > > imized out>, mode=3D3D3D<optimized out>, zpp=3D3D3Dzpp@entry > =3D3D3D0xfffffe04=3D > > 10f1cbc8, =3D3D > > > > cr=3D3D3D0xfffff80140fb1100, flag=3D3D3D<optimized out>, vsecp=3D3D= 3D0x0, > mnt=3D > > _ns=3D3D3D0x0) > > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vno=3D > > ps_o=3D3D > > > > s.c:1146 > > > > > #18 0xffffffff81faea57 in zfs_freebsd_create > (ap=3D3D3D0xfffffe0410=3D > > f1cda0=3D3D > > > > ) at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:=3D > > 4618 > > > > > #19 0xffffffff8111aa9a in VOP_MKNOD_APV > (vop=3D3D3D0xffffffff822e71=3D > > a0 <zf=3D3D > > > > s_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1cda0) at vnode_if.= c:372 > > > > > #20 0xffffffff80c50207 in VOP_MKNOD (dvp=3D3D3D<unavailable>, > cnp=3D > > =3D3D3D0xfff=3D3D > > > > ffe0410f1cd50, vap=3D3D3D0xfffffe0410f1cc20, vpp=3D3D3D<optimized o= ut>) > at =3D > > ./vnode_=3D3D > > > > if.h:188 > > > > > #21 kern_mkfifoat (td=3D3D3D0xfffffe046acd8e40, fd=3D3D3D-100= , > path=3D > > =3D3D3D0x12772=3D3D > > > > f073500 <error: Cannot access memory at address 0x12772f073500>, > pathse=3D > > g=3D3D3D=3D3D > > > > UIO_USERSPACE, mode=3D3D3D<optimized out>) at > /usr/src/sys/kern/vfs_sysca=3D > > lls.c:=3D3D > > > > 1492 > > > > > #22 0xffffffff81049398 in syscallenter (td=3D3D3D<optimized > out>) a=3D > > t /usr=3D3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #23 amd64_ =3DE6=3D90=3DAC=3DEE=3D8A=3D80 syscall > (td=3D3D3D0xfffffe046acd8e=3D > > 40, traced=3D3D3D0) at /usr/src/s=3D3D > > > > ys/amd64/amd64/trap.c:1199 > > > > > #24 <signal handler called> > > > > > > > > > > Frame 7 is trying to acquire the ZFS teardown inactive lock, whic= h > is > > > > > held by a process which is performing a ZFS rollback and is > waiting f=3D > > or > > > > > the transaction to sync: > > > > > > > > > > #0 sched_switch (td=3D3D3Dtd@entry=3D3D3D0xfffffe0422ef8560, > flags=3D > > =3D3D3Dflags@e=3D3D > > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3D3Dflags@entry > =3D3D3D25=3D > > 9) at /u=3D3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D3D0xfffff8011b83=3D > > d540, =3D3D > > > > pri=3D3D3D0) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D3D<unavailable>, w=3D > > chan@e=3D3D > > > > ntry=3D3D3D0xfffff8011b83d540, pri=3D3D3D<unavailable>, pri@entry= =3D3D3D0) > at=3D > > /usr/src/=3D3D > > > > sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80ad7f75 in _cv_wait (cvp=3D3D3Dcvp@entry > =3D3D3D0xfffff=3D > > 8011b83d=3D3D > > > > 540, lock=3D3D3Dlock@entry=3D3D3D0xfffff8011b83d4d0) at > /usr/src/sys/kern/k=3D > > ern_cond=3D3D > > > > var.c:146 > > > > > #5 0xffffffff820b42fb in txg_wait_synced_impl (dp=3D3D3Ddp@e= ntry > =3D > > =3D3D3D0xfff=3D3D > > > > ff8011b83d000, txg=3D3D3D8585097, wait_sig=3D3D3Dwait_sig@entry=3D3= D3D0) > at /=3D > > usr/src/sy=3D3D > > > > s/contrib/openzfs/module/zfs/txg.c:726 > > > > > #6 0xffffffff820b3cab in txg_wait_synced > (dp=3D3D3D<unavailable>, =3D > > dp@ent=3D3D > > > > ry=3D3D3D0xfffff8011b83d000, txg=3D3D3D<unavailable>) at > /usr/src/sys/contr=3D > > ib/openz=3D3D > > > > fs/module/zfs/txg.c:736 > > > > > #7 0xffffffff8206d5b5 in dsl_sync_task_common > (pool=3D3D3Dpool@ent=3D > > ry=3D3D3D0=3D3D > > > > xfffffe0401d15000 "zroot/poudriere/jails/13amd64-default-ref/15", > check=3D > > func=3D3D > > > > =3D3D3D<optimized out>, syncfunc=3D3D3D0xffffffff8203fbc0 > <dsl_dataset_roll=3D > > back_syn=3D3D > > > > c>, sigfunc=3D3D3Dsigfunc@entry=3D3D3D0x0, arg=3D3D3Darg@entry > =3D3D3D0xfffffe02=3D > > fb827a90, > > > > > blocks_modified=3D3D3Dblocks_modified@entry=3D3D3D1, > space_check=3D > > =3D3D3DZFS_SP=3D3D > > > > ACE_CHECK_RESERVED, early=3D3D3D0) at > /usr/src/sys/contrib/openzfs/module=3D > > /zfs/d=3D3D > > > > sl_synctask.c:93 > > > > > #8 0xffffffff8206d3c7 in dsl_sync_task > (pool=3D3D3D<unavailable>, =3D > > pool@e=3D3D > > > > ntry=3D3D3D0xfffffe0401d15000 > "zroot/poudriere/jails/13amd64-default-ref/=3D > > 15", c=3D3D > > > > heckfunc=3D3D3D<unavailable>, syncfunc=3D3D3D<unavailable>, > arg=3D3D3D<unavai=3D > > lable>, ar=3D3D > > > > g@entry=3D3D3D0xfffffe02fb827a90, blocks_modified=3D3D3D<unavailabl= e>, > > > > > blocks_modified@entry=3D3D3D1, > space_check=3D3D3D<unavailable>, s=3D > > pace_che=3D3D > > > > ck@entry=3D3D3DZFS_SPACE_CHECK_RESERVED) at > /usr/src/sys/contrib/openzfs/=3D > > module=3D3D > > > > /zfs/dsl_synctask.c:132 > > > > > #9 0xffffffff8204075b in dsl_dataset_rollback > (fsname=3D3D3D<unava=3D > > ilable=3D3D > > > > >, fsname@entry=3D3D3D0xfffffe0401d15000 > "zroot/poudriere/jails/13amd64-d=3D > > efault=3D3D > > > > -ref/15", tosnap=3D3D3D<optimized out>, owner=3D3D3D<optimized out>= , > result=3D > > =3D3D3Dresul=3D3D > > > > t@entry=3D3D3D0xfffff81c826a9ea0) > > > > > at > /usr/src/sys/contrib/openzfs/module/zfs/dsl_dataset.c:3261 > > > > > #10 0xffffffff82168dd9 in zfs_ioc_rollback > (fsname=3D3D3D0xfffffe04=3D > > 01d150=3D3D > > > > 00 "zroot/poudriere/jails/13amd64-default-ref/15", fsname@entry > =3D3D3D<er=3D > > ror re=3D3D > > > > ading variable: value is not available>, innvl=3D3D3D<unavailable>, > innvl=3D > > @entry=3D3D > > > > =3D3D3D<error reading variable: value is not available>, > > > > > outnvl=3D3D3D0xfffff81c826a9ea0, outnvl@entry=3D3D3D<erro= r > readin=3D > > g variab=3D3D > > > > le: value is not available>) at > /usr/src/sys/contrib/openzfs/module/zfs=3D > > /zfs=3D3D > > > > _ioctl.c:4405 > > > > > #11 0xffffffff82164522 in zfsdev_ioctl_common > (vecnum=3D3D3Dvecnum@=3D > > entry=3D3D > > > > =3D3D3D25, zc=3D3D3Dzc@entry=3D3D3D0xfffffe0401d15000, flag=3D3D3Df= lag@entry > =3D3D=3D > > 3D0) at /usr/s=3D3D > > > > rc/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:7798 > > > > > #12 0xffffffff81f97fca in zfsdev_ioctl (dev=3D3D3D<optimized > out>, =3D > > zcmd=3D3D > > > > =3D3D3D<unavailable>, zcmd@entry=3D3D3D<error reading variable: val= ue > is no=3D > > t availa=3D3D > > > > ble>, arg=3D3D3D0xfffffe02fb827d50 "\017", arg@entry=3D3D3D<error > reading v=3D > > ariable:=3D3D > > > > value is not available>, flag=3D3D3D<optimized out>, > td=3D3D3D<optimized o=3D > > ut>) > > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/kmod_co=3D > > re.c=3D3D > > > > :168 > > > > > #13 0xffffffff809d6212 in devfs_ioctl > (ap=3D3D3D0xfffffe02fb827c50)=3D > > at /u=3D3D > > > > sr/src/sys/fs/devfs/devfs_vnops.c:935 > > > > > #14 0xffffffff80c585f2 in vn_ioctl (fp=3D3D3D0xfffff8052cdd80= f0, > co=3D > > m=3D3D3D<o=3D3D > > > > ptimized out>, data=3D3D3D0xfffffe02fb827d50, > active_cred=3D3D3D0xfffff8012=3D > > 2ab1e00,=3D3D > > > > td=3D3D3D<unavailable>) at /usr/src/sys/kern/vfs_vnops.c:1704 > > > > > #15 0xffffffff809d68ee in devfs_ioctl_f (fp=3D3D3D<unavailabl= e>, > fp=3D > > @entry=3D3D > > > > =3D3D3D<error reading variable: value is not available>, > com=3D3D3D<unavail=3D > > able>, c=3D3D > > > > om@entry=3D3D3D<error reading variable: value is not available>, > data=3D3D3=3D > > D<unavai=3D3D > > > > lable>, data@entry=3D3D3D<error reading variable: value is not > available>=3D > > , > > > > > cred=3D3D3D<unavailable>, cred@entry=3D3D3D<error reading > variabl=3D > > e: value=3D3D > > > > is not available>, td=3D3D3D<unavailable>, td@entry=3D3D3D<error > reading v=3D > > ariable:=3D3D > > > > value is not available>) at /usr/src/sys/fs/devfs/devfs_vnops.c:86= 6 > > > > > #16 0xffffffff80bc57e6 in fo_ioctl (fp=3D3D3D0xfffff8052cdd80= f0, > co=3D > > m=3D3D3D32=3D3D > > > > 22821401, data=3D3D3D<unavailable>, active_cred=3D3D3D<unavailable>= , > td=3D3D3=3D > > D0xfffffe0=3D3D > > > > 422ef8560) at /usr/src/sys/sys/file.h:367 > > > > > #17 kern_ioctl (td=3D3D3Dtd@entry=3D3D3D0xfffffe0422ef8560, > fd=3D3D3D4,=3D > > com=3D3D3Dcom=3D3D > > > > @entry=3D3D3D3222821401, data=3D3D3D<unavailable>, data@entry > =3D3D3D0xfffffe0=3D > > 2fb827d50 =3D3D > > > > "\017") at /usr/src/sys/kern/sys_generic.c:807 > > > > > #18 0xffffffff80bc54f2 in sys_ioctl > (td=3D3D3D0xfffffe0422ef8560, u=3D > > ap=3D3D3D0=3D3D > > > > xfffffe0422ef8960) at /usr/src/sys/kern/sys_generic.c:715 > > > > > #19 0xffffffff81049398 in syscallenter (td=3D3D3D<optimized > out>) a=3D > > t /usr=3D3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #20 amd64_syscall (td=3D3D3D0xfffffe0422ef8560, traced=3D3D3D= 0) at > /u=3D > > sr/src/s=3D3D > > > > ys/amd64/amd64/trap.c:1199 > > > [...] > > > > > > The backtrace looks different though it certainly smells like > PR/271945. > > > > > > I've had similar to PR/271945 panics on an amd64 with a mirrored zpoo= l > wi=3D > > th > > > four vdevs running poudriere with AMD64 jails. My other amd64 with a > > > mirrored zpool with two vdevs using i386 jails has no such issue. All > oth=3D > > er > > > workloads are unaffected. > > > > > > On the affected machine running poudriere bulk with -J N:1 circumvent= s > th=3D > > e > > > issue. So far. There were two openzfs cherry-picks this morning. I > intend > > > to try them against a full bulk build later today. > > > > > > > > > -- > > > Cheers, > > > Cy Schubert <Cy.Schubert@cschubert.com> > > > FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org > > > NTP: <cy@nwtime.org> Web: https://nwtime.org > > > > > > e^(i*pi)+1=3D3D0 > > > > > > > > > --000000000000b7db4b06029d5ea5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto">Spoke too soon still seeing zfs lockups under heavy poudr= iere workload after the MFVs.=C2=A0 Regression time matches what has been r= eported here.</div><div><br><div class=3D"gmail_quote"><div dir=3D"ltr" cla= ss=3D"gmail_attr">On Thu, Aug 10, 2023 at 4:33 PM Cy Schubert <<a href= =3D"mailto:Cy.Schubert@cschubert.com">Cy.Schubert@cschubert.com</a>> wro= te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border= -left-color:rgb(204,204,204)">I haven't experienced any problems (yet) = either.<br> <br> <br> -- <br> Cheers,<br> Cy Schubert <<a href=3D"mailto:Cy.Schubert@cschubert.com" target=3D"_bla= nk">Cy.Schubert@cschubert.com</a>><br> FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 <a href= =3D"https://FreeBSD.org" rel=3D"noreferrer" target=3D"_blank">https://FreeB= SD.org</a><br> NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:cy@nwtim= e.org" target=3D"_blank">cy@nwtime.org</a>>=C2=A0 =C2=A0 Web:=C2=A0 <a h= ref=3D"https://nwtime.org" rel=3D"noreferrer" target=3D"_blank">https://nwt= ime.org</a><br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 e^(i*pi)+1=3D0<br> <br> <br> In message <CAK7dMtDJQtaai3_6VjEkwVwW5JN6e8v=3DkKTOPffp371xb=3DORUg@mail= .gmail.c<br> om><br> , Kevin Bowling writes:<br> > The two MFVs on head have improved/fixed stability with poudriere for<= br> > me 48 core bare metal.<br> ><br> > On Thu, Aug 10, 2023 at 6:37=3DE2=3D80=3DAFAM Cy Schubert <Cy.Schub= ert@cschubert.=3D<br> > com> wrote:<br> > ><br> > > In message <CAK7dMtDJeuf8rjWbsNEZABUfeqpjUyCHzuOL9AAhKk93sy+PK= g@mail.gmai=3D<br> > l.c<br> > > om><br> > > , Kevin Bowling writes:<br> > > > Possibly <a href=3D"https://github.com/openzfs/zfs/commit/2c= b992a99ccadb78d97049b4=3D" rel=3D"noreferrer" target=3D"_blank">https://git= hub.com/openzfs/zfs/commit/2cb992a99ccadb78d97049b4=3D</a><br> > 0bd4=3D3D<br> > > > 42eb4fdc549d<br> > > ><br> > > > On Tue, Aug 8, 2023 at 10:08=3D3DE2=3D3D80=3D3DAFAM Dag-Erli= ng Sm=3D3DC3=3D3DB8rg=3D<br> > rav <des@freeb=3D3D<br> > > > <a href=3D"http://sd.org" rel=3D"noreferrer" target=3D"_blan= k">sd.org</a>> wrote:<br> > > > ><br> > > > > At some point between 42d088299c (4 May) and f0c9703301= (26 June), a<br> > > > > deadlock was introduced in ZFS.=C2=A0 It is still prese= nt as of 9c2823bae9=3D<br> >=C2=A0 (4<br> > > > > August) and is 100% reproducable just by starting poudr= iere bulk in a<br> > > > > 16-core VM and waiting a few hours until deadlkres kick= s in.=C2=A0 In the<br> > > > > latest instance, deadlkres complained about a bash proc= ess:<br> > > > ><br> > > > >=C2=A0 =C2=A0 =C2=A0#0=C2=A0 sched_switch (td=3D3D3Dtd@e= ntry=3D3D3D0xfffffe02fb1d8000, flags=3D<br> > =3D3D3Dflags@e=3D3D<br> > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299<br> > > > >=C2=A0 =C2=A0 =C2=A0#1=C2=A0 0xffffffff80b5a0a3 in mi_sw= itch (flags=3D3D3Dflags@entry=3D3D3D25=3D<br> > 9) at /u=3D3D<br> > > > sr/src/sys/kern/kern_synch.c:550<br> > > > >=C2=A0 =C2=A0 =C2=A0#2=C2=A0 0xffffffff80babcb4 in sleep= q_switch (wchan=3D3D3D0xfffff818543a=3D<br> > 9e70, =3D3D<br> > > > pri=3D3D3D64) at /usr/src/sys/kern/subr_sleepqueue.c:609<br> > > > >=C2=A0 =C2=A0 =C2=A0#3=C2=A0 0xffffffff80babb8c in sleep= q_wait (wchan=3D3D3D<unavailable>, p=3D<br> > ri=3D3D3D<=3D3D<br> > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660<= br> > > > >=C2=A0 =C2=A0 =C2=A0#4=C2=A0 0xffffffff80b1c1b0 in sleep= lk (lk=3D3D3Dlk@entry=3D3D3D0xfffff818=3D<br> > 543a9e70=3D3D<br> > > > , flags=3D3D3Dflags@entry=3D3D3D2121728, ilk=3D3D3Dilk@entry= =3D3D3D0x0, wmesg=3D<br> > =3D3D3Dwmesg@entry=3D3D<br> > > > =3D3D3D0xffffffff8222a054 "zfs", pri=3D3D3D<opt= imized out>, pri@entry=3D3D3D6=3D<br> > 4, timo=3D3D3D=3D3D<br> > > > timo@entry=3D3D3D6, queue=3D3D3D1) at /usr/src/sys/kern/kern= _lock.c:310<br> > > > >=C2=A0 =C2=A0 =C2=A0#5=C2=A0 0xffffffff80b1a23f in lockm= gr_slock_hard (lk=3D3D3D0xfffff81854=3D<br> > 3a9e70=3D3D<br> > > > , flags=3D3D3D2121728, ilk=3D3D3D<optimized out>, file= =3D3D3D0xffffffff812544=3D<br> > fb "/usr/s=3D3D<br> > > > rc/sys/kern/vfs_subr.c", line=3D3D3D3057, lwa=3D3D3D0x0= ) at /usr/src/sys/ke=3D<br> > rn/kern_=3D3D<br> > > > lock.c:705<br> > > > >=C2=A0 =C2=A0 =C2=A0#6=C2=A0 0xffffffff80c59ec3 in VOP_L= OCK1 (vp=3D3D3D0xfffff818543a9e00, f=3D<br> > lags=3D3D<br> > > > =3D3D3D2105344, file=3D3D3D0xffffffff812544fb "/usr/src= /sys/kern/vfs_subr.c=3D<br> > ", line=3D3D<br> > > > =3D3D3D3057) at ./vnode_if.h:1120<br> > > > >=C2=A0 =C2=A0 =C2=A0#7=C2=A0 _vn_lock (vp=3D3D3Dvp@entry= =3D3D3D0xfffff818543a9e00, flags=3D3D3D2=3D<br> > 105344, fi=3D3D<br> > > > le=3D3D3D<unavailable>, line=3D3D3D<unavailable>= , line@entry=3D3D3D3057) at /=3D<br> > usr/src/sy=3D3D<br> > > > s/kern/vfs_vnops.c:1815<br> > > > >=C2=A0 =C2=A0 =C2=A0#8=C2=A0 0xffffffff80c4173d in vget_= finish (vp=3D3D3D0xfffff818543a9e00,=3D<br> >=C2=A0 flags=3D3D<br> > > > =3D3D3D<unavailable>, vs=3D3D3Dvs@entry=3D3D3DVGET_USE= COUNT) at /usr/src/sys/=3D<br> > kern/vfs_s=3D3D<br> > > > ubr.c:3057<br> > > > >=C2=A0 =C2=A0 =C2=A0#9=C2=A0 0xffffffff80c1c9b7 in cache= _lookup (dvp=3D3D3Ddvp@entry=3D3D3D0xf=3D<br> > ffff802c=3D3D<br> > > > d02ac40, vpp=3D3D3Dvpp@entry=3D3D3D0xfffffe046b20ac30, cnp= =3D3D3Dcnp@entry=3D3D=3D<br> > 3D0xfffffe04=3D3D<br> > > > 6b20ac58, tsp=3D3D3Dtsp@entry=3D3D3D0x0, ticksp=3D3D3Dticksp= @entry=3D3D3D0x0) a=3D<br> > t /usr/src/s=3D3D<br> > > > ys/kern/vfs_cache.c:2086<br> > > > >=C2=A0 =C2=A0 =C2=A0#10 0xffffffff80c2150c in vfs_cache_= lookup (ap=3D3D3D<optimized out=3D<br> > >) at =3D3D<br> > > > /usr/src/sys/kern/vfs_cache.c:3068<br> > > > >=C2=A0 =C2=A0 =C2=A0#11 0xffffffff80c32c37 in VOP_LOOKUP= (dvp=3D3D3D0xfffff802cd02ac40,=3D<br> >=C2=A0 vpp=3D3D<br> > > > =3D3D3D0xfffffe046b20ac30, cnp=3D3D3D0xfffffe046b20ac58) at = ./vnode_if.h:69<br> > > > >=C2=A0 =C2=A0 =C2=A0#12 vfs_lookup (ndp=3D3D3Dndp@entry= =3D3D3D0xfffffe046b20abd8) at /usr=3D<br> > /src/sys=3D3D<br> > > > /kern/vfs_lookup.c:1266<br> > > > >=C2=A0 =C2=A0 =C2=A0#13 0xffffffff80c31ce1 in namei (ndp= =3D3D3Dndp@entry=3D3D3D0xfffffe04=3D<br> > 6b20abd8=3D3D<br> > > > ) at /usr/src/sys/kern/vfs_lookup.c:689<br> > > > >=C2=A0 =C2=A0 =C2=A0#14 0xffffffff80c52090 in kern_stata= t (td=3D3D3D0xfffffe02fb1d8000,=3D<br> >=C2=A0 flag=3D3D<br> > > > =3D3D3D<optimized out>, fd=3D3D3D-100, path=3D3D3D0xa7= 5b480e070 <error: Canno=3D<br> > t access m=3D3D<br> > > > emory at address 0xa75b480e070>, pathseg=3D3D3Dpathseg@en= try=3D3D3DUIO_USER=3D<br> > SPACE, s=3D3D<br> > > > bp=3D3D3Dsbp@entry=3D3D3D0xfffffe046b20ad18)<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/v= fs_syscalls.c:2441<br> > > > >=C2=A0 =C2=A0 =C2=A0#15 0xffffffff80c52797 in sys_fstata= t (td=3D3D3D<unavailable>, uap=3D<br> > =3D3D3D0xff=3D3D<br> > > > fffe02fb1d8400) at /usr/src/sys/kern/vfs_syscalls.c:2419<br> > > > >=C2=A0 =C2=A0 =C2=A0#16 0xffffffff81049398 in syscallent= er (td=3D3D3D<optimized out>) a=3D<br> > t /usr=3D3D<br> > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190<br> > > > >=C2=A0 =C2=A0 =C2=A0#17 amd64_syscall (td=3D3D3D0xfffffe= 02fb1d8000, traced=3D3D3D0) at /u=3D<br> > sr/src/s=3D3D<br> > > > ys/amd64/amd64/trap.c:1199<br> > > > >=C2=A0 =C2=A0 =C2=A0#18 <signal handler called><br= > > > > ><br> > > > > The lock it is trying to acquire in frame 5 belongs to = another bash<br> > > > > process which is in the process of creating a fifo:<br> > > > ><br> > > > >=C2=A0 =C2=A0 =C2=A0#0=C2=A0 sched_switch (td=3D3D3Dtd@e= ntry=3D3D3D0xfffffe046acd8e40, flags=3D<br> > =3D3D3Dflags@e=3D3D<br> > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299<br> > > > >=C2=A0 =C2=A0 =C2=A0#1=C2=A0 0xffffffff80b5a0a3 in mi_sw= itch (flags=3D3D3Dflags@entry=3D3D3D25=3D<br> > 9) at /u=3D3D<br> > > > sr/src/sys/kern/kern_synch.c:550<br> > > > >=C2=A0 =C2=A0 =C2=A0#2=C2=A0 0xffffffff80babcb4 in sleep= q_switch (wchan=3D3D3D0xfffff8018acb=3D<br> > f154, =3D3D<br> > > > pri=3D3D3D87) at /usr/src/sys/kern/subr_sleepqueue.c:609<br> > > > >=C2=A0 =C2=A0 =C2=A0#3=C2=A0 0xffffffff80babb8c in sleep= q_wait (wchan=3D3D3D<unavailable>, p=3D<br> > ri=3D3D3D<=3D3D<br> > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660<= br> > > > >=C2=A0 =C2=A0 =C2=A0#4=C2=A0 0xffffffff80b59606 in _slee= p (ident=3D3D3Dident@entry=3D3D3D0xfff=3D<br> > ff8018ac=3D3D<br> > > > bf154, lock=3D3D3Dlock@entry=3D3D3D0xfffff8018acbf120, prior= ity=3D3D3Dpriorit=3D<br> > y@entry=3D3D3D=3D3D<br> > > > 87, wmesg=3D3D3D0xffffffff8223af0e "zfs teardown inacti= ve", sbt=3D3D3Dsbt@e=3D<br> > ntry=3D3D3D0=3D3D<br> > > > , pr=3D3D3Dpr@entry=3D3D3D0, flags=3D3D3D256)<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/k= ern_synch.c:225<br> > > > >=C2=A0 =C2=A0 =C2=A0#5=C2=A0 0xffffffff80b45dc0 in rms_r= lock_fallback (rms=3D3D3D0xfffff8018=3D<br> > acbf12=3D3D<br> > > > 0) at /usr/src/sys/kern/kern_rmlock.c:1015<br> > > > >=C2=A0 =C2=A0 =C2=A0#6=C2=A0 0xffffffff80b45c93 in rms_r= lock (rms=3D3D3D<unavailable>, rms@e=3D<br> > ntry=3D3D<br> > > > =3D3D3D0xfffff8018acbf120) at /usr/src/sys/kern/kern_rmlock.= c:1036<br> > > > >=C2=A0 =C2=A0 =C2=A0#7=C2=A0 0xffffffff81fb147b in zfs_f= reebsd_reclaim (ap=3D3D3D<optimized =3D<br> > out>) =3D3D<br> > > > at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vn= ops_os.c:51=3D<br> > 64<br> > > > >=C2=A0 =C2=A0 =C2=A0#8=C2=A0 0xffffffff8111d245 in VOP_R= ECLAIM_APV (vop=3D3D3D0xffffffff822e=3D<br> > 71a0 <=3D3D<br> > > > zfs_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1c9c8) = at vnode_if.c:2180<br> > > > >=C2=A0 =C2=A0 =C2=A0#9=C2=A0 0xffffffff80c43569 in VOP_R= ECLAIM (vp=3D3D3D0xfffff802cdbaca80)=3D<br> >=C2=A0 at ./=3D3D<br> > > > vnode_if.h:1084<br> > > > >=C2=A0 =C2=A0 =C2=A0#10 vgonel (vp=3D3D3Dvp@entry=3D3D3D= 0xfffff802cdbaca80) at /usr/src/s=3D<br> > ys/kern/=3D3D<br> > > > vfs_subr.c:4143<br> > > > >=C2=A0 =C2=A0 =C2=A0#11 0xffffffff80c3ef61 in vtryrecycl= e (vp=3D3D3D0xfffff802cdbaca80)=3D<br> >=C2=A0 at /u=3D3D<br> > > > sr/src/sys/kern/vfs_subr.c:1693<br> > > > >=C2=A0 =C2=A0 =C2=A0#12 vnlru_free_impl (count=3D3D3Dcou= nt@entry=3D3D3D1, mnt_op=3D3D3Dmnt_=3D<br> > op@entry=3D3D<br> > > > =3D3D3D0x0, mvp=3D3D3D0xfffff8010864da00) at /usr/src/sys/ke= rn/vfs_subr.c:1=3D<br> > 344<br> > > > >=C2=A0 =C2=A0 =C2=A0#13 0xfffff =C3=A3=C2=B5=C2=B6=E2=80= =93,=C2=A0 =C2=A0 fff80c49553 in vnlru_free_locked (count=3D3D3D1) at /usr= =3D<br> > /src/s=3D3D<br> > > > ys/kern/vfs_subr.c:1357<br> > > > >=C2=A0 =C2=A0 =C2=A0#14 vn_alloc_hard (mp=3D3D3Dmp@entry= =3D3D3D0x0) at /usr/src/sys/kern/=3D<br> > vfs_subr=3D3D<br> > > > .c:1744<br> > > > >=C2=A0 =C2=A0 =C2=A0#15 0xffffffff80c3f6f0 in vn_alloc (= mp=3D3D3D0x0) at /usr/src/sys/a=3D<br> > md64/i=3D3D<br> > > > nclude/atomic.h:375<br> > > > >=C2=A0 =C2=A0 =C2=A0#16 getnewvnode_reserve () at /usr/s= rc/sys/kern/vfs_subr.c:1888<br> > > > >=C2=A0 =C2=A0 =C2=A0#17 0xffffffff81faa072 in zfs_create= (dzp=3D3D3D0xfffff812200261d0,=3D<br> >=C2=A0 name=3D3D<br> > > > =3D3D3D0xfffff8011b8ac805 "sh-np.yPbxoo", vap=3D3D= 3D0xfffffe0410f1cc20, exc=3D<br> > l=3D3D3D<opt=3D3D<br> > > > imized out>, mode=3D3D3D<optimized out>, zpp=3D3D3D= zpp@entry=3D3D3D0xfffffe04=3D<br> > 10f1cbc8, =3D3D<br> > > > cr=3D3D3D0xfffff80140fb1100, flag=3D3D3D<optimized out>= ;, vsecp=3D3D3D0x0, mnt=3D<br> > _ns=3D3D3D0x0)<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/contri= b/openzfs/module/os/freebsd/zfs/zfs_vno=3D<br> > ps_o=3D3D<br> > > > s.c:1146<br> > > > >=C2=A0 =C2=A0 =C2=A0#18 0xffffffff81faea57 in zfs_freebs= d_create (ap=3D3D3D0xfffffe0410=3D<br> > f1cda0=3D3D<br> > > > ) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_= vnops_os.c:=3D<br> > 4618<br> > > > >=C2=A0 =C2=A0 =C2=A0#19 0xffffffff8111aa9a in VOP_MKNOD_= APV (vop=3D3D3D0xffffffff822e71=3D<br> > a0 <zf=3D3D<br> > > > s_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1cda0) at= vnode_if.c:372<br> > > > >=C2=A0 =C2=A0 =C2=A0#20 0xffffffff80c50207 in VOP_MKNOD = (dvp=3D3D3D<unavailable>, cnp=3D<br> > =3D3D3D0xfff=3D3D<br> > > > ffe0410f1cd50, vap=3D3D3D0xfffffe0410f1cc20, vpp=3D3D3D<o= ptimized out>) at =3D<br> > ./vnode_=3D3D<br> > > > if.h:188<br> > > > >=C2=A0 =C2=A0 =C2=A0#21 kern_mkfifoat (td=3D3D3D0xfffffe= 046acd8e40, fd=3D3D3D-100, path=3D<br> > =3D3D3D0x12772=3D3D<br> > > > f073500 <error: Cannot access memory at address 0x12772f0= 73500>, pathse=3D<br> > g=3D3D3D=3D3D<br> > > > UIO_USERSPACE, mode=3D3D3D<optimized out>) at /usr/src= /sys/kern/vfs_sysca=3D<br> > lls.c:=3D3D<br> > > > 1492<br> > > > >=C2=A0 =C2=A0 =C2=A0#22 0xffffffff81049398 in syscallent= er (td=3D3D3D<optimized out>) a=3D<br> > t /usr=3D3D<br> > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190<br> > > > >=C2=A0 =C2=A0 =C2=A0#23 amd64_=C2=A0 =C2=A0 =3DE6=3D90= =3DAC=3DEE=3D8A=3D80 syscall (td=3D3D3D0xfffffe046acd8e=3D<br> > 40, traced=3D3D3D0) at /usr/src/s=3D3D<br> > > > ys/amd64/amd64/trap.c:1199<br> > > > >=C2=A0 =C2=A0 =C2=A0#24 <signal handler called><br= > > > > ><br> > > > > Frame 7 is trying to acquire the ZFS teardown inactive = lock, which is<br> > > > > held by a process which is performing a ZFS rollback an= d is waiting f=3D<br> > or<br> > > > > the transaction to sync:<br> > > > ><br> > > > >=C2=A0 =C2=A0 =C2=A0#0=C2=A0 sched_switch (td=3D3D3Dtd@e= ntry=3D3D3D0xfffffe0422ef8560, flags=3D<br> > =3D3D3Dflags@e=3D3D<br> > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299<br> > > > >=C2=A0 =C2=A0 =C2=A0#1=C2=A0 0xffffffff80b5a0a3 in mi_sw= itch (flags=3D3D3Dflags@entry=3D3D3D25=3D<br> > 9) at /u=3D3D<br> > > > sr/src/sys/kern/kern_synch.c:550<br> > > > >=C2=A0 =C2=A0 =C2=A0#2=C2=A0 0xffffffff80babcb4 in sleep= q_switch (wchan=3D3D3D0xfffff8011b83=3D<br> > d540, =3D3D<br> > > > pri=3D3D3D0) at /usr/src/sys/kern/subr_sleepqueue.c:609<br> > > > >=C2=A0 =C2=A0 =C2=A0#3=C2=A0 0xffffffff80babb8c in sleep= q_wait (wchan=3D3D3D<unavailable>, w=3D<br> > chan@e=3D3D<br> > > > ntry=3D3D3D0xfffff8011b83d540, pri=3D3D3D<unavailable>= , pri@entry=3D3D3D0) at=3D<br> >=C2=A0 /usr/src/=3D3D<br> > > > sys/kern/subr_sleepqueue.c:660<br> > > > >=C2=A0 =C2=A0 =C2=A0#4=C2=A0 0xffffffff80ad7f75 in _cv_w= ait (cvp=3D3D3Dcvp@entry=3D3D3D0xfffff=3D<br> > 8011b83d=3D3D<br> > > > 540, lock=3D3D3Dlock@entry=3D3D3D0xfffff8011b83d4d0) at /usr= /src/sys/kern/k=3D<br> > ern_cond=3D3D<br> > > > var.c:146<br> > > > >=C2=A0 =C2=A0 =C2=A0#5=C2=A0 0xffffffff820b42fb in txg_w= ait_synced_impl (dp=3D3D3Ddp@entry=3D<br> > =3D3D3D0xfff=3D3D<br> > > > ff8011b83d000, txg=3D3D3D8585097, wait_sig=3D3D3Dwait_sig@en= try=3D3D3D0) at /=3D<br> > usr/src/sy=3D3D<br> > > > s/contrib/openzfs/module/zfs/txg.c:726<br> > > > >=C2=A0 =C2=A0 =C2=A0#6=C2=A0 0xffffffff820b3cab in txg_w= ait_synced (dp=3D3D3D<unavailable>, =3D<br> > dp@ent=3D3D<br> > > > ry=3D3D3D0xfffff8011b83d000, txg=3D3D3D<unavailable>) = at /usr/src/sys/contr=3D<br> > ib/openz=3D3D<br> > > > fs/module/zfs/txg.c:736<br> > > > >=C2=A0 =C2=A0 =C2=A0#7=C2=A0 0xffffffff8206d5b5 in dsl_s= ync_task_common (pool=3D3D3Dpool@ent=3D<br> > ry=3D3D3D0=3D3D<br> > > > xfffffe0401d15000 "zroot/poudriere/jails/13amd64-defaul= t-ref/15", check=3D<br> > func=3D3D<br> > > > =3D3D3D<optimized out>, syncfunc=3D3D3D0xffffffff8203f= bc0 <dsl_dataset_roll=3D<br> > back_syn=3D3D<br> > > > c>, sigfunc=3D3D3Dsigfunc@entry=3D3D3D0x0, arg=3D3D3Darg@= entry=3D3D3D0xfffffe02=3D<br> > fb827a90,<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0blocks_modified=3D3D3D= blocks_modified@entry=3D3D3D1, space_check=3D<br> > =3D3D3DZFS_SP=3D3D<br> > > > ACE_CHECK_RESERVED, early=3D3D3D0) at /usr/src/sys/contrib/o= penzfs/module=3D<br> > /zfs/d=3D3D<br> > > > sl_synctask.c:93<br> > > > >=C2=A0 =C2=A0 =C2=A0#8=C2=A0 0xffffffff8206d3c7 in dsl_s= ync_task (pool=3D3D3D<unavailable>, =3D<br> > pool@e=3D3D<br> > > > ntry=3D3D3D0xfffffe0401d15000 "zroot/poudriere/jails/13= amd64-default-ref/=3D<br> > 15", c=3D3D<br> > > > heckfunc=3D3D3D<unavailable>, syncfunc=3D3D3D<unava= ilable>, arg=3D3D3D<unavai=3D<br> > lable>, ar=3D3D<br> > > > g@entry=3D3D3D0xfffffe02fb827a90, blocks_modified=3D3D3D<= unavailable>,<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0blocks_modified@entry= =3D3D3D1, space_check=3D3D3D<unavailable>, s=3D<br> > pace_che=3D3D<br> > > > ck@entry=3D3D3DZFS_SPACE_CHECK_RESERVED) at /usr/src/sys/con= trib/openzfs/=3D<br> > module=3D3D<br> > > > /zfs/dsl_synctask.c:132<br> > > > >=C2=A0 =C2=A0 =C2=A0#9=C2=A0 0xffffffff8204075b in dsl_d= ataset_rollback (fsname=3D3D3D<unava=3D<br> > ilable=3D3D<br> > > > >, fsname@entry=3D3D3D0xfffffe0401d15000 "zroot/poud= riere/jails/13amd64-d=3D<br> > efault=3D3D<br> > > > -ref/15", tosnap=3D3D3D<optimized out>, owner=3D3= D3D<optimized out>, result=3D<br> > =3D3D3Dresul=3D3D<br> > > > t@entry=3D3D3D0xfffff81c826a9ea0)<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/contri= b/openzfs/module/zfs/dsl_dataset.c:3261<br> > > > >=C2=A0 =C2=A0 =C2=A0#10 0xffffffff82168dd9 in zfs_ioc_ro= llback (fsname=3D3D3D0xfffffe04=3D<br> > 01d150=3D3D<br> > > > 00 "zroot/poudriere/jails/13amd64-default-ref/15",= fsname@entry=3D3D3D<er=3D<br> > ror re=3D3D<br> > > > ading variable: value is not available>, innvl=3D3D3D<= unavailable>, innvl=3D<br> > @entry=3D3D<br> > > > =3D3D3D<error reading variable: value is not available>= ;,<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0outnvl=3D3D3D0xfffff81= c826a9ea0, outnvl@entry=3D3D3D<error readin=3D<br> > g variab=3D3D<br> > > > le: value is not available>) at /usr/src/sys/contrib/open= zfs/module/zfs=3D<br> > /zfs=3D3D<br> > > > _ioctl.c:4405<br> > > > >=C2=A0 =C2=A0 =C2=A0#11 0xffffffff82164522 in zfsdev_ioc= tl_common (vecnum=3D3D3Dvecnum@=3D<br> > entry=3D3D<br> > > > =3D3D3D25, zc=3D3D3Dzc@entry=3D3D3D0xfffffe0401d15000, flag= =3D3D3Dflag@entry=3D3D=3D<br> > 3D0) at /usr/s=3D3D<br> > > > rc/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:7798<br> > > > >=C2=A0 =C2=A0 =C2=A0#12 0xffffffff81f97fca in zfsdev_ioc= tl (dev=3D3D3D<optimized out>, =3D<br> > zcmd=3D3D<br> > > > =3D3D3D<unavailable>, zcmd@entry=3D3D3D<error readi= ng variable: value is no=3D<br> > t availa=3D3D<br> > > > ble>, arg=3D3D3D0xfffffe02fb827d50 "\017", arg@= entry=3D3D3D<error reading v=3D<br> > ariable:=3D3D<br> > > >=C2=A0 value is not available>, flag=3D3D3D<optimized o= ut>, td=3D3D3D<optimized o=3D<br> > ut>)<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/contri= b/openzfs/module/os/freebsd/zfs/kmod_co=3D<br> > re.c=3D3D<br> > > > :168<br> > > > >=C2=A0 =C2=A0 =C2=A0#13 0xffffffff809d6212 in devfs_ioct= l (ap=3D3D3D0xfffffe02fb827c50)=3D<br> >=C2=A0 at /u=3D3D<br> > > > sr/src/sys/fs/devfs/devfs_vnops.c:935<br> > > > >=C2=A0 =C2=A0 =C2=A0#14 0xffffffff80c585f2 in vn_ioctl (= fp=3D3D3D0xfffff8052cdd80f0, co=3D<br> > m=3D3D3D<o=3D3D<br> > > > ptimized out>, data=3D3D3D0xfffffe02fb827d50, active_cred= =3D3D3D0xfffff8012=3D<br> > 2ab1e00,=3D3D<br> > > >=C2=A0 td=3D3D3D<unavailable>) at /usr/src/sys/kern/vfs= _vnops.c:1704<br> > > > >=C2=A0 =C2=A0 =C2=A0#15 0xffffffff809d68ee in devfs_ioct= l_f (fp=3D3D3D<unavailable>, fp=3D<br> > @entry=3D3D<br> > > > =3D3D3D<error reading variable: value is not available>= ;, com=3D3D3D<unavail=3D<br> > able>, c=3D3D<br> > > > om@entry=3D3D3D<error reading variable: value is not avai= lable>, data=3D3D3=3D<br> > D<unavai=3D3D<br> > > > lable>, data@entry=3D3D3D<error reading variable: valu= e is not available>=3D<br> > ,<br> > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cred=3D3D3D<unavail= able>, cred@entry=3D3D3D<error reading variabl=3D<br> > e: value=3D3D<br> > > >=C2=A0 is not available>, td=3D3D3D<unavailable>, td= @entry=3D3D3D<error reading v=3D<br> > ariable:=3D3D<br> > > >=C2=A0 value is not available>) at /usr/src/sys/fs/devfs/d= evfs_vnops.c:866<br> > > > >=C2=A0 =C2=A0 =C2=A0#16 0xffffffff80bc57e6 in fo_ioctl (= fp=3D3D3D0xfffff8052cdd80f0, co=3D<br> > m=3D3D3D32=3D3D<br> > > > 22821401, data=3D3D3D<unavailable>, active_cred=3D3D3D= <unavailable>, td=3D3D3=3D<br> > D0xfffffe0=3D3D<br> > > > 422ef8560) at /usr/src/sys/sys/file.h:367<br> > > > >=C2=A0 =C2=A0 =C2=A0#17 kern_ioctl (td=3D3D3Dtd@entry=3D= 3D3D0xfffffe0422ef8560, fd=3D3D3D4,=3D<br> >=C2=A0 com=3D3D3Dcom=3D3D<br> > > > @entry=3D3D3D3222821401, data=3D3D3D<unavailable>, dat= a@entry=3D3D3D0xfffffe0=3D<br> > 2fb827d50 =3D3D<br> > > > "\017") at /usr/src/sys/kern/sys_generic.c:807<br> > > > >=C2=A0 =C2=A0 =C2=A0#18 0xffffffff80bc54f2 in sys_ioctl = (td=3D3D3D0xfffffe0422ef8560, u=3D<br> > ap=3D3D3D0=3D3D<br> > > > xfffffe0422ef8960) at /usr/src/sys/kern/sys_generic.c:715<br= > > > > >=C2=A0 =C2=A0 =C2=A0#19 0xffffffff81049398 in syscallent= er (td=3D3D3D<optimized out>) a=3D<br> > t /usr=3D3D<br> > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190<br> > > > >=C2=A0 =C2=A0 =C2=A0#20 amd64_syscall (td=3D3D3D0xfffffe= 0422ef8560, traced=3D3D3D0) at /u=3D<br> > sr/src/s=3D3D<br> > > > ys/amd64/amd64/trap.c:1199<br> > > [...]<br> > ><br> > > The backtrace looks different though it certainly smells like PR/= 271945.<br> > ><br> > > I've had similar to PR/271945 panics on an amd64 with a mirro= red zpool wi=3D<br> > th<br> > > four vdevs running poudriere with AMD64 jails. My other amd64 wit= h a<br> > > mirrored zpool with two vdevs using i386 jails has no such issue.= All oth=3D<br> > er<br> > > workloads are unaffected.<br> > ><br> > > On the affected machine running poudriere bulk with -J N:1 circum= vents th=3D<br> > e<br> > > issue. So far. There were two openzfs cherry-picks this morning. = I intend<br> > > to try them against a full bulk build later today.<br> > ><br> > ><br> > > --<br> > > Cheers,<br> > > Cy Schubert <<a href=3D"mailto:Cy.Schubert@cschubert.com" targ= et=3D"_blank">Cy.Schubert@cschubert.com</a>><br> > > FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0= <a href=3D"https://FreeBSD.org" rel=3D"noreferrer" target=3D"_blank">https= ://FreeBSD.org</a><br> > > NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<<a href=3D"mailt= o:cy@nwtime.org" target=3D"_blank">cy@nwtime.org</a>>=C2=A0 =C2=A0 Web:= =C2=A0 <a href=3D"https://nwtime.org" rel=3D"noreferrer" target=3D"_blank">= https://nwtime.org</a><br>; > ><br> > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0e^(i*pi)+1=3D3D0<br> > ><br> > ><br> <br> <br> </blockquote></div></div> --000000000000b7db4b06029d5ea5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAK7dMtDZP9gZcxvo2QPxHSVXOoLyaDsUTmTA-yeTTQuNfmm-kA>