Date: Thu, 10 Aug 2023 20:32:51 -0700 From: Kevin Bowling <kevin.bowling@kev009.com> To: Cy Schubert <Cy.Schubert@cschubert.com> Cc: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= <des@freebsd.org>, current@freebsd.org Subject: Re: ZFS deadlock in 14 Message-ID: <CAK7dMtDZP9gZcxvo2QPxHSVXOoLyaDsUTmTA-yeTTQuNfmm-kA@mail.gmail.com> In-Reply-To: <20230810233312.0E10AF4@slippy.cwsent.com> References: <86leeltqcb.fsf@ltc.des.no> <CAK7dMtDJeuf8rjWbsNEZABUfeqpjUyCHzuOL9AAhKk93sy%2BPKg@mail.gmail.com> <20230810133745.D0EC0178@slippy.cwsent.com> <CAK7dMtDJQtaai3_6VjEkwVwW5JN6e8v=kKTOPffp371xb=ORUg@mail.gmail.com> <20230810233312.0E10AF4@slippy.cwsent.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Spoke too soon still seeing zfs lockups under heavy poudriere workload after the MFVs. Regression time matches what has been reported here. On Thu, Aug 10, 2023 at 4:33 PM Cy Schubert <Cy.Schubert@cschubert.com> wrote: > I haven't experienced any problems (yet) either. > > > -- > Cheers, > Cy Schubert <Cy.Schubert@cschubert.com> > FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org > NTP: <cy@nwtime.org> Web: https://nwtime.org > > e^(i*pi)+1=0 > > > In message > <CAK7dMtDJQtaai3_6VjEkwVwW5JN6e8v=kKTOPffp371xb=ORUg@mail.gmail.c > om> > , Kevin Bowling writes: > > The two MFVs on head have improved/fixed stability with poudriere for > > me 48 core bare metal. > > > > On Thu, Aug 10, 2023 at 6:37=E2=80=AFAM Cy Schubert > <Cy.Schubert@cschubert.= > > com> wrote: > > > > > > In message > <CAK7dMtDJeuf8rjWbsNEZABUfeqpjUyCHzuOL9AAhKk93sy+PKg@mail.gmai= > > l.c > > > om> > > > , Kevin Bowling writes: > > > > Possibly > https://github.com/openzfs/zfs/commit/2cb992a99ccadb78d97049b4= > > 0bd4=3D > > > > 42eb4fdc549d > > > > > > > > On Tue, Aug 8, 2023 at 10:08=3DE2=3D80=3DAFAM Dag-Erling > Sm=3DC3=3DB8rg= > > rav <des@freeb=3D > > > > sd.org> wrote: > > > > > > > > > > At some point between 42d088299c (4 May) and f0c9703301 (26 June), > a > > > > > deadlock was introduced in ZFS. It is still present as of > 9c2823bae9= > > (4 > > > > > August) and is 100% reproducable just by starting poudriere bulk > in a > > > > > 16-core VM and waiting a few hours until deadlkres kicks in. In > the > > > > > latest instance, deadlkres complained about a bash process: > > > > > > > > > > #0 sched_switch (td=3D3Dtd@entry=3D3D0xfffffe02fb1d8000, > flags= > > =3D3Dflags@e=3D > > > > ntry=3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3Dflags@entry > =3D3D25= > > 9) at /u=3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D0xfffff818543a= > > 9e70, =3D > > > > pri=3D3D64) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D<unavailable>, p= > > ri=3D3D<=3D > > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80b1c1b0 in sleeplk (lk=3D3Dlk@entry > =3D3D0xfffff818= > > 543a9e70=3D > > > > , flags=3D3Dflags@entry=3D3D2121728, ilk=3D3Dilk@entry=3D3D0x0, > wmesg= > > =3D3Dwmesg@entry=3D > > > > =3D3D0xffffffff8222a054 "zfs", pri=3D3D<optimized out>, pri@entry > =3D3D6= > > 4, timo=3D3D=3D > > > > timo@entry=3D3D6, queue=3D3D1) at /usr/src/sys/kern/kern_lock.c:310 > > > > > #5 0xffffffff80b1a23f in lockmgr_slock_hard > (lk=3D3D0xfffff81854= > > 3a9e70=3D > > > > , flags=3D3D2121728, ilk=3D3D<optimized out>, > file=3D3D0xffffffff812544= > > fb "/usr/s=3D > > > > rc/sys/kern/vfs_subr.c", line=3D3D3057, lwa=3D3D0x0) at > /usr/src/sys/ke= > > rn/kern_=3D > > > > lock.c:705 > > > > > #6 0xffffffff80c59ec3 in VOP_LOCK1 > (vp=3D3D0xfffff818543a9e00, f= > > lags=3D > > > > =3D3D2105344, file=3D3D0xffffffff812544fb > "/usr/src/sys/kern/vfs_subr.c= > > ", line=3D > > > > =3D3D3057) at ./vnode_if.h:1120 > > > > > #7 _vn_lock (vp=3D3Dvp@entry=3D3D0xfffff818543a9e00, > flags=3D3D2= > > 105344, fi=3D > > > > le=3D3D<unavailable>, line=3D3D<unavailable>, line@entry=3D3D3057) > at /= > > usr/src/sy=3D > > > > s/kern/vfs_vnops.c:1815 > > > > > #8 0xffffffff80c4173d in vget_finish > (vp=3D3D0xfffff818543a9e00,= > > flags=3D > > > > =3D3D<unavailable>, vs=3D3Dvs@entry=3D3DVGET_USECOUNT) at > /usr/src/sys/= > > kern/vfs_s=3D > > > > ubr.c:3057 > > > > > #9 0xffffffff80c1c9b7 in cache_lookup (dvp=3D3Ddvp@entry > =3D3D0xf= > > ffff802c=3D > > > > d02ac40, vpp=3D3Dvpp@entry=3D3D0xfffffe046b20ac30, cnp=3D3Dcnp@entry > =3D= > > 3D0xfffffe04=3D > > > > 6b20ac58, tsp=3D3Dtsp@entry=3D3D0x0, ticksp=3D3Dticksp@entry=3D3D0x0) > a= > > t /usr/src/s=3D > > > > ys/kern/vfs_cache.c:2086 > > > > > #10 0xffffffff80c2150c in vfs_cache_lookup (ap=3D3D<optimized > out= > > >) at =3D > > > > /usr/src/sys/kern/vfs_cache.c:3068 > > > > > #11 0xffffffff80c32c37 in VOP_LOOKUP > (dvp=3D3D0xfffff802cd02ac40,= > > vpp=3D > > > > =3D3D0xfffffe046b20ac30, cnp=3D3D0xfffffe046b20ac58) at > ./vnode_if.h:69 > > > > > #12 vfs_lookup (ndp=3D3Dndp@entry=3D3D0xfffffe046b20abd8) at > /usr= > > /src/sys=3D > > > > /kern/vfs_lookup.c:1266 > > > > > #13 0xffffffff80c31ce1 in namei (ndp=3D3Dndp@entry > =3D3D0xfffffe04= > > 6b20abd8=3D > > > > ) at /usr/src/sys/kern/vfs_lookup.c:689 > > > > > #14 0xffffffff80c52090 in kern_statat > (td=3D3D0xfffffe02fb1d8000,= > > flag=3D > > > > =3D3D<optimized out>, fd=3D3D-100, path=3D3D0xa75b480e070 <error: > Canno= > > t access m=3D > > > > emory at address 0xa75b480e070>, pathseg=3D3Dpathseg@entry > =3D3DUIO_USER= > > SPACE, s=3D > > > > bp=3D3Dsbp@entry=3D3D0xfffffe046b20ad18) > > > > > at /usr/src/sys/kern/vfs_syscalls.c:2441 > > > > > #15 0xffffffff80c52797 in sys_fstatat (td=3D3D<unavailable>, > uap= > > =3D3D0xff=3D > > > > fffe02fb1d8400) at /usr/src/sys/kern/vfs_syscalls.c:2419 > > > > > #16 0xffffffff81049398 in syscallenter (td=3D3D<optimized > out>) a= > > t /usr=3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #17 amd64_syscall (td=3D3D0xfffffe02fb1d8000, traced=3D3D0) at > /u= > > sr/src/s=3D > > > > ys/amd64/amd64/trap.c:1199 > > > > > #18 <signal handler called> > > > > > > > > > > The lock it is trying to acquire in frame 5 belongs to another bash > > > > > process which is in the process of creating a fifo: > > > > > > > > > > #0 sched_switch (td=3D3Dtd@entry=3D3D0xfffffe046acd8e40, > flags= > > =3D3Dflags@e=3D > > > > ntry=3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3Dflags@entry > =3D3D25= > > 9) at /u=3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D0xfffff8018acb= > > f154, =3D > > > > pri=3D3D87) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D<unavailable>, p= > > ri=3D3D<=3D > > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80b59606 in _sleep (ident=3D3Dident@entry > =3D3D0xfff= > > ff8018ac=3D > > > > bf154, lock=3D3Dlock@entry=3D3D0xfffff8018acbf120, > priority=3D3Dpriorit= > > y@entry=3D3D=3D > > > > 87, wmesg=3D3D0xffffffff8223af0e "zfs teardown inactive", > sbt=3D3Dsbt@e= > > ntry=3D3D0=3D > > > > , pr=3D3Dpr@entry=3D3D0, flags=3D3D256) > > > > > at /usr/src/sys/kern/kern_synch.c:225 > > > > > #5 0xffffffff80b45dc0 in rms_rlock_fallback > (rms=3D3D0xfffff8018= > > acbf12=3D > > > > 0) at /usr/src/sys/kern/kern_rmlock.c:1015 > > > > > #6 0xffffffff80b45c93 in rms_rlock (rms=3D3D<unavailable>, > rms@e= > > ntry=3D > > > > =3D3D0xfffff8018acbf120) at /usr/src/sys/kern/kern_rmlock.c:1036 > > > > > #7 0xffffffff81fb147b in zfs_freebsd_reclaim > (ap=3D3D<optimized = > > out>) =3D > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:51= > > 64 > > > > > #8 0xffffffff8111d245 in VOP_RECLAIM_APV > (vop=3D3D0xffffffff822e= > > 71a0 <=3D > > > > zfs_vnodeops>, a=3D3Da@entry=3D3D0xfffffe0410f1c9c8) at > vnode_if.c:2180 > > > > > #9 0xffffffff80c43569 in VOP_RECLAIM > (vp=3D3D0xfffff802cdbaca80)= > > at ./=3D > > > > vnode_if.h:1084 > > > > > #10 vgonel (vp=3D3Dvp@entry=3D3D0xfffff802cdbaca80) at > /usr/src/s= > > ys/kern/=3D > > > > vfs_subr.c:4143 > > > > > #11 0xffffffff80c3ef61 in vtryrecycle > (vp=3D3D0xfffff802cdbaca80)= > > at /u=3D > > > > sr/src/sys/kern/vfs_subr.c:1693 > > > > > #12 vnlru_free_impl (count=3D3Dcount@entry=3D3D1, > mnt_op=3D3Dmnt_= > > op@entry=3D > > > > =3D3D0x0, mvp=3D3D0xfffff8010864da00) at > /usr/src/sys/kern/vfs_subr.c:1= > > 344 > > > > > #13 0xfffff 㵶–, fff80c49553 in vnlru_free_locked > (count=3D3D1) at /usr= > > /src/s=3D > > > > ys/kern/vfs_subr.c:1357 > > > > > #14 vn_alloc_hard (mp=3D3Dmp@entry=3D3D0x0) at > /usr/src/sys/kern/= > > vfs_subr=3D > > > > .c:1744 > > > > > #15 0xffffffff80c3f6f0 in vn_alloc (mp=3D3D0x0) at > /usr/src/sys/a= > > md64/i=3D > > > > nclude/atomic.h:375 > > > > > #16 getnewvnode_reserve () at /usr/src/sys/kern/vfs_subr.c:1888 > > > > > #17 0xffffffff81faa072 in zfs_create > (dzp=3D3D0xfffff812200261d0,= > > name=3D > > > > =3D3D0xfffff8011b8ac805 "sh-np.yPbxoo", vap=3D3D0xfffffe0410f1cc20, > exc= > > l=3D3D<opt=3D > > > > imized out>, mode=3D3D<optimized out>, zpp=3D3Dzpp@entry > =3D3D0xfffffe04= > > 10f1cbc8, =3D > > > > cr=3D3D0xfffff80140fb1100, flag=3D3D<optimized out>, vsecp=3D3D0x0, > mnt= > > _ns=3D3D0x0) > > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vno= > > ps_o=3D > > > > s.c:1146 > > > > > #18 0xffffffff81faea57 in zfs_freebsd_create > (ap=3D3D0xfffffe0410= > > f1cda0=3D > > > > ) at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:= > > 4618 > > > > > #19 0xffffffff8111aa9a in VOP_MKNOD_APV > (vop=3D3D0xffffffff822e71= > > a0 <zf=3D > > > > s_vnodeops>, a=3D3Da@entry=3D3D0xfffffe0410f1cda0) at vnode_if.c:372 > > > > > #20 0xffffffff80c50207 in VOP_MKNOD (dvp=3D3D<unavailable>, > cnp= > > =3D3D0xfff=3D > > > > ffe0410f1cd50, vap=3D3D0xfffffe0410f1cc20, vpp=3D3D<optimized out>) > at = > > ./vnode_=3D > > > > if.h:188 > > > > > #21 kern_mkfifoat (td=3D3D0xfffffe046acd8e40, fd=3D3D-100, > path= > > =3D3D0x12772=3D > > > > f073500 <error: Cannot access memory at address 0x12772f073500>, > pathse= > > g=3D3D=3D > > > > UIO_USERSPACE, mode=3D3D<optimized out>) at > /usr/src/sys/kern/vfs_sysca= > > lls.c:=3D > > > > 1492 > > > > > #22 0xffffffff81049398 in syscallenter (td=3D3D<optimized > out>) a= > > t /usr=3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #23 amd64_ =E6=90=AC=EE=8A=80 syscall > (td=3D3D0xfffffe046acd8e= > > 40, traced=3D3D0) at /usr/src/s=3D > > > > ys/amd64/amd64/trap.c:1199 > > > > > #24 <signal handler called> > > > > > > > > > > Frame 7 is trying to acquire the ZFS teardown inactive lock, which > is > > > > > held by a process which is performing a ZFS rollback and is > waiting f= > > or > > > > > the transaction to sync: > > > > > > > > > > #0 sched_switch (td=3D3Dtd@entry=3D3D0xfffffe0422ef8560, > flags= > > =3D3Dflags@e=3D > > > > ntry=3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3Dflags@entry > =3D3D25= > > 9) at /u=3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D0xfffff8011b83= > > d540, =3D > > > > pri=3D3D0) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D<unavailable>, w= > > chan@e=3D > > > > ntry=3D3D0xfffff8011b83d540, pri=3D3D<unavailable>, pri@entry=3D3D0) > at= > > /usr/src/=3D > > > > sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80ad7f75 in _cv_wait (cvp=3D3Dcvp@entry > =3D3D0xfffff= > > 8011b83d=3D > > > > 540, lock=3D3Dlock@entry=3D3D0xfffff8011b83d4d0) at > /usr/src/sys/kern/k= > > ern_cond=3D > > > > var.c:146 > > > > > #5 0xffffffff820b42fb in txg_wait_synced_impl (dp=3D3Ddp@entry > = > > =3D3D0xfff=3D > > > > ff8011b83d000, txg=3D3D8585097, wait_sig=3D3Dwait_sig@entry=3D3D0) > at /= > > usr/src/sy=3D > > > > s/contrib/openzfs/module/zfs/txg.c:726 > > > > > #6 0xffffffff820b3cab in txg_wait_synced > (dp=3D3D<unavailable>, = > > dp@ent=3D > > > > ry=3D3D0xfffff8011b83d000, txg=3D3D<unavailable>) at > /usr/src/sys/contr= > > ib/openz=3D > > > > fs/module/zfs/txg.c:736 > > > > > #7 0xffffffff8206d5b5 in dsl_sync_task_common > (pool=3D3Dpool@ent= > > ry=3D3D0=3D > > > > xfffffe0401d15000 "zroot/poudriere/jails/13amd64-default-ref/15", > check= > > func=3D > > > > =3D3D<optimized out>, syncfunc=3D3D0xffffffff8203fbc0 > <dsl_dataset_roll= > > back_syn=3D > > > > c>, sigfunc=3D3Dsigfunc@entry=3D3D0x0, arg=3D3Darg@entry > =3D3D0xfffffe02= > > fb827a90, > > > > > blocks_modified=3D3Dblocks_modified@entry=3D3D1, > space_check= > > =3D3DZFS_SP=3D > > > > ACE_CHECK_RESERVED, early=3D3D0) at > /usr/src/sys/contrib/openzfs/module= > > /zfs/d=3D > > > > sl_synctask.c:93 > > > > > #8 0xffffffff8206d3c7 in dsl_sync_task > (pool=3D3D<unavailable>, = > > pool@e=3D > > > > ntry=3D3D0xfffffe0401d15000 > "zroot/poudriere/jails/13amd64-default-ref/= > > 15", c=3D > > > > heckfunc=3D3D<unavailable>, syncfunc=3D3D<unavailable>, > arg=3D3D<unavai= > > lable>, ar=3D > > > > g@entry=3D3D0xfffffe02fb827a90, blocks_modified=3D3D<unavailable>, > > > > > blocks_modified@entry=3D3D1, > space_check=3D3D<unavailable>, s= > > pace_che=3D > > > > ck@entry=3D3DZFS_SPACE_CHECK_RESERVED) at > /usr/src/sys/contrib/openzfs/= > > module=3D > > > > /zfs/dsl_synctask.c:132 > > > > > #9 0xffffffff8204075b in dsl_dataset_rollback > (fsname=3D3D<unava= > > ilable=3D > > > > >, fsname@entry=3D3D0xfffffe0401d15000 > "zroot/poudriere/jails/13amd64-d= > > efault=3D > > > > -ref/15", tosnap=3D3D<optimized out>, owner=3D3D<optimized out>, > result= > > =3D3Dresul=3D > > > > t@entry=3D3D0xfffff81c826a9ea0) > > > > > at > /usr/src/sys/contrib/openzfs/module/zfs/dsl_dataset.c:3261 > > > > > #10 0xffffffff82168dd9 in zfs_ioc_rollback > (fsname=3D3D0xfffffe04= > > 01d150=3D > > > > 00 "zroot/poudriere/jails/13amd64-default-ref/15", fsname@entry > =3D3D<er= > > ror re=3D > > > > ading variable: value is not available>, innvl=3D3D<unavailable>, > innvl= > > @entry=3D > > > > =3D3D<error reading variable: value is not available>, > > > > > outnvl=3D3D0xfffff81c826a9ea0, outnvl@entry=3D3D<error > readin= > > g variab=3D > > > > le: value is not available>) at > /usr/src/sys/contrib/openzfs/module/zfs= > > /zfs=3D > > > > _ioctl.c:4405 > > > > > #11 0xffffffff82164522 in zfsdev_ioctl_common > (vecnum=3D3Dvecnum@= > > entry=3D > > > > =3D3D25, zc=3D3Dzc@entry=3D3D0xfffffe0401d15000, flag=3D3Dflag@entry > =3D= > > 3D0) at /usr/s=3D > > > > rc/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:7798 > > > > > #12 0xffffffff81f97fca in zfsdev_ioctl (dev=3D3D<optimized > out>, = > > zcmd=3D > > > > =3D3D<unavailable>, zcmd@entry=3D3D<error reading variable: value > is no= > > t availa=3D > > > > ble>, arg=3D3D0xfffffe02fb827d50 "\017", arg@entry=3D3D<error > reading v= > > ariable:=3D > > > > value is not available>, flag=3D3D<optimized out>, > td=3D3D<optimized o= > > ut>) > > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/kmod_co= > > re.c=3D > > > > :168 > > > > > #13 0xffffffff809d6212 in devfs_ioctl > (ap=3D3D0xfffffe02fb827c50)= > > at /u=3D > > > > sr/src/sys/fs/devfs/devfs_vnops.c:935 > > > > > #14 0xffffffff80c585f2 in vn_ioctl (fp=3D3D0xfffff8052cdd80f0, > co= > > m=3D3D<o=3D > > > > ptimized out>, data=3D3D0xfffffe02fb827d50, > active_cred=3D3D0xfffff8012= > > 2ab1e00,=3D > > > > td=3D3D<unavailable>) at /usr/src/sys/kern/vfs_vnops.c:1704 > > > > > #15 0xffffffff809d68ee in devfs_ioctl_f (fp=3D3D<unavailable>, > fp= > > @entry=3D > > > > =3D3D<error reading variable: value is not available>, > com=3D3D<unavail= > > able>, c=3D > > > > om@entry=3D3D<error reading variable: value is not available>, > data=3D3= > > D<unavai=3D > > > > lable>, data@entry=3D3D<error reading variable: value is not > available>= > > , > > > > > cred=3D3D<unavailable>, cred@entry=3D3D<error reading > variabl= > > e: value=3D > > > > is not available>, td=3D3D<unavailable>, td@entry=3D3D<error > reading v= > > ariable:=3D > > > > value is not available>) at /usr/src/sys/fs/devfs/devfs_vnops.c:866 > > > > > #16 0xffffffff80bc57e6 in fo_ioctl (fp=3D3D0xfffff8052cdd80f0, > co= > > m=3D3D32=3D > > > > 22821401, data=3D3D<unavailable>, active_cred=3D3D<unavailable>, > td=3D3= > > D0xfffffe0=3D > > > > 422ef8560) at /usr/src/sys/sys/file.h:367 > > > > > #17 kern_ioctl (td=3D3Dtd@entry=3D3D0xfffffe0422ef8560, > fd=3D3D4,= > > com=3D3Dcom=3D > > > > @entry=3D3D3222821401, data=3D3D<unavailable>, data@entry > =3D3D0xfffffe0= > > 2fb827d50 =3D > > > > "\017") at /usr/src/sys/kern/sys_generic.c:807 > > > > > #18 0xffffffff80bc54f2 in sys_ioctl > (td=3D3D0xfffffe0422ef8560, u= > > ap=3D3D0=3D > > > > xfffffe0422ef8960) at /usr/src/sys/kern/sys_generic.c:715 > > > > > #19 0xffffffff81049398 in syscallenter (td=3D3D<optimized > out>) a= > > t /usr=3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #20 amd64_syscall (td=3D3D0xfffffe0422ef8560, traced=3D3D0) at > /u= > > sr/src/s=3D > > > > ys/amd64/amd64/trap.c:1199 > > > [...] > > > > > > The backtrace looks different though it certainly smells like > PR/271945. > > > > > > I've had similar to PR/271945 panics on an amd64 with a mirrored zpool > wi= > > th > > > four vdevs running poudriere with AMD64 jails. My other amd64 with a > > > mirrored zpool with two vdevs using i386 jails has no such issue. All > oth= > > er > > > workloads are unaffected. > > > > > > On the affected machine running poudriere bulk with -J N:1 circumvents > th= > > e > > > issue. So far. There were two openzfs cherry-picks this morning. I > intend > > > to try them against a full bulk build later today. > > > > > > > > > -- > > > Cheers, > > > Cy Schubert <Cy.Schubert@cschubert.com> > > > FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org > > > NTP: <cy@nwtime.org> Web: https://nwtime.org > > > > > > e^(i*pi)+1=3D0 > > > > > > > > > [-- Attachment #2 --] <div dir="auto">Spoke too soon still seeing zfs lockups under heavy poudriere workload after the MFVs. Regression time matches what has been reported here.</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 10, 2023 at 4:33 PM Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com">Cy.Schubert@cschubert.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">I haven't experienced any problems (yet) either.<br> <br> <br> -- <br> Cheers,<br> Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com" target="_blank">Cy.Schubert@cschubert.com</a>><br> FreeBSD UNIX: <cy@FreeBSD.org> Web: <a href="https://FreeBSD.org" rel="noreferrer" target="_blank">https://FreeBSD.org</a><br>; NTP: <<a href="mailto:cy@nwtime.org" target="_blank">cy@nwtime.org</a>> Web: <a href="https://nwtime.org" rel="noreferrer" target="_blank">https://nwtime.org</a><br>; <br> e^(i*pi)+1=0<br> <br> <br> In message <CAK7dMtDJQtaai3_6VjEkwVwW5JN6e8v=kKTOPffp371xb=ORUg@mail.gmail.c<br> om><br> , Kevin Bowling writes:<br> > The two MFVs on head have improved/fixed stability with poudriere for<br> > me 48 core bare metal.<br> ><br> > On Thu, Aug 10, 2023 at 6:37=E2=80=AFAM Cy Schubert <Cy.Schubert@cschubert.=<br> > com> wrote:<br> > ><br> > > In message <CAK7dMtDJeuf8rjWbsNEZABUfeqpjUyCHzuOL9AAhKk93sy+PKg@mail.gmai=<br> > l.c<br> > > om><br> > > , Kevin Bowling writes:<br> > > > Possibly <a href="https://github.com/openzfs/zfs/commit/2cb992a99ccadb78d97049b4=" rel="noreferrer" target="_blank">https://github.com/openzfs/zfs/commit/2cb992a99ccadb78d97049b4=</a><br>; > 0bd4=3D<br> > > > 42eb4fdc549d<br> > > ><br> > > > On Tue, Aug 8, 2023 at 10:08=3DE2=3D80=3DAFAM Dag-Erling Sm=3DC3=3DB8rg=<br> > rav <des@freeb=3D<br> > > > <a href="http://sd.org" rel="noreferrer" target="_blank">sd.org</a>> wrote:<br> > > > ><br> > > > > At some point between 42d088299c (4 May) and f0c9703301 (26 June), a<br> > > > > deadlock was introduced in ZFS. It is still present as of 9c2823bae9=<br> > (4<br> > > > > August) and is 100% reproducable just by starting poudriere bulk in a<br> > > > > 16-core VM and waiting a few hours until deadlkres kicks in. In the<br> > > > > latest instance, deadlkres complained about a bash process:<br> > > > ><br> > > > > #0 sched_switch (td=3D3Dtd@entry=3D3D0xfffffe02fb1d8000, flags=<br> > =3D3Dflags@e=3D<br> > > > ntry=3D3D259) at /usr/src/sys/kern/sched_ule.c:2299<br> > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3Dflags@entry=3D3D25=<br> > 9) at /u=3D<br> > > > sr/src/sys/kern/kern_synch.c:550<br> > > > > #2 0xffffffff80babcb4 in sleepq_switch (wchan=3D3D0xfffff818543a=<br> > 9e70, =3D<br> > > > pri=3D3D64) at /usr/src/sys/kern/subr_sleepqueue.c:609<br> > > > > #3 0xffffffff80babb8c in sleepq_wait (wchan=3D3D<unavailable>, p=<br> > ri=3D3D<=3D<br> > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660<br> > > > > #4 0xffffffff80b1c1b0 in sleeplk (lk=3D3Dlk@entry=3D3D0xfffff818=<br> > 543a9e70=3D<br> > > > , flags=3D3Dflags@entry=3D3D2121728, ilk=3D3Dilk@entry=3D3D0x0, wmesg=<br> > =3D3Dwmesg@entry=3D<br> > > > =3D3D0xffffffff8222a054 "zfs", pri=3D3D<optimized out>, pri@entry=3D3D6=<br> > 4, timo=3D3D=3D<br> > > > timo@entry=3D3D6, queue=3D3D1) at /usr/src/sys/kern/kern_lock.c:310<br> > > > > #5 0xffffffff80b1a23f in lockmgr_slock_hard (lk=3D3D0xfffff81854=<br> > 3a9e70=3D<br> > > > , flags=3D3D2121728, ilk=3D3D<optimized out>, file=3D3D0xffffffff812544=<br> > fb "/usr/s=3D<br> > > > rc/sys/kern/vfs_subr.c", line=3D3D3057, lwa=3D3D0x0) at /usr/src/sys/ke=<br> > rn/kern_=3D<br> > > > lock.c:705<br> > > > > #6 0xffffffff80c59ec3 in VOP_LOCK1 (vp=3D3D0xfffff818543a9e00, f=<br> > lags=3D<br> > > > =3D3D2105344, file=3D3D0xffffffff812544fb "/usr/src/sys/kern/vfs_subr.c=<br> > ", line=3D<br> > > > =3D3D3057) at ./vnode_if.h:1120<br> > > > > #7 _vn_lock (vp=3D3Dvp@entry=3D3D0xfffff818543a9e00, flags=3D3D2=<br> > 105344, fi=3D<br> > > > le=3D3D<unavailable>, line=3D3D<unavailable>, line@entry=3D3D3057) at /=<br> > usr/src/sy=3D<br> > > > s/kern/vfs_vnops.c:1815<br> > > > > #8 0xffffffff80c4173d in vget_finish (vp=3D3D0xfffff818543a9e00,=<br> > flags=3D<br> > > > =3D3D<unavailable>, vs=3D3Dvs@entry=3D3DVGET_USECOUNT) at /usr/src/sys/=<br> > kern/vfs_s=3D<br> > > > ubr.c:3057<br> > > > > #9 0xffffffff80c1c9b7 in cache_lookup (dvp=3D3Ddvp@entry=3D3D0xf=<br> > ffff802c=3D<br> > > > d02ac40, vpp=3D3Dvpp@entry=3D3D0xfffffe046b20ac30, cnp=3D3Dcnp@entry=3D=<br> > 3D0xfffffe04=3D<br> > > > 6b20ac58, tsp=3D3Dtsp@entry=3D3D0x0, ticksp=3D3Dticksp@entry=3D3D0x0) a=<br> > t /usr/src/s=3D<br> > > > ys/kern/vfs_cache.c:2086<br> > > > > #10 0xffffffff80c2150c in vfs_cache_lookup (ap=3D3D<optimized out=<br> > >) at =3D<br> > > > /usr/src/sys/kern/vfs_cache.c:3068<br> > > > > #11 0xffffffff80c32c37 in VOP_LOOKUP (dvp=3D3D0xfffff802cd02ac40,=<br> > vpp=3D<br> > > > =3D3D0xfffffe046b20ac30, cnp=3D3D0xfffffe046b20ac58) at ./vnode_if.h:69<br> > > > > #12 vfs_lookup (ndp=3D3Dndp@entry=3D3D0xfffffe046b20abd8) at /usr=<br> > /src/sys=3D<br> > > > /kern/vfs_lookup.c:1266<br> > > > > #13 0xffffffff80c31ce1 in namei (ndp=3D3Dndp@entry=3D3D0xfffffe04=<br> > 6b20abd8=3D<br> > > > ) at /usr/src/sys/kern/vfs_lookup.c:689<br> > > > > #14 0xffffffff80c52090 in kern_statat (td=3D3D0xfffffe02fb1d8000,=<br> > flag=3D<br> > > > =3D3D<optimized out>, fd=3D3D-100, path=3D3D0xa75b480e070 <error: Canno=<br> > t access m=3D<br> > > > emory at address 0xa75b480e070>, pathseg=3D3Dpathseg@entry=3D3DUIO_USER=<br> > SPACE, s=3D<br> > > > bp=3D3Dsbp@entry=3D3D0xfffffe046b20ad18)<br> > > > > at /usr/src/sys/kern/vfs_syscalls.c:2441<br> > > > > #15 0xffffffff80c52797 in sys_fstatat (td=3D3D<unavailable>, uap=<br> > =3D3D0xff=3D<br> > > > fffe02fb1d8400) at /usr/src/sys/kern/vfs_syscalls.c:2419<br> > > > > #16 0xffffffff81049398 in syscallenter (td=3D3D<optimized out>) a=<br> > t /usr=3D<br> > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190<br> > > > > #17 amd64_syscall (td=3D3D0xfffffe02fb1d8000, traced=3D3D0) at /u=<br> > sr/src/s=3D<br> > > > ys/amd64/amd64/trap.c:1199<br> > > > > #18 <signal handler called><br> > > > ><br> > > > > The lock it is trying to acquire in frame 5 belongs to another bash<br> > > > > process which is in the process of creating a fifo:<br> > > > ><br> > > > > #0 sched_switch (td=3D3Dtd@entry=3D3D0xfffffe046acd8e40, flags=<br> > =3D3Dflags@e=3D<br> > > > ntry=3D3D259) at /usr/src/sys/kern/sched_ule.c:2299<br> > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3Dflags@entry=3D3D25=<br> > 9) at /u=3D<br> > > > sr/src/sys/kern/kern_synch.c:550<br> > > > > #2 0xffffffff80babcb4 in sleepq_switch (wchan=3D3D0xfffff8018acb=<br> > f154, =3D<br> > > > pri=3D3D87) at /usr/src/sys/kern/subr_sleepqueue.c:609<br> > > > > #3 0xffffffff80babb8c in sleepq_wait (wchan=3D3D<unavailable>, p=<br> > ri=3D3D<=3D<br> > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660<br> > > > > #4 0xffffffff80b59606 in _sleep (ident=3D3Dident@entry=3D3D0xfff=<br> > ff8018ac=3D<br> > > > bf154, lock=3D3Dlock@entry=3D3D0xfffff8018acbf120, priority=3D3Dpriorit=<br> > y@entry=3D3D=3D<br> > > > 87, wmesg=3D3D0xffffffff8223af0e "zfs teardown inactive", sbt=3D3Dsbt@e=<br> > ntry=3D3D0=3D<br> > > > , pr=3D3Dpr@entry=3D3D0, flags=3D3D256)<br> > > > > at /usr/src/sys/kern/kern_synch.c:225<br> > > > > #5 0xffffffff80b45dc0 in rms_rlock_fallback (rms=3D3D0xfffff8018=<br> > acbf12=3D<br> > > > 0) at /usr/src/sys/kern/kern_rmlock.c:1015<br> > > > > #6 0xffffffff80b45c93 in rms_rlock (rms=3D3D<unavailable>, rms@e=<br> > ntry=3D<br> > > > =3D3D0xfffff8018acbf120) at /usr/src/sys/kern/kern_rmlock.c:1036<br> > > > > #7 0xffffffff81fb147b in zfs_freebsd_reclaim (ap=3D3D<optimized =<br> > out>) =3D<br> > > > at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:51=<br> > 64<br> > > > > #8 0xffffffff8111d245 in VOP_RECLAIM_APV (vop=3D3D0xffffffff822e=<br> > 71a0 <=3D<br> > > > zfs_vnodeops>, a=3D3Da@entry=3D3D0xfffffe0410f1c9c8) at vnode_if.c:2180<br> > > > > #9 0xffffffff80c43569 in VOP_RECLAIM (vp=3D3D0xfffff802cdbaca80)=<br> > at ./=3D<br> > > > vnode_if.h:1084<br> > > > > #10 vgonel (vp=3D3Dvp@entry=3D3D0xfffff802cdbaca80) at /usr/src/s=<br> > ys/kern/=3D<br> > > > vfs_subr.c:4143<br> > > > > #11 0xffffffff80c3ef61 in vtryrecycle (vp=3D3D0xfffff802cdbaca80)=<br> > at /u=3D<br> > > > sr/src/sys/kern/vfs_subr.c:1693<br> > > > > #12 vnlru_free_impl (count=3D3Dcount@entry=3D3D1, mnt_op=3D3Dmnt_=<br> > op@entry=3D<br> > > > =3D3D0x0, mvp=3D3D0xfffff8010864da00) at /usr/src/sys/kern/vfs_subr.c:1=<br> > 344<br> > > > > #13 0xfffff 㵶–, fff80c49553 in vnlru_free_locked (count=3D3D1) at /usr=<br> > /src/s=3D<br> > > > ys/kern/vfs_subr.c:1357<br> > > > > #14 vn_alloc_hard (mp=3D3Dmp@entry=3D3D0x0) at /usr/src/sys/kern/=<br> > vfs_subr=3D<br> > > > .c:1744<br> > > > > #15 0xffffffff80c3f6f0 in vn_alloc (mp=3D3D0x0) at /usr/src/sys/a=<br> > md64/i=3D<br> > > > nclude/atomic.h:375<br> > > > > #16 getnewvnode_reserve () at /usr/src/sys/kern/vfs_subr.c:1888<br> > > > > #17 0xffffffff81faa072 in zfs_create (dzp=3D3D0xfffff812200261d0,=<br> > name=3D<br> > > > =3D3D0xfffff8011b8ac805 "sh-np.yPbxoo", vap=3D3D0xfffffe0410f1cc20, exc=<br> > l=3D3D<opt=3D<br> > > > imized out>, mode=3D3D<optimized out>, zpp=3D3Dzpp@entry=3D3D0xfffffe04=<br> > 10f1cbc8, =3D<br> > > > cr=3D3D0xfffff80140fb1100, flag=3D3D<optimized out>, vsecp=3D3D0x0, mnt=<br> > _ns=3D3D0x0)<br> > > > > at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vno=<br> > ps_o=3D<br> > > > s.c:1146<br> > > > > #18 0xffffffff81faea57 in zfs_freebsd_create (ap=3D3D0xfffffe0410=<br> > f1cda0=3D<br> > > > ) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:=<br> > 4618<br> > > > > #19 0xffffffff8111aa9a in VOP_MKNOD_APV (vop=3D3D0xffffffff822e71=<br> > a0 <zf=3D<br> > > > s_vnodeops>, a=3D3Da@entry=3D3D0xfffffe0410f1cda0) at vnode_if.c:372<br> > > > > #20 0xffffffff80c50207 in VOP_MKNOD (dvp=3D3D<unavailable>, cnp=<br> > =3D3D0xfff=3D<br> > > > ffe0410f1cd50, vap=3D3D0xfffffe0410f1cc20, vpp=3D3D<optimized out>) at =<br> > ./vnode_=3D<br> > > > if.h:188<br> > > > > #21 kern_mkfifoat (td=3D3D0xfffffe046acd8e40, fd=3D3D-100, path=<br> > =3D3D0x12772=3D<br> > > > f073500 <error: Cannot access memory at address 0x12772f073500>, pathse=<br> > g=3D3D=3D<br> > > > UIO_USERSPACE, mode=3D3D<optimized out>) at /usr/src/sys/kern/vfs_sysca=<br> > lls.c:=3D<br> > > > 1492<br> > > > > #22 0xffffffff81049398 in syscallenter (td=3D3D<optimized out>) a=<br> > t /usr=3D<br> > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190<br> > > > > #23 amd64_ =E6=90=AC=EE=8A=80 syscall (td=3D3D0xfffffe046acd8e=<br> > 40, traced=3D3D0) at /usr/src/s=3D<br> > > > ys/amd64/amd64/trap.c:1199<br> > > > > #24 <signal handler called><br> > > > ><br> > > > > Frame 7 is trying to acquire the ZFS teardown inactive lock, which is<br> > > > > held by a process which is performing a ZFS rollback and is waiting f=<br> > or<br> > > > > the transaction to sync:<br> > > > ><br> > > > > #0 sched_switch (td=3D3Dtd@entry=3D3D0xfffffe0422ef8560, flags=<br> > =3D3Dflags@e=3D<br> > > > ntry=3D3D259) at /usr/src/sys/kern/sched_ule.c:2299<br> > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3Dflags@entry=3D3D25=<br> > 9) at /u=3D<br> > > > sr/src/sys/kern/kern_synch.c:550<br> > > > > #2 0xffffffff80babcb4 in sleepq_switch (wchan=3D3D0xfffff8011b83=<br> > d540, =3D<br> > > > pri=3D3D0) at /usr/src/sys/kern/subr_sleepqueue.c:609<br> > > > > #3 0xffffffff80babb8c in sleepq_wait (wchan=3D3D<unavailable>, w=<br> > chan@e=3D<br> > > > ntry=3D3D0xfffff8011b83d540, pri=3D3D<unavailable>, pri@entry=3D3D0) at=<br> > /usr/src/=3D<br> > > > sys/kern/subr_sleepqueue.c:660<br> > > > > #4 0xffffffff80ad7f75 in _cv_wait (cvp=3D3Dcvp@entry=3D3D0xfffff=<br> > 8011b83d=3D<br> > > > 540, lock=3D3Dlock@entry=3D3D0xfffff8011b83d4d0) at /usr/src/sys/kern/k=<br> > ern_cond=3D<br> > > > var.c:146<br> > > > > #5 0xffffffff820b42fb in txg_wait_synced_impl (dp=3D3Ddp@entry=<br> > =3D3D0xfff=3D<br> > > > ff8011b83d000, txg=3D3D8585097, wait_sig=3D3Dwait_sig@entry=3D3D0) at /=<br> > usr/src/sy=3D<br> > > > s/contrib/openzfs/module/zfs/txg.c:726<br> > > > > #6 0xffffffff820b3cab in txg_wait_synced (dp=3D3D<unavailable>, =<br> > dp@ent=3D<br> > > > ry=3D3D0xfffff8011b83d000, txg=3D3D<unavailable>) at /usr/src/sys/contr=<br> > ib/openz=3D<br> > > > fs/module/zfs/txg.c:736<br> > > > > #7 0xffffffff8206d5b5 in dsl_sync_task_common (pool=3D3Dpool@ent=<br> > ry=3D3D0=3D<br> > > > xfffffe0401d15000 "zroot/poudriere/jails/13amd64-default-ref/15", check=<br> > func=3D<br> > > > =3D3D<optimized out>, syncfunc=3D3D0xffffffff8203fbc0 <dsl_dataset_roll=<br> > back_syn=3D<br> > > > c>, sigfunc=3D3Dsigfunc@entry=3D3D0x0, arg=3D3Darg@entry=3D3D0xfffffe02=<br> > fb827a90,<br> > > > > blocks_modified=3D3Dblocks_modified@entry=3D3D1, space_check=<br> > =3D3DZFS_SP=3D<br> > > > ACE_CHECK_RESERVED, early=3D3D0) at /usr/src/sys/contrib/openzfs/module=<br> > /zfs/d=3D<br> > > > sl_synctask.c:93<br> > > > > #8 0xffffffff8206d3c7 in dsl_sync_task (pool=3D3D<unavailable>, =<br> > pool@e=3D<br> > > > ntry=3D3D0xfffffe0401d15000 "zroot/poudriere/jails/13amd64-default-ref/=<br> > 15", c=3D<br> > > > heckfunc=3D3D<unavailable>, syncfunc=3D3D<unavailable>, arg=3D3D<unavai=<br> > lable>, ar=3D<br> > > > g@entry=3D3D0xfffffe02fb827a90, blocks_modified=3D3D<unavailable>,<br> > > > > blocks_modified@entry=3D3D1, space_check=3D3D<unavailable>, s=<br> > pace_che=3D<br> > > > ck@entry=3D3DZFS_SPACE_CHECK_RESERVED) at /usr/src/sys/contrib/openzfs/=<br> > module=3D<br> > > > /zfs/dsl_synctask.c:132<br> > > > > #9 0xffffffff8204075b in dsl_dataset_rollback (fsname=3D3D<unava=<br> > ilable=3D<br> > > > >, fsname@entry=3D3D0xfffffe0401d15000 "zroot/poudriere/jails/13amd64-d=<br> > efault=3D<br> > > > -ref/15", tosnap=3D3D<optimized out>, owner=3D3D<optimized out>, result=<br> > =3D3Dresul=3D<br> > > > t@entry=3D3D0xfffff81c826a9ea0)<br> > > > > at /usr/src/sys/contrib/openzfs/module/zfs/dsl_dataset.c:3261<br> > > > > #10 0xffffffff82168dd9 in zfs_ioc_rollback (fsname=3D3D0xfffffe04=<br> > 01d150=3D<br> > > > 00 "zroot/poudriere/jails/13amd64-default-ref/15", fsname@entry=3D3D<er=<br> > ror re=3D<br> > > > ading variable: value is not available>, innvl=3D3D<unavailable>, innvl=<br> > @entry=3D<br> > > > =3D3D<error reading variable: value is not available>,<br> > > > > outnvl=3D3D0xfffff81c826a9ea0, outnvl@entry=3D3D<error readin=<br> > g variab=3D<br> > > > le: value is not available>) at /usr/src/sys/contrib/openzfs/module/zfs=<br> > /zfs=3D<br> > > > _ioctl.c:4405<br> > > > > #11 0xffffffff82164522 in zfsdev_ioctl_common (vecnum=3D3Dvecnum@=<br> > entry=3D<br> > > > =3D3D25, zc=3D3Dzc@entry=3D3D0xfffffe0401d15000, flag=3D3Dflag@entry=3D=<br> > 3D0) at /usr/s=3D<br> > > > rc/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:7798<br> > > > > #12 0xffffffff81f97fca in zfsdev_ioctl (dev=3D3D<optimized out>, =<br> > zcmd=3D<br> > > > =3D3D<unavailable>, zcmd@entry=3D3D<error reading variable: value is no=<br> > t availa=3D<br> > > > ble>, arg=3D3D0xfffffe02fb827d50 "\017", arg@entry=3D3D<error reading v=<br> > ariable:=3D<br> > > > value is not available>, flag=3D3D<optimized out>, td=3D3D<optimized o=<br> > ut>)<br> > > > > at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/kmod_co=<br> > re.c=3D<br> > > > :168<br> > > > > #13 0xffffffff809d6212 in devfs_ioctl (ap=3D3D0xfffffe02fb827c50)=<br> > at /u=3D<br> > > > sr/src/sys/fs/devfs/devfs_vnops.c:935<br> > > > > #14 0xffffffff80c585f2 in vn_ioctl (fp=3D3D0xfffff8052cdd80f0, co=<br> > m=3D3D<o=3D<br> > > > ptimized out>, data=3D3D0xfffffe02fb827d50, active_cred=3D3D0xfffff8012=<br> > 2ab1e00,=3D<br> > > > td=3D3D<unavailable>) at /usr/src/sys/kern/vfs_vnops.c:1704<br> > > > > #15 0xffffffff809d68ee in devfs_ioctl_f (fp=3D3D<unavailable>, fp=<br> > @entry=3D<br> > > > =3D3D<error reading variable: value is not available>, com=3D3D<unavail=<br> > able>, c=3D<br> > > > om@entry=3D3D<error reading variable: value is not available>, data=3D3=<br> > D<unavai=3D<br> > > > lable>, data@entry=3D3D<error reading variable: value is not available>=<br> > ,<br> > > > > cred=3D3D<unavailable>, cred@entry=3D3D<error reading variabl=<br> > e: value=3D<br> > > > is not available>, td=3D3D<unavailable>, td@entry=3D3D<error reading v=<br> > ariable:=3D<br> > > > value is not available>) at /usr/src/sys/fs/devfs/devfs_vnops.c:866<br> > > > > #16 0xffffffff80bc57e6 in fo_ioctl (fp=3D3D0xfffff8052cdd80f0, co=<br> > m=3D3D32=3D<br> > > > 22821401, data=3D3D<unavailable>, active_cred=3D3D<unavailable>, td=3D3=<br> > D0xfffffe0=3D<br> > > > 422ef8560) at /usr/src/sys/sys/file.h:367<br> > > > > #17 kern_ioctl (td=3D3Dtd@entry=3D3D0xfffffe0422ef8560, fd=3D3D4,=<br> > com=3D3Dcom=3D<br> > > > @entry=3D3D3222821401, data=3D3D<unavailable>, data@entry=3D3D0xfffffe0=<br> > 2fb827d50 =3D<br> > > > "\017") at /usr/src/sys/kern/sys_generic.c:807<br> > > > > #18 0xffffffff80bc54f2 in sys_ioctl (td=3D3D0xfffffe0422ef8560, u=<br> > ap=3D3D0=3D<br> > > > xfffffe0422ef8960) at /usr/src/sys/kern/sys_generic.c:715<br> > > > > #19 0xffffffff81049398 in syscallenter (td=3D3D<optimized out>) a=<br> > t /usr=3D<br> > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190<br> > > > > #20 amd64_syscall (td=3D3D0xfffffe0422ef8560, traced=3D3D0) at /u=<br> > sr/src/s=3D<br> > > > ys/amd64/amd64/trap.c:1199<br> > > [...]<br> > ><br> > > The backtrace looks different though it certainly smells like PR/271945.<br> > ><br> > > I've had similar to PR/271945 panics on an amd64 with a mirrored zpool wi=<br> > th<br> > > four vdevs running poudriere with AMD64 jails. My other amd64 with a<br> > > mirrored zpool with two vdevs using i386 jails has no such issue. All oth=<br> > er<br> > > workloads are unaffected.<br> > ><br> > > On the affected machine running poudriere bulk with -J N:1 circumvents th=<br> > e<br> > > issue. So far. There were two openzfs cherry-picks this morning. I intend<br> > > to try them against a full bulk build later today.<br> > ><br> > ><br> > > --<br> > > Cheers,<br> > > Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com" target="_blank">Cy.Schubert@cschubert.com</a>><br> > > FreeBSD UNIX: <cy@FreeBSD.org> Web: <a href="https://FreeBSD.org" rel="noreferrer" target="_blank">https://FreeBSD.org</a><br>; > > NTP: <<a href="mailto:cy@nwtime.org" target="_blank">cy@nwtime.org</a>> Web: <a href="https://nwtime.org" rel="noreferrer" target="_blank">https://nwtime.org</a><br>; > ><br> > > e^(i*pi)+1=3D0<br> > ><br> > ><br> <br> <br> </blockquote></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAK7dMtDZP9gZcxvo2QPxHSVXOoLyaDsUTmTA-yeTTQuNfmm-kA>