Date: Mon, 21 Nov 2022 15:33:36 +0100 From: Kristof Provost <kp@FreeBSD.org> To: =?utf-8?q?Corvin_K=C3=B6hne?= <corvink@FreeBSD.org> Cc: John Baldwin <jhb@FreeBSD.org>, src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: Re: git: c0f35dbf19c3 - main - vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs. Message-ID: <8EB1182A-8493-47F1-ACC3-F78685CD2846@FreeBSD.org> In-Reply-To: <39b67bfb-6533-74ef-2bdb-782fb162951f@FreeBSD.org> References: <202211181826.2AIIQssl030757@gitrepo.freebsd.org> <B7E34E6F-924F-412E-88CC-ED4A65936001@FreeBSD.org> <39b67bfb-6533-74ef-2bdb-782fb162951f@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 21 Nov 2022, at 15:18, Corvin K=C3=B6hne wrote: > On 11/21/2022 2:15 PM, Kristof Provost wrote: >> >> On 18 Nov 2022, at 19:26, John Baldwin wrote: >> >> The branch main has been updated by jhb: >> >> URL: >> https://cgit.FreeBSD.org/src/commit/?id=3Dc0f35dbf19c3c8825bd2b321d= 8efd582807d1940 >> <https://cgit.FreeBSD.org/src/commit/?id=3Dc0f35dbf19c3c8825bd2b32= 1d8efd582807d1940> >> >> commit c0f35dbf19c3c8825bd2b321d8efd582807d1940 >> Author: John Baldwin <jhb@FreeBSD.org> >> AuthorDate: 2022-11-18 18:05:10 +0000 >> Commit: John Baldwin <jhb@FreeBSD.org> >> CommitDate: 2022-11-18 18:25:38 +0000 >> >> vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs. >> >> Retire the boot_state member of struct vlapic and instead use a >> cpuset >> in the VM to track vCPUs waiting for STARTUP IPIs. INIT IPIs add >> vCPUs to this set, and STARTUP IPIs remove vCPUs from the set. >> STARTUP IPIs are only reported to userland for vCPUs that were >> removed >> from the set. >> >> In particular, this permits a subsequent change to allocate vCPUs = on >> demand when the vCPU may not be allocated until after a STARTUP >> IPI is >> reported to userland. >> >> Reviewed by: corvink, markj >> Differential Revision: https://reviews.freebsd.org/D37173 >> --- >> sys/amd64/include/vmm.h | 3 +++ >> sys/amd64/vmm/io/vlapic.c | 46 >> ++++++++++-------------------------------- >> sys/amd64/vmm/io/vlapic_priv.h | 7 ------- >> sys/amd64/vmm/vmm.c | 27 +++++++++++++++++++++++++ >> 4 files changed, 41 insertions(+), 42 deletions(-) >> >> >> I=E2=80=99m seeing a panic starting a bhyve guest, and I think this co= mmit might be the responsible one: >> >> |login: panic: acquiring blockable sleep lock with spinlock or critica= l section held (sleep mutex) vm rendezvous lock @ /usr/src/sys/amd64/vmm/= vmm.c:2508 cpuid =3D 14 time =3D 1669035212 KDB: stack backtrace: db_trac= e_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe015f2dd530 v= panic() at vpanic+0x151/frame 0xfffffe015f2dd580 panic() at panic+0x43/fr= ame 0xfffffe015f2dd5e0 witness_checkorder() at witness_checkorder+0xd3e/f= rame 0xfffffe015f2dd7a0 __mtx_lock_flags() at __mtx_lock_flags+0x94/frame= 0xfffffe015f2dd7f0 vm_start_cpus() at vm_start_cpus+0x31/frame 0xfffffe0= 15f2dd820 vlapic_icrlo_write_handler() at vlapic_icrlo_write_handler+0x30= c/frame 0xfffffe015f2dd8a0 vmx_run() at vmx_run+0x20ed/frame 0xfffffe015f= 2dd9c0 vm_run() at vm_run+0x1d2/frame 0xfffffe015f2ddaa0 vmmdev_ioctl() a= t vmmdev_ioctl+0x644/frame 0xfffffe015f2ddb40 devfs_ioctl() at devfs_ioct= l+0xcd/frame 0xfffffe015f2ddb90 vn_ioctl() at vn_ioctl+0x131/frame 0xffff= fe015f2ddca0 devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe015f2ddc= c0 kern_ioctl() at kern_ioctl+0x202/frame 0xfffffe015f2ddd30 sys_ioctl() = at sys_ioctl+0x12a/frame 0xfffffe015f2dde00 amd64_syscall() at amd64_sysc= all+0x12e/frame 0xfffffe015f2ddf30 fast_syscall_common() at fast_syscall_= common+0xf8/frame 0xfffffe015f2ddf30 --- syscall (54, FreeBSD ELF64, ioct= l), rip =3D 0x3dd9cbd7f94a, rsp =3D 0x3ddc1d44ae58, rbp =3D 0x3ddc1d44af1= 0 --- | >> >> And kgdb=E2=80=99s backtrace: >> >> |(kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:= 59 #1 dump_savectx () at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xfffff= fff80bec678 in dumpsys (di=3D0x0) at /usr/src/sys/x86/include/dump.h:87 #= 3 doadump (textdump=3Dtextdump@entry=3D0) at /usr/src/sys/kern/kern_shutd= own.c:434 #4 0xffffffff804b403a in db_dump (dummy=3D<optimized out>, dumm= y2=3D<unavailable>, dummy3=3D<unavailable>, dummy4=3D<unavailable>) at /u= sr/src/sys/ddb/db_command.c:593 #5 0xffffffff804b3e40 in db_command (last= _cmdp=3D<optimized out>, cmd_table=3D<optimized out>, dopager=3Dtrue) at = /usr/src/sys/ddb/db_command.c:506 #6 0xffffffff804b3b0d in db_command_loo= p () at /usr/src/sys/ddb/db_command.c:553 #7 0xffffffff804b71a6 in db_tra= p (type=3D<optimized out>, code=3D<optimized out>) at /usr/src/sys/ddb/db= _main.c:270 #8 0xffffffff80c3b89e in kdb_trap (type=3Dtype@entry=3D3, cod= e=3D<unavailable>, code@entry=3D0, tf=3Dtf@entry=3D0xfffffe015f2dd470) at= /usr/src/sys/kern/subr_kdb.c:745 #9 0xffffffff810ce577 in trap (frame=3D= 0xfffffe015f2dd470) at /usr/src/sys/amd64/amd64/trap.c:611 #10 <signal ha= ndler called> #11 kdb_enter (why=3D<optimized out>, msg=3D<optimized out>= ) at /usr/src/sys/kern/subr_kdb.c:509 #12 0xffffffff80bec822 in vpanic (f= mt=3D<optimized out>, ap=3Dap@entry=3D0xfffffe015f2dd5c0) at /usr/src/sys= /kern/kern_shutdown.c:967 #13 0xffffffff80bec5c3 in panic (fmt=3D0xffffff= ff81e8ce70 <cnputs_mtx> "\314:)\201\377\377\377\377") at /usr/src/sys/ker= n/kern_shutdown.c:903 #14 0xffffffff80c5ea4e in witness_checkorder (lock=3D= 0xfffff804d585d138, flags=3D<optimized out>, file=3D<optimized out>, line= =3D2508, interlock=3D<optimized out>) at /usr/src/sys/kern/subr_witness.c= :1202 #15 0xffffffff80bc6d04 in __mtx_lock_flags (c=3D0xfffff804d585d150,= opts=3D0, file=3D0xffffffff8322542d "/usr/src/sys/amd64/vmm/vmm.c", line= =3D2508) at /usr/src/sys/kern/kern_mutex.c:278 #16 0xffffffff83204101 in = vm_start_cpus (vm=3D0xfffff804d585d000, tostart=3Dtostart@entry=3D0xfffff= e015f2dd858) at /usr/src/sys/amd64/vmm/vmm.c:2508 #17 0xffffffff83211b5c = in vlapic_icrlo_write_handler (vlapic=3Dvlapic@entry=3D0xfffff8048dc14a80= , retu=3Dretu@entry=3D0xfffffe015f2dd950) at /usr/src/sys/amd64/vmm/io/vl= apic.c:1172 #18 0xffffffff8321977d in vmx_handle_apic_write (vcpu=3D0xfff= ff8048db3ac00, vlapic=3D0xfffff8048dc14a80, qual=3D768) at /usr/src/sys/a= md64/vmm/intel/vmx.c:2184 #19 vmx_exit_process (vmx=3D0xfffff804d585b000,= vcpu=3D0xfffff8048db3ac00, vmexit=3D0xfffff804a7c1a688) at /usr/src/sys/= amd64/vmm/intel/vmx.c:2767 #20 vmx_run (vcpui=3D0xfffff8048db3ac00, rip=3D= <optimized out>, pmap=3D0xfffffe003dce1530, evinfo=3D0xfffffe015f2dd9d8) = at /usr/src/sys/amd64/vmm/intel/vmx.c:3174 #21 0xffffffff83202572 in vm_r= un (vcpu=3Dvcpu@entry=3D0xfffff804a7c1a600, vme_user=3Dvme_user@entry=3D0= xfffff80032601b08) at /usr/src/sys/amd64/vmm/vmm.c:1873 #22 0xffffffff832= 05ab4 in vmmdev_ioctl (cdev=3D<optimized out>, cmd=3D3230692865, data=3D<= optimized out>, fflag=3D<optimized out>, td=3D<optimized out>) at /usr/sr= c/sys/amd64/vmm/vmm_dev.c:565 #23 0xffffffff80a7be7d in devfs_ioctl (ap=3D= 0xfffffe015f2ddba8) at /usr/src/sys/fs/devfs/devfs_vnops.c:933 #24 0xffff= ffff80cf6421 in vn_ioctl (fp=3D0xfffff8000ce2cb40, com=3D<optimized out>,= data=3D0xfffff80032601b00, active_cred=3D0xfffff8000c4d0800, td=3D0x10) = at /usr/src/sys/kern/vfs_vnops.c:1699 #25 0xffffffff80a7c52e in devfs_ioc= tl_f (fp=3D0xffffffff81e8ce70 <cnputs_mtx>, com=3D0, data=3D0xffffffff812= a0000, cred=3D0x1, td=3D0x10) at /usr/src/sys/fs/devfs/devfs_vnops.c:864 = #26 0xffffffff80c64672 in fo_ioctl (fp=3D0xfffff8000ce2cb40, com=3D323069= 2865, data=3D0x1d0, active_cred=3D0x1, td=3D<optimized out>) at /usr/src/= sys/sys/file.h:365 #27 kern_ioctl (td=3Dtd@entry=3D0xfffffe0160936000, fd= =3D<optimized out>, com=3Dcom@entry=3D3230692865, data=3D0x1d0 <error: Ca= nnot access memory at address 0x1d0>, data@entry=3D0xfffff80032601b00 "")= at /usr/src/sys/kern/sys_generic.c:803 #28 0xffffffff80c643ba in sys_ioc= tl (td=3D0xfffffe0160936000, uap=3D0xfffffe01609363f8) at /usr/src/sys/ke= rn/sys_generic.c:711 #29 0xffffffff810cf3be in syscallenter (td=3D<optimi= zed out>) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189 #30 a= md64_syscall (td=3D0xfffffe0160936000, traced=3D0) at /usr/src/sys/amd64/= amd64/trap.c:1200 #31 <signal handler called> #32 0x00003dd9cbd7f94a in ?= ? () Backtrace stopped: Cannot access memory at address 0x3ddc1d44ae58 (k= gdb) fr 16 #16 0xffffffff83204101 in vm_start_cpus (vm=3D0xfffff804d585d0= 00, tostart=3Dtostart@entry=3D0xfffffe015f2dd858) at /usr/src/sys/amd64/v= mm/vmm.c:2508 2508 mtx_lock(&vm->rendezvous_mtx); | >> >> I believe WITNESS is upset that we=E2=80=99re going to potentially sle= ep doing mtx_lock(&vm->rendezvous_mtx); in vm_start_cpus() when we=E2=80=99= ve done critical_enter() in vm_run(). >> >> Best regards, >> Kristof >> > > Can you please test https://reviews.freebsd.org/D37452? > The VM boots correctly with that patch. Best regards, Kristof
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8EB1182A-8493-47F1-ACC3-F78685CD2846>