From nobody Mon Nov 21 14:33:36 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NG8yb1Brnz4j9sJ; Mon, 21 Nov 2022 14:33:43 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NG8yb0cRmz4HTQ; Mon, 21 Nov 2022 14:33:43 +0000 (UTC) (envelope-from kp@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1669041223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C4Qv4qWIGpS9niqJyVtpDkwTqFr/gwtVSoou5OdJBlk=; b=AtbYBPsJGCiBAQ8GnMOBQeopiiO50VlIoNEZPEWadkk4pXQOpb34YsrMzFHfUpuBOoIW67 y0+exDD0wKw3O4foXcD8lZMfEfn22wDqvTyU7c2vuP2W4tvSNRRKPWnazW1acrH7iRlB8q dlqDOsk8okrumUdIwGUtDSvBEkpHQ+CGGrC66xZm937UXfUwIK/Fj0GDFY1BoAFyD36Tq1 ouOGjYeJGVYUV/H9PWHHNsFAqH56G6AGbY6kJOBZVRWu145nyW+BBZgIVqyPO9pJz6DvFb R71OYnwO25P1cIStY5u6Jtgb6uyiPOX+g7XX+pRxbIAe2Yz2K2keGjBhvOykuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1669041223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C4Qv4qWIGpS9niqJyVtpDkwTqFr/gwtVSoou5OdJBlk=; b=tbGJx3FdLLsX9GMVUgR/Xq0q8e6kEVRLSfksxaB777uhgmoJxMi31VM6Tk3BMVhPhz6L0B l5soKA4PJ7v0w88bzzWsXI3gYQ3fQmD+KtnCyPsf5DMHB6dxiNnU62VykgOulzbUoAko1U T7kjcaci0wHn71tuhlDDd/+jvDmep9Rkm06pjppPkk9DyE8BAdZIbPet05dW7DrBFsSFVs N60+Ef2O0ucmHzcaTAYHyNlNzV7HOqNW1Vc+qXdIIb3Gahv8q1NMXANRLTHwCrT0NxdZAU 37+reSkDLeBY6Wf1FfkiaMv93cnxNf0ZtaCoXYB4uXCO1hW8tQ730Jv/XKu1fw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1669041223; a=rsa-sha256; cv=none; b=k1sweppvZGEM1N+54nOq+vQpUjMXVd/kyVLRe4fZA+iC/VL20tIk+bE6UTjsA1H73Yi3yN DozbNiN51vXK+P/Qn979/QhiZMGpuOw8+DYPQFIc/dJydL8JxIzCQzyY12cYRmib6MtKsQ gxRPx3X2TnON0vzifvLvpiLrLRC2dy/myhtD69ErwDYz7MpITef31WkAK+Jb/1+wu/QDk8 nIyt49qvnWAyIWv0H2Cck010ZNto7YUM9zx4FcloMMCAVDnQV1CIP1CFOtcnAliuFodYoM cbXtIJKhcsImkv61XhuQHwFCSKy7iVNCnQtCmrcV3r/dBY3mT6TUuvuK98oOkg== Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.codepro.be", Issuer "R3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NG8yZ55h6z1Fll; Mon, 21 Nov 2022 14:33:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 935EE44FF1; Mon, 21 Nov 2022 15:33:40 +0100 (CET) From: Kristof Provost To: =?utf-8?q?Corvin_K=C3=B6hne?= Cc: John Baldwin , src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: Re: git: c0f35dbf19c3 - main - vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs. Date: Mon, 21 Nov 2022 15:33:36 +0100 X-Mailer: MailMate (1.14r5918) Message-ID: <8EB1182A-8493-47F1-ACC3-F78685CD2846@FreeBSD.org> In-Reply-To: <39b67bfb-6533-74ef-2bdb-782fb162951f@FreeBSD.org> References: <202211181826.2AIIQssl030757@gitrepo.freebsd.org> <39b67bfb-6533-74ef-2bdb-782fb162951f@FreeBSD.org> List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-ThisMailContainsUnwantedMimeParts: N On 21 Nov 2022, at 15:18, Corvin K=C3=B6hne wrote: > On 11/21/2022 2:15 PM, Kristof Provost wrote: >> >> On 18 Nov 2022, at 19:26, John Baldwin wrote: >> >> The branch main has been updated by jhb: >> >> URL: >> https://cgit.FreeBSD.org/src/commit/?id=3Dc0f35dbf19c3c8825bd2b321d= 8efd582807d1940 >> >> >> commit c0f35dbf19c3c8825bd2b321d8efd582807d1940 >> Author: John Baldwin >> AuthorDate: 2022-11-18 18:05:10 +0000 >> Commit: John Baldwin >> CommitDate: 2022-11-18 18:25:38 +0000 >> >> vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs. >> >> Retire the boot_state member of struct vlapic and instead use a >> cpuset >> in the VM to track vCPUs waiting for STARTUP IPIs. INIT IPIs add >> vCPUs to this set, and STARTUP IPIs remove vCPUs from the set. >> STARTUP IPIs are only reported to userland for vCPUs that were >> removed >> from the set. >> >> In particular, this permits a subsequent change to allocate vCPUs = on >> demand when the vCPU may not be allocated until after a STARTUP >> IPI is >> reported to userland. >> >> Reviewed by: corvink, markj >> Differential Revision: https://reviews.freebsd.org/D37173 >> --- >> sys/amd64/include/vmm.h | 3 +++ >> sys/amd64/vmm/io/vlapic.c | 46 >> ++++++++++-------------------------------- >> sys/amd64/vmm/io/vlapic_priv.h | 7 ------- >> sys/amd64/vmm/vmm.c | 27 +++++++++++++++++++++++++ >> 4 files changed, 41 insertions(+), 42 deletions(-) >> >> >> I=E2=80=99m seeing a panic starting a bhyve guest, and I think this co= mmit might be the responsible one: >> >> |login: panic: acquiring blockable sleep lock with spinlock or critica= l section held (sleep mutex) vm rendezvous lock @ /usr/src/sys/amd64/vmm/= vmm.c:2508 cpuid =3D 14 time =3D 1669035212 KDB: stack backtrace: db_trac= e_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe015f2dd530 v= panic() at vpanic+0x151/frame 0xfffffe015f2dd580 panic() at panic+0x43/fr= ame 0xfffffe015f2dd5e0 witness_checkorder() at witness_checkorder+0xd3e/f= rame 0xfffffe015f2dd7a0 __mtx_lock_flags() at __mtx_lock_flags+0x94/frame= 0xfffffe015f2dd7f0 vm_start_cpus() at vm_start_cpus+0x31/frame 0xfffffe0= 15f2dd820 vlapic_icrlo_write_handler() at vlapic_icrlo_write_handler+0x30= c/frame 0xfffffe015f2dd8a0 vmx_run() at vmx_run+0x20ed/frame 0xfffffe015f= 2dd9c0 vm_run() at vm_run+0x1d2/frame 0xfffffe015f2ddaa0 vmmdev_ioctl() a= t vmmdev_ioctl+0x644/frame 0xfffffe015f2ddb40 devfs_ioctl() at devfs_ioct= l+0xcd/frame 0xfffffe015f2ddb90 vn_ioctl() at vn_ioctl+0x131/frame 0xffff= fe015f2ddca0 devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe015f2ddc= c0 kern_ioctl() at kern_ioctl+0x202/frame 0xfffffe015f2ddd30 sys_ioctl() = at sys_ioctl+0x12a/frame 0xfffffe015f2dde00 amd64_syscall() at amd64_sysc= all+0x12e/frame 0xfffffe015f2ddf30 fast_syscall_common() at fast_syscall_= common+0xf8/frame 0xfffffe015f2ddf30 --- syscall (54, FreeBSD ELF64, ioct= l), rip =3D 0x3dd9cbd7f94a, rsp =3D 0x3ddc1d44ae58, rbp =3D 0x3ddc1d44af1= 0 --- | >> >> And kgdb=E2=80=99s backtrace: >> >> |(kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:= 59 #1 dump_savectx () at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xfffff= fff80bec678 in dumpsys (di=3D0x0) at /usr/src/sys/x86/include/dump.h:87 #= 3 doadump (textdump=3Dtextdump@entry=3D0) at /usr/src/sys/kern/kern_shutd= own.c:434 #4 0xffffffff804b403a in db_dump (dummy=3D, dumm= y2=3D, dummy3=3D, dummy4=3D) at /u= sr/src/sys/ddb/db_command.c:593 #5 0xffffffff804b3e40 in db_command (last= _cmdp=3D, cmd_table=3D, dopager=3Dtrue) at = /usr/src/sys/ddb/db_command.c:506 #6 0xffffffff804b3b0d in db_command_loo= p () at /usr/src/sys/ddb/db_command.c:553 #7 0xffffffff804b71a6 in db_tra= p (type=3D, code=3D) at /usr/src/sys/ddb/db= _main.c:270 #8 0xffffffff80c3b89e in kdb_trap (type=3Dtype@entry=3D3, cod= e=3D, code@entry=3D0, tf=3Dtf@entry=3D0xfffffe015f2dd470) at= /usr/src/sys/kern/subr_kdb.c:745 #9 0xffffffff810ce577 in trap (frame=3D= 0xfffffe015f2dd470) at /usr/src/sys/amd64/amd64/trap.c:611 #10 #11 kdb_enter (why=3D, msg=3D= ) at /usr/src/sys/kern/subr_kdb.c:509 #12 0xffffffff80bec822 in vpanic (f= mt=3D, ap=3Dap@entry=3D0xfffffe015f2dd5c0) at /usr/src/sys= /kern/kern_shutdown.c:967 #13 0xffffffff80bec5c3 in panic (fmt=3D0xffffff= ff81e8ce70 "\314:)\201\377\377\377\377") at /usr/src/sys/ker= n/kern_shutdown.c:903 #14 0xffffffff80c5ea4e in witness_checkorder (lock=3D= 0xfffff804d585d138, flags=3D, file=3D, line= =3D2508, interlock=3D) at /usr/src/sys/kern/subr_witness.c= :1202 #15 0xffffffff80bc6d04 in __mtx_lock_flags (c=3D0xfffff804d585d150,= opts=3D0, file=3D0xffffffff8322542d "/usr/src/sys/amd64/vmm/vmm.c", line= =3D2508) at /usr/src/sys/kern/kern_mutex.c:278 #16 0xffffffff83204101 in = vm_start_cpus (vm=3D0xfffff804d585d000, tostart=3Dtostart@entry=3D0xfffff= e015f2dd858) at /usr/src/sys/amd64/vmm/vmm.c:2508 #17 0xffffffff83211b5c = in vlapic_icrlo_write_handler (vlapic=3Dvlapic@entry=3D0xfffff8048dc14a80= , retu=3Dretu@entry=3D0xfffffe015f2dd950) at /usr/src/sys/amd64/vmm/io/vl= apic.c:1172 #18 0xffffffff8321977d in vmx_handle_apic_write (vcpu=3D0xfff= ff8048db3ac00, vlapic=3D0xfffff8048dc14a80, qual=3D768) at /usr/src/sys/a= md64/vmm/intel/vmx.c:2184 #19 vmx_exit_process (vmx=3D0xfffff804d585b000,= vcpu=3D0xfffff8048db3ac00, vmexit=3D0xfffff804a7c1a688) at /usr/src/sys/= amd64/vmm/intel/vmx.c:2767 #20 vmx_run (vcpui=3D0xfffff8048db3ac00, rip=3D= , pmap=3D0xfffffe003dce1530, evinfo=3D0xfffffe015f2dd9d8) = at /usr/src/sys/amd64/vmm/intel/vmx.c:3174 #21 0xffffffff83202572 in vm_r= un (vcpu=3Dvcpu@entry=3D0xfffff804a7c1a600, vme_user=3Dvme_user@entry=3D0= xfffff80032601b08) at /usr/src/sys/amd64/vmm/vmm.c:1873 #22 0xffffffff832= 05ab4 in vmmdev_ioctl (cdev=3D, cmd=3D3230692865, data=3D<= optimized out>, fflag=3D, td=3D) at /usr/sr= c/sys/amd64/vmm/vmm_dev.c:565 #23 0xffffffff80a7be7d in devfs_ioctl (ap=3D= 0xfffffe015f2ddba8) at /usr/src/sys/fs/devfs/devfs_vnops.c:933 #24 0xffff= ffff80cf6421 in vn_ioctl (fp=3D0xfffff8000ce2cb40, com=3D,= data=3D0xfffff80032601b00, active_cred=3D0xfffff8000c4d0800, td=3D0x10) = at /usr/src/sys/kern/vfs_vnops.c:1699 #25 0xffffffff80a7c52e in devfs_ioc= tl_f (fp=3D0xffffffff81e8ce70 , com=3D0, data=3D0xffffffff812= a0000, cred=3D0x1, td=3D0x10) at /usr/src/sys/fs/devfs/devfs_vnops.c:864 = #26 0xffffffff80c64672 in fo_ioctl (fp=3D0xfffff8000ce2cb40, com=3D323069= 2865, data=3D0x1d0, active_cred=3D0x1, td=3D) at /usr/src/= sys/sys/file.h:365 #27 kern_ioctl (td=3Dtd@entry=3D0xfffffe0160936000, fd= =3D, com=3Dcom@entry=3D3230692865, data=3D0x1d0 , data@entry=3D0xfffff80032601b00 "")= at /usr/src/sys/kern/sys_generic.c:803 #28 0xffffffff80c643ba in sys_ioc= tl (td=3D0xfffffe0160936000, uap=3D0xfffffe01609363f8) at /usr/src/sys/ke= rn/sys_generic.c:711 #29 0xffffffff810cf3be in syscallenter (td=3D) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189 #30 a= md64_syscall (td=3D0xfffffe0160936000, traced=3D0) at /usr/src/sys/amd64/= amd64/trap.c:1200 #31 #32 0x00003dd9cbd7f94a in ?= ? () Backtrace stopped: Cannot access memory at address 0x3ddc1d44ae58 (k= gdb) fr 16 #16 0xffffffff83204101 in vm_start_cpus (vm=3D0xfffff804d585d0= 00, tostart=3Dtostart@entry=3D0xfffffe015f2dd858) at /usr/src/sys/amd64/v= mm/vmm.c:2508 2508 mtx_lock(&vm->rendezvous_mtx); | >> >> I believe WITNESS is upset that we=E2=80=99re going to potentially sle= ep doing mtx_lock(&vm->rendezvous_mtx); in vm_start_cpus() when we=E2=80=99= ve done critical_enter() in vm_run(). >> >> Best regards, >> Kristof >> > > Can you please test https://reviews.freebsd.org/D37452? > The VM boots correctly with that patch. Best regards, Kristof