From nobody Mon Oct 14 22:52:09 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XSCD55snYz5ZJ2X for ; Mon, 14 Oct 2024 22:52:21 +0000 (UTC) (envelope-from SRS0=YIY8=RK=freebsd.org=grehan@iredmail.onthenet.com.au) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XSCD201lTz44PJ for ; Mon, 14 Oct 2024 22:52:15 +0000 (UTC) (envelope-from SRS0=YIY8=RK=freebsd.org=grehan@iredmail.onthenet.com.au) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nt.com.au header.s=dkim header.b=jdJbE1vf; spf=pass (mx1.freebsd.org: domain of "SRS0=YIY8=RK=freebsd.org=grehan@iredmail.onthenet.com.au" designates 203.13.68.12 as permitted sender) smtp.mailfrom="SRS0=YIY8=RK=freebsd.org=grehan@iredmail.onthenet.com.au"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 930C820B49A6 for ; Tue, 15 Oct 2024 08:52:09 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 8CDD62127CB9 for ; Tue, 15 Oct 2024 08:52:09 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=nt.com.au; h= content-transfer-encoding:content-type:content-type:in-reply-to :from:from:content-language:references:to:subject:subject :user-agent:mime-version:date:date:message-id; s=dkim; t= 1728946329; x=1731538330; bh=E+VvW8Rx9ayBaZ9PmlakWAO6cfMME/GTdGh XkV8IFsY=; b=jdJbE1vfTHynz6n9iA8ATraimRLlHzlJC+26y5emw1gkEy7CYH4 w3TA7NcjpVUm4b4mLki0J7w8/5spDgmCfmmjkzgPbxRD2LwZiyLa4GVUk5Wu1Tc4 k8I6Tlza76D4rzbPfnqtqrsvGY0NXbyXZnOvWJeEKtbMlpEa/JvuVK84= Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ibX1fXl7u5Y7 for ; Tue, 15 Oct 2024 08:52:09 +1000 (AEST) Received: from [192.168.1.101] (otn-120-29-24-249.broadband.onthenet.net [120.29.24.249]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 68ED42127CB7; Tue, 15 Oct 2024 08:52:09 +1000 (AEST) Message-ID: <106b8500-a0ef-4095-af20-8c0f110ea739@freebsd.org> Date: Tue, 15 Oct 2024 08:52:09 +1000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Running Mezzano in bhyve To: Vasily Postnicov Cc: freebsd-virtualization@freebsd.org References: <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> Content-Language: en-US From: Peter Grehan In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.4 cv=Fu4D/Xrq c=1 sm=1 tr=0 ts=670da099 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=0UtLtLgMz4NvdyIsuxvgLw==:17 a=IkcTkHD0fZMA:10 a=DAUX931o1VcA:10 a=bi0XHdcepdgA:10 a=H7MLjYanAAAA:8 a=1Wr8Lj4pOnmtSuG4wvkA:9 a=QEXdDO2ut3YA:10 a=bgSlMGY6Cwri7Y-TTqhu:22 X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; FORGED_SENDER(0.30)[grehan@freebsd.org,SRS0=YIY8=RK=freebsd.org=grehan@iredmail.onthenet.com.au]; R_SPF_ALLOW(-0.20)[+ip4:203.13.68.0/24]; R_DKIM_ALLOW(-0.20)[nt.com.au:s=dkim]; RCVD_IN_DNSWL_LOW(-0.10)[203.13.68.12:from]; RWL_MAILSPIKE_GOOD(-0.10)[203.13.68.12:from]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[grehan@freebsd.org,SRS0=YIY8=RK=freebsd.org=grehan@iredmail.onthenet.com.au]; DKIM_TRACE(0.00)[nt.com.au:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/24, country:AU]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4XSCD201lTz44PJ X-Spamd-Bar: --- > 1) The problem with PIT. Can be solved as you proposed or by > patching Mezzano. The bhyve patch would be the best option for that: it's useful for other older o/s's (DOS). > 2) Mezzano assumes that Intel AHCI controllers report no more than 6 > ports. Can be solved by patching Mezzano or defining MAX_PORTS to be > 6 in usr.sbin/bhyve/pci_ahci.c A Mezzano patch would be best for that. The bhyve man page has an example with 8 disks attached so reducing the limit to 6 could hit existing users. > 3) According to > https://wiki.osdev.org/PCI#Message_Signaled_Interrupts > , interrupt > line config register must be RW. Bhyve does not support writing to > it. I do not know a correct fix, this [1] workaround helps, however. Bhyve does support writing to that - your patch disables that, and my guess is that when Mezzano sees this as zero (ie invalid) it then looks for the irq line via the ACPI MADT (or other means). A quick look at Mezzano shows that it is still using the 8259 PIC for interrupts. At the minimum it should be using the IOAPIC, or excessive interrupt sharing will result, and possibly incorrect behaviour when this happens. I think IOAPIC support could be added without a large amount of effort, compared to e.g. MSI/MSI-x. > 4) Finally, I had a random deadlock in interrupt handling for the > virtio-net device. Likewise, I do not know how to fix it correctly, > but this [2] patch helped. Hmmm that seems strange: MSI interrupts aren't generated if they haven't been setup/enabled by a guest. Commenting out the lock/unlock code would seem to indicate a larger bug in play. Would it possible to get some tracing on that segment of code e.g. a dtrace log ? > Do you have any ideas how to make proper patches for bhyve from > these workarounds? The first one can be put in a phab diff, which I'll do. I think there's still some more work involved for the others. later, Peter. From nobody Tue Oct 15 05:15:32 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XSMkW4HhJz5YKBF for ; Tue, 15 Oct 2024 05:15:47 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XSMkV5RSMz4jMn; Tue, 15 Oct 2024 05:15:46 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-53a007743e7so169784e87.1; Mon, 14 Oct 2024 22:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728969345; x=1729574145; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=4Y9rl7zgdw70ybhpilllPsqgzyakyFdwnn0BRXZkM1c=; b=KK+WpZC3SfCzVnVe+5LS9kuqqO8seAxM4c8IxodwcU66dBgsNbEQrbYZn3SVTngl2P skyV1E2cCUO2GHZ8eCyY7Wry8O5ik/0UdhLHvoqnOEqLcPI5VGZgGhhgvz5WYH/bFmBf lzxA0GS6InNMVr8/IHsVTH73I17lccItsSdZQuPLwreha4M3C9jLuOYcd4f43NyeCXpR bOcD45KaLgPx1SigHjBkl6TuQzkw398RH1vAUbeEDjCHCsFtJBInxgiav2Ff/uDw5QeU FWeYqOUvRUdPOO4Raq2yenj32XTraOZHu1dHoBZjYu0EU4QHks3CpMF3CileX/nEiTQB /ADQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728969345; x=1729574145; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4Y9rl7zgdw70ybhpilllPsqgzyakyFdwnn0BRXZkM1c=; b=Aeggu/kzaMYajuLevV8p8XM4VOEa/epwkv9/deu24nVzV6OBL1bCcFMVvnGjs4U2MA dCiToK0ezSev2RKamV0OkqbamEnd3IdCYEiliKy6U43aU3Mc2JQLA3r0rcVnhrf+Y65b EyKAQSLVp1mhlP0R8qRvy5VGNzqTfU9Mpo5lDapsChZqKkSv8KfHeHenw8CyHc1YGGAG Um1/imowXvffdNtr9gteM2TJiGsV3Zb4GJSSeA0UYWN4xLIGJImidkXAX0tHtVIfyI4h lUQq1fyvqxZQm1X/zgKqhPizOMUCZ0QdSi34xBAlShFG2ufJ+K1WhqvP59DXXi6wmah+ edxg== X-Gm-Message-State: AOJu0YxV+R/fWRA2e9Lw6029mnYWhfs/hcWXWODU0R9pWrCDjjPpOpY8 aj5NQfAE7jpgtdS+YlyZ6afcpiEShGbn7sbtOU/CHzQic40Mr6Q755CJ607rN5snjimnLs2F48o 8H7vjnApHi1dJrDv8t9u+C/WPSzLkTeAbt4E= X-Google-Smtp-Source: AGHT+IGx7V3gs1ufm2/citaRI/dI3EgL082dJkuEESHjddMjegigI5ayBpjPhKSI/4IQ4E47jPac1SZI6/kS/2AHP9A= X-Received: by 2002:a05:6512:2811:b0:539:f4ab:5638 with SMTP id 2adb3069b0e04-539f4ab57c9mr2595338e87.60.1728969344072; Mon, 14 Oct 2024 22:15:44 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> <106b8500-a0ef-4095-af20-8c0f110ea739@freebsd.org> In-Reply-To: <106b8500-a0ef-4095-af20-8c0f110ea739@freebsd.org> From: Vasily Postnicov Date: Tue, 15 Oct 2024 05:15:32 +0000 Message-ID: Subject: Re: Running Mezzano in bhyve To: Peter Grehan Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="00000000000094c86f06247d0be4" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4XSMkV5RSMz4jMn X-Spamd-Bar: ---- --00000000000094c86f06247d0be4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Regarding items 3) and 4): 3) Indeed, bhyve does not explicitly forbid writing to 0x3c. I meant the following. The interrupt line is set is pci_emul.c in bhyve: pci_set_cfgdata8(pi, PCIR_INTLINE, pirq_irq(ii->ii_pirq_pin)); Bhyve asserts interrupts with pci_irq_assert in amd64/pci_irq.c. We need this line: vm_isa_assert_irq(pi->pi_vmctx, pirq->reg & PIRQ_IRQ, pi->pi_lintr.ioapic_irq); pirq->reg & PIRQ_IRQ is literally the same as pirq_irq(ii->ii_pirq_pin). Now, if something (e.g. UEFI firmware, bootloader) writes to PCIR_INTLINE bhyve will still send interrupts with the number that was there before the write, while the OS will expect an interrupt with the new number. I treat this as a bug in bhyve (but it affects nobody, because newer OSes do not use the 8259 interrupt controller). 4) It's commenting the lock what makes an effect. I commented pci_generate_msi just in case because it's not needed for Mezzano, but runs protected by the mutex which is now gone. This is a backtrace and thread list when bhyve hangs up if the mutex is not commented out: (lldb) bt * thread #1, name =3D 'mevent', stop reason =3D signal SIGSTOP * frame #0: 0x000011adeaa37e2a libthr.so.3`_umtx_op_err at _umtx_op_err.S:38 frame #1: 0x000011adeaa479c0 libthr.so.3`__thr_umutex_lock(mtx=3D0x0000378ecca00888, id=3D101223) at thr_umtx.c:79:3 frame #2: 0x000011adeaa40eea libthr.so.3`mutex_lock_sleep(curthread=3D0x0000378ecc412000, m=3D0x0000378ecca00888, abstime=3D0x0000000000000000) at thr_mutex.c:699:9 frame #3: 0x000011adeaa3ed8f libthr.so.3`__Tthr_mutex_lock [inlined] mutex_lock_common(m=3D0x0000378ecca00888, abstime=3D0x0000000000000000, cvattach=3Dfalse, rb_onlist=3Dfalse) at thr_mutex.c:733:9 frame #4: 0x000011adeaa3ed4d libthr.so.3`__Tthr_mutex_lock(mutex=3D) at thr_mutex.c:752:9 frame #5: 0x000011a5c43e7b06 bhyve`vi_interrupt(vs=3D0x0000378ecc4b8000= , isr=3D'\x01', msix_idx=3D65535) at virtio.h:358:3 frame #6: 0x000011a5c43e6c86 bhyve`vq_interrupt(vs=3D0x0000378ecc4b8000= , vq=3D0x0000378ecc4b8038) at virtio.h:376:2 frame #7: 0x000011a5c43e6c44 bhyve`vq_endchains(vq=3D0x0000378ecc4b8038= , used_all_avail=3D0) at virtio.c:512:3 frame #8: 0x000011a5c43db348 bhyve`pci_vtnet_rx(sc=3D0x0000378ecc4b8000= ) at pci_virtio_net.c:271:4 frame #9: 0x000011a5c43dab53 bhyve`pci_vtnet_rx_callback(fd=3D6, type=3DEVF_READ, param=3D0x0000378ecc4b8000) at pci_virtio_net.c:403:2 frame #10: 0x000011a5c43bb9f8 bhyve`mevent_handle(kev=3D0x000011ade4451200, numev=3D1) at mevent.c:273:3 frame #11: 0x000011a5c43bb5d7 bhyve`mevent_dispatch at mevent.c:549:3 frame #12: 0x000011a5c43aed4b bhyve`main(argc=3D1, argv=3D0x000011ade4453418) at bhyverun.c:1052:2 frame #13: 0x000011adec6c1a6a libc.so.7`__libc_start1(argc=3D24, argv=3D0x000011ade4453360, env=3D0x000011ade4453428, cleanup=3D, mainX=3D(bhyve`main at bhyverun.c:694)) at libc_start1.c:157:7 frame #14: 0x000011a5c43a80cd bhyve`_start at crt1_s.S:83 (lldb) frame select 5 frame #5: 0x000011a5c43e7b06 bhyve`vi_interrupt(vs=3D0x0000378ecc4b8000, isr=3D'\x01', msix_idx=3D65535) at virtio.h:358:3 355 if (pci_msix_enabled(vs->vs_pi)) 356 pci_generate_msix(vs->vs_pi, msix_idx); 357 else { -> 358 VS_LOCK(vs); 359 vs->vs_isr |=3D isr; 360 pci_generate_msi(vs->vs_pi, 0); 361 #ifdef __amd64__ (lldb) thread list Process 3185 stopped * thread #1: tid =3D 101223, 0x000011adeaa37e2a libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'mevent', stop reason =3D signal SIGSTOP thread #2: tid =3D 101868, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-0', stop reason =3D signal SIGSTOP thread #3: tid =3D 101869, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-1', stop reason =3D signal SIGSTOP thread #4: tid =3D 101870, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-2', stop reason =3D signal SIGSTOP thread #5: tid =3D 101871, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-3', stop reason =3D signal SIGSTOP thread #6: tid =3D 101872, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-4', stop reason =3D signal SIGSTOP thread #7: tid =3D 101873, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-5', stop reason =3D signal SIGSTOP thread #8: tid =3D 101874, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-6', stop reason =3D signal SIGSTOP thread #9: tid =3D 101875, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-7', stop reason =3D signal SIGSTOP thread #10: tid =3D 101876, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err a= t _umtx_op_err.S:38, name =3D 'vtnet-5:0 tx', stop reason =3D signal SIGSTOP thread #11: tid =3D 101877, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err a= t _umtx_op_err.S:38, name =3D 'hda-audio-output', stop reason =3D signal SIGS= TOP thread #12: tid =3D 101878, 0x000011adec7752ea libc.so.7`__sys_accept at _accept.S:4, name =3D 'rfb', stop reason =3D signal SIGSTOP thread #13: tid =3D 101879, 0x000011adec7726aa libc.so.7`__sys_ioctl at ioctl.S:4, name =3D 'vcpu 0', stop reason =3D signal SIGSTOP thread #14: tid =3D 101880, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err a= t _umtx_op_err.S:38, name =3D 'vcpu 1', stop reason =3D signal SIGSTOP I think implementing IOAPIC in MEzzano is the best option indeed, but I have a little experience. I'll see what I can do. =D0=BF=D0=BD, 14 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 22:52, Pet= er Grehan : > > 1) The problem with PIT. Can be solved as you proposed or by > > patching Mezzano. The bhyve patch would be the best option for that: > it's useful for > other older o/s's (DOS). > > > 2) Mezzano assumes that Intel AHCI controllers report no more than 6 > > ports. Can be solved by patching Mezzano or defining MAX_PORTS to be > > 6 in usr.sbin/bhyve/pci_ahci.c > > A Mezzano patch would be best for that. The bhyve man page has an > example with 8 disks attached so reducing the limit to 6 could hit > existing users. > > > 3) According to > > https://wiki.osdev.org/PCI#Message_Signaled_Interrupts > > , interrupt > > line config register must be RW. Bhyve does not support writing to > > it. I do not know a correct fix, this [1] workaround helps, however. > > Bhyve does support writing to that - your patch disables that, and my > guess is that when Mezzano sees this as zero (ie invalid) it then looks > for the irq line via the ACPI MADT (or other means). > > A quick look at Mezzano shows that it is still using the 8259 PIC for > interrupts. At the minimum it should be using the IOAPIC, or excessive > interrupt sharing will result, and possibly incorrect behaviour when > this happens. I think IOAPIC support could be added without a large > amount of effort, compared to e.g. MSI/MSI-x. > > > 4) Finally, I had a random deadlock in interrupt handling for the > > virtio-net device. Likewise, I do not know how to fix it correctly, > > but this [2] patch helped. > > Hmmm that seems strange: MSI interrupts aren't generated if they > haven't been setup/enabled by a guest. Commenting out the lock/unlock > code would seem to indicate a larger bug in play. Would it possible to > get some tracing on that segment of code e.g. a dtrace log ? > > > Do you have any ideas how to make proper patches for bhyve from > > these workarounds? > > The first one can be put in a phab diff, which I'll do. I think there's > still some more work involved for the others. > > later, > > Peter. > > > > --00000000000094c86f06247d0be4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Regarding items 3) and 4):

3) Indeed, b= hyve does not explicitly forbid writing to 0x3c. I meant the following. The= interrupt line is set is pci_emul.c in bhyve:
=C2=A0pci_set_cfgd= ata8(pi, PCIR_INTLINE, pirq_irq(ii->ii_pirq_pin));
Bhyve a= sserts interrupts with pci_irq_assert in amd64/pci_irq.c. We need this line= : vm_isa_assert_irq(pi->pi_vmctx, pirq->reg & PIRQ_IRQ, pi->pi= _lintr.ioapic_irq);
pirq->reg & PIRQ_IRQ is literally the = same as pirq_irq(ii->ii_pirq_pin). Now, if something (e.g. UEFI firmware= , bootloader) writes to PCIR_INTLINE bhyve will still send interrupts with = the number that was there before the write, while the OS will expect an int= errupt with the new number. I treat this as a bug in bhyve (but it affects = nobody, because newer OSes do not use the 8259 interrupt controller).
=

4) It's commenting the lock what makes an effect. I= commented pci_generate_msi just in case because it's not needed for Me= zzano, but runs protected by the mutex which is now gone.
This is= a backtrace and thread list when bhyve hangs up if the mutex is not commen= ted out:

(lldb) bt
* thread #1, name =3D 'm= event', stop reason =3D signal SIGSTOP
=C2=A0 * frame #0: 0x000011ad= eaa37e2a libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
=C2=A0 =C2=A0 fra= me #1: 0x000011adeaa479c0 libthr.so.3`__thr_umutex_lock(mtx=3D0x0000378ecca= 00888, id=3D101223) at thr_umtx.c:79:3
=C2=A0 =C2=A0 frame #2: 0x000011a= deaa40eea libthr.so.3`mutex_lock_sleep(curthread=3D0x0000378ecc412000, m=3D= 0x0000378ecca00888, abstime=3D0x0000000000000000) at thr_mutex.c:699:9
= =C2=A0 =C2=A0 frame #3: 0x000011adeaa3ed8f libthr.so.3`__Tthr_mutex_lock [i= nlined] mutex_lock_common(m=3D0x0000378ecca00888, abstime=3D0x0000000000000= 000, cvattach=3Dfalse, rb_onlist=3Dfalse) at thr_mutex.c:733:9
=C2=A0 = =C2=A0 frame #4: 0x000011adeaa3ed4d libthr.so.3`__Tthr_mutex_lock(mutex=3D&= lt;unavailable>) at thr_mutex.c:752:9
=C2=A0 =C2=A0 frame #5: 0x00001= 1a5c43e7b06 bhyve`vi_interrupt(vs=3D0x0000378ecc4b8000, isr=3D'\x01'= ;, msix_idx=3D65535) at virtio.h:358:3
=C2=A0 =C2=A0 frame #6: 0x000011a= 5c43e6c86 bhyve`vq_interrupt(vs=3D0x0000378ecc4b8000, vq=3D0x0000378ecc4b80= 38) at virtio.h:376:2
=C2=A0 =C2=A0 frame #7: 0x000011a5c43e6c44 bhyve`v= q_endchains(vq=3D0x0000378ecc4b8038, used_all_avail=3D0) at virtio.c:512:3<= br>=C2=A0 =C2=A0 frame #8: 0x000011a5c43db348 bhyve`pci_vtnet_rx(sc=3D0x000= 0378ecc4b8000) at pci_virtio_net.c:271:4
=C2=A0 =C2=A0 frame #9: 0x00001= 1a5c43dab53 bhyve`pci_vtnet_rx_callback(fd=3D6, type=3DEVF_READ, param=3D0x= 0000378ecc4b8000) at pci_virtio_net.c:403:2
=C2=A0 =C2=A0 frame #10: 0x0= 00011a5c43bb9f8 bhyve`mevent_handle(kev=3D0x000011ade4451200, numev=3D1) at= mevent.c:273:3
=C2=A0 =C2=A0 frame #11: 0x000011a5c43bb5d7 bhyve`mevent= _dispatch at mevent.c:549:3
=C2=A0 =C2=A0 frame #12: 0x000011a5c43aed4b = bhyve`main(argc=3D1, argv=3D0x000011ade4453418) at bhyverun.c:1052:2
=C2= =A0 =C2=A0 frame #13: 0x000011adec6c1a6a libc.so.7`__libc_start1(argc=3D24,= argv=3D0x000011ade4453360, env=3D0x000011ade4453428, cleanup=3D<unavail= able>, mainX=3D(bhyve`main at bhyverun.c:694)) at libc_start1.c:157:7=C2=A0 =C2=A0 frame #14: 0x000011a5c43a80cd bhyve`_start at crt1_s.S:83

(lldb) frame select 5
frame #5: 0x000011a5c43= e7b06 bhyve`vi_interrupt(vs=3D0x0000378ecc4b8000, isr=3D'\x01', msi= x_idx=3D65535) at virtio.h:358:3
=C2=A0 =C2=A0355 if (pci_msix_enabled= (vs->vs_pi))
=C2=A0 =C2=A0356 pci_generate_msix(vs->vs_pi, msix= _idx);
=C2=A0 =C2=A0357 else {
-> 358 VS_LOCK(vs);
=C2=A0 = =C2=A0359 vs->vs_isr |=3D isr;
=C2=A0 =C2=A0360 pci_generate_ms= i(vs->vs_pi, 0);
=C2=A0 =C2=A0361 #ifdef __amd64__
(lldb) thread list
Process 3185 stopped
* thread #1: tid = =3D 101223, 0x000011adeaa37e2a libthr.so.3`_umtx_op_err at _umtx_op_err.S:3= 8, name =3D 'mevent', stop reason =3D signal SIGSTOP
=C2=A0 thre= ad #2: tid =3D 101868, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx= _op_err.S:38, name =3D 'blk-3:0-0', stop reason =3D signal SIGSTOP<= br>=C2=A0 thread #3: tid =3D 101869, 0x000011adeaa37e2c libthr.so.3`_umtx_o= p_err at _umtx_op_err.S:38, name =3D 'blk-3:0-1', stop reason =3D s= ignal SIGSTOP
=C2=A0 thread #4: tid =3D 101870, 0x000011adeaa37e2c libth= r.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-2', sto= p reason =3D signal SIGSTOP
=C2=A0 thread #5: tid =3D 101871, 0x000011ad= eaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3= :0-3', stop reason =3D signal SIGSTOP
=C2=A0 thread #6: tid =3D 1018= 72, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name = =3D 'blk-3:0-4', stop reason =3D signal SIGSTOP
=C2=A0 thread #7= : tid =3D 101873, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_e= rr.S:38, name =3D 'blk-3:0-5', stop reason =3D signal SIGSTOP
= =C2=A0 thread #8: tid =3D 101874, 0x000011adeaa37e2c libthr.so.3`_umtx_op_e= rr at _umtx_op_err.S:38, name =3D 'blk-3:0-6', stop reason =3D sign= al SIGSTOP
=C2=A0 thread #9: tid =3D 101875, 0x000011adeaa37e2c libthr.s= o.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'blk-3:0-7', stop r= eason =3D signal SIGSTOP
=C2=A0 thread #10: tid =3D 101876, 0x000011adea= a37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, name =3D 'vtnet-5= :0 tx', stop reason =3D signal SIGSTOP
=C2=A0 thread #11: tid =3D 10= 1877, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38, nam= e =3D 'hda-audio-output', stop reason =3D signal SIGSTOP
=C2=A0 = thread #12: tid =3D 101878, 0x000011adec7752ea libc.so.7`__sys_accept at _a= ccept.S:4, name =3D 'rfb', stop reason =3D signal SIGSTOP
=C2=A0= thread #13: tid =3D 101879, 0x000011adec7726aa libc.so.7`__sys_ioctl at io= ctl.S:4, name =3D 'vcpu 0', stop reason =3D signal SIGSTOP
=C2= =A0 thread #14: tid =3D 101880, 0x000011adeaa37e2c libthr.so.3`_umtx_op_err= at _umtx_op_err.S:38, name =3D 'vcpu 1', stop reason =3D signal SI= GSTOP

I think implementing IOAPIC in MEzzano i= s the best option indeed, but I have a little experience. I'll see what= I can do.

=D0=BF=D0=BD, 14 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. = =D0=B2 22:52, Peter Grehan <grehan= @freebsd.org>:
> 1) The problem with PI= T. Can be solved as you proposed or by
> patching Mezzano. The bhyve patch would be the best option for that: i= t's useful for
other older o/s's (DOS).

> 2) Mezzano assumes that Intel AHCI controllers report no more than 6 <= br> > ports. Can be solved by patching Mezzano or defining MAX_PORTS to be <= br> > 6 in usr.sbin/bhyve/pci_ahci.c

=C2=A0 A Mezzano patch would be best for that. The bhyve man page has an example with 8 disks attached so reducing the limit to 6 could hit
existing users.

> 3) According to
> https://wiki.osdev.org/PCI#Message_Signal= ed_Interrupts
> <https://wiki.osdev.org/PCI#Message_Si= gnaled_Interrupts>, interrupt
> line config register must be RW. Bhyve does not support writing to > it. I do not know a correct fix, this [1] workaround helps, however.
=C2=A0 Bhyve does support writing to that - your patch disables that, and m= y
guess is that when Mezzano sees this as zero (ie invalid) it then looks
for the irq line via the ACPI MADT (or other means).

=C2=A0 A quick look at Mezzano shows that it is still using the 8259 PIC fo= r
interrupts. At the minimum it should be using the IOAPIC, or excessive
interrupt sharing will result, and possibly incorrect behaviour when
this happens. I think IOAPIC support could be added without a large
amount of effort, compared to e.g. MSI/MSI-x.

> 4) Finally, I had a random deadlock in interrupt handling for the
> virtio-net device. Likewise, I do not know how to fix it correctly, > but this [2] patch helped.

=C2=A0 Hmmm that seems strange: MSI interrupts aren't generated if they=
haven't been setup/enabled by a guest. Commenting out the lock/unlock code would seem to indicate a larger bug in play. Would it possible to
get some tracing on that segment of code e.g. a dtrace log ?

> Do you have any ideas how to make proper patches for bhyve from
> these workarounds?

=C2=A0 The first one can be put in a phab diff, which I'll do. I think = there's
still some more work involved for the others.

later,

Peter.



--00000000000094c86f06247d0be4-- From nobody Tue Oct 15 07:28:34 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XSQgk1ZSsz5YVdL for ; Tue, 15 Oct 2024 07:28:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XSQgj6ZMhz42hK for ; Tue, 15 Oct 2024 07:28:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728977313; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=rlvkLAydTZzEOQZ24FaUsNYf4WraI636PosushJCM+I=; b=Devkcj3+vDRAV/Rddh/69Rjb7smQG+w+COxO2ElIHdc6gZ/K6K0XHR7C35ZV/4X9k1U9Ws GV8b6J9wTXhrxtPsmSLwChv+HngCeo9TfDtl/ad4zaN0HY/a7jp5J/5jdWcaNtAHg/beeC OHDj6L+YLu5I+xB/EUAysHjfb+41G953CC5ZURERrljwzrHg32GeloVa0e/r53ksfGkdlI isGukBamWeLJ+gfX1bdmeu5KeDs8YKZUca1n51MetmJU94p8mVppVbTMPfYAzZ8ueiZcjU HePLe53q9BetdDj2buK47zjLyTawAOKzZ2V8o7XFUpC4fHZGiZF8mVkK1VCoWw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1728977313; a=rsa-sha256; cv=none; b=CQGuhvtSrjhNvkZWTqVSkg9r/w8ZjvAyJno1k8643Mlv8Zqnc7qI4e0Io7TUptYkz+ivT2 I8ZbFVkAa08l28UlCdvhp7hkhkJ3ckZQomhxbsghNUiWsjz0cl2cPZaoePXQ3ynJJFeV/7 ZS01eXWcnHMKr6kTqzD9eNIOIJ4khG0sykngCmMU7tB7biZUeSfcx6g1fHVPMhnrNGg6q4 a0RnQ6Y7XCahiWStbQeqvNz/ArKkr4uRK4Y8pJDr5CyJ9pYuiD406M+ivNT3ZR1P6AMSUq hVYTux+oLa0jV6bmOoGxUMgfQl5kMFHiAPg5WMZ0RVGI3P3z68gOO1qWjQTlgQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XSQgj6B31z1BdC for ; Tue, 15 Oct 2024 07:28:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 49F7SXBc066641 for ; Tue, 15 Oct 2024 07:28:33 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 49F7SX9a066640 for virtualization@FreeBSD.org; Tue, 15 Oct 2024 07:28:33 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 282098] bhyve is eating my memory and running far beyond the configured limits Date: Tue, 15 Oct 2024 07:28:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: elk-freebsd@wetznet.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282098 Bug ID: 282098 Summary: bhyve is eating my memory and running far beyond the configured limits Product: Base System Version: 13.3-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: elk-freebsd@wetznet.de Looks like there is a memory leak in bhyve.=20 I running Truenas 13.3 which is on Freebsd 13.3-RELEASE-p4 on AMD Ryzen 5 P= RO 4650G bhyve is configured for two VMs one with 4GB (Windows) and one with 8GB (li= nux) But If I start memory intensive actions inside the linux VM inside a docker this raised the memory usage far beyond the 8GB.. 14,6GB was the last I saw yesterday and this morning the VMs were killed by Truenas cos of OOM. Inside the vm the memory is stable at max 8GB. Swap is used sometimes as we= ll internal the vm. The vm is configured with this xml ... 1_Jellyfin 615af2a2-84e0-434e-b544-b4c9e7c0872d Jellyfin Jellyfin 8388608 8388608 8 hvm /usr/local/share/uefi-firmware/BHYVE_UEFI.fd ..... But the bhyve VM Jellyfin is using 14G at this moment Wed Sep 18 14:24:00 CEST 2024 UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 2106 1 10 20 0 14774616 14149064 kqread SC - 1689:51.13=20 bhyve: 1_Jellyfin (bhyve) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Oct 18 06:31:07 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XVFGK36XSz5Yn92 for ; Fri, 18 Oct 2024 06:31:21 +0000 (UTC) (envelope-from SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XVFGG39ZLz4b4k for ; Fri, 18 Oct 2024 06:31:15 +0000 (UTC) (envelope-from SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nt.com.au header.s=dkim header.b=VfWUtYUQ; spf=pass (mx1.freebsd.org: domain of "SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au" designates 203.13.68.12 as permitted sender) smtp.mailfrom="SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 92BA120B4981 for ; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 865472127CB8 for ; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=nt.com.au; h= content-transfer-encoding:content-type:content-type:in-reply-to :content-language:references:to:subject:subject:from:from :user-agent:mime-version:date:date:message-id; s=dkim; t= 1729233068; x=1731825069; bh=W0rXtUgnBUUjFb8kGjEaeaI1tKbiPqSnjUz 9VVe+gno=; b=VfWUtYUQ9mER8TsJ33rVeA12at55xi0ube1/qUbRZbrJFJsQMtL TVr3AGXdsuu45doPjaIdQ+mpPdo4GO0ZQkZSV71u+NYWvXq7bpPL++C0GiDW0+YB 5yhUjs3/TsaX3mwaeu1Nxh48UB9dPv7QyPNXRvAoJ8FJm+AzpCW0kIfs= Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id XKryYq-mBM5y for ; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) Received: from [192.168.1.101] (otn-120-29-24-249.broadband.onthenet.net [120.29.24.249]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 5384D2127CB7; Fri, 18 Oct 2024 16:31:08 +1000 (AEST) Message-ID: <2540b716-8f7e-4ff8-a582-613977bdb8c0@freebsd.org> Date: Fri, 18 Oct 2024 16:31:07 +1000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Peter Grehan Subject: Re: Running Mezzano in bhyve To: Vasily Postnicov Cc: freebsd-virtualization@freebsd.org References: <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> <106b8500-a0ef-4095-af20-8c0f110ea739@freebsd.org> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.4 cv=Fu4D/Xrq c=1 sm=1 tr=0 ts=671200ac a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=0UtLtLgMz4NvdyIsuxvgLw==:17 a=IkcTkHD0fZMA:10 a=DAUX931o1VcA:10 a=bi0XHdcepdgA:10 a=a4NEJbfMAAAA:8 a=QyXUC8HyAAAA:8 a=2LeWwqssq1ttuCmCoZgA:9 a=QEXdDO2ut3YA:10 X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[grehan@freebsd.org,SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au]; R_SPF_ALLOW(-0.20)[+ip4:203.13.68.0/24]; R_DKIM_ALLOW(-0.20)[nt.com.au:s=dkim]; RCVD_IN_DNSWL_LOW(-0.10)[203.13.68.12:from]; RWL_MAILSPIKE_GOOD(-0.10)[203.13.68.12:from]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[grehan@freebsd.org,SRS0=6dos=RO=freebsd.org=grehan@iredmail.onthenet.com.au]; DKIM_TRACE(0.00)[nt.com.au:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/24, country:AU]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4XVFGG39ZLz4b4k X-Spamd-Bar: --- > Regarding items 3) and 4): > > 3) Indeed, bhyve does not explicitly forbid writing to 0x3c. I meant > the following. The interrupt line is set is pci_emul.c in bhyve: > pci_set_cfgdata8(pi, PCIR_INTLINE, pirq_irq(ii->ii_pirq_pin)); Bhyve > asserts interrupts with pci_irq_assert in amd64/pci_irq.c. We need > this line: vm_isa_assert_irq(pi->pi_vmctx, pirq->reg & PIRQ_IRQ, > pi->pi_lintr.ioapic_irq); pirq->reg & PIRQ_IRQ is literally the same > as pirq_irq(ii->ii_pirq_pin). Now, if something (e.g. UEFI firmware, > bootloader) writes to PCIR_INTLINE bhyve will still send interrupts > with the number that was there before the write, while the OS will > expect an interrupt with the new number. I treat this as a bug in > bhyve (but it affects nobody, because newer OSes do not use the 8259 > interrupt controller). Ah yes, I think that's correct. I'm assuming that you're using EFI to boot, and bhyve/EFI recently changed to have PCI setup done by EFI and not in bhyve as before. Sounds like this is an extreme corner case that was broken by that. Not surprising: any o/s that supports EFI boot could be expected to support IOAPIC interrupts at a minimum. > 4) It's commenting the lock what makes an effect. I commented > pci_generate_msi just in case because it's not needed for Mezzano, > but runs protected by the mutex which is now gone. This is a > backtrace and thread list when bhyve hangs up if the mutex is not > commented out: I suspect this might be related to the above. > I think implementing IOAPIC in MEzzano is the best option indeed, > but I have a little experience. I'll see what I can do. See figures 3-2 through 3-5 in the Intel MP spec for an explanation of how it all fits together https://web.archive.org/web/20121002210153/http://download.intel.com/design/archives/processors/pro/docs/24201606.pdf later, Peter. From nobody Sun Oct 20 21:00:52 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XWrSk3byhz5ZmZ8 for ; Sun, 20 Oct 2024 21:00:54 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XWrSh6qVXz4mxR for ; Sun, 20 Oct 2024 21:00:52 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1729458053; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FcxGPGQseSFsunMHaWjHqWNh2wYv1MrzBOXXWxID7Ts=; b=Z9JBGS77SJpnxOizyCeSKzKrMYi1Mzkth2d0jodUr+gjhCk2KaNIf+7KMfaD+Ymo3UtXhH PgJysC3eGG00UzfkaW7ARmGLZ/Zgi2euU2qbRX8J/okXilKx+otKZKAPwtV0i0hbv8BwaR /tFu12M66Sk8+IK+mQsBruIpbY2OFrFK3qpOjOmRE+69GZbWJiTHd806Th1r1K+ALdoM8Y M6/wcYOY0gmr/jlYygYIK4qRHkO4kwTH13LmJ4tp86NQpM+6wd9UPevFjeKJfZO0BpVx9c mXUbfIqpE1ZM5iDlLtobyVrpkYeVgYf0L7bHevfg01goNkkUQJBnLUJSmOmdlQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1729458053; a=rsa-sha256; cv=none; b=mTaMYAR/Npw7tpQYiGvAx8ybfFARbSGExLnEwRn4Qr02U2h5y279JhoEA2uBmJVn89bOqD /C6fLlIwOb2eNCdLn0vi4TPrs3uMpkoyHReDr+LORGa/tHxx3B2TT+IWw/JN11QcSXlgeZ QKziuZzy11t06cihd1IOCnKn5XFA73sf77HS362Ghb73BqggAF7QMMLogCUtnShfg6Q9xf qrAZkiKyC+0gPI3LIzHlfBxkT5rm2Qy49LinP8V/37DsLD7xqY3v2VwU0FRlRgGnJvDlf6 lgliLbTvej8hqdMJz4vAqNxVuxrwIzwlQ2GQnCReb5PDit+cDzTn7IXjPU2kHQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XWrSh58gTz1GH6 for ; Sun, 20 Oct 2024 21:00:52 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 49KL0q0n047842 for ; Sun, 20 Oct 2024 21:00:52 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 49KL0qEW047841 for virtualization@FreeBSD.org; Sun, 20 Oct 2024 21:00:52 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202410202100.49KL0qEW047841@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 20 Oct 2024 21:00:52 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17294580523.bFbDEe7b2.41595" Content-Transfer-Encoding: 7bit --17294580523.bFbDEe7b2.41595 Date: Sun, 20 Oct 2024 21:00:52 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 264267 | UEFI Booting on Azure Generation 2 VMs crashes 1 problems total for which you should take action. --17294580523.bFbDEe7b2.41595 Date: Sun, 20 Oct 2024 21:00:52 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    264267 | UEFI Booting on Azure Generation 2 VMs crashes

1 problems total for which you should take action.
--17294580523.bFbDEe7b2.41595--