From nobody Sat Jan 17 17:19:47 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dtk4N1fXxz6P16K for ; Sat, 17 Jan 2026 17:20:04 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dtk4M6qKBz3g5K for ; Sat, 17 Jan 2026 17:20:03 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x542.google.com with SMTP id 4fb4d7f45d1cf-64b8123c333so4770756a12.3 for ; Sat, 17 Jan 2026 09:20:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1768670397; x=1769275197; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=4EM47Lu6Ph/FvGxY8fKoxSEL7etgrDeW2u/LunTJn1U=; b=RplqrvaYlJWB5jYScTjdvXVQqa9MW2qPLv4u/48JP8jCn68wvX5Tqud2DsckeCx2rn LWL9lamKShvxhraoGVk+B9Z7bzFEzSFXgJwMOukGPLbqYwAhcv5XeePJN80y2Hdrk4Ec qGnSknT/GVddZJnPvurM0NiSmLIGRRnvrwpsALSk9MKLrOyUdsuXs07OW5lvYIJAmlIq ke9E1hKg3gWlAu+tv1LOtQPQMEee9lRLd+0FzXyxJ5jFGwEL1yx0DAY6yeN4HmdvLs92 1hDUekTZiX1Sc9P1o7Jd6IjKv3JERPaVRNWgvZk3/gLuemrAeMAVbCKVGp8fpjitpvNg mvYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768670397; x=1769275197; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=4EM47Lu6Ph/FvGxY8fKoxSEL7etgrDeW2u/LunTJn1U=; b=fq6Rrmw8ZB6xn56oOiYrOiPe+YZw2xxb31b2xl8KJyvWP6b1dCxFPVSQtANziutajh SOJhJ9udh9nAcQYgzYuzOkPYPPVFpje4AKWkmAI+amzpKUjJop04g+sJEW9/1UKPvzRY MNdyvjDhFWdfuwPZGyiO7Zcd+J4qcJNaM3Z9Almmm6CqAcfeJJuXwlXhAayErSjjgyqa ldtixpQINHiGZhfFjEFwD5ef4QGaT3wiVriqXrQ2Iwb+c53Gk2FDpzyQ3F+KWb1HobOf a54y97dWkmzqvTx+T8i+xYgcYDReOM9960vcYDmTRkNtFucWJxd8ZVWIi0KxIFPfqhcD V4hA== X-Gm-Message-State: AOJu0Ywz11w/ptDyNUaqWsuGGy3KzxTIhIFGQgqIp4o7kqD587V0+lYS Qr7VrBCJmjGnmhkHzzJqy/b1GXOTeAtDOgXja/q5BI40yHYv5o3B8jTN/Dll6DFaMt0= X-Gm-Gg: AY/fxX5kZP5Tp3CAjhpr+1gDFmt/FL+N4th3rLrBkCYYDGJKG4aokBjpAs+YK66Av7R UmGVdr7badP6ngR5yvFf0yCSspT9+effAcQEcsg9rLdmb+MszyxAO6rx7ZH2tMkauQbwxqZnUzk 7tTDuJ/YC9yUqFRKO8dpqnHt+TaIDlmWYUlmMeRGaXJN6UEKr6XmP58+ejynt8T0L/IdnxsuWiL oSY6VI5wuVLkYqTJL+/XXAHFtEBop/9EHxW/UJ1trYzeIu6et65emsQ1bh+cCW15kweiN5E/MTb EgfFt7X7X9zJkYr0XhzHOErLgCDsXyOgki9qZP6B8f6n4iRF9daKlxH2LqpHy7WRp+FnFqqZeX7 TFpgylHIggjwSCul9g1JSd2dOmM5hyGRqCjo6eIWW9/ETp21epvhkZwSTWNxG7PsAtU+8WV4rHX bOqbjH+mTI8KlDxxkFcXZoPEDyBSXjyFGPxh08cjXFZlro3cCQizTj1R3Ef8/BYJSLe8/XwITuz 5Y7fw== X-Received: by 2002:a05:6402:2346:b0:649:9268:1f43 with SMTP id 4fb4d7f45d1cf-654bb32c88fmr4329982a12.21.1768670396796; Sat, 17 Jan 2026 09:19:56 -0800 (PST) Received: from tulp (2001-1c02-1a15-3000-c218-03ff-feb5-6cc4.cable.dynamic.v6.ziggo.nl. [2001:1c02:1a15:3000:c218:3ff:feb5:6cc4]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-654534c8694sm5681867a12.25.2026.01.17.09.19.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Jan 2026 09:19:55 -0800 (PST) Date: Sat, 17 Jan 2026 18:19:47 +0100 From: Roman Bogorodskiy To: Mark Johnston Cc: freebsd-virtualization@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Quartz64: vmm: No vgic found Message-ID: References: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dtk4M6qKBz3g5K Mark Johnston wrote: > On Thu, Jan 15, 2026 at 06:42:10PM +0100, Roman Bogorodskiy wrote: > > Hi, > > > > Loading of the vmm(4) module shows errors like: > > > > Jan 15 17:32:10 generic kernel: vmm: No vgic found > > Jan 15 17:32:10 generic kernel: module_register_init: MOD_LOAD (vmm, 0xffff0000e12931f0, 0) error 19 > > > > The module is listed in kldstat: > > > > $ kldstat |grep vmm > > 7 1 0xffff0000e1268000 4f000 vmm.ko > > $ > > This is due to a bug. vmm.ko should unload itself after the error you > observed above: https://reviews.freebsd.org/D54750 Thanks for the fix. > > But bhyve does not start: > > > > # bhyve -o bootrom=/usr/local/share/u-boot/u-boot-bhyve-arm64/u-boot.bin -s 1,virtio-blk,./FreeBSD-15.0-RELEASE-arm64-aarch64-ufs.raw -o console=stdio vm0 > > bhyve: vm_openf: No such file or directory > > # > > > > Host system is: > > > > Pine64 Quartz64 Model A > > FreeBSD 15.0-RELEASE arm64 > > > > gic0: mem 0xfd400000-0xfd40ffff,0xfd460000-0xfd4dffff irq 12 on ofwbus0 > > gic0: using for IPIs > > > > CPU 0: ARM Cortex-A55 r2p0 affinity: 0 0 > > Cache Type = <64 byte CWG,64 byte ERG,64 byte D-cacheline,VIPT I-cache,64 byte I-cacheline> > > Instruction Set Attributes 0 = > > Instruction Set Attributes 1 = > > Instruction Set Attributes 2 = <> > > Processor Features 0 = > > Processor Features 1 = > > Processor Features 2 = <> > > Memory Model Features 0 = > > Memory Model Features 1 = > > Memory Model Features 2 = <32bit CCIDX,48bit VA,IESB,UAO,CnP> > > Memory Model Features 3 = <> > > Memory Model Features 4 = <> > > Debug Features 0 = > > Debug Features 1 = <> > > Auxiliary Features 0 = <> > > Auxiliary Features 1 = <> > > AArch32 Instruction Set Attributes 5 = > > AArch32 Media and VFP Features 0 = > > AArch32 Media and VFP Features 1 = > > CPU 1: ARM Cortex-A55 r2p0 affinity: 1 0 > > CPU 2: ARM Cortex-A55 r2p0 affinity: 2 0 > > CPU 3: ARM Cortex-A55 r2p0 affinity: 3 0 > > > > Does the board need something in addition to GICv3 that is missing that > > is needed to make vmm(4) work, or is that a configuration or vmm(4) > > issue? > > The vgic driver is refusing to attach, apparently because we don't > detect virtualization support when probing the controller. When booting > from a FDT we look for an "interrupts" property to determine this, but > it's not clear to me whether that's totally right. Can you share the > device tree you're using with this board? Not sure what's the right way to share the device tree, so I used the command I found on wiki (https://wiki.freebsd.org/FlattenedDeviceTree), that is: sysctl -b hw.fdt.dtb | dtc -I dtb It's lengthy, so I uploaded it here: https://people.freebsd.org/~novel/misc/quartz64_devicetree.txt And some more details. Until today I was using u-boot, sysutils/u-boot-quartz64-a specifically. Then I tried using Tianocore EDK2. Apparently, it supports three modes for devices: - ACPI - ACPI + devicetree - Devicetree The devicetree mode behaves the same way as u-boot, i.e. the vgic fails to attach. In the ACPI (and ACPI + devicetree) mode I can actually start a bhyve VM. However, in this mode Ethernet interface is not detected. Earlier it was available as: eqos0: mem 0xfe010000-0xfe01ffff irq 28,29 on ofwbus0 and now it's completely missing, though I can see it in the device tree. Does it get confused by ACPI presence? Roman