Date: Thu, 11 Jan 2024 15:34:44 +0100 From: Mario Marietto <marietto2008@gmail.com> To: freebsd-hackers <freebsd-hackers@freebsd.org>, FreeBSD virtualization <freebsd-virtualization@freebsd.org>, FreeBSD Mailing List <freebsd-questions@freebsd.org> Subject: Unknown option "XENHVM" for Arm 32 and for arm 64 ; missing code detected. Message-ID: <CA%2B1FSigxFsitie=RwbLMXw4YEMFxFX%2BGSnMopi9ayj5gfS2hYw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hello. I'm trying to boot FreeBSD for arm32 bit as DomU on my ARM Chromebook. Basically,with the help of the xen developers I have patched the FreeBSD kernel and I've been able to boot it as if it was a zImage format file. I've enabled all the xen options inside the Linux kernel file used for the domU and this is what happened when I tried to boot the FreeBSD kernel file : mario@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebook/freebsd-xen/domU-freebsd# ./start-freebsd Parsing config from freebsd.cfg libxl: debug: libxl_create.c:2081:do_domain_create: ao 0x442780: create: how=(nil) callback=(nil) poller=0x43cc50 libxl: detail: libxl_create.c:662:libxl__domain_make: passthrough: disabled libxl: debug: libxl_arm.c:148:libxl__arch_domain_prepare_config: Configure the domain libxl: debug: libxl_arm.c:151:libxl__arch_domain_prepare_config: - Allocate 0 SPIs libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown specification=xen libxl: debug: libxl_device.c:452:libxl__device_disk_set_backend: Disk vdev=xvda, using backend phy libxl: debug: libxl_create.c:1342:initiate_domain_create: Domain 1:running bootloader libxl: debug: libxl_bootloader.c:417:libxl__bootloader_run: Domain 1:no bootloader configured, using user supplied kernel libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x43d8f0: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline="console=hvc0", features="" domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x40008+0x966 at 0xb5972000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x40008000 -> 0x4096e000 (pfn 0x40008 + 0x966 pages) domainbuilder: detail: xc_dom_load_zimage_kernel: called domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 0x40008000-0x4096e000 domainbuilder: detail: xc_dom_load_zimage_kernel: copy 9851212 bytes from blob 0xb62d8000 to dst 0xb5972000 domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x48000+0x1 at 0xb6f0a000 domainbuilder: detail: xc_dom_alloc_segment: devicetree : 0x48000000 -> 0x48001000 (pfn 0x48000 + 0x1 pages) domainbuilder: detail: alloc_magic_pages: called domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x48001000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: bootearly: doing nothing domainbuilder: detail: start_info_arm: called domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 58 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 9620 kB domainbuilder: detail: domU mmap : 9628 kB domainbuilder: detail: vcpu_arm32: called domainbuilder: detail: Initial state CPSR 0x400001d3 PC 0x40008000 domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[0] -> d0 0x39000 domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[1] -> d0 0x39001 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=phy specification=xen libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: register slotnum=3 libxl: debug: libxl_create.c:2120:do_domain_create: ao 0x442780: inprogress: poller=0x43cc50, flags=i libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/1/51712/state libxl: debug: libxl_event.c:1054:devstate_callback: backend /local/domain/0/backend/vbd/1/51712/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/1/51712/state libxl: debug: libxl_event.c:1051:devstate_callback: backend /local/domain/0/backend/vbd/1/51712/state wanted state 2 ok libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: deregister slotnum=3 libxl: debug: libxl_device.c:1150:device_backend_callback: Domain 1:calling device_backend_cleanup libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x4410e4: deregister unregistered libxl: debug: libxl_linux.c:194:libxl__hotplug_disk: Domain 1:Args and environment ready libxl: debug: libxl_device.c:1251:device_hotplug: Domain 1:calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_device.c:1252:device_hotplug: Domain 1:extra args: libxl: debug: libxl_device.c:1260:device_hotplug: Domain 1:env: libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: script: /etc/xen/scripts/block libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_TYPE: vbd libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_PATH: backend/vbd/1/51712 libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_BASE_PATH: backend libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block add libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x441168: deregister unregistered libxl: debug: libxl_linux.c:211:libxl__get_hotplug_script_info: Domain 1:num_exec 1, not running hotplug scripts libxl: debug: libxl_device.c:1236:device_hotplug: Domain 1:No hotplug script to execute libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x441168: deregister unregistered libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: register slotnum=3 libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: event epath=/local/domain/1/console/tty libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: event epath=/local/domain/1/console/tty libxl: debug: libxl_event.c:2403:libxl__ao_progress_report: ao 0x442780: progress report: ignored libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: deregister slotnum=3 libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x43e7cc: deregister unregistered libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0 libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy libxl: debug: libxl_domain.c:704:libxl_domain_unpause: Domain 1:ao 0x442780: create: how=(nil) callback=(nil) poller=0x43cc50 libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0 libxl: debug: libxl_domain.c:712:libxl_domain_unpause: Domain 1:ao 0x442780: inprogress: poller=0x43cc50, flags=ic libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy xencall:buffer: debug: total allocations:98 total releases:98 xencall:buffer: debug: current allocations:0 maximum allocations:3 xencall:buffer: debug: cache current size:3 xencall:buffer: debug: cache hits:86 misses:3 toobig:9 xencall:buffer: debug: total allocations:0 total releases:0 xencall:buffer: debug: current allocations:0 maximum allocations:0 xencall:buffer: debug: cache current size:0 xencall:buffer: debug: cache hits:0 misses:0 toobig:0 FROZEN. I see these two processes enabled...but I don't know how to use FreeBSD... # ps ax 2606 ? Ssl 0:00 xl -vvvv create freebsd.cfg 2607 pts/0 Sl+ 0:00 /usr/lib/xen-4.17/bin/xenconsole 1 --num 0 --type pv now the fun begins...because,based on what say the xen developers : ... the ps output, it seems that ``xl create`` completed and you have the console open. So the freeze you mention is just because your FreeBSD guest is not outputting anything. As mentioned earlier, I don't think a lot of testing has been done for 32-bit Arm FreeBSD. So it is quite possible that there are some pieces of code missing. The first step is to check if the FreeBSD kernel was built with Xen options. If they are, then you will need to find out where FreeBSD is stuck (or why the console is not enabled). Unfortunately, we don't have any support to use GDB on the guest kernel. So you will have to modify FreeBSD a bit to check where it freezes. Assuming you have a debug build of the hypervisor, then you can sprinkle the FreeBSD boot code with the assembly instruction 'hvc 0xfffc'. When this is reached, this will issue an hypercall that will print on the Xen console that the given instruction has reached (the PC will be printed). There are other useful 'hvc' calls implemented by Xen for low level debugging. You can look at do_debug_trap() in Xen code. I've added this to GENERIC kernel of FreeBSD guest : # Xen HVM Guest Optimizations # NOTE: XENHVM depends on xenpci. They must be added or removed together. options XENHVM # Xen HVM kernel infrastructure device xenpci # Xen HVM Hypervisor services driver Unfortunately,it is not accepted : unknown option "XENHVM" I've found these parameters ONLY on the file /usr/src/sys/conf/files.amd64 : x86/xen/pv.c optional xenhvm x86/xen/pvcpu_enum.c optional xenhvm x86/xen/xen_pci_bus.c optional xenhvm Look at the prefix x86. It is not for arm. It seems there is a piece of code that hasn't been written. Is there a chance to add it ? Does anyone want to help ? No arch/arm64 does not contain Xen in KERNCONF either. Surprisingly files.x86 invokes a different set of files. I don't know what build it is. x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xenhvm x86/xen/xen_apic.c optional xenhvm x86/xen/xenpv.c optional xenhvm x86/xen/xen_msi.c optional xenhvm x86/xen/xen_nexus.c optional xenhvm -- Mario. [-- Attachment #2 --] <div dir="ltr"><div>Hello.</div><div> <br> I'm trying to boot FreeBSD for arm32 bit as DomU on my ARM Chromebook. Basically,with the help of the xen developers I have patched the FreeBSD kernel and I've been able to boot it as if it was a zImage format file. <br></div><div><br></div><div>I've enabled all the xen options inside the Linux kernel file used for the domU and this is what happened when I tried to boot the FreeBSD kernel file :<br><div> <div><br></div><div><br></div> <div dir="ltr"> <pre dir="ltr"><code>mario@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebook/freebsd-xen/domU-freebsd# ./start-freebsd Parsing config from freebsd.cfg libxl: debug: libxl_create.c:2081:do_domain_create: ao 0x442780: create: how=(nil) callback=(nil) poller=0x43cc50 libxl: detail: libxl_create.c:662:libxl__domain_make: passthrough: disabled libxl: debug: libxl_arm.c:148:libxl__arch_domain_prepare_config: Configure the domain libxl: debug: libxl_arm.c:151:libxl__arch_domain_prepare_config: - Allocate 0 SPIs libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown specification=xen libxl: debug: libxl_device.c:452:libxl__device_disk_set_backend: Disk vdev=xvda, using backend phy libxl: debug: libxl_create.c:1342:initiate_domain_create: Domain 1:running bootloader libxl: debug: libxl_bootloader.c:417:libxl__bootloader_run: Domain 1:no bootloader configured, using user supplied kernel libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x43d8f0: deregister unregistered domainbuilder: detail: xc_dom_allocate: cmdline="console=hvc0", features="" domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x40008+0x966 at 0xb5972000 domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x40008000 -> 0x4096e000 (pfn 0x40008 + 0x966 pages) domainbuilder: detail: xc_dom_load_zimage_kernel: called domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 0x40008000-0x4096e000 domainbuilder: detail: xc_dom_load_zimage_kernel: copy 9851212 bytes from blob 0xb62d8000 to dst 0xb5972000 domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x48000+0x1 at 0xb6f0a000 domainbuilder: detail: xc_dom_alloc_segment: devicetree : 0x48000000 -> 0x48001000 (pfn 0x48000 + 0x1 pages) domainbuilder: detail: alloc_magic_pages: called domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x48001000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: bootearly: doing nothing domainbuilder: detail: start_info_arm: called domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 58 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 9620 kB domainbuilder: detail: domU mmap : 9628 kB domainbuilder: detail: vcpu_arm32: called domainbuilder: detail: Initial state CPSR 0x400001d3 PC 0x40008000 domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[0] -> d0 0x39000 domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[1] -> d0 0x39001 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=phy specification=xen libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: register slotnum=3 libxl: debug: libxl_create.c:2120:do_domain_create: ao 0x442780: inprogress: poller=0x43cc50, flags=i libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/1/51712/state libxl: debug: libxl_event.c:1054:devstate_callback: backend /local/domain/0/backend/vbd/1/51712/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/1/51712/state libxl: debug: libxl_event.c:1051:devstate_callback: backend /local/domain/0/backend/vbd/1/51712/state wanted state 2 ok libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: deregister slotnum=3 libxl: debug: libxl_device.c:1150:device_backend_callback: Domain 1:calling device_backend_cleanup libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x4410e4: deregister unregistered libxl: debug: libxl_linux.c:194:libxl__hotplug_disk: Domain 1:Args and environment ready libxl: debug: libxl_device.c:1251:device_hotplug: Domain 1:calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_device.c:1252:device_hotplug: Domain 1:extra args: libxl: debug: libxl_device.c:1260:device_hotplug: Domain 1:env: libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: script: /etc/xen/scripts/block libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_TYPE: vbd libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_PATH: backend/vbd/1/51712 libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_BASE_PATH: backend libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block add libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x441168: deregister unregistered libxl: debug: libxl_linux.c:211:libxl__get_hotplug_script_info: Domain 1:num_exec 1, not running hotplug scripts libxl: debug: libxl_device.c:1236:device_hotplug: Domain 1:No hotplug script to execute libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x441168: deregister unregistered libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: register slotnum=3 libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: event epath=/local/domain/1/console/tty libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: event epath=/local/domain/1/console/tty libxl: debug: libxl_event.c:2403:libxl__ao_progress_report: ao 0x442780: progress report: ignored libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: deregister slotnum=3 libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch w=0x43e7cc: deregister unregistered libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0 libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy libxl: debug: libxl_domain.c:704:libxl_domain_unpause: Domain 1:ao 0x442780: create: how=(nil) callback=(nil) poller=0x43cc50 libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0 libxl: debug: libxl_domain.c:712:libxl_domain_unpause: Domain 1:ao 0x442780: inprogress: poller=0x43cc50, flags=ic libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy xencall:buffer: debug: total allocations:98 total releases:98 xencall:buffer: debug: current allocations:0 maximum allocations:3 xencall:buffer: debug: cache current size:3 xencall:buffer: debug: cache hits:86 misses:3 toobig:9 xencall:buffer: debug: total allocations:0 total releases:0 xencall:buffer: debug: current allocations:0 maximum allocations:0 xencall:buffer: debug: cache current size:0 xencall:buffer: debug: cache hits:0 misses:0 toobig:0 FROZEN.</code></pre> </div> </div><br> I see these two processes enabled...but I don't know how to use FreeBSD...<br> <br> <div> <div></div> <div dir="ltr"> <pre dir="ltr"><code># ps ax 2606 ? Ssl 0:00 xl -vvvv create freebsd.cfg 2607 pts/0 Sl+ 0:00 /usr/lib/xen-4.17/bin/xenconsole 1 --num 0 --type pv</code></pre> </div> </div><br> now the fun begins...because,based on what say the xen developers :<br> <br> <blockquote> <div> <div> ... the ps output, it seems that ``xl create`` completed and you have the console open. So the freeze you mention is just because your FreeBSD guest is not outputting anything. As mentioned earlier, I don't think a lot of testing has been done for 32-bit Arm FreeBSD. So it is quite possible that there are some pieces of code missing. The first step is to check if the FreeBSD kernel was built with Xen options. If they are, then you will need to find out where FreeBSD is stuck (or why the console is not enabled). Unfortunately, we don't have any support to use GDB on the guest kernel. So you will have to modify FreeBSD a bit to check where it freezes. Assuming you have a debug build of the hypervisor, then you can sprinkle the FreeBSD boot code with the assembly instruction 'hvc 0xfffc'. When this is reached, this will issue an hypercall that will print on the Xen console that the given instruction has reached (the PC will be printed). There are other useful 'hvc' calls implemented by Xen for low level debugging. You can look at do_debug_trap() in Xen code. </div> </div> </blockquote><br></div><div>I've added this to GENERIC kernel of FreeBSD guest :</div> <br> <div> <div></div> <div dir="ltr"> <pre dir="ltr"><code># Xen HVM Guest Optimizations <br># NOTE: XENHVM depends on xenpci. They must be added or removed together.<br> options XENHVM # Xen HVM kernel infrastructure device xenpci # Xen HVM Hypervisor services driver<br><br></code></pre><pre><code>Unfortunately,it is not accepted :<br><br></code></pre><pre><code>unknown option "XENHVM"<br><br></code></pre><pre><code><br>I've found these parameters ONLY on the file </code><tt style="color:rgb(0,122,0)">/usr/src/sys/conf/files.amd64</tt><code> :<br><br></code></pre><div><div dir="ltr"> <pre dir="ltr"><code>x86/xen/pv.c optional xenhvm x86/xen/pvcpu_enum.c optional xenhvm x86/xen/xen_pci_bus.c optional xenhvm</code></pre> </div> </div><br></div><div>Look at the prefix x86. It is not for arm. It seems there is a piece of code that hasn't been written. Is there a chance to add it ? Does anyone want to help ?</div><div><br></div><div>No arch/arm64 does not contain Xen in KERNCONF either.<br> <br> Surprisingly <tt style="color:rgb(0,122,0)">files.x86</tt> invokes a different set of files. I don't know what build it is.<br> <br> <div> <div></div> <div dir="ltr"> <pre dir="ltr"><code>x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xenhvm x86/xen/xen_apic.c optional xenhvm x86/xen/xenpv.c optional xenhvm x86/xen/xen_msi.c optional xenhvm x86/xen/xen_nexus.c optional xenhvm</code></pre></div></div></div></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Mario.<br></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSigxFsitie=RwbLMXw4YEMFxFX%2BGSnMopi9ayj5gfS2hYw>