Date: Mon, 17 Aug 2020 10:04:07 +0900 From: KIRIYAMA Kazuhiko <kiri@truefc.org> To: freebsd-virtualization@freebsd.org Cc: kiri@truefc.org Subject: Re: Why qemu-system-aarch64 so late to process ? Message-ID: <202008170104.07H148Tm059096@kx.truefc.org> In-Reply-To: <202008110123.07B1NChc032219@kx.truefc.org> References: <202008110123.07B1NChc032219@kx.truefc.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi
Disk access performance is about 10 times between bhyve and
qemu:
root@jdtpkxb:~ # diskinfo -vt vtbd0p2
vtbd0p2
512 # sectorsize
31138512896 # mediasize in bytes (29G)
60817408 # mediasize in sectors
131072 # stripesize
20480 # stripeoffset
3785 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
# Disk descr.
BHYVE-438B-D187-47E7 # Disk ident.
# Attachment
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Seek times:
Full stroke: 250 iter in 0.019028 sec = 0.076 msec
Half stroke: 250 iter in 0.018842 sec = 0.075 msec
Quarter stroke: 500 iter in 0.032337 sec = 0.065 msec
Short forward: 400 iter in 0.027096 sec = 0.068 msec
Short backward: 400 iter in 0.027531 sec = 0.069 msec
Seq outer: 2048 iter in 0.106816 sec = 0.052 msec
Seq inner: 2048 iter in 0.113622 sec = 0.055 msec
Transfer rates:
outside: 102400 kbytes in 0.080248 sec = 1276044 kbytes/sec
middle: 102400 kbytes in 0.062225 sec = 1645641 kbytes/sec
inside: 102400 kbytes in 0.062555 sec = 1636959 kbytes/sec
root@jdtpkxb:~ # gpart show vtbd0
=> 40 67108784 vtbd0 GPT (32G)
40 1024 1 freebsd-boot (512K)
1064 60817408 2 freebsd-ufs (29G)
60818472 6290352 3 freebsd-swap (3.0G)
root@jdtpkxb:~ #
root@lpbkpkx:~ # diskinfo -vt vtbd0p2
vtbd0p2
512 # sectorsize
31138512896 # mediasize in bytes (29G)
60817408 # mediasize in sectors
0 # stripesize
272650240 # stripeoffset
60334 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
# Disk descr.
# Disk ident.
# Attachment
No # TRIM/UNMAP support
Unknown # Rotation rate in RPM
Seek times:
Full stroke: 250 iter in 0.127904 sec = 0.512 msec
Half stroke: 250 iter in 0.126985 sec = 0.508 msec
Quarter stroke: 500 iter in 0.285858 sec = 0.572 msec
Short forward: 400 iter in 0.206051 sec = 0.515 msec
Short backward: 400 iter in 0.201722 sec = 0.504 msec
Seq outer: 2048 iter in 1.050497 sec = 0.513 msec
Seq inner: 2048 iter in 1.019406 sec = 0.498 msec
Transfer rates:
outside: 102400 kbytes in 0.505535 sec = 202558 kbytes/sec
middle: 102400 kbytes in 0.620144 sec = 165123 kbytes/sec
inside: 102400 kbytes in 0.615600 sec = 166342 kbytes/sec
root@lpbkpkx:~ # gpart show vtbd0
=> 40 67108784 vtbd0 GPT (32G)
40 532480 1 freebsd-boot (260M)
532520 60817408 2 freebsd-ufs (29G)
61349928 5758896 3 freebsd-swap (2.7G)
root@lpbkpkx:~ #
Does it caused by disk cache ?
On Tue, 11 Aug 2020 10:23:12 +0900,
KIRIYAMA Kazuhiko wrote:
>
> Hi, all
>
> I'm working on qemu-system-aarch64 and try to build
> packages, but it's too late to compile stuffs. I've compare
> to bhyve package building procedure, then about 70 times
> later than bhyve one. I don't know why so much late on
> qemu-system-aarch64.
>
> Following results are devel/binutils package building
> process between bhyve and qemu-system-aarch64.
>
> === devel/binutils package building (bhyve) ===
>
> VM environments:
>
> root@vm:~ # cat /vm/jdtpkxb/jdtpkxb.conf
> loader="bhyveload"
> cpu=4
> memory=8GB
> network0_type="virtio-net"
> network0_switch="local"
> disk0_type="virtio-blk"
> disk0_name="disk0.img"
> storage=32G
> memory=8G
> cpu=4
> uuid="200bdead-cd7e-11ea-8cb7-0cc47ab3cfd4"
> network0_mac="58:9c:fc:02:0f:18"
> root@vm:~ # tail -35 /vm/jdtpkxb/vm-bhyve.log
> Jul 27 06:24:14: [primary disk dev: file]
> Jul 27 06:24:14: initialising network device tap1
> Jul 27 06:24:14: setting mtu of tap1 to 9000
> Jul 27 06:24:14: adding tap1 -> vm-local (local addm)
> Jul 27 06:24:14: bring up tap1 -> vm-local (local addm)
> Jul 27 06:24:14: booting
> Jul 27 06:24:14: bhyveload -c /dev/nmdm-jdtpkxb.1A -m 8GB -e autoboot_delay=3 -d /vm/jdtpkxb/disk0.img jdtpkxb
> Jul 27 06:24:18: [bhyve options: -c 4 -m 8GB -AHP -U 200bdead-cd7e-11ea-8cb7-0cc47ab3cfd4 -u]
> Jul 27 06:24:18: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/vm/jdtpkxb/disk0.img -s 5:0,virtio-net,tap1,mac=58:9c:fc:02:0f:18]
> Jul 27 06:24:18: [bhyve console: -l com1,/dev/nmdm-jdtpkxb.1A]
> Jul 27 06:24:18: starting bhyve (run 1)
> Jul 28 21:44:41: bhyve exited with status 1
> Jul 28 21:44:41: destroying network device tap1
> Jul 28 21:44:42: stopped
> Aug 10 09:58:51: initialising
> Aug 10 09:58:51: [loader: bhyveload]
> Aug 10 09:58:51: [cpu: 4]
> Aug 10 09:58:51: [memory: 8GB]
> Aug 10 09:58:51: [hostbridge: standard]
> Aug 10 09:58:51: [com ports: com1]
> Aug 10 09:58:51: [uuid: 200bdead-cd7e-11ea-8cb7-0cc47ab3cfd4]
> Aug 10 09:58:51: [utctime: yes]
> Aug 10 09:58:51: [debug mode: no]
> Aug 10 09:58:51: [primary disk: disk0.img]
> Aug 10 09:58:51: [primary disk dev: file]
> Aug 10 09:58:51: initialising network device tap2
> Aug 10 09:58:51: setting mtu of tap2 to 9000
> Aug 10 09:58:51: adding tap2 -> vm-local (local addm)
> Aug 10 09:58:51: bring up tap2 -> vm-local (local addm)
> Aug 10 09:58:51: booting
> Aug 10 09:58:51: bhyveload -c /dev/nmdm-jdtpkxb.1A -m 8GB -e autoboot_delay=3 -d /vm/jdtpkxb/disk0.img jdtpkxb
> Aug 10 09:58:54: [bhyve options: -c 4 -m 8GB -AHP -U 200bdead-cd7e-11ea-8cb7-0cc47ab3cfd4 -u]
> Aug 10 09:58:54: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/vm/jdtpkxb/disk0.img -s 5:0,virtio-net,tap2,mac=58:9c:fc:02:0f:18]
> Aug 10 09:58:54: [bhyve console: -l com1,/dev/nmdm-jdtpkxb.1A]
> Aug 10 09:58:54: starting bhyve (run 1)
> root@vm:~ #
>
> Package building process in VM:
>
> root@jdtpkxb:~ # top -bd1
> last pid: 54340; load averages: 2.91, 1.23, 0.61 up 0+05:28:59 15:27:58
> 37 processes: 5 running, 32 sleeping
> CPU: 0.3% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.4% idle
> Mem: 208M Active, 125M Inact, 838M Wired, 692M Buf, 6749M Free
> Swap: 3071M Total, 3071M Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 54299 root 1 75 0 154M 108M RUN 0 0:01 10.99% cc
> 54315 root 1 74 0 124M 84M CPU3 3 0:01 8.98% cc
> 54323 root 1 74 0 157M 113M CPU2 2 0:01 0.00% cc
> 16066 root 1 20 0 21M 9828K select 0 0:01 0.00% sshd
> 624 ntpd 1 20 0 21M 6420K select 2 0:01 0.00% ntpd
> 54339 root 1 73 0 132M 87M CPU1 1 0:01 0.00% cc
> 17898 root 1 52 0 19M 7540K wait 2 0:00 0.00% sh
> 16070 root 1 52 0 19M 7372K wait 1 0:00 0.00% sh
> 665 root 1 20 0 18M 7036K select 0 0:00 0.00% sendmail
> 52975 root 1 52 0 13M 3596K select 0 0:00 0.00% gmake
> 672 root 1 20 0 13M 2832K nanslp 0 0:00 0.00% cron
> 495 root 1 20 0 13M 2836K select 1 0:00 0.00% syslogd
> 25104 root 1 52 0 15M 4592K wait 3 0:00 0.00% gmake
> 25091 root 1 52 0 15M 4332K wait 3 0:00 0.00% gmake
> 23169 root 1 52 0 13M 3200K wait 2 0:00 0.00% make
> 761 root 1 20 0 14M 4448K pause 3 0:00 0.00% tcsh
> 52886 root 1 52 0 13M 3492K wait 0 0:00 0.00% gmake
> 728 root 1 39 0 21M 9440K select 3 0:00 0.00% sshd
>
> root@jdtpkxb:~ # uname -a
> FreeBSD jdtpkxb 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r363381M: Thu Jul 23 22:48:19 JST 2020 root@msrvkxb:/usr/obj/usr/src/amd64.amd64/sys/XIJ amd64
> root@jdtpkxb:~ # svnlite info /usr/ports
> Path: /usr/ports
> Working Copy Root Path: /usr/ports
> URL: http://svn.freebsd.org/ports/head
> Relative URL: ^/head
> Repository Root: http://svn.freebsd.org/ports
> Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
> Revision: 542063
> Node Kind: directory
> Schedule: normal
> Last Changed Author: glewis
> Last Changed Rev: 542063
> Last Changed Date: 2020-07-12 11:13:27 +0900 (Sun, 12 Jul 2020)
>
> root@jdtpkxb:~ #
>
> Package building result:
>
> :===> Building binutils-2.33.1_2,1 start 2020.08.10 15:25:33
> Updating database digests format: .......... done
> Checking integrity... done (0 conflicting)
> Deinstallation has been requested for the following 233 packages (of 0 packages in the universe):
> :
> (snip)
> :
> install -m 0644 /var/ports/work/var/ports/jdtpkxb/devel/binutils/work-native/binutils-2.33.1/includ
> e/plugin-api.h /var/ports/work/var/ports/jdtpkxb/devel/binutils/work-native/stage/usr/local/include/
> /bin/rm -f /var/ports/work/var/ports/jdtpkxb/devel/binutils/work-native/stage/usr/local/include/ansi
> decl.h
> /usr/bin/sed -i.bak '/#include "ansidecl.h"/d' /var/ports/work/var/ports/jdtpkxb/devel/binutils/work
> -native/stage/usr/local/include/bfd.h
> ====> Compressing man pages (compress-man)
> ===> Building package for binutils-2.33.1_2,1
> :<=== Building binutils-2.33.1_2,1 end 2020.08.10 15:32:44 (00:07:11)
>
>
> === devel/binutils package building (qemu-system-aarch64) ===
>
> VM environments:
>
> root@vm:~ # cat /vm/lpbkpkx/lpbkpkx.conf
> loader="qemu-system-aarch64"
> machine=arm64
> machine_arc=aarch64
> cpu=4
> memory=8GB
> network0_type="virtio-net-device"
> network0_switch="local"
> disk0_type="virtio-blk-device"
> disk0_name="disk0.img"
> storage=32G
> qemu_telnet_port="5044"
> qemu_monitor_port="5045"
> ssh_host="192.168.1.233"
> ssh_port="22"
> network0_mac="58:9c:fc:0b:1d:a6"
> uuid="7f22b8ab-c0bd-11ea-a00d-0cc47ab3cfd4"
> root@vm:~ # tail -35 /vm/lpbkpkx/vm-bhyve.log
> Aug 10 09:58:38: setting mtu of tap1 to 9000
> Aug 10 09:58:38: adding tap1 -> vm-local (local addm)
> Aug 10 09:58:38: bring up tap1 -> vm-local (local addm)
> Aug 10 09:58:38: booting
> Aug 10 09:58:38: [qemu standard options: -machine virt -cpu cortex-a57 -smp 4 -m 8G -name lpbkpkx -uuid 7f22b8ab-c0bd-11ea-a00d-0cc47ab3cfd4]
> Aug 10 09:58:38: [qemu block device options: -device virtio-blk-device,drive=hd0 -drive if=none,format=raw,file=/vm/lpbkpkx/disk0.img,id=hd0]
> Aug 10 09:58:38: [qemu display options: -nographic]
> Aug 10 09:58:38: [qemu network options: -netdev tap,id=net0,ifname=tap1 -device virtio-net-device,netdev=net0,mac=58:9c:fc:0b:1d:a6]
> Aug 10 09:58:38: [qemu debug/expert options: -monitor telnet::5045,server,nowait,nodelay -serial telnet::5044,server,nowait,nodelay -bios /vm/.config/QEMU_EFI.fd]
> Aug 10 09:58:38: starting qemu (run 1)
> Aug 10 10:04:16: qemu exited with status 0
> Aug 10 10:04:16: destroying network device tap1
> Aug 10 10:04:16: stopped
> Aug 10 10:04:29: initialising
> Aug 10 10:04:29: [uefi: QEMU_EFI.fd]
> Aug 10 10:04:29: [cpu: cortex-a57]
> Aug 10 10:04:29: [smp: 4]
> Aug 10 10:04:29: [memory: 8GB]
> Aug 10 10:04:29: [hostbridge: standard]
> Aug 10 10:04:29: [uuid: 7f22b8ab-c0bd-11ea-a00d-0cc47ab3cfd4]
> Aug 10 10:04:29: [utctime: yes]
> Aug 10 10:04:29: [debug mode: no]
> Aug 10 10:04:29: [primary disk: disk0.img]
> Aug 10 10:04:29: [primary disk dev: file]
> Aug 10 10:04:29: initialising network device tap1
> Aug 10 10:04:29: setting mtu of tap1 to 9000
> Aug 10 10:04:29: adding tap1 -> vm-local (local addm)
> Aug 10 10:04:29: bring up tap1 -> vm-local (local addm)
> Aug 10 10:04:29: booting
> Aug 10 10:04:29: [qemu standard options: -machine virt -cpu cortex-a57 -smp 4 -m 8G -name lpbkpkx -uuid 7f22b8ab-c0bd-11ea-a00d-0cc47ab3cfd4]
> Aug 10 10:04:29: [qemu block device options: -device virtio-blk-device,drive=hd0 -drive if=none,format=raw,file=/vm/lpbkpkx/disk0.img,id=hd0]
> Aug 10 10:04:29: [qemu display options: -nographic]
> Aug 10 10:04:29: [qemu network options: -netdev tap,id=net0,ifname=tap1 -device virtio-net-device,netdev=net0,mac=58:9c:fc:0b:1d:a6]
> Aug 10 10:04:29: [qemu debug/expert options: -monitor telnet::5045,server,nowait,nodelay -serial telnet::5044,server,nowait,nodelay -bios /vm/.config/QEMU_EFI.fd]
> Aug 10 10:04:29: starting qemu (run 1)
> root@vm:~ #
>
> Package building process in VM:
>
> root@lpbkpkx:~ # top -bd1
> last pid: 31159; load averages: 4.48, 4.58, 4.37 up 0+06:07:42 16:12:45
> 36 processes: 5 running, 31 sleeping
> CPU: 4.3% user, 0.0% nice, 7.4% system, 0.9% interrupt, 87.5% idle
> Mem: 33M Active, 329M Inact, 406M Wired, 302M Buf, 7203M Free
> Swap: 2812M Total, 2812M Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 31123 root 1 85 0 0B 16K CPU1 1 0:04 43.46% cc
> 30320 root 1 80 0 13M 3180K RUN 2 0:17 30.18% sh
> 31137 root 1 81 0 87M 51M CPU0 0 0:03 29.49% cc
> 31109 root 1 52 0 85M 49M wait 3 0:02 8.79% cc
> 25892 root 1 52 0 13M 3760K wait 2 0:39 3.66% sh
> 873 admin 1 21 0 20M 9224K select 2 0:01 1.76% sshd
> 26808 root 1 52 0 13M 3640K wait 2 0:36 1.37% sh
> 878 root 1 21 0 14M 4376K pause 0 0:02 1.07% tcsh
> 8761 root 1 20 0 20M 9652K select 0 0:22 0.88% sshd
> 28889 root 1 52 0 13M 3100K select 0 0:05 0.88% gmake
> 769 ntpd 1 20 0 21M 5896K select 3 0:34 0.10% ntpd
> 18670 root 1 52 0 13M 3408K wait 1 0:51 0.00% sh
> 10643 root 1 52 0 16M 7076K wait 0 0:16 0.00% sh
> 8765 root 1 52 0 16M 6928K wait 3 0:11 0.00% sh
> 810 root 1 20 0 17M 6440K select 3 0:09 0.00% sendmail
> 817 root 1 20 0 13M 2712K nanslp 0 0:03 0.00% cron
> 575 root 1 20 0 13M 2712K select 1 0:02 0.00% syslogd
> 15754 root 1 35 0 15M 4456K select 2 0:02 0.00% gmake
>
> root@lpbkpkx:~ # uname -a
> FreeBSD lpbkpkx 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r362853: Thu Jul 2 04:21:45 UTC 2020 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64
> root@lpbkpkx:~ #
>
> Package building result:
>
> :===> Building binutils-2.33.1_2,1 start 2020.08.10 15:25:20
> Updating database digests format: .......... done
> Checking integrity... done (0 conflicting)
> Deinstallation has been requested for the following 16 packages (of 0 packages in the universe):
> :
> (snip)
> :
> install -m 0644 /var/ports/work/var/ports/lpbkpkx/devel/binutils/work-native/binutils-2.33.1/include/plugin-api.h /var/ports/work/var/ports/lpbkpkx/devel/binutils/work-native/stage/usr/local/include/
> /bin/rm -f /var/ports/work/var/ports/lpbkpkx/devel/binutils/work-native/stage/usr/local/include/ansidecl.h
> /usr/bin/sed -i.bak '/#include "ansidecl.h"/d' /var/ports/work/var/ports/lpbkpkx/devel/binutils/work-native/stage/usr/local/include/bfd.h
> ====> Compressing man pages (compress-man)
> ===> Building package for binutils-2.33.1_2,1
> :<=== Building binutils-2.33.1_2,1 end 2020.08.10 23:26:11 (08:00:51)
>
>
> Best Regards
>
> ---
> Kazuhiko Kiriyama <kiri@truefc.org>
> _______________________________________________
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202008170104.07H148Tm059096>