Date: Tue, 27 Feb 2024 13:21:43 +0300 From: Vitaliy Gusev <gusev.vitaliy@gmail.com> To: Matthew Grooms <mgrooms@shrew.net> Cc: virtualization@freebsd.org Subject: Re: bhyve disk performance issue Message-ID: <BE794E98-7B69-4626-BB66-B56F23D6A67E@gmail.com> In-Reply-To: <e45c95df-4858-48aa-a274-ba1bf8e599d5@shrew.net> References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> <f01a9bca-7023-40c0-93f2-8cdbe4cd8078@tubnor.net> <edb80fff-561b-4dc5-95ee-204e0c6d95df@shrew.net> <a07d070b-4dc1-40c9-bc80-163cd59a5bfc@Duedinghausen.eu> <e45c95df-4858-48aa-a274-ba1bf8e599d5@shrew.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hi, > On 23 Feb 2024, at 18:37, Matthew Grooms <mgrooms@shrew.net> wrote: > >> ... > The problem occurs when an image file is used on either ZFS or UFS. The problem also occurs when the virtual disk is backed by a raw disk partition or a ZVOL. This issue isn't related to a specific underlying filesystem. > Do I understand right, you ran testing inside VM inside guest VM on ext4 filesystem? If so you should be aware about additional overhead in comparison when you were running tests on the hosts. I would suggest to run fio (or even dd) on raw disk device inside VM, i.e. without filesystem at all. Just do not forget do “echo 3 > /proc/sys/vm/drop_caches” in Linux Guest VM before you run tests. Could you also give more information about: 1. What results did you get (decode bonnie++ output)? 2. What results expecting? 3. VM configuration, virtio-blk disk size, etc. 4. Full command for tests (including size of test-set), bhyve, etc. 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you should try 4K. 6. Linux has several read-ahead options for IO schedule, and it could be related too. Additionally could also you play with “sync=disabled” volume/zvol option? Of course it is only for write testing. —— Vitaliy [-- Attachment #2 --] <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi,<div><br></div><div><div><br><blockquote type="cite"><div>On 23 Feb 2024, at 18:37, Matthew Grooms <mgrooms@shrew.net> wrote:</div><br class="Apple-interchange-newline"><div><div><blockquote type="cite">...</blockquote>The problem occurs when an image file is used on either ZFS or UFS. The problem also occurs when the virtual disk is backed by a raw disk partition or a ZVOL. This issue isn't related to a specific underlying filesystem.<br><br></div></div></blockquote><div><br></div>Do I understand right, you ran testing inside VM inside guest VM on ext4 filesystem? If so you should be aware about additional overhead in comparison when you were running tests on the hosts.</div><div><br></div><div>I would suggest to run fio (or even dd) on raw disk device inside VM, i.e. without filesystem at all. Just do not forget do “<span style="font-family: Menlo; font-size: 12px; background-color: rgb(231, 238, 238);">echo 3 > /proc/sys/vm/drop_caches</span>” in Linux Guest VM before you run tests. </div><div><br></div><div>Could you also give more information about:</div><div><br></div><div> 1. What results did you get (decode bonnie++ output)?</div><div> 2. What results expecting?</div><div> 3. VM configuration, virtio-blk disk size, etc.</div><div> 4. Full command for tests (including size of test-set), bhyve, etc.</div><div> 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you should try 4K.</div><div> 6. Linux has several read-ahead options for IO schedule, and it could be related too.</div><div><br></div><div>Additionally could also you play with “sync=disabled” volume/zvol option? Of course it is only for write testing.</div><div><br></div><div>——</div><div>Vitaliy</div><div><br></div></div></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BE794E98-7B69-4626-BB66-B56F23D6A67E>