Date: Wed, 28 Feb 2024 15:31:39 -0600 From: Matthew Grooms <mgrooms@shrew.net> To: Vitaliy Gusev <gusev.vitaliy@gmail.com> Cc: virtualization@freebsd.org Subject: Re: bhyve disk performance issue Message-ID: <3738b08a-7841-4d18-9439-5f1c73a5c9e1@shrew.net> In-Reply-To: <3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com> References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> <f01a9bca-7023-40c0-93f2-8cdbe4cd8078@tubnor.net> <edb80fff-561b-4dc5-95ee-204e0c6d95df@shrew.net> <a07d070b-4dc1-40c9-bc80-163cd59a5bfc@Duedinghausen.eu> <e45c95df-4858-48aa-a274-ba1bf8e599d5@shrew.net> <BE794E98-7B69-4626-BB66-B56F23D6A67E@gmail.com> <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> <1DAEB435-A613-4A04-B63F-D7AF7A0B7C0A@gmail.com> <b353b39a-56d3-4757-a607-3c612944b509@shrew.net> <3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------uA3ah2p5aRXCMQ7fXwU07F9H Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/28/24 15:02, Vitaliy Gusev wrote: > > >> On 28 Feb 2024, at 23:03, Matthew Grooms <mgrooms@shrew.net> wrote: >> >> ... >> The virtual disks were provisioned with either a 128G disk image or a >> 1TB raw partition, so I don't think space was an issue. >> >> Trim is definitely not an issue. I'm using a tiny fraction of the >> 32TB array have tried both heavily under-provisioned HW RAID10 and SW >> RAID10 using GEOM. The latter was tested after sending full trim >> resets to all drives individually. >> > It could be then TRIM/UNMAP is not used, zvol (for the instance) > becomes full for the while. ZFS considers it as all blocks are used > and write operations could have troubles. I believe it was recently > fixed. > > Also look at this one: > > GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk > > The problem of UNMAP that it could have unpredictable slowdown at any > time. So I would suggest to check results with enabled and disabled > UNMAP in a guest. > Yes. I'm aware of issues associated with TRIM/UNMAP, but that's not implemented by any hardware RAID vendor that I'm aware of. I tested with both hardware and software RAID10. The issue I'm reporting is present in both cases. I'm quite certain this has nothing to do with TRIM/UNMAP. -Matthew --------------uA3ah2p5aRXCMQ7fXwU07F9H Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <div class="moz-cite-prefix">On 2/28/24 15:02, Vitaliy Gusev wrote:<br> </div> <blockquote type="cite" cite="mid:3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <br id="lineBreakAtBeginningOfMessage"> <div><br> <blockquote type="cite"> <div>On 28 Feb 2024, at 23:03, Matthew Grooms <a class="moz-txt-link-rfc2396E" href="mailto:mgrooms@shrew.net"><mgrooms@shrew.net></a> wrote:</div> <br class="Apple-interchange-newline"> <div> <meta charset="UTF-8"> <div class="moz-cite-prefix" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">...</div> </div> </blockquote> <blockquote type="cite"> <div> <div class="moz-cite-prefix" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">The virtual disks were provisioned with either a 128G disk image or a 1TB raw partition, so I don't think space was an issue.</div> <p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Trim is definitely not an issue. I'm using a tiny fraction of the 32TB array have tried both heavily under-provisioned HW RAID10 and SW RAID10 using GEOM. The latter was tested after sending full trim resets to all drives individually.</p> </div> </blockquote> <div>It could be then TRIM/UNMAP is not used, zvol (for the instance) becomes full for the while. ZFS considers it as all blocks are used and write operations could have troubles. I believe it was recently fixed.</div> <div><br> </div> <div>Also look at this one:</div> <div><br> </div> <div> GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk</div> <div><br> </div> <div>The problem of UNMAP that it could have unpredictable slowdown at any time. So I would suggest to check results with enabled and disabled UNMAP in a guest.</div> <br> </div> </blockquote> <p>Yes. I'm aware of issues associated with TRIM/UNMAP, but that's not implemented by any hardware RAID vendor that I'm aware of. I tested with both hardware and software RAID10. The issue I'm reporting is present in both cases. I'm quite certain this has nothing to do with TRIM/UNMAP.<br> <br> -Matthew<br> </p> </body> </html> --------------uA3ah2p5aRXCMQ7fXwU07F9H--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3738b08a-7841-4d18-9439-5f1c73a5c9e1>