From nobody Mon Feb 26 17:01:29 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tk6Mt35jQz5Bmdj for ; Mon, 26 Feb 2024 17:01:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tk6Mt1v39z4YbV for ; Mon, 26 Feb 2024 17:01:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708966890; a=rsa-sha256; cv=none; b=MQGC7Tve7hf8KWBLn1j5xaXvB7cnAjkr+4tPj68rzEF54NGGdOyqIZHZRHuNmVBFVDfLN0 vbmRTi6YlKqnyrUYXankOX+H0mSSz09DPVWCUry5JmjbwHUBOt/opYFxsMgtNzFNhR+FjI 2N37NzrW4o+92HLB1TnCL2rh9qBbEE0jv/jUjOblrGN4BaXgAkcv3V+JpLwYLfUul8RBSA YazJYKmOfPqT0+VT7Ys+CQHfCXTtdL2NvB1hxcPOKKmdIyIQG6vO+uEJfhiaQuVyiif9h0 g0ALJmLS4GTZQsXeEOnZ4jtZjLSbov6k3C7WTW7qtEGQlKteXTCSlrMwXUxkqA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708966890; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QdBQr/gOmu8Y4hEwjNEx7Mb41X2bgI6c5pcQZq9xM6I=; b=R6/57lUE2K/3iccMbCPcdhMC4KJqLdTXwDUMlEcsejWJwRCxz128OwLc7Vqb8UFNgTqdHN lOoDkQMa9hS9u52lhFLNifiH0fLhdQHThcYhx3+b2A6AP8OSsEW+WqWG7qSSk3fn7CTggV 3HnQLV+lrvK7/HGMDm08IdnFFVxOvmdLu0opOscn9Avob0xrbNF0J/QFAw9eZAr9vIQ4aG mhKgmrjUKQjXKrI9/5DdI1IZZhZuVqEXulHh+rJZcV0J4Azsv3Hi72Y85rNDfeJ39VttuQ NhZYeNe2XYHtbRy5W/lATYS+N6bm328Uy91ECiSWbthOiYvk9wS/ffZRyXjQ+A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tk6Mt119XzcRT for ; Mon, 26 Feb 2024 17:01:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41QH1UXS068000 for ; Mon, 26 Feb 2024 17:01:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41QH1UEm067999 for virtualization@FreeBSD.org; Mon, 26 Feb 2024 17:01:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 277326] FreeBSD guest on QEMU with Hyper-V enlightenment on is no longer able to detect Hyper-V devices Date: Mon, 26 Feb 2024 17:01:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277326 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |virtualization@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 10:21:43 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkYSc45fZz5CtZD for ; Tue, 27 Feb 2024 10:22:08 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkYSb6GPhz4qxv for ; Tue, 27 Feb 2024 10:22:07 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=icCT19zs; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of gusev.vitaliy@gmail.com designates 2a00:1450:4864:20::132 as permitted sender) smtp.mailfrom=gusev.vitaliy@gmail.com Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-513143d3c40so94989e87.3 for ; Tue, 27 Feb 2024 02:22:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709029325; x=1709634125; darn=freebsd.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=u35HOfZx09c0Jkx1DaHouJbTlLDHGu3cYqXhRUVbGIw=; b=icCT19zsC+ZKBGyrBNJToUHkzFnWbFi2LIPyPluBVqwBNmTVbY6VLsUAMciheRVrXO t03biV4rKNMX1mSee/qTDPVv5QInssA0cUMWslxCwXu2hT731gHTpdXRo307Do+3PhlX ZbuV/TYgoZZ7vlFWEMuDtQ6yKRn0g48P4JkxV4ClvyTTSy8Iyvu9Y+0uJ10vtB8Qp1T5 fRE0f0XcAU3CHvtFnj+B42M7E2FolJplDUAb2inMxDDlze6N6FA6aZ+vAA69ulJ3o0C8 5uO1XfCQ5x3crADpjuW7Devjaeae6nzoe6Z6M6J4FK0MlcyLCAchHvc3CTW5L40eZzTJ xH1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709029325; x=1709634125; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u35HOfZx09c0Jkx1DaHouJbTlLDHGu3cYqXhRUVbGIw=; b=cTJRQ7ZVW+ldIplltKQ+egmrs+6VDnEMmrAWA9xDSdy8FRZ3YL5ujPds5BDS8RLQjJ XnAgXsqJ6CwL8liMZMIL4TrWjUGC23A1jXOGf7gyH/f8uxwLBg8mSrGKB5r/+zBw59uB C9pFIfiGf3nXve5JtuFwsMdOIOI0oPiLB5NCAcOgYyG1cbUJPITyhsgXeU1i7OgKkSuD 8c9d7IfPJhElAle3u2JNxH34wFSqwwztQcUK23JboOPbm/UU2P5F4gY/h/N3DJTZkPMT iszS/z85Lh/OLm5zOSM4lJiRoTHfhBGFDgjfTupPyM+3jDmW+Hdg6/rgS3/JqBZsw47v l1lA== X-Gm-Message-State: AOJu0Yx13yreQMz16SJNTaTzf0qeZm46jrr4efqX6LlZapWpV0FMcyWS cFDDEz9e8dDC6pZlUH/EhQBI+TZtxUBamOU1vYp6QgHMTTWeJZvQiBlJFmr6oh2jmw== X-Google-Smtp-Source: AGHT+IHttSmbTXvL1KiiSWxtGaUrv1BZ0niufj5YCqPffhupgstsD3uRJB/xpDzcOIUnhSFiNgFuaw== X-Received: by 2002:a05:6512:234d:b0:512:d89e:946 with SMTP id p13-20020a056512234d00b00512d89e0946mr7427163lfu.44.1709029324554; Tue, 27 Feb 2024 02:22:04 -0800 (PST) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id u7-20020a056512040700b00513131a218fsm70990lfk.97.2024.02.27.02.22.04 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Feb 2024 02:22:04 -0800 (PST) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: bhyve disk performance issue Date: Tue, 27 Feb 2024 13:21:43 +0300 In-Reply-To: Cc: virtualization@freebsd.org To: Matthew Grooms References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.48)[-0.483]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::132:from] X-Rspamd-Queue-Id: 4TkYSb6GPhz4qxv --Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, > On 23 Feb 2024, at 18:37, Matthew Grooms wrote: >=20 >> ... > The problem occurs when an image file is used on either ZFS or UFS. = The problem also occurs when the virtual disk is backed by a raw disk = partition or a ZVOL. This issue isn't related to a specific underlying = filesystem. >=20 Do I understand right, you ran testing inside VM inside guest VM on = ext4 filesystem? If so you should be aware about additional overhead in = comparison when you were running tests on the hosts. I would suggest to run fio (or even dd) on raw disk device inside VM, = i.e. without filesystem at all. Just do not forget do =E2=80=9Cecho 3 > = /proc/sys/vm/drop_caches=E2=80=9D in Linux Guest VM before you run = tests.=20 Could you also give more information about: 1. What results did you get (decode bonnie++ output)? 2. What results expecting? 3. VM configuration, virtio-blk disk size, etc. 4. Full command for tests (including size of test-set), bhyve, etc. 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you should = try 4K. 6. Linux has several read-ahead options for IO schedule, and it could = be related too. Additionally could also you play with =E2=80=9Csync=3Ddisabled=E2=80=9D = volume/zvol option? Of course it is only for write testing. =E2=80=94=E2=80=94 Vitaliy --Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi,


On 23 Feb 2024, at 18:37, Matthew Grooms = <mgrooms@shrew.net> wrote:

...
The problem occurs when an image file is = used on either ZFS or UFS. The problem also occurs when the virtual disk = is backed by a raw disk partition or a ZVOL. This issue isn't related to = a specific underlying = filesystem.


Do I = understand right, you ran testing inside VM inside guest VM  on = ext4 filesystem? If so you should be aware about additional overhead in = comparison when you were running tests on the = hosts.

I would suggest to run fio (or even dd) = on raw disk device inside VM, i.e. without filesystem at all.  Just = do not forget do =E2=80=9Cecho 3 > = /proc/sys/vm/drop_caches=E2=80=9D in Linux Guest VM before you = run tests. 

Could you also give more = information about:

 1. What results did = you get (decode bonnie++ output)?
 2. What results = expecting?
 3. VM configuration, virtio-blk disk size, = etc.
 4. Full command for tests (including size of = test-set), bhyve, etc.
 5. Did you pass virtio-blk as 512 = or 4K ? If 512, probably you should try 4K.
 6. Linux has = several read-ahead options for IO schedule, and it could be related = too.

Additionally could also you play with = =E2=80=9Csync=3Ddisabled=E2=80=9D volume/zvol option? Of course it is = only for write = testing.

=E2=80=94=E2=80=94
Vitaliy

= --Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6-- From nobody Tue Feb 27 15:21:05 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tkh5Y5VX6z5BtFf for ; Tue, 27 Feb 2024 15:21:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tkh5Y3HtZz4WGW for ; Tue, 27 Feb 2024 15:21:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709047265; a=rsa-sha256; cv=none; b=HBq7u61cH3tpHzM7OHSj6CC4FQeDAtenFVA32KYrcv/C3f6swn6Dwaxi89hpv2s1WiL+xp E6aYd3U2+4YgrkM2Naoqjzs69rK7aPVKNqZJPRCsWjJcRpOhSSjhhiZtiM6S35co2N4LxL jX5k+qKkvU/VaQShn+dX0ELxCfeCm9UKiCW/wGJeCZgc++Zfo5L4H2E8DCuPj4xvZk3RC6 Yzi5VG3qoPuU3W9+cQpzgmtlk3Oad6kVH6WRH9ThMwT1Yt16T4taPTE9WCfxmyq2jwrA97 4/IEPK5iIKh29aom2lOIaLY1ZsIqZtlZqLjwNxm1LZ+A3WREsw9eMx/8Wv4u7A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709047265; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MrHCYHYgfcy4jJXq4kqI1gKecaC6SqQyhDhBkv0XI8s=; b=NlSAPE1pGY15KvJk6V2U44WQ8gWGeJVqqDp3P5guYIbl2eV10vQ+3o2CdNb+mCbzPa59tD xrTzamSqBesNmE+p8t1p+wjbXXe6+GN2zd3YS/AjkqwbRNAEnL+HtQJdizUsOMF42npkj3 uCk5Yvgs80KgcfrJU37txiF+C+SCqYUfIELpx0HJsQI2c4j6Vl+icHYWMr/ozEkgql9Cna 4X0w/SGSM3hLFN6kE7Iq+KNp/5iU+Bev8mruf1MiZEh1uiMUdFK2VRfT6O6e0lCDeWbvTe OTDKmetVmjrjSKJXaqEQ7GoKxt18Na9SV3UYKq+pMvJy1O7nT1sX2wOJhfHe2A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tkh5Y2N9mzH0Q for ; Tue, 27 Feb 2024 15:21:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RFL543046524 for ; Tue, 27 Feb 2024 15:21:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RFL5X5046523 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 15:21:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 277326] FreeBSD guest on QEMU with Hyper-V enlightenment on is no longer able to detect Hyper-V devices Date: Tue, 27 Feb 2024 15:21:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zlei@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277326 --- Comment #4 from Zhenlei Huang --- (In reply to Roger Pau Monn=C3=A9 from comment #3) > Hm, I guess I'm a bit confused. While debugging PR 274810, I see the behavior, that FreeBSD guests report HyperV rather than KVM when Hyper-V enlightenment is enabled. > If the hypervisor is KVM, shouldn't it be detected as KVM rather than Hyp= erV? I think KVM should be reported. But it seems everything goes right even Hyp= erV is reported. > What's the benefit of using the HyperV enlightenments when running on QEM= U/KVM? I think Microsoft people are more familiar with that. When debugging PR 274= 810 I read briefly the design of Hyper-V virtual devices. I think the design is pretty clear and performant. I see lots of comparing of vm_guest in hyperv device drivers. I guess they should be removed to correctly probe Hyper-V devices, to correctly support Hyper-V enlightenment feature. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 15:25:08 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkhBD4G2jz5Btr7 for ; Tue, 27 Feb 2024 15:25:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkhBD2dqDz4WtZ for ; Tue, 27 Feb 2024 15:25:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709047508; a=rsa-sha256; cv=none; b=oJqTPlwaLFh4cLQPI1NdpPn7zswOXuM0bJRBk/ql7Jkl/44R4DWUlNuwY87jbn6gzk2xI3 yrrHskHtkqP7Pmny1OiU4gwQDSdFcX93hWQMgmsj+GSz+j9R8I3QgUHjmGDh4fUUXqYDZv HK0DpfDz79lh1rhdH4D63sGGgMNuTr6wOH+DA8yI5XmQOune+oNCMM25i0B3hsFuIw5FRL SZ67claPit/9i8UGQ2ooPiVD5cNBYqZyhg9QB7ipUzVBNA/EkalZkLBwjiopyQrkPE5y0r q/T2grsNHDIowbR8znL06OOszMmGVQRbmUMwOi0WQV7C7GFhS2acTS0gwFXDzg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709047508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9NtQyaXc/XJIYHfpsXYr4wFBCfgbBlaEq4QS+y1jikE=; b=mNDO2Nlfn597XE94PoLkC8KB3Tas/Vzr2+GVbbFRXMRERYb80k1stHfrYqXV2Usbx8NhwL KvGKieLD1ck4YJsQzlYPDqu08GvBz6fGXl3Piurgps4K3F/473yF64fqxcydp4l45QnH+8 yiu1cSr8OEBz91YxQRH8WRoCFf4q5F3PM1k1CCmzj6mZ8N1byvt3xEP/zkFOrignPwcfZE yVx36aoedXmu5EnvIlgXyix6q9kIX3SkI4gzMbpzHE8E0iSmevEkEczjgHSGd4JZ3kxZuB gN+hqNQiUCKP3cpEIBzO0ldn6Zl4iOMazkFgvxrpCLBXi4Kf0FRqwLcjm13W9g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TkhBD1d0GzHwM for ; Tue, 27 Feb 2024 15:25:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RFP8fo068754 for ; Tue, 27 Feb 2024 15:25:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RFP8ob068753 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 15:25:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 277326] FreeBSD guest on QEMU with Hyper-V enlightenment on is no longer able to detect Hyper-V devices Date: Tue, 27 Feb 2024 15:25:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zlei@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277326 --- Comment #5 from Zhenlei Huang --- (In reply to Zhenlei Huang from comment #4) > I see lots of comparing of vm_guest in hyperv device drivers. I guess the= y should be removed to correctly probe Hyper-V devices, to correctly suppor= t Hyper-V enlightenment feature. In case KVM should be reported when Hyper-V enlightenmen is enabled. # grep -r 'VM_GUEST_HV' sys/dev/hyperv sys/dev/hyperv/storvsc/hv_storvsc_drv_freebsd.c: if (vm_guest =3D=3D VM_GUEST_HV) { sys/dev/hyperv/vmbus/vmbus.c: if (device_get_unit(parent) !=3D 0 || vm_gu= est !=3D VM_GUEST_HV || sys/dev/hyperv/vmbus/vmbus.c: if (device_get_unit(dev) !=3D 0 || vm_guest= !=3D VM_GUEST_HV || sys/dev/hyperv/vmbus/vmbus.c: if (vm_guest !=3D VM_GUEST_HV || sc =3D=3D = NULL) sys/dev/hyperv/vmbus/hyperv.c: if (vm_guest =3D=3D VM_GUEST_HV) sys/dev/hyperv/vmbus/hyperv.c: if (vm_guest !=3D VM_GUEST_HV) sys/dev/hyperv/vmbus/vmbus_res.c: if (device_get_unit(dev) !=3D 0 || vm_guest !=3D VM_GUEST_HV || sys/dev/hyperv/vmbus/x86/hyperv_x86.c: if (vm_guest !=3D VM_GUEST_HV) sys/dev/hyperv/vmbus/aarch64/hyperv_aarch64.c: vm_guest =3D VM_GUEST_HV; sys/dev/hyperv/hvsock/hv_sock.c: if (vm_guest !=3D VM_GUEST_HV) sys/dev/hyperv/netvsc/if_hn.c: if (vm_guest !=3D VM_GUEST_HV) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 16:52:27 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tkk6z62dqz5C2D0 for ; Tue, 27 Feb 2024 16:52:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tkk6z4K7lz4hFH for ; Tue, 27 Feb 2024 16:52:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709052747; a=rsa-sha256; cv=none; b=JTRpQBr1lF64xAL5vSaZ4qDjbU5bpoIwes0DgEP4lW0IG58r5tgh3f/ZEGNYPZGjhKDcdm EJcI91+6VYxwGBVHyzrBf12xITKfJEEjem33XCk/0Id47+je5Dm7XVNrs1lf2Jgw500Ijh ngXl8brbKCU/2whMt71UT7R1bVXlUnCtmBPNWy9JIvBGqnA1GcFck6r/krrNA2vbnECR2M 40AD+MLkM37ag+O6d7GG+x9l0ZC7P73LC5C2uNXey8neJCPshyvZb9KDEhnQjE8GgOqPaO XLGej7XQ4pwvIT/fWfqrDUzevl5lGVkdALsLvzDdYXBWcK3bD0fv/PdhM4v6Yw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709052747; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TEWGR0WE+ka56ypSdy4pI/6GirG5OkGXM71OVWSWAxw=; b=rURt5oM+1g6MfTFEOWlGqlHbIMqqM5EVxbeAZO5SMGwszDmD6FyqPj9HxqcKyat60r6B32 U7F5w5ekYZALK5MNR892LzKYnxcF6eXSMNOEgsqZnUln+TWA/6ohjo5HXwgb1xKkDgqEbJ 7jP15Z7FFoIApMduIohYkJeNMVHwSz48Kth7guwVPa8FOsu7cV4jNm8oVaFpDY3qnTepo9 MEQ5SC+fNRDZDLKGr0UvUA99Z6+fTiAF9HkLWNifemk98qNZgVi2HwcZHKjUEIO6xbnFKt P53XKmrFVq1cqk3dKR4nQ9jsEPc1gZeRlMYDQ7DGby7IJmERf8oSUmbtPrNNKg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tkk6z3H8lzKkq for ; Tue, 27 Feb 2024 16:52:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RGqRME027152 for ; Tue, 27 Feb 2024 16:52:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RGqRMF027151 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 16:52:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 277326] FreeBSD guest on QEMU with Hyper-V enlightenment on is no longer able to detect Hyper-V devices Date: Tue, 27 Feb 2024 16:52:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: 000.fbsd@quip.cz X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277326 Miroslav Lachman <000.fbsd@quip.cz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |000.fbsd@quip.cz --- Comment #6 from Miroslav Lachman <000.fbsd@quip.cz> --- According to this document https://www.qemu.org/docs/master/system/i386/hyperv.html some thing looks like Hyper-V, but for others you can tell it's a KVM hypervisor. "When any set of the Hyper-V enlightenments is enabled, QEMU changes hyperv= isor identification (CPUID 0x40000000..0x4000000A) to Hyper-V. KVM identification and features are kept in leaves 0x40000100..0x40000101." --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 17:16:44 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tkkg05vRgz5C4PJ for ; Tue, 27 Feb 2024 17:16:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tkkg029bvz4k5K for ; Tue, 27 Feb 2024 17:16:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709054204; a=rsa-sha256; cv=none; b=GEoQkmO2yNf2h4+xs3MS3hlbM3t2qOGqD7mxsYGwOyXNQQXYJ+RxS0q7ZzIlr9H+TcBP2M q4oUF4UThhgLgjRlRMdz7TiCOa5r35/CZMKPR01dwV582p4d5wVVeiqdiaqrFx/D/mouTt YfdR+3530x6RPMA1rYsZ7YofNwxPVub3L4VUmHvXyiIHHIwCAEZa2itv81ZhPgrghmy1un /xuZlyJXyJWeTC7pHVJRGHvEIXFMJOnfJdcxjt2Oh/cAoW9osVQ+kd/plmJ26pAI8FAHbJ cXoAse+J8t1OqMgX8eM+ia6bS23MvGcL9sJq3glC+huxSsqfkUZZu+0l8ga2Ag== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709054204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o7pURxXozOm2lyXbkSXo3xaBBSl8Eap2UMx2U5OM/cE=; b=F6lRf4FJ0+BbIPA2FT2NpSyLwB0EAwhv3LW218v8wc39HylpK+xCQQzG3a0BiJPVHhCymx ppiDFIrzwIAhGdl3kbA8kF09i2P+Cj3Gs4umhu/SZlD9eAaGIVbn77r1FvnW4MdtXiZK7t fyc0WYbzsMzBaKwMcfdZ9HhW/ol0pNnuExa+zMxNy+/jhdfvakt9vPxQuUU9+3nagTfS3z YfUkNfJIYvA7DB+0NSalh66m/nVpgU6HYtT2QGkCQVGr3bUG0JlWSxXDOeWIMPn2zhemLY KMtD49422q1t0XZV9eIs3zN1PUdM+nuS8rd438L0uwQhZYpetYdIEU8+MNF1aw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tkkg01DjpzLsl for ; Tue, 27 Feb 2024 17:16:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RHGiLu048543 for ; Tue, 27 Feb 2024 17:16:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RHGirC048542 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 17:16:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 277326] FreeBSD guest on QEMU with Hyper-V enlightenment on is no longer able to detect Hyper-V devices Date: Tue, 27 Feb 2024 17:16:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: royger@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277326 --- Comment #7 from Roger Pau Monn=C3=A9 --- (In reply to Miroslav Lachman from comment #6) That's the same on Xen, the first CPUID signature is the HyperV one, but the following one is the Xen one, and we want to use the Xen one, as the Viridi= an set of features emulated by Xen is not very complete and just tested against Windows guests. This was already discussed in the differential revision of= the change for Xen: https://reviews.freebsd.org/D43508 So the question boils down to whether FreeBSD wants to treat KVM with HyperV extensions as HyperV or as KVM. I think the latter, as KVM is the real hypervisor, disguising as HyperV. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 18:11:07 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tklsl58rBz5C93W for ; Tue, 27 Feb 2024 18:11:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tklsl1ltgz4rJM for ; Tue, 27 Feb 2024 18:11:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709057467; a=rsa-sha256; cv=none; b=snKErFbdNNa0viOljO30SWbZL566PMDUTKjyAz5/5AwLRrLuJmdbwO0/ZhD2GCK7UIpNvz hIkcmarQUsNsqawYkMYJiMqfBleALuD80zxChJprhvNHaKWKndsDz3jqawGaiTwyRoDBiC wKHHPmFdEvbdUQkn8C6Q3KqUOEUzgn4n/oGRU3m05lRu1IMnpkpWL9FtEASeHPu8K65uqx zY7ORWQ/V9laX3O8mp9neRSeNNq6QGnCrzk8a0vdErF62IzMA17Vc/dokofEJpuRILqtVT luvFc5L1LF2QnqdjycRzPTEJ8Lw7ahQAV2+vsRwvAIu580u00ucI+Rq3uc1wqw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709057467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G30UmP2c0pZrBXOklvZO8hep4wzu32JcpLbhpopl+CQ=; b=fNqEBLLS30OBUzgWfezuvtHVXe0BtsIGauRz92fyAE0lZ8gllhFmrzjBZ/6mXhg7FVck0s NCaM6VH4oeVzVTl94JbykU8SHAGcOD00XZ3mnqbDkYphG6IZzZNRxW4TaiOUSqSR8C6LQw TVm7wTKLakIp+eoYOTzgVExvgtzMn8sve6/8d2jYSIYqq1vdwaczSYByN8p/Y04dTHr1gP iSR92wOXQIJcerp2YreCf8+JMnVDd80bN5kwNyG12PT+BnJSPrAcJ4REb+lvZhFM76zuh4 e5tCieX3PVH7LLu3aPqGxMlSB4NMY1XtEQOgcluTtJDe256x/9nwaWFdpFxq+Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tklsl0rGkzMrw for ; Tue, 27 Feb 2024 18:11:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RIB75u033486 for ; Tue, 27 Feb 2024 18:11:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RIB7Nl033483 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 18:11:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 276524] Setting LUN block size in ctl.conf to 512 will use volblocksize instead Date: Tue, 27 Feb 2024 18:11:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276524 --- Comment #18 from Alexander Motin --- Format Device mode page reports physical sector size, not physical block si= ze.=20 That concept is used for bad block marking/relocation and appeared very ear= ly in SCSI specs, long before logical blocks being formalized in SBC-3. Since Format Device mode page is obsolete since SBC-2 specification, released alm= ost 20 years ago, it is hard to guess what would be its meaning in today's term= s, but I suppose physical sector smaller than physical block would not make se= nse. So I'll update the code to report physical block size there instead of log= ical block size. But after that I have a strong itch to remove Format Device mo= de page together with Rigid Disk Geometry mode page, obsoleted for the same 20 years. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 18:26:39 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkmCh0SLNz5CBgd for ; Tue, 27 Feb 2024 18:26:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkmCg66Prz4w3R for ; Tue, 27 Feb 2024 18:26:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709058399; a=rsa-sha256; cv=none; b=GwhNHjTiRqD9SIAEoJSjJaXVnndY5C62lzZuZ6uDg3BTacnAEeZL6OOJzhsRqZqcy12ZWy oNcl61yPvImZaMIo59tg4KYbcJ0yEK9dbrKRDlE22ybyf2Jd5TWnTBcZ7j0xx3wroShn3U PyJ7tFt7rroqQfz9MWjMBVD8dsdgJfj/CZwu91pH0iPlSwwTzMZmjCjZB57P1fFzNvk7cL ZFWEAY1iCGCCQrt5Vx86isyNIhjA6nP1TG8yPqNBB7uUOkK4TIorEHDRpxqDQ8rS6GQ2aG zeItYDbDAPu/xAIb3G9km8Iorp0KWO43fwB8TpxvBSh98k1mfhXBdL2St65mKw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709058399; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J+3KBlQT7isDrEGQDYRyalx7AuGscCKy8nTEd4bh8ZY=; b=j1H9KZEbEha2onelUIsVVdpYiklwS+SAIyXZLe6SZADh1d8yOqcS2YUxIj5TOmZtKmRFT5 Kufy3z9UFggbU8+yDoWxzCEkYdX0JxIjfNuDPjN+tUOmJhcsLx5TKYuPEi8vnY3XykT2uV FajKSXi39T14+t0PvC07ritZBRCjT6aGtXZEUlaL7wfZwTJfL5Afhclsst48B8NoORwXN1 kDsAA2BeF4k1fbhPQ6cLw3uUuC/wbO6P5UKbD4YVRxGKPxuuMYK+UKB8PMEKKzsGvp1kqB n8zAejj2PwA3j9TVotS3E6OMiWUTAjlQOtUVQN5pgMuXLyofbe5VFiWzyJzO5g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TkmCg56z0zNXZ for ; Tue, 27 Feb 2024 18:26:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RIQdOq015149 for ; Tue, 27 Feb 2024 18:26:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RIQdgi015147 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 18:26:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 276524] Setting LUN block size in ctl.conf to 512 will use volblocksize instead Date: Tue, 27 Feb 2024 18:26:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276524 --- Comment #19 from Alexander Motin --- Thinking more, those physical sectors should somehow fit into disk CHS geometry, based on logical sectors, and I don't want to get even close to t= hat legacy disaster. I think I'll just remove it all together. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 18:53:04 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tkmp90Hrnz5CFFS for ; Tue, 27 Feb 2024 18:53:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tkmp86LDGz51xJ for ; Tue, 27 Feb 2024 18:53:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709059984; a=rsa-sha256; cv=none; b=HEKIofL8JFVMUxGKf+rcP/zIQ1s47XshHbo6rXlXGM+uUXHSrSpiFUHIAKgvFIykWAyDn2 f9ca4oCZdc8d++wqAuied6nG8wqNomHg687kGgNGdO/L9x0N9q5shSHEIMYkLY7YcWoj46 BG11hzu/VzvTKXzxJ4CYLGSRQAfPV4OiYBOYPDNI/R42vjUcKz5DnmJSW7VypMzbN0Gp48 N0jlUFhU8VjcWOo9rXsNf/Rj20DfeuM+fN+Al8R7Eib5e7na77Fq7GQPXYNl8py0Ag1Fvk WxggiFk7ApQP049neytF6H3RvkoYDc4YKLjxkGfGi1X7L04FhjYo02N7s5RCgQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709059984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P0lG9/ZBtvwHGC3KkxQxbhKTKzUIfBLGmQupcEe2L04=; b=WbWQZIc+LZ7nS1sQVtVcbM/AFiCPMT25JVIacd8JIJddnoB7tt3n8A0lX9nZTnmijTvNzl wHHBEFJafz1Bh2WjHA04Hwx4z/dIJnx5YFHijcdzKnRc6/KPrGKyxHpP7RoNYyW1+TRHWV wIdorsNseKuPW0YhzABcjixD10yQPRVF2yw26pARZTbFNdaZyaGLRuyD2L6KuTyym60Ubc APAaBw0JhL51SxGROS3gO0zTcx7GRIR4h9jMoEk6zpnxVTO84NOuvp78tIZqgRqPUcl1L5 HKN56/W3iCXHdeej+y39ZJvW9U1/cVUVvcB7G5kGISnZtGCs9fe7F92WiQzpXQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tkmp85Jj8zP2K for ; Tue, 27 Feb 2024 18:53:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RIr4Vv058301 for ; Tue, 27 Feb 2024 18:53:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RIr4HA058299 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 18:53:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 276524] Setting LUN block size in ctl.conf to 512 will use volblocksize instead Date: Tue, 27 Feb 2024 18:53:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276524 --- Comment #20 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D7c667affb7b09fcdcb81f6f2635e9dfab= 7bc1fa8 commit 7c667affb7b09fcdcb81f6f2635e9dfab7bc1fa8 Author: Alexander Motin AuthorDate: 2024-02-27 18:28:44 +0000 Commit: Alexander Motin CommitDate: 2024-02-27 18:28:44 +0000 CTL: Drop Format Device and Rigid Disk Geometry mode pages Those mode pages are obsolete since SBC-2 specification almost 20 years ago. First I was trying to understand possible relations between physical block and physical sector terms in different specs. Then was thinking about possible relations to device CHS geometry and compatibility issues. Finally I just decided that none of it worth the efforts and should rest in piece. PR: 276524 sys/cam/ctl/ctl.c | 194 ------------------------------------------= ---- sys/cam/ctl/ctl_private.h | 63 --------------- 2 files changed, 257 deletions(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Feb 27 19:03:35 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tkn2H73MNz5CFly for ; Tue, 27 Feb 2024 19:03:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tkn2H5FgPz52xQ for ; Tue, 27 Feb 2024 19:03:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709060615; a=rsa-sha256; cv=none; b=rozG93Woi8V1bAE/jUWour6MBg+TKFWWYd+axkf6+3rkQYmJyQnE81VEYCZ0+/qKfSEkSW 5tG2tQvE6ymXE63JP/6ekxjmnJKKW/BU3OUOLvf8cphF46ieXqMJh7DjjF2QpXhbq0oh6O BMOBygrQPXFI7HX0YrRn0nHxFC6vQQKFhF7pqnNC7Sf2uqvFdGCJ6i2PiYS0WV4p+603WE vlbmLzAlA9CnNU/t+7h+zF9bmBMw5y4s5ak9vs7msJu7AtAZrGjO3uBe9g2Nhn1xQMX1ne GOFnNuXkxtpVQEuKL098t2F9GC4nYanetQSwGUc/w/C9598J9UTSfsUxKwlNXQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709060615; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5KcU0VNYBezplXLmX7xwDZA4N6O8TGiH+BpcFItlzGw=; b=QWK+QaChPKYlKkjDI/Jt+tcL9fwlPYsbYjpFq5Ynw5ItAglGTXUJqJk7jrZsF9bVHTAxb6 qCqKF8dAnq0CGHrW6auwDEVHRY2u9JmwR3lNJ3CEB5p/w1os1tA5GORzd5rafA+cwGxBOq jhYQYpWVsKYi/n1IagJ6rBJfGCtP/QS3IOiFYcO0X8Vg1AWcdBBEyaRoac0mEuRBh+sDYj spj21kbWj0NqD+AL25J6ojYVsE6zavC4aKIclJWvjcGDAfMFOJLwwVNphVvU81yoPa0C7Q V8APj87ufEDyXnU02mS2/+hEkfGvVvfuyO9TniRRqg9a80mcwbfUQwYLAZ6I3g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tkn2H4Ky5zPYw for ; Tue, 27 Feb 2024 19:03:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41RJ3ZQB011607 for ; Tue, 27 Feb 2024 19:03:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41RJ3Z7x011602 for virtualization@FreeBSD.org; Tue, 27 Feb 2024 19:03:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 276524] Setting LUN block size in ctl.conf to 512 will use volblocksize instead Date: Tue, 27 Feb 2024 19:03:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276524 Alexander Motin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |Not A Bug --- Comment #21 from Alexander Motin --- > Where did 4K come from? man blockdev says: --getbsz Print the blocksize in bytes. This size does not describe device topology. It=E2=80=99s the size used internally by the ke= rnel and it may be modified (for example) by filesystem driver on mount. To me it sounds like they just invented it out of thin air. I don't see anything else actionable here, so I am closing the PR. I you t= hink I am wrong, I am open to hear more arguments. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Feb 28 18:29:44 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlNDx4Wc5z5C4BS for ; Wed, 28 Feb 2024 18:29:53 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlNDw1HqCz4DMN for ; Wed, 28 Feb 2024 18:29:52 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=rJFaMgHh; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41SITih6095085 for ; Wed, 28 Feb 2024 12:29:44 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1709144984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=gXRs2Fh+Ei/m2sI+avvuZYdIzUV6M/DV80XK/QYcBLs=; b=rJFaMgHhlDGdYV2Wjx753Cplzdv5xu/kxs9tGNQhDZdeWiP8RkfljbEcChzVwM9JxUuBr7 urstcP5txXT+scMKm0Z1PyLSib1MkWKNCiiFIfEdlFd0FNATUQRR5LIAXpG7IGhU8UFH54 UQ8Zx8TOfRs0vGd2VmlMI2GzcuTI7Rygqb2iFhHbklU6Tmc/6wtye/n+RPOLPgrMNWzgSU kSAdr67ZIdROcsyw8AfSF0kGEJ9T3e1Yuk/hQ8d6kVVM50h4yQGPrwP+BoOn84EupX/7pq YcET6zQUR/miEXQz/qxLm+JoWY1PvWYYCUQ6FBlIqJRhNMhlShbEtUocYq33Gw== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id A43463AB37 for ; Wed, 28 Feb 2024 12:29:44 -0600 (CST) Content-Type: multipart/alternative; boundary="------------W0i8tGWHhvTJGsHdz01WWBm5" Message-ID: <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> Date: Wed, 28 Feb 2024 12:29:44 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> From: Matthew Grooms In-Reply-To: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.42 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.93)[-0.927]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] X-Rspamd-Queue-Id: 4TlNDw1HqCz4DMN This is a multi-part message in MIME format. --------------W0i8tGWHhvTJGsHdz01WWBm5 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/27/24 04:21, Vitaliy Gusev wrote: > Hi, > > >> On 23 Feb 2024, at 18:37, Matthew Grooms wrote: >> >>> ... >> The problem occurs when an image file is used on either ZFS or UFS. >> The problem also occurs when the virtual disk is backed by a raw disk >> partition or a ZVOL. This issue isn't related to a specific >> underlying filesystem. >> > > Do I understand right, you ran testing inside VM inside guest VM  on > ext4 filesystem? If so you should be aware about additional overhead > in comparison when you were running tests on the hosts. > Hi Vitaliy, I appreciate you providing the feedback and suggestions. I spent over a week trying as many combinations of host and guest options as possible to narrow this issue down to a specific host storage or a guest device model option. Unfortunately the problem occurred with every combination I tested while running Linux as the guest. Note, I only tested RHEL8 & RHEL9 compatible distributions ( Alma & Rocky ). The problem did not occur when I ran FreeBSD as the guest. The problem did not occur when I ran KVM in the host and Linux as the guest. > I would suggest to run fio (or even dd) on raw disk device inside VM, > i.e. without filesystem at all.  Just do not forget do “echo 3 > > /proc/sys/vm/drop_caches” in Linux Guest VM before you run tests. The two servers I was using to test with are are no longer available. However, I'll have two more identical servers arriving in the next week or so. I'll try to run additional tests and report back here. I used bonnie++ as that was easily installed from the package repos on all the systems I tested. > > Could you also give more information about: > >  1. What results did you get (decode bonnie++ output)? If you look back at this email thread, there are many examples of running bonnie++ on the guest. I first ran the tests on the host system using Linux + ext4 and FreeBSD 14 + UFS & ZFS to get a baseline of performance. Then I ran bonnie++ tests using bhyve as the hypervisor and Linux & FreeBSD as the guest. The combination of host and guest storage options included ... 1) block device + virtio blk 2) block device + nvme 3) UFS disk image + virtio blk 4) UFS disk image + nvme 5) ZFS disk image + virtio blk 6) ZFS disk image + nvme 7) ZVOL + virtio blk 8) ZVOL + nvme In every instance, I observed the Linux guest disk IO often perform very well for some time after the guest was first booted. Then the performance of the guest would drop to a fraction of the original performance. The benchmark test was run every 5 or 10 minutes in a cron job. Sometimes the guest would perform well for up to an hour before performance would drop off. Most of the time it would only perform well for a few cycles ( 10 - 30 mins ) before performance would drop off. The only way to restore the performance was to reboot the guest. Once I determined that the problem was not specific to a particular host or guest storage option, I switched my testing to only use a block device as backing storage on the host to avoid hitting any system disk caches. Here is the test script I used in the cron job ... #!/bin/sh FNAME='output.txt' echo ================================================================================ >> $FNAME echo Begin @ `/usr/bin/date` >> $FNAME echo >> $FNAME /usr/sbin/bonnie++ 2>&1 | /usr/bin/grep -v 'done\|,' >> $FNAME echo >> $FNAME echo End @ `/usr/bin/date` >> $FNAME As you can see, I'm calling bonnie++ with the system defaults. That uses a data set size that's 2x the guest RAM in an attempt to minimize the effect of filesystem cache on results. Here is an example of the output that bonnie++ produces ... Version  2.00 ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP linux-blk    63640M  694k  99  1.6g  99  737m  76  985k  99 1.3g  69 +++++ +++ Latency             11579us     535us   11889us    8597us 21819us    8238us Version  2.00       ------Sequential Create------ --------Random Create-------- linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency              7620us     126us    1648us     151us 15us     633us --------------------------------- speed drop --------------------------------- Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP linux-blk    63640M  676k  99  451m  99  314m  93  951k  99 402m  99 15167 530 Latency             11902us    8959us   24711us   10185us 20884us    5831us Version  2.00       ------Sequential Create------ --------Random Create-------- linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP                  16     0  96 +++++ +++ +++++ +++     0  96 +++++ +++     0  75 Latency               343us     165us    1636us     113us 55us    1836us In the example above, the benchmark test repeated about 20 times with results that were similar to the performance shown above the dotted line ( ~ 1.6g/s seq write and 1.3g/s seq read ). After that, the performance dropped to what's shown below the dotted line which is less than 1/4 the original speed ( ~ 451m/s seq write and 402m/s seq read ). >  2. What results expecting? > What I expect is that, when I perform the same test with the same parameters, the results would stay more or less consistent over time. This is true when KVM is used as the hypervisor on the same hardware and guest options. That said, I'm not worried about bhyve being consistently slower than kvm or a FreeBSD guest being consistently slower than a Linux guest. I'm concerned that the performance drop over time is indicative of an issue with how bhyve interacts with non-freebsd guests. >  3. VM configuration, virtio-blk disk size, etc. >  4. Full command for tests (including size of test-set), bhyve, etc. I believe this was answered above. Please let me know if you have additional questions. > >  5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you should > try 4K. > The testing performed was not exclusively with virtio-blk. >  6. Linux has several read-ahead options for IO schedule, and it could > be related too. > I suppose it's possible that bhyve could be somehow causing the disk scheduler in the Linux guest to act differently. I'll see if I can figure out how to disable that in future tests. > Additionally could also you play with “sync=disabled” volume/zvol > option? Of course it is only for write testing. The testing performed was not exclusively with zvols. Once I have more hardware available, I'll try to report back with more testing. It may be interesting to also see how a Windows guest performs compared to Linux & FreeBSD. I suspect that this issue may only be triggered when a fast disk array is in use on the host. My tests use a 16x SSD RAID 10 array. It's also quite possible that the disk IO slowdown is only a symptom of another issue that's triggered by the disk IO test ( please see end of my last post related to scheduler priority observations ). All I can say for sure is that ... 1) There is a problem and it's reproducible across multiple hosts 2) It affects RHEL8 & RHEL9 guests but not FreeBSD guests 3) It is not specific to any host or guest storage option Thanks, -Matthew --------------W0i8tGWHhvTJGsHdz01WWBm5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/27/24 04:21, Vitaliy Gusev wrote:
Hi,


On 23 Feb 2024, at 18:37, Matthew Grooms <mgrooms@shrew.net> wrote:

...
The problem occurs when an image file is used on either ZFS or UFS. The problem also occurs when the virtual disk is backed by a raw disk partition or a ZVOL. This issue isn't related to a specific underlying filesystem.


Do I understand right, you ran testing inside VM inside guest VM  on ext4 filesystem? If so you should be aware about additional overhead in comparison when you were running tests on the hosts.

Hi Vitaliy,

I appreciate you providing the feedback and suggestions. I spent over a week trying as many combinations of host and guest options as possible to narrow this issue down to a specific host storage or a guest device model option. Unfortunately the problem occurred with every combination I tested while running Linux as the guest. Note, I only tested RHEL8 & RHEL9 compatible distributions ( Alma & Rocky ). The problem did not occur when I ran FreeBSD as the guest. The problem did not occur when I ran KVM in the host and Linux as the guest.

I would suggest to run fio (or even dd) on raw disk device inside VM, i.e. without filesystem at all.  Just do not forget do “echo 3 > /proc/sys/vm/drop_caches” in Linux Guest VM before you run tests.

The two servers I was using to test with are are no longer available. However, I'll have two more identical servers arriving in the next week or so. I'll try to run additional tests and report back here. I used bonnie++ as that was easily installed from the package repos on all the systems I tested.


Could you also give more information about:

 1. What results did you get (decode bonnie++ output)?

If you look back at this email thread, there are many examples of running bonnie++ on the guest. I first ran the tests on the host system using Linux + ext4 and FreeBSD 14 + UFS & ZFS to get a baseline of performance. Then I ran bonnie++ tests using bhyve as the hypervisor and Linux & FreeBSD as the guest. The combination of host and guest storage options included ...

1) block device + virtio blk
2) block device + nvme
3) UFS disk image + virtio blk
4) UFS disk image + nvme
5) ZFS disk image + virtio blk
6) ZFS disk image + nvme
7) ZVOL + virtio blk
8) ZVOL + nvme

In every instance, I observed the Linux guest disk IO often perform very well for some time after the guest was first booted. Then the performance of the guest would drop to a fraction of the original performance. The benchmark test was run every 5 or 10 minutes in a cron job. Sometimes the guest would perform well for up to an hour before performance would drop off. Most of the time it would only perform well for a few cycles ( 10 - 30 mins ) before performance would drop off. The only way to restore the performance was to reboot the guest. Once I determined that the problem was not specific to a particular host or guest storage option, I switched my testing to only use a block device as backing storage on the host to avoid hitting any system disk caches.

Here is the test script I used in the cron job ...

#!/bin/sh
FNAME='output.txt'
echo ================================================================================ >> $FNAME
echo Begin @ `/usr/bin/date` >> $FNAME
echo >> $FNAME
/usr/sbin/bonnie++ 2>&1 | /usr/bin/grep -v 'done\|,' >> $FNAME
echo >> $FNAME
echo End @ `/usr/bin/date` >> $FNAME
As you can see, I'm calling bonnie++ with the system defaults. That uses a data set size that's 2x the guest RAM in an attempt to minimize the effect of filesystem cache on results. Here is an example of the output that bonnie++ produces ...

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  694k  99  1.6g  99  737m  76  985k  99  1.3g  69 +++++ +++
Latency             11579us     535us   11889us    8597us   21819us    8238us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              7620us     126us    1648us     151us      15us     633us

--------------------------------- speed drop ---------------------------------

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  676k  99  451m  99  314m  93  951k  99  402m  99 15167 530
Latency             11902us    8959us   24711us   10185us   20884us    5831us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  96 +++++ +++ +++++ +++     0  96 +++++ +++     0  75
Latency               343us     165us    1636us     113us      55us    1836us

In the example above, the benchmark test repeated about 20 times with results that were similar to the performance shown above the dotted line ( ~ 1.6g/s seq write and 1.3g/s seq read ). After that, the performance dropped to what's shown below the dotted line which is less than 1/4 the original speed ( ~ 451m/s seq write and 402m/s seq read ).

 2. What results expecting?

What I expect is that, when I perform the same test with the same parameters, the results would stay more or less consistent over time. This is true when KVM is used as the hypervisor on the same hardware and guest options. That said, I'm not worried about bhyve being consistently slower than kvm or a FreeBSD guest being consistently slower than a Linux guest. I'm concerned that the performance drop over time is indicative of an issue with how bhyve interacts with non-freebsd guests.

 3. VM configuration, virtio-blk disk size, etc.
 4. Full command for tests (including size of test-set), bhyve, etc.

I believe this was answered above. Please let me know if you have additional questions.


 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you should try 4K.

The testing performed was not exclusively with virtio-blk.

 6. Linux has several read-ahead options for IO schedule, and it could be related too.

I suppose it's possible that bhyve could be somehow causing the disk scheduler in the Linux guest to act differently. I'll see if I can figure out how to disable that in future tests.

Additionally could also you play with “sync=disabled” volume/zvol option? Of course it is only for write testing.

The testing performed was not exclusively with zvols.

Once I have more hardware available, I'll try to report back with more testing. It may be interesting to also see how a Windows guest performs compared to Linux & FreeBSD. I suspect that this issue may only be triggered when a fast disk array is in use on the host. My tests use a 16x SSD RAID 10 array. It's also quite possible that the disk IO slowdown is only a symptom of another issue that's triggered by the disk IO test ( please see end of my last post related to scheduler priority observations ). All I can say for sure is that ...

1) There is a problem and it's reproducible across multiple hosts
2) It affects RHEL8 & RHEL9 guests but not FreeBSD guests
3) It is not specific to any host or guest storage option

Thanks,

-Matthew

--------------W0i8tGWHhvTJGsHdz01WWBm5-- From nobody Wed Feb 28 19:31:06 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlPc23DMrz5CHVb for ; Wed, 28 Feb 2024 19:31:30 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlPc16qzGz4PkR for ; Wed, 28 Feb 2024 19:31:29 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2d2505352e6so888581fa.3 for ; Wed, 28 Feb 2024 11:31:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709148688; x=1709753488; darn=freebsd.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=Rc2Q9glJ978b46nPDzeonzO79RJwcZ2ldWtxkxfLT2g=; b=Pia8Fkl1tKcxzI8X0vaJ4rvCgOqvLy4CKm2wCwF5SUoGIa16a6jQoXpwSpqGnVfPv+ 3S79SPO9BVHIC0sdO85T2FDdgR9//Jnh/Vd+dO5UHnAp76fsU0i32bpkU66MXlpi9glt NI49PX6lmZMJ+EuwCxEpf9MC9JKxotAkPqjup6WKcphMq203uS0fHHU03wTbYArqnYKg 0vCUQbuUcQBIwuzezzz6nq5XTiiTEVLO7XVAzxkHaSh4Pw49s4Si0ml+4Ic3sQTBt7Hb uebgtfK9FKmzW9ddl0afThFpk75+6rlMOY1lKAfpuDjTBVvIcB9QZmAvXPF2mIqLQkgM 431A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709148688; x=1709753488; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Rc2Q9glJ978b46nPDzeonzO79RJwcZ2ldWtxkxfLT2g=; b=UvM7CYTql3xp6aZmOshpI8BeRcKFxFrMIrlkzGMs3p/hFjKhrFG9S7P5OwSYE4M+9q bSnBpiO2Rrra8Z8BvF1GkINTnkmIJjYWy0Bq4KLivqsl54zYQRzMd9poLjSqHdcvyjpc 04KYPa7JmqTMcspKAnWJ2zyvnsLwOaxm5+BU79i5C3j9+YxrM8YIBDfozoD6iwcZMPxb z3RXzfYIBkhskFbML2RtoVFlSm3k+F0fiokW3vBkvXWbrqIJgnhZAupkQSLD/yWRhCZt ColmQPN54Mqw6d8uB1UCO4A63FoAGw2CnJxZmSpnUi977eBVwvABK4q/RHMUKT47iv2u gulA== X-Gm-Message-State: AOJu0YxbQdgQwgRBr6dbFoYTdxOTnhYsaBmni+TeFf9I2qACDmU39Sl5 unJ9uGyMHzZ5Ej86OKOOGqW8jc8fYI88yu4PxV3xkZp3mtIz+cJFLaIG3sMSHGQ1pA== X-Google-Smtp-Source: AGHT+IEd++0ZNDtGg9MxUo030h7GPNPM9Uf6N0b3wRDwiqzUYZqHXK9uu+x2mhpS8S2aO4z1z9tzTg== X-Received: by 2002:a2e:a274:0:b0:2d2:39e6:fa3f with SMTP id k20-20020a2ea274000000b002d239e6fa3fmr7515481ljm.31.1709148687745; Wed, 28 Feb 2024 11:31:27 -0800 (PST) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id l18-20020a2e9092000000b002d0ca6e0f9fsm12075ljg.15.2024.02.28.11.31.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Feb 2024 11:31:27 -0800 (PST) From: Vitaliy Gusev Message-Id: <1DAEB435-A613-4A04-B63F-D7AF7A0B7C0A@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_4C6B768C-854E-40D3-81D0-E10177927920" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: bhyve disk performance issue Date: Wed, 28 Feb 2024 22:31:06 +0300 In-Reply-To: <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> Cc: virtualization@freebsd.org To: Matthew Grooms References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4TlPc16qzGz4PkR --Apple-Mail=_4C6B768C-854E-40D3-81D0-E10177927920 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, Matthew. I still do not know what command line was used for bhyve. I couldn't = find it through the thread, sorry. And I couldn't find virtual disk size = that you used. Could you, please, simplify bonnie++ output, it is hard to decode due to = alignment and use exact numbers for: READ seq - I see you had 1.6GB/s for the good time and ~500MB/s for the = worst. WRITE seq - ... If you have slow results both for the read and write operations, you = probably should perform testing only for READs and do not do anything = until READs are fine. Again, if you have slow performance for Ext4 Filesystem in guest VM = placed on the passed disk image, you should try to test on the raw disk = image, i.e. without Ext4, because it could be related. If you run test inside VM on a filesystem, you can have deal with = filesystem bottlenecks, bugs, fragmentation etc. Do you want to fix them = all? I don=E2=80=99t think so. For example, if you pass disk image 40G and create Ext4 filesystem, and = during testing the filesystem becomes full over 80%, I/O could be = performed not so fine. You probably should eliminate that guest filesystem behaviour when you = meet IO performance slowdown. Also, please look at the TRIM operations when you perform WRITE testing. = It could be also related to the slow write I/O. =E2=80=94=E2=80=94 Vitaliy > On 28 Feb 2024, at 21:29, Matthew Grooms wrote: >=20 > On 2/27/24 04:21, Vitaliy Gusev wrote: >> Hi, >>=20 >>=20 >>> On 23 Feb 2024, at 18:37, Matthew Grooms = wrote: >>>=20 >>>> ... >>> The problem occurs when an image file is used on either ZFS or UFS. = The problem also occurs when the virtual disk is backed by a raw disk = partition or a ZVOL. This issue isn't related to a specific underlying = filesystem. >>>=20 >>=20 >> Do I understand right, you ran testing inside VM inside guest VM on = ext4 filesystem? If so you should be aware about additional overhead in = comparison when you were running tests on the hosts. >>=20 > Hi Vitaliy, >=20 > I appreciate you providing the feedback and suggestions. I spent over = a week trying as many combinations of host and guest options as possible = to narrow this issue down to a specific host storage or a guest device = model option. Unfortunately the problem occurred with every combination = I tested while running Linux as the guest. Note, I only tested RHEL8 & = RHEL9 compatible distributions ( Alma & Rocky ). The problem did not = occur when I ran FreeBSD as the guest. The problem did not occur when I = ran KVM in the host and Linux as the guest. >=20 >> I would suggest to run fio (or even dd) on raw disk device inside VM, = i.e. without filesystem at all. Just do not forget do =E2=80=9Cecho 3 > = /proc/sys/vm/drop_caches=E2=80=9D in Linux Guest VM before you run = tests. > The two servers I was using to test with are are no longer available. = However, I'll have two more identical servers arriving in the next week = or so. I'll try to run additional tests and report back here. I used = bonnie++ as that was easily installed from the package repos on all the = systems I tested. >=20 >>=20 >> Could you also give more information about: >>=20 >> 1. What results did you get (decode bonnie++ output)? > If you look back at this email thread, there are many examples of = running bonnie++ on the guest. I first ran the tests on the host system = using Linux + ext4 and FreeBSD 14 + UFS & ZFS to get a baseline of = performance. Then I ran bonnie++ tests using bhyve as the hypervisor and = Linux & FreeBSD as the guest. The combination of host and guest storage = options included ... >=20 > 1) block device + virtio blk > 2) block device + nvme > 3) UFS disk image + virtio blk > 4) UFS disk image + nvme > 5) ZFS disk image + virtio blk > 6) ZFS disk image + nvme > 7) ZVOL + virtio blk > 8) ZVOL + nvme >=20 > In every instance, I observed the Linux guest disk IO often perform = very well for some time after the guest was first booted. Then the = performance of the guest would drop to a fraction of the original = performance. The benchmark test was run every 5 or 10 minutes in a cron = job. Sometimes the guest would perform well for up to an hour before = performance would drop off. Most of the time it would only perform well = for a few cycles ( 10 - 30 mins ) before performance would drop off. The = only way to restore the performance was to reboot the guest. Once I = determined that the problem was not specific to a particular host or = guest storage option, I switched my testing to only use a block device = as backing storage on the host to avoid hitting any system disk caches. >=20 > Here is the test script I used in the cron job ... >=20 > #!/bin/sh > FNAME=3D'output.txt' >=20 > echo = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >> $FNAME > echo Begin @ `/usr/bin/date` >> $FNAME > echo >> $FNAME > /usr/sbin/bonnie++ 2>&1 | /usr/bin/grep -v 'done\|,' >> $FNAME > echo >> $FNAME > echo End @ `/usr/bin/date` >> $FNAME >=20 > As you can see, I'm calling bonnie++ with the system defaults. That = uses a data set size that's 2x the guest RAM in an attempt to minimize = the effect of filesystem cache on results. Here is an example of the = output that bonnie++ produces ... >=20 > Version 2.00 ------Sequential Output------ --Sequential Input- = --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- = --Seeks-- > Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP = /sec %CP > linux-blk 63640M 694k 99 1.6g 99 737m 76 985k 99 1.3g 69 = +++++ +++ > Latency 11579us 535us 11889us 8597us 21819us = 8238us > Version 2.00 ------Sequential Create------ --------Random = Create-------- > linux-blk -Create-- --Read--- -Delete-- -Create-- --Read--- = -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP = /sec %CP > 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ = +++++ +++ > Latency 7620us 126us 1648us 151us 15us = 633us >=20 > --------------------------------- speed drop = --------------------------------- >=20 > Version 2.00 ------Sequential Output------ --Sequential Input- = --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- = --Seeks-- > Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP = /sec %CP > linux-blk 63640M 676k 99 451m 99 314m 93 951k 99 402m 99 = 15167 530 > Latency 11902us 8959us 24711us 10185us 20884us = 5831us > Version 2.00 ------Sequential Create------ --------Random = Create-------- > linux-blk -Create-- --Read--- -Delete-- -Create-- --Read--- = -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP = /sec %CP > 16 0 96 +++++ +++ +++++ +++ 0 96 +++++ +++ = 0 75 > Latency 343us 165us 1636us 113us 55us = 1836us >=20 > In the example above, the benchmark test repeated about 20 times with = results that were similar to the performance shown above the dotted line = ( ~ 1.6g/s seq write and 1.3g/s seq read ). After that, the performance = dropped to what's shown below the dotted line which is less than 1/4 the = original speed ( ~ 451m/s seq write and 402m/s seq read ). >=20 >> 2. What results expecting? >>=20 > What I expect is that, when I perform the same test with the same = parameters, the results would stay more or less consistent over time. = This is true when KVM is used as the hypervisor on the same hardware and = guest options. That said, I'm not worried about bhyve being consistently = slower than kvm or a FreeBSD guest being consistently slower than a = Linux guest. I'm concerned that the performance drop over time is = indicative of an issue with how bhyve interacts with non-freebsd guests. >=20 >> 3. VM configuration, virtio-blk disk size, etc. >> 4. Full command for tests (including size of test-set), bhyve, etc. > I believe this was answered above. Please let me know if you have = additional questions. >=20 >>=20 >> 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you = should try 4K. >>=20 > The testing performed was not exclusively with virtio-blk. >=20 >=20 >> 6. Linux has several read-ahead options for IO schedule, and it = could be related too. >>=20 > I suppose it's possible that bhyve could be somehow causing the disk = scheduler in the Linux guest to act differently. I'll see if I can = figure out how to disable that in future tests. >=20 >=20 >> Additionally could also you play with =E2=80=9Csync=3Ddisabled=E2=80=9D= volume/zvol option? Of course it is only for write testing. > The testing performed was not exclusively with zvols. >=20 >=20 > Once I have more hardware available, I'll try to report back with more = testing. It may be interesting to also see how a Windows guest performs = compared to Linux & FreeBSD. I suspect that this issue may only be = triggered when a fast disk array is in use on the host. My tests use a = 16x SSD RAID 10 array. It's also quite possible that the disk IO = slowdown is only a symptom of another issue that's triggered by the disk = IO test ( please see end of my last post related to scheduler priority = observations ). All I can say for sure is that ... >=20 > 1) There is a problem and it's reproducible across multiple hosts > 2) It affects RHEL8 & RHEL9 guests but not FreeBSD guests > 3) It is not specific to any host or guest storage option >=20 > Thanks, >=20 > -Matthew >=20 --Apple-Mail=_4C6B768C-854E-40D3-81D0-E10177927920 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi,  Matthew.

I still do not know what command line was used for bhyve. I =  couldn't find it through the thread, sorry. And = I couldn't find virtual = disk size that you used.

Could you, = please, simplify bonnie++ output, it is hard to decode due to alignment = and use exact numbers for:

READ seq =  - I see you had 1.6GB/s for the good time and ~500MB/s for the = worst.
WRITE seq =  - ...

If you = have slow results both for the read and write operations, you probably = should perform testing only for READs and do not do anything = until READs are fine.

Again, if = you have slow performance for Ext4 Filesystem in guest VM placed on the = passed disk image, you should try to test on the raw disk image, i.e. without Ext4, because it = could be related.

If you run = test inside VM on a filesystem, you can have deal with filesystem = bottlenecks, bugs, fragmentation etc. Do you want to fix them all? I = don=E2=80=99t think so.

For = example, if you pass disk image 40G and create Ext4 filesystem, and = during testing the filesystem becomes full over 80%, I/O could be = performed not so fine.

You = probably should eliminate that guest filesystem behaviour when you meet = IO performance slowdown.

Also, = please look at the TRIM operations when you perform WRITE testing. It = could be also related to the slow write I/O.

=E2=80=94=E2=80=94
Vitaliy

On= 28 Feb 2024, at 21:29, Matthew Grooms <mgrooms@shrew.net> = wrote:

=20 =20
On 2/27/24 04:21, Vitaliy Gusev = wrote:
Hi,


On 23 Feb 2024, at 18:37, Matthew Grooms <mgrooms@shrew.net> = wrote:

...
The problem occurs when an image file is used on either ZFS or UFS. The problem also occurs when the virtual disk is backed by a raw disk partition or a ZVOL. This issue isn't related to a specific underlying = filesystem.


Do I understand right, you ran testing inside VM inside guest VM  on ext4 filesystem? If so you should be aware about additional overhead in comparison when you were running tests on the hosts.

Hi Vitaliy,

I appreciate you providing the feedback and suggestions. I spent over a week trying as many combinations of host and guest options as possible to narrow this issue down to a specific host storage or a guest device model option. Unfortunately the problem occurred with every combination I tested while running Linux as the guest. Note, I only tested RHEL8 & RHEL9 compatible distributions ( Alma & Rocky ). The problem did not occur when I ran FreeBSD as the guest. The problem did not occur when I ran KVM in the host and Linux as the guest.

I would suggest to run fio (or even dd) on raw disk device inside VM, i.e. without filesystem at all.  Just do not = forget do =E2=80=9Cecho 3 > /proc/sys/vm/drop_caches=E2=80=9D in Linux = Guest VM before you run tests.

The two servers I was using to test with are are no = longer available. However, I'll have two more identical servers arriving in the next week or so. I'll try to run additional tests and report back here. I used bonnie++ as that was easily installed from the package repos on all the systems I tested.


Could you also give more information about:

 1. What results did you get (decode bonnie++ = output)?

If you look back at this email thread, there are = many examples of running bonnie++ on the guest. I first ran the tests on the host system using Linux + ext4 and FreeBSD 14 + UFS & ZFS to get a baseline of performance. Then I ran bonnie++ tests using bhyve as the hypervisor and Linux & FreeBSD as the guest. The combination of host and guest storage options included ...

1) block device + virtio blk
2) block device + nvme
3) UFS disk image + virtio blk
4) UFS disk image + nvme
5) ZFS disk image + virtio blk
6) ZFS disk image + nvme
7) ZVOL + virtio blk
8) ZVOL + nvme

In every instance, I observed the Linux guest disk IO often perform very well for some time after the guest was first booted. Then the performance of the guest would drop to a fraction of the original performance. The benchmark test was run every 5 or 10 minutes in a cron job. Sometimes the guest would perform well for up to an hour before performance would drop off. Most of the time it would only perform well for a few cycles ( 10 - 30 mins ) before performance would drop off. The only way to restore the performance was to reboot the guest. Once I determined that the problem was not specific to a particular host or guest storage option, I switched my testing to only use a block device as backing storage on the host to avoid hitting any system disk caches.

Here is the test script I used in the cron job ...

#!/bin/sh
FNAME=3D'output.txt'
echo = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >> $FNAME
echo Begin @ `/usr/bin/date` >> $FNAME
echo >> $FNAME
/usr/sbin/bonnie++ 2>&1 | /usr/bin/grep -v 'done\|,' >> $FNAME
echo >> $FNAME
echo End @ `/usr/bin/date` >> $FNAME
As you can see, I'm calling bonnie++ with the system defaults. That uses a data set size that's 2x the guest RAM in an attempt to minimize the effect of filesystem cache on results. Here is an example of the output that bonnie++ produces ...

Version  = 2.00       ------Sequential Output------ --Sequential Input- = --Random-
=             &n= bsp;       -Per Chr- --Block-- -Rewrite- = -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec = %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  694k  99  = 1.6g  99  737m  76  985k  99  1.3g  69 +++++ +++
= Latency           &= nbsp; 11579us     535us   = 11889us    8597us   21819us    8238us
Version  2.00       = ------Sequential Create------ --------Random Create--------
= linux-blk           = -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
=             &n= bsp; files  /sec %CP  /sec %CP  /sec %CP  /sec = %CP  /sec %CP  /sec %CP
=             &n= bsp;    16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
= Latency           &= nbsp;  7620us     126us    = 1648us     151us      15us     633us

--------------------------------- speed drop ---------------------------------

Version  2.00       = ------Sequential Output------ --Sequential Input- --Random-
=             &n= bsp;       -Per Chr- --Block-- -Rewrite- = -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec = %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  676k  99  = 451m  99  314m  93  951k  99  402m  99 15167 530
= Latency           &= nbsp; 11902us    8959us   24711us   = 10185us   20884us    5831us
Version  2.00       = ------Sequential Create------ --------Random Create--------
= linux-blk           = -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
=             &n= bsp; files  /sec %CP  /sec %CP  /sec %CP  /sec = %CP  /sec %CP  /sec %CP
=             &n= bsp;    16     0  96 +++++ +++ = +++++ +++     0  96 +++++ +++     0  75
= Latency           &= nbsp;   343us     165us    = 1636us     113us      55us    1836us

In the example above, the benchmark test repeated about 20 times with results that were similar to the performance shown above the dotted line ( ~ 1.6g/s seq write and 1.3g/s seq read ). After that, the performance dropped to what's shown below the dotted line which is less than 1/4 the original speed ( ~ 451m/s seq write and 402m/s seq read ).

 2. What results expecting?

What I expect is that, when I perform the same test = with the same parameters, the results would stay more or less consistent over time. This is true when KVM is used as the hypervisor on the = same hardware and guest options. That said, I'm not worried about bhyve being consistently slower than kvm or a FreeBSD guest being consistently slower than a Linux guest. I'm concerned that the performance drop over time is indicative of an issue with how bhyve interacts with non-freebsd guests.

 3. VM configuration, virtio-blk disk size, etc.
 4. Full command for tests (including size of = test-set), bhyve, etc.

I believe this was answered above. Please let me = know if you have additional questions.


 5. Did you pass virtio-blk as 512 or 4K ? If 512, = probably you should try 4K.

The testing performed was not exclusively with = virtio-blk.

 6. Linux has several read-ahead options for IO = schedule, and it could be related too.

I suppose it's possible that bhyve could be somehow = causing the disk scheduler in the Linux guest to act differently. I'll see if I can figure out how to disable that in future tests.

Additionally could also you play with =E2=80=9Csync=3Ddisable= d=E2=80=9D volume/zvol option? Of course it is only for write = testing.

The testing performed was not exclusively with = zvols.

Once I have more hardware available, I'll try to report back = with more testing. It may be interesting to also see how a Windows guest performs compared to Linux & FreeBSD. I suspect that this issue may only be triggered when a fast disk array is in use on the host. My tests use a 16x SSD RAID 10 array. It's also quite possible that the disk IO slowdown is only a symptom of another issue that's triggered by the disk IO test ( please see end of my last post related to scheduler priority observations ). All I can say for sure is that ...

1) There is a problem and it's reproducible across multiple = hosts
2) It affects RHEL8 & RHEL9 guests but not FreeBSD guests
3) It is not specific to any host or guest storage option

Thanks,

-Matthew


= --Apple-Mail=_4C6B768C-854E-40D3-81D0-E10177927920-- From nobody Wed Feb 28 20:03:03 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlQJY4NBjz5CL1W for ; Wed, 28 Feb 2024 20:03:09 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlQJX0gb9z4ScH for ; Wed, 28 Feb 2024 20:03:08 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=f0FPLzee; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41SK335X095945; Wed, 28 Feb 2024 14:03:03 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1709150583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=WbPs/dABj2oQJ16a+ufhwtsOujfdOtSkCUuhOGQsImE=; b=f0FPLzeeAcPNLYGAr6mODeGLRTnQP8poUevtPRoW0oeT2gLrOU9nQr63MiyaWHHi9YcCaa 5z6ePb1Ppg1FiWTkVsHwFNGFjOEz1F0tE2+edEmsqJHuy+YvEhBGIvIQKu0YeTcNa/mcyM vcquF2h1erYqCWhWLBDoX9k7hyzCfzkWkkDiSJeGBLe0MMsZv21xlYSt4f77nvjSK8cVdV ViH4qrzAPEUH+s6oNw1PDEB1CTg+ClqmWGt+LHzg5qspTukxAUxUeNLPy5kaA+8v7mpevy MMuF9qD3r4Y4q1Pg+WgT0rKlyxB1Gci2SqgaAgmuLXDaCVHe6ZXvTpYEtOjd0A== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id A47B43AB37; Wed, 28 Feb 2024 14:03:03 -0600 (CST) Content-Type: multipart/alternative; boundary="------------6Zic9841NtG2gfiqcz3dfntV" Message-ID: Date: Wed, 28 Feb 2024 14:03:03 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: Vitaliy Gusev Cc: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> <1DAEB435-A613-4A04-B63F-D7AF7A0B7C0A@gmail.com> From: Matthew Grooms In-Reply-To: <1DAEB435-A613-4A04-B63F-D7AF7A0B7C0A@gmail.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[shrew.net]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[shrew.net:+] X-Rspamd-Queue-Id: 4TlQJX0gb9z4ScH This is a multi-part message in MIME format. --------------6Zic9841NtG2gfiqcz3dfntV Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/28/24 13:31, Vitaliy Gusev wrote: > Hi,  Matthew. > HI Vitaliy, Thanks for the pointers. > I still do not know what command line was used for bhyve. I  couldn't > find it through the thread, sorry. And I couldn't find virtual disk > size that you used. > Sorry about that. I'll try to get you the exact command line invocation used to launch the guest process once I have test hardware again. > > Could you, please, simplify bonnie++ output, it is hard to decode due > to alignment and use exact numbers for: > > READ seq  - I see you had 1.6GB/s for the good time and ~500MB/s for > the worst. > WRITE seq  - ... > I summarized the output for you. Here it is again: Fast: ~ 1.6g/s seq write and 1.3g/s seq read Slow: ~ 451m/s seq write and 402m/s seq read > If you have slow results both for the read and write operations, you > probably should perform testing _only_ for READs and do not do > anything until READs are fine. > > Again, if you have slow performance for Ext4 Filesystem in guest VM > placed on the passed disk image, you should try to test on the raw > disk image, i.e. without Ext4, because it could be related. > > If you run test inside VM on a filesystem, you can have deal with > filesystem bottlenecks, bugs, fragmentation etc. Do you want to fix > them all? I don’t think so. > > For example, if you pass disk image 40G and create Ext4 filesystem, > and during testing the filesystem becomes full over 80%, I/O could be > performed not so fine. > > You probably should eliminate that guest filesystem behaviour when you > meet IO performance slowdown. > > Also, please look at the TRIM operations when you perform WRITE > testing. It could be also related to the slow write I/O. > The virtual disks were provisioned with either a 128G disk image or a 1TB raw partition, so I don't think space was an issue. Trim is definitely not an issue. I'm using a tiny fraction of the 32TB array have tried both heavily under-provisioned HW RAID10 and SW RAID10 using GEOM. The latter was tested after sending full trim resets to all drives individually. I will try to incorporate the rest of your feedback into my next round of testing. If I can find a benchmark tool that works with a raw block device, that would be ideal. Thanks, -Matthew > —— > Vitaliy > >> On 28 Feb 2024, at 21:29, Matthew Grooms wrote: >> >> On 2/27/24 04:21, Vitaliy Gusev wrote: >>> Hi, >>> >>> >>>> On 23 Feb 2024, at 18:37, Matthew Grooms wrote: >>>> >>>>> ... >>>> The problem occurs when an image file is used on either ZFS or UFS. >>>> The problem also occurs when the virtual disk is backed by a raw >>>> disk partition or a ZVOL. This issue isn't related to a specific >>>> underlying filesystem. >>>> >>> >>> Do I understand right, you ran testing inside VM inside guest VM  on >>> ext4 filesystem? If so you should be aware about additional overhead >>> in comparison when you were running tests on the hosts. >>> >> Hi Vitaliy, >> >> I appreciate you providing the feedback and suggestions. I spent over >> a week trying as many combinations of host and guest options as >> possible to narrow this issue down to a specific host storage or a >> guest device model option. Unfortunately the problem occurred with >> every combination I tested while running Linux as the guest. Note, I >> only tested RHEL8 & RHEL9 compatible distributions ( Alma & Rocky ). >> The problem did not occur when I ran FreeBSD as the guest. The >> problem did not occur when I ran KVM in the host and Linux as the guest. >> >>> I would suggest to run fio (or even dd) on raw disk device inside >>> VM, i.e. without filesystem at all.  Just do not forget do “echo 3 > >>> /proc/sys/vm/drop_caches” in Linux Guest VM before you run tests. >> >> The two servers I was using to test with are are no longer available. >> However, I'll have two more identical servers arriving in the next >> week or so. I'll try to run additional tests and report back here. I >> used bonnie++ as that was easily installed from the package repos on >> all the systems I tested. >> >>> >>> Could you also give more information about: >>> >>>  1. What results did you get (decode bonnie++ output)? >> >> If you look back at this email thread, there are many examples of >> running bonnie++ on the guest. I first ran the tests on the host >> system using Linux + ext4 and FreeBSD 14 + UFS & ZFS to get a >> baseline of performance. Then I ran bonnie++ tests using bhyve as the >> hypervisor and Linux & FreeBSD as the guest. The combination of host >> and guest storage options included ... >> >> 1) block device + virtio blk >> 2) block device + nvme >> 3) UFS disk image + virtio blk >> 4) UFS disk image + nvme >> 5) ZFS disk image + virtio blk >> 6) ZFS disk image + nvme >> 7) ZVOL + virtio blk >> 8) ZVOL + nvme >> >> In every instance, I observed the Linux guest disk IO often perform >> very well for some time after the guest was first booted. Then the >> performance of the guest would drop to a fraction of the original >> performance. The benchmark test was run every 5 or 10 minutes in a >> cron job. Sometimes the guest would perform well for up to an hour >> before performance would drop off. Most of the time it would only >> perform well for a few cycles ( 10 - 30 mins ) before performance >> would drop off. The only way to restore the performance was to reboot >> the guest. Once I determined that the problem was not specific to a >> particular host or guest storage option, I switched my testing to >> only use a block device as backing storage on the host to avoid >> hitting any system disk caches. >> >> Here is the test script I used in the cron job ... >> >> #!/bin/sh >> FNAME='output.txt' >> >> echo >> ================================================================================ >> >> $FNAME >> echo Begin @ `/usr/bin/date` >> $FNAME >> echo >> $FNAME >> /usr/sbin/bonnie++ 2>&1 | /usr/bin/grep -v 'done\|,' >> $FNAME >> echo >> $FNAME >> echo End @ `/usr/bin/date` >> $FNAME >> >> As you can see, I'm calling bonnie++ with the system defaults. That >> uses a data set size that's 2x the guest RAM in an attempt to >> minimize the effect of filesystem cache on results. Here is an >> example of the output that bonnie++ produces ... >> >> Version 2.00       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP  >> /sec %CP >> linux-blk    63640M  694k  99  1.6g  99  737m  76 985k  99  1.3g  69 >> +++++ +++ >> Latency             11579us     535us   11889us 8597us   21819us    >> 8238us >> Version  2.00       ------Sequential Create------ --------Random >> Create-------- >> linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP  >> /sec %CP >>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ >> +++++ +++ >> Latency              7620us     126us 1648us     151us      15us     >> 633us >> >> --------------------------------- speed drop >> --------------------------------- >> >> Version  2.00       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP  >> /sec %CP >> linux-blk    63640M  676k  99  451m  99  314m  93 951k  99  402m  99 >> 15167 530 >> Latency             11902us    8959us   24711us 10185us   20884us    >> 5831us >> Version  2.00       ------Sequential Create------ --------Random >> Create-------- >> linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP /sec %CP  /sec %CP  >> /sec %CP >>                  16     0  96 +++++ +++ +++++ +++     0  96 +++++ >> +++     0  75 >> Latency               343us     165us 1636us     113us      55us    >> 1836us >> >> In the example above, the benchmark test repeated about 20 times with >> results that were similar to the performance shown above the dotted >> line ( ~ 1.6g/s seq write and 1.3g/s seq read ). After that, the >> performance dropped to what's shown below the dotted line which is >> less than 1/4 the original speed ( ~ 451m/s seq write and 402m/s seq >> read ). >> >>>  2. What results expecting? >>> >> What I expect is that, when I perform the same test with the same >> parameters, the results would stay more or less consistent over >> time. This is true when KVM is used as the hypervisor on the same >> hardware and guest options. That said, I'm not worried about bhyve >> being consistently slower than kvm or a FreeBSD guest being >> consistently slower than a Linux guest. I'm concerned that the >> performance drop over time is indicative of an issue with how bhyve >> interacts with non-freebsd guests. >> >>>  3. VM configuration, virtio-blk disk size, etc. >>>  4. Full command for tests (including size of test-set), bhyve, etc. >> >> I believe this was answered above. Please let me know if you have >> additional questions. >> >>> >>>  5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you >>> should try 4K. >>> >> The testing performed was not exclusively with virtio-blk. >> >>>  6. Linux has several read-ahead options for IO schedule, and it >>> could be related too. >>> >> I suppose it's possible that bhyve could be somehow causing the disk >> scheduler in the Linux guest to act differently. I'll see if I can >> figure out how to disable that in future tests. >> >>> Additionally could also you play with “sync=disabled” volume/zvol >>> option? Of course it is only for write testing. >> >> The testing performed was not exclusively with zvols. >> >> Once I have more hardware available, I'll try to report back with >> more testing. It may be interesting to also see how a Windows guest >> performs compared to Linux & FreeBSD. I suspect that this issue may >> only be triggered when a fast disk array is in use on the host. My >> tests use a 16x SSD RAID 10 array. It's also quite possible that the >> disk IO slowdown is only a symptom of another issue that's triggered >> by the disk IO test ( please see end of my last post related to >> scheduler priority observations ). All I can say for sure is that ... >> >> 1) There is a problem and it's reproducible across multiple hosts >> 2) It affects RHEL8 & RHEL9 guests but not FreeBSD guests >> 3) It is not specific to any host or guest storage option >> >> Thanks, >> >> -Matthew >> > --------------6Zic9841NtG2gfiqcz3dfntV Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/28/24 13:31, Vitaliy Gusev wrote:
Hi,  Matthew.

HI Vitaliy,

Thanks for the pointers.

I still do not know what command line was used for bhyve. I  couldn't find it through the thread, sorry. And I couldn't find virtual disk size that you used.

Sorry about that. I'll try to get you the exact command line invocation used to launch the guest process once I have test hardware again.


Could you, please, simplify bonnie++ output, it is hard to decode due to alignment and use exact numbers for:

READ seq  - I see you had 1.6GB/s for the good time and ~500MB/s for the worst.
WRITE seq  - ...

I summarized the output for you. Here it is again:

Fast: ~ 1.6g/s seq write and 1.3g/s seq read
Slow: ~ 451m/s seq write and 402m/s seq read

If you have slow results both for the read and write operations, you probably should perform testing only for READs and do not do anything until READs are fine.

Again, if you have slow performance for Ext4 Filesystem in guest VM placed on the passed disk image, you should try to test on the raw disk image, i.e. without Ext4, because it could be related.

If you run test inside VM on a filesystem, you can have deal with filesystem bottlenecks, bugs, fragmentation etc. Do you want to fix them all? I don’t think so.

For example, if you pass disk image 40G and create Ext4 filesystem, and during testing the filesystem becomes full over 80%, I/O could be performed not so fine.

You probably should eliminate that guest filesystem behaviour when you meet IO performance slowdown.

Also, please look at the TRIM operations when you perform WRITE testing. It could be also related to the slow write I/O.

The virtual disks were provisioned with either a 128G disk image or a 1TB raw partition, so I don't think space was an issue.

Trim is definitely not an issue. I'm using a tiny fraction of the 32TB array have tried both heavily under-provisioned HW RAID10 and SW RAID10 using GEOM. The latter was tested after sending full trim resets to all drives individually.

I will try to incorporate the rest of your feedback into my next round of testing. If I can find a benchmark tool that works with a raw block device, that would be ideal.

Thanks,

-Matthew


——
Vitaliy

On 28 Feb 2024, at 21:29, Matthew Grooms <mgrooms@shrew.net> wrote:

On 2/27/24 04:21, Vitaliy Gusev wrote:
Hi,


On 23 Feb 2024, at 18:37, Matthew Grooms <mgrooms@shrew.net> wrote:

...
The problem occurs when an image file is used on either ZFS or UFS. The problem also occurs when the virtual disk is backed by a raw disk partition or a ZVOL. This issue isn't related to a specific underlying filesystem.


Do I understand right, you ran testing inside VM inside guest VM  on ext4 filesystem? If so you should be aware about additional overhead in comparison when you were running tests on the hosts.

Hi Vitaliy,

I appreciate you providing the feedback and suggestions. I spent over a week trying as many combinations of host and guest options as possible to narrow this issue down to a specific host storage or a guest device model option. Unfortunately the problem occurred with every combination I tested while running Linux as the guest. Note, I only tested RHEL8 & RHEL9 compatible distributions ( Alma & Rocky ). The problem did not occur when I ran FreeBSD as the guest. The problem did not occur when I ran KVM in the host and Linux as the guest.

I would suggest to run fio (or even dd) on raw disk device inside VM, i.e. without filesystem at all.  Just do not forget do “echo 3 > /proc/sys/vm/drop_caches” in Linux Guest VM before you run tests.

The two servers I was using to test with are are no longer available. However, I'll have two more identical servers arriving in the next week or so. I'll try to run additional tests and report back here. I used bonnie++ as that was easily installed from the package repos on all the systems I tested.


Could you also give more information about:

 1. What results did you get (decode bonnie++ output)?

If you look back at this email thread, there are many examples of running bonnie++ on the guest. I first ran the tests on the host system using Linux + ext4 and FreeBSD 14 + UFS & ZFS to get a baseline of performance. Then I ran bonnie++ tests using bhyve as the hypervisor and Linux & FreeBSD as the guest. The combination of host and guest storage options included ...

1) block device + virtio blk
2) block device + nvme
3) UFS disk image + virtio blk
4) UFS disk image + nvme
5) ZFS disk image + virtio blk
6) ZFS disk image + nvme
7) ZVOL + virtio blk
8) ZVOL + nvme

In every instance, I observed the Linux guest disk IO often perform very well for some time after the guest was first booted. Then the performance of the guest would drop to a fraction of the original performance. The benchmark test was run every 5 or 10 minutes in a cron job. Sometimes the guest would perform well for up to an hour before performance would drop off. Most of the time it would only perform well for a few cycles ( 10 - 30 mins ) before performance would drop off. The only way to restore the performance was to reboot the guest. Once I determined that the problem was not specific to a particular host or guest storage option, I switched my testing to only use a block device as backing storage on the host to avoid hitting any system disk caches.

Here is the test script I used in the cron job ...

#!/bin/sh
FNAME='output.txt'
echo ================================================================================ >> $FNAME
echo Begin @ `/usr/bin/date` >> $FNAME
echo >> $FNAME
/usr/sbin/bonnie++ 2>&1 | /usr/bin/grep -v 'done\|,' >> $FNAME
echo >> $FNAME
echo End @ `/usr/bin/date` >> $FNAME
As you can see, I'm calling bonnie++ with the system defaults. That uses a data set size that's 2x the guest RAM in an attempt to minimize the effect of filesystem cache on results. Here is an example of the output that bonnie++ produces ...

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  694k  99  1.6g  99  737m  76  985k  99  1.3g  69 +++++ +++
Latency             11579us     535us   11889us    8597us   21819us    8238us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              7620us     126us    1648us     151us      15us     633us

--------------------------------- speed drop ---------------------------------

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
linux-blk    63640M  676k  99  451m  99  314m  93  951k  99  402m  99 15167 530
Latency             11902us    8959us   24711us   10185us   20884us    5831us
Version  2.00       ------Sequential Create------ --------Random Create--------
linux-blk           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  96 +++++ +++ +++++ +++     0  96 +++++ +++     0  75
Latency               343us     165us    1636us     113us      55us    1836us

In the example above, the benchmark test repeated about 20 times with results that were similar to the performance shown above the dotted line ( ~ 1.6g/s seq write and 1.3g/s seq read ). After that, the performance dropped to what's shown below the dotted line which is less than 1/4 the original speed ( ~ 451m/s seq write and 402m/s seq read ).

 2. What results expecting?

What I expect is that, when I perform the same test with the same parameters, the results would stay more or less consistent over time. This is true when KVM is used as the hypervisor on the same hardware and guest options. That said, I'm not worried about bhyve being consistently slower than kvm or a FreeBSD guest being consistently slower than a Linux guest. I'm concerned that the performance drop over time is indicative of an issue with how bhyve interacts with non-freebsd guests.

 3. VM configuration, virtio-blk disk size, etc.
 4. Full command for tests (including size of test-set), bhyve, etc.

I believe this was answered above. Please let me know if you have additional questions.


 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you should try 4K.

The testing performed was not exclusively with virtio-blk.

 6. Linux has several read-ahead options for IO schedule, and it could be related too.

I suppose it's possible that bhyve could be somehow causing the disk scheduler in the Linux guest to act differently. I'll see if I can figure out how to disable that in future tests.

Additionally could also you play with “sync=disabled” volume/zvol option? Of course it is only for write testing.

The testing performed was not exclusively with zvols.

Once I have more hardware available, I'll try to report back with more testing. It may be interesting to also see how a Windows guest performs compared to Linux & FreeBSD. I suspect that this issue may only be triggered when a fast disk array is in use on the host. My tests use a 16x SSD RAID 10 array. It's also quite possible that the disk IO slowdown is only a symptom of another issue that's triggered by the disk IO test ( please see end of my last post related to scheduler priority observations ). All I can say for sure is that ...

1) There is a problem and it's reproducible across multiple hosts
2) It affects RHEL8 & RHEL9 guests but not FreeBSD guests
3) It is not specific to any host or guest storage option

Thanks,

-Matthew


--------------6Zic9841NtG2gfiqcz3dfntV-- From nobody Wed Feb 28 21:02:45 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlRdq3BDgz5CQ6H for ; Wed, 28 Feb 2024 21:03:11 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlRdq0MXpz4bMk for ; Wed, 28 Feb 2024 21:03:11 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x233.google.com with SMTP id 38308e7fff4ca-2d22b8801b9so2575811fa.0 for ; Wed, 28 Feb 2024 13:03:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709154188; x=1709758988; darn=freebsd.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=6+qw8Y8RFPP1xZcM5Sr47TFZSWREudJ+M26tgVxLYMQ=; b=joiYaZzYz20Ao1yur/hsRhPVPppTBOLyJAC5+JPMW31gBmKt+CVl6gMae+Kui0JyqO sXsUXDKc8GI/+8XYGVjtok5LEVKsP5yWgoIFTHEeGw2CefSZ3ZBYC32Cn2cwCtn4j6V+ lQckD4PDqvtPyh6bQWN0Vw7MPdJyTXeBMYO9aTx5/wDodPC59BF/3MZ0GGSHujwKLayl MhAhTpTf68qCcRWV3YYufxrIxgc9BlY3khML4FkgHSC7eV/I6Y1225cJmdiHt93W28n1 wzQx+lRRZcuMZjsCFQBh5Mw6ktPS/jeG7KFT6BvcWES2PD3ElJ+hU1JZgHpU0yfpd1V9 m/JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709154188; x=1709758988; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6+qw8Y8RFPP1xZcM5Sr47TFZSWREudJ+M26tgVxLYMQ=; b=b6Jx/P1SrGyJZEjg53Dm3SI9zj+VPt+oJ8dZdDmcEobK2B7p4ek5ieSNSxYJk8qSvU xaR2rJ+sQX9PB9nOSYUREdn5R2o9nisjQDGQJCJJIBFlunTrO9G/fK7Q9jcUA9pKjnER 1nJKZO9fWo/iui6NMAc7QFTcpkt4Bkw8IkTxtgsJnzS9sh/iYyOcwHZuq/SeeiK15Unc LjLSHIIbAlM10qM2EtRXpENdcSTUMF0Wusk46rRoHiASqDbFBiffGYlvcfTml+6g3hrb nN+qOVveAd/og210NxFyskN6xyxawKXN95Z4Aq0SEhA02yEqM8EInn7+JXv6+m5yjAY0 h7Bw== X-Gm-Message-State: AOJu0YzViRNZWjUlXXPv31fyCBCJsduRqRLrbBPJZ0CbjfkACQabXrOt xACOw84+or07ZXYFm62whMnyHi8MKreuUfjz763sdy4UPSlfU5JspC65MoliVi99Ow== X-Google-Smtp-Source: AGHT+IHfkvc3C8omuO2YHHFokZsxAaQt/fn5ur2D0xBxJkewAhWznKKP536WBOY77FiNF9qZvGEdhA== X-Received: by 2002:a05:6512:2216:b0:512:9e78:998c with SMTP id h22-20020a056512221600b005129e78998cmr102196lfu.9.1709154187554; Wed, 28 Feb 2024 13:03:07 -0800 (PST) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id f7-20020a056512360700b00512e3924539sm36727lfs.309.2024.02.28.13.03.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Feb 2024 13:03:07 -0800 (PST) From: Vitaliy Gusev Message-Id: <3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_8118B541-F589-4E79-AF6C-3E98D8AADC93" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: bhyve disk performance issue Date: Thu, 29 Feb 2024 00:02:45 +0300 In-Reply-To: Cc: virtualization@freebsd.org To: Matthew Grooms References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> <1DAEB435-A613-4A04-B63F-D7AF7A0B7C0A@gmail.com> X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4TlRdq0MXpz4bMk --Apple-Mail=_8118B541-F589-4E79-AF6C-3E98D8AADC93 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 28 Feb 2024, at 23:03, Matthew Grooms wrote: >=20 > ... > The virtual disks were provisioned with either a 128G disk image or a = 1TB raw partition, so I don't think space was an issue. > Trim is definitely not an issue. I'm using a tiny fraction of the 32TB = array have tried both heavily under-provisioned HW RAID10 and SW RAID10 = using GEOM. The latter was tested after sending full trim resets to all = drives individually. >=20 It could be then TRIM/UNMAP is not used, zvol (for the instance) becomes = full for the while. ZFS considers it as all blocks are used and write = operations could have troubles. I believe it was recently fixed. Also look at this one: GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk The problem of UNMAP that it could have unpredictable slowdown at any = time. So I would suggest to check results with enabled and disabled = UNMAP in a guest. > I will try to incorporate the rest of your feedback into my next round = of testing. If I can find a benchmark tool that works with a raw block = device, that would be ideal. >=20 >=20 Use =E2=80=9Cdd=E2=80=9D as the first step for read testing; ~# dd if=3D/dev/nvme0n2 of=3D/dev/null bs=3D1M status=3Dprogress = flag=3Ddirect ~# dd if=3D/dev/nvme0n2 of=3D/dev/null bs=3D1M status=3Dprogress Compare results with directio and without. =E2=80=9Cfio=E2=80=9D tool.=20 =20 1) write prepare: ~# fio --name=3Dprep --rw=3Dwrite --verify=3Dcrc32 --loop=3D1 = --numjobs=3D2 --time_based --thread --bs=3D1M --iodepth=3D32 = --ioengine=3Dlibaio --direct=3D1 --group_reporting --size=3D20G = --filename=3D/dev/nvme0n2 2) read test: ~# fio --name=3Dreadtest --rw=3Dread =E2=80=94loop=3D30 = --numjobs=3D2 --time_based --thread =E2=80=94bs=3D256K --iodepth=3D32 = --ioengine=3Dlibaio --direct=3D1 --group_reporting --size=3D20G = --filename=3D/dev/nvme0n2 =20 =E2=80=94 Vitaliy =20 > Thanks, >=20 > -Matthew >=20 >=20 >=20 >> =E2=80=94=E2=80=94 >> Vitaliy >>=20 >>> On 28 Feb 2024, at 21:29, Matthew Grooms = wrote: >>>=20 >>> On 2/27/24 04:21, Vitaliy Gusev wrote: >>>> Hi, >>>>=20 >>>>=20 >>>>> On 23 Feb 2024, at 18:37, Matthew Grooms = wrote: >>>>>=20 >>>>>> ... >>>>> The problem occurs when an image file is used on either ZFS or = UFS. The problem also occurs when the virtual disk is backed by a raw = disk partition or a ZVOL. This issue isn't related to a specific = underlying filesystem. >>>>>=20 >>>>=20 >>>> Do I understand right, you ran testing inside VM inside guest VM = on ext4 filesystem? If so you should be aware about additional overhead = in comparison when you were running tests on the hosts. >>>>=20 >>> Hi Vitaliy, >>>=20 >>> I appreciate you providing the feedback and suggestions. I spent = over a week trying as many combinations of host and guest options as = possible to narrow this issue down to a specific host storage or a guest = device model option. Unfortunately the problem occurred with every = combination I tested while running Linux as the guest. Note, I only = tested RHEL8 & RHEL9 compatible distributions ( Alma & Rocky ). The = problem did not occur when I ran FreeBSD as the guest. The problem did = not occur when I ran KVM in the host and Linux as the guest. >>>=20 >>>> I would suggest to run fio (or even dd) on raw disk device inside = VM, i.e. without filesystem at all. Just do not forget do =E2=80=9Cecho = 3 > /proc/sys/vm/drop_caches=E2=80=9D in Linux Guest VM before you run = tests.=20 >>> The two servers I was using to test with are are no longer = available. However, I'll have two more identical servers arriving in the = next week or so. I'll try to run additional tests and report back here. = I used bonnie++ as that was easily installed from the package repos on = all the systems I tested. >>>=20 >>>>=20 >>>> Could you also give more information about: >>>>=20 >>>> 1. What results did you get (decode bonnie++ output)? >>> If you look back at this email thread, there are many examples of = running bonnie++ on the guest. I first ran the tests on the host system = using Linux + ext4 and FreeBSD 14 + UFS & ZFS to get a baseline of = performance. Then I ran bonnie++ tests using bhyve as the hypervisor and = Linux & FreeBSD as the guest. The combination of host and guest storage = options included ... >>>=20 >>> 1) block device + virtio blk >>> 2) block device + nvme >>> 3) UFS disk image + virtio blk >>> 4) UFS disk image + nvme >>> 5) ZFS disk image + virtio blk >>> 6) ZFS disk image + nvme >>> 7) ZVOL + virtio blk >>> 8) ZVOL + nvme >>>=20 >>> In every instance, I observed the Linux guest disk IO often perform = very well for some time after the guest was first booted. Then the = performance of the guest would drop to a fraction of the original = performance. The benchmark test was run every 5 or 10 minutes in a cron = job. Sometimes the guest would perform well for up to an hour before = performance would drop off. Most of the time it would only perform well = for a few cycles ( 10 - 30 mins ) before performance would drop off. The = only way to restore the performance was to reboot the guest. Once I = determined that the problem was not specific to a particular host or = guest storage option, I switched my testing to only use a block device = as backing storage on the host to avoid hitting any system disk caches. >>>=20 >>> Here is the test script I used in the cron job ... >>>=20 >>> #!/bin/sh >>> FNAME=3D'output.txt' >>>=20 >>> echo = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >> $FNAME >>> echo Begin @ `/usr/bin/date` >> $FNAME >>> echo >> $FNAME >>> /usr/sbin/bonnie++ 2>&1 | /usr/bin/grep -v 'done\|,' >> $FNAME >>> echo >> $FNAME >>> echo End @ `/usr/bin/date` >> $FNAME >>>=20 >>> As you can see, I'm calling bonnie++ with the system defaults. That = uses a data set size that's 2x the guest RAM in an attempt to minimize = the effect of filesystem cache on results. Here is an example of the = output that bonnie++ produces ... >>>=20 >>> Version 2.00 ------Sequential Output------ --Sequential = Input- --Random- >>> -Per Chr- --Block-- -Rewrite- -Per Chr- = --Block-- --Seeks-- >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec = %CP /sec %CP >>> linux-blk 63640M 694k 99 1.6g 99 737m 76 985k 99 1.3g = 69 +++++ +++ >>> Latency 11579us 535us 11889us 8597us 21819us = 8238us >>> Version 2.00 ------Sequential Create------ --------Random = Create-------- >>> linux-blk -Create-- --Read--- -Delete-- -Create-- = --Read--- -Delete-- >>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec = %CP /sec %CP >>> 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ = +++ +++++ +++ >>> Latency 7620us 126us 1648us 151us 15us = 633us >>>=20 >>> --------------------------------- speed drop = --------------------------------- >>>=20 >>> Version 2.00 ------Sequential Output------ --Sequential = Input- --Random- >>> -Per Chr- --Block-- -Rewrite- -Per Chr- = --Block-- --Seeks-- >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec = %CP /sec %CP >>> linux-blk 63640M 676k 99 451m 99 314m 93 951k 99 402m = 99 15167 530 >>> Latency 11902us 8959us 24711us 10185us 20884us = 5831us >>> Version 2.00 ------Sequential Create------ --------Random = Create-------- >>> linux-blk -Create-- --Read--- -Delete-- -Create-- = --Read--- -Delete-- >>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec = %CP /sec %CP >>> 16 0 96 +++++ +++ +++++ +++ 0 96 +++++ = +++ 0 75 >>> Latency 343us 165us 1636us 113us 55us = 1836us >>>=20 >>> In the example above, the benchmark test repeated about 20 times = with results that were similar to the performance shown above the dotted = line ( ~ 1.6g/s seq write and 1.3g/s seq read ). After that, the = performance dropped to what's shown below the dotted line which is less = than 1/4 the original speed ( ~ 451m/s seq write and 402m/s seq read ).=20= >>>=20 >>>> 2. What results expecting? >>>>=20 >>> What I expect is that, when I perform the same test with the same = parameters, the results would stay more or less consistent over time. = This is true when KVM is used as the hypervisor on the same hardware and = guest options. That said, I'm not worried about bhyve being consistently = slower than kvm or a FreeBSD guest being consistently slower than a = Linux guest. I'm concerned that the performance drop over time is = indicative of an issue with how bhyve interacts with non-freebsd guests. >>>=20 >>>> 3. VM configuration, virtio-blk disk size, etc. >>>> 4. Full command for tests (including size of test-set), bhyve, = etc. >>> I believe this was answered above. Please let me know if you have = additional questions. >>>=20 >>>>=20 >>>> 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you = should try 4K. >>>>=20 >>> The testing performed was not exclusively with virtio-blk. >>>=20 >>>=20 >>>> 6. Linux has several read-ahead options for IO schedule, and it = could be related too. >>>>=20 >>> I suppose it's possible that bhyve could be somehow causing the disk = scheduler in the Linux guest to act differently. I'll see if I can = figure out how to disable that in future tests. >>>=20 >>>=20 >>>> Additionally could also you play with =E2=80=9Csync=3Ddisabled=E2=80=9D= volume/zvol option? Of course it is only for write testing. >>> The testing performed was not exclusively with zvols. >>>=20 >>>=20 >>> Once I have more hardware available, I'll try to report back with = more testing. It may be interesting to also see how a Windows guest = performs compared to Linux & FreeBSD. I suspect that this issue may only = be triggered when a fast disk array is in use on the host. My tests use = a 16x SSD RAID 10 array. It's also quite possible that the disk IO = slowdown is only a symptom of another issue that's triggered by the disk = IO test ( please see end of my last post related to scheduler priority = observations ). All I can say for sure is that ... >>>=20 >>> 1) There is a problem and it's reproducible across multiple hosts >>> 2) It affects RHEL8 & RHEL9 guests but not FreeBSD guests >>> 3) It is not specific to any host or guest storage option >>>=20 >>> Thanks, >>>=20 >>> -Matthew >>>=20 --Apple-Mail=_8118B541-F589-4E79-AF6C-3E98D8AADC93 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 28 Feb 2024, at 23:03, Matthew Grooms = <mgrooms@shrew.net> wrote:

...
The virtual disks were provisioned with either a = 128G disk image or a 1TB raw partition, so I don't think space was an = issue.

Trim is definitely not an issue. I'm using a tiny fraction of the = 32TB array have tried both heavily under-provisioned HW RAID10 and SW = RAID10 using GEOM. The latter was tested after sending full trim resets = to all drives individually.

It could be then = TRIM/UNMAP is not used, zvol (for the instance) becomes full for the = while. ZFS considers it as all blocks are used and write operations = could  have troubles. I believe it was recently = fixed.

Also look at this = one:

    = GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk

The problem of UNMAP that it could have unpredictable slowdown = at any time. So I would suggest to check results with enabled and = disabled UNMAP in a guest.

I will try to = incorporate the rest of your feedback into my next round of testing. If = I can find a benchmark tool that works with a raw block device, that = would be ideal.

Use =E2=80=9Cdd=E2=80=9D= as the first step for read testing;

  =  ~# dd if=3D/dev/nvme0n2 of=3D/dev/null bs=3D1M status=3Dprogress = flag=3Ddirect
   ~# dd if=3D/dev/nvme0n2 = of=3D/dev/null bs=3D1M status=3Dprogress

Compare = results with directio and without.

=E2=80=9Cfio=E2= =80=9D tool. 
 
  1) write = prepare:

      =  ~# fio  --name=3Dprep= --rw=3Dwrite --verify=3Dcrc32 --loop=3D1 = --numjobs=3D2  = --time_based --thread  --bs=3D1M = --iodepth=3D32  --ioengine=3Dlibaio --direct=3D1  --group_reporting  = --size=3D20G  --filename=3D/dev/nvme0n2


 2)  read test:

    =   ~# fio  --name=3Dreadtest --rw=3Dread =E2=80=94loop=3D30 = --numjobs=3D2  = --time_based --thread  =E2=80=94bs=3D256K --iodepth=3D32  --ioengine=3Dlibaio = --direct=3D1  --group_reporting  = --size=3D20G  --filename=3D/dev/nvme0n2
    =  
=E2=80=94
Vitaliy  

Thanks,

-Matthew


=E2=80=94=E2=80=94
Vitaliy

On 28 Feb 2024, at 21:29, Matthew Grooms <mgrooms@shrew.net> wrote:

On 2/27/24 04:21, Vitaliy Gusev = wrote:
Hi,


On 23 Feb 2024, at = 18:37, Matthew Grooms <mgrooms@shrew.net> wrote:

...
The problem occurs when an image file is = used on either ZFS or UFS. The problem also occurs when the virtual disk = is backed by a raw disk partition or a ZVOL. This issue isn't related to = a specific underlying = filesystem.


Do I = understand right, you ran testing inside VM inside guest VM  on = ext4 filesystem? If so you should be aware about additional overhead in = comparison when you were running tests on the = hosts.

Hi Vitaliy,

I = appreciate you providing the feedback and suggestions. I spent over a = week trying as many combinations of host and guest options as possible = to narrow this issue down to a specific host storage or a guest device = model option. Unfortunately the problem occurred with every combination = I tested while running Linux as the guest. Note, I only tested RHEL8 = & RHEL9 compatible distributions ( Alma & Rocky ). The problem = did not occur when I ran FreeBSD as the guest. The problem did not occur = when I ran KVM in the host and Linux as the guest.

I = would suggest to run fio (or even dd) on raw disk device inside VM, i.e. = without filesystem at all.  Just do not forget do =E2=80=9Cecho 3 > /proc/sys/vm/drop_caches=E2=80=9D in = Linux Guest VM before you run tests. 
=

The two servers I was using to test with are are no longer available. = However, I'll have two more identical servers arriving in the next week = or so. I'll try to run additional tests and report back here. I used = bonnie++ as that was easily installed from the package repos on all the = systems I tested.


=
Could you also give more information = about:

 1. What results did you get = (decode bonnie++ output)?

If you look back at = this email thread, there are many examples of running bonnie++ on the = guest. I first ran the tests on the host system using Linux + ext4 and = FreeBSD 14 + UFS & ZFS to get a baseline of performance. Then I ran = bonnie++ tests using bhyve as the hypervisor and Linux & FreeBSD as = the guest. The combination of host and guest storage options included = ...

1) block device + virtio blk
2) block device + nvme
3) = UFS disk image + virtio blk
4) UFS disk image + nvme
5) ZFS disk = image + virtio blk
6) ZFS disk image + nvme
7) ZVOL + virtio = blk
8) ZVOL + nvme

In every instance, I observed the Linux = guest disk IO often perform very well for some time after the guest was = first booted. Then the performance of the guest would drop to a fraction = of the original performance. The benchmark test was run every 5 or 10 = minutes in a cron job. Sometimes the guest would perform well for up to = an hour before performance would drop off. Most of the time it would = only perform well for a few cycles ( 10 - 30 mins ) before performance = would drop off. The only way to restore the performance was to reboot = the guest. Once I determined that the problem was not specific to a = particular host or guest storage option, I switched my testing to only = use a block device as backing storage on the host to avoid hitting any = system disk caches.

Here is the test script I used in the cron = job ...

#!/bin/sh
FNAME=3D'output.txt'

echo = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >> $FNAME
echo Begin @ `/usr/bin/date` >> = $FNAME
echo >> $FNAME
/usr/sbin/bonnie++ 2>&1 | = /usr/bin/grep -v 'done\|,' >> $FNAME
echo >> = $FNAME
echo End @ `/usr/bin/date` >> = $FNAME

As you can see, I'm calling bonnie++ with = the system defaults. That uses a data set size that's 2x the guest RAM = in an attempt to minimize the effect of filesystem cache on results. = Here is an example of the output that bonnie++ produces ...

Version  = 2.00       ------Sequential Output------ = --Sequential Input- = --Random-
          &= nbsp;         -Per Chr- --Block-- = -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size = etc        /sec %CP  /sec = %CP  /sec %CP  /sec %CP  /sec %CP  /sec = %CP
linux-blk    63640M  694k  99  = 1.6g  99  737m  76  985k  99  1.3g  = 69 +++++ = +++
Latency          =    11579us     535us   = 11889us    8597us   21819us    = 8238us
Version  2.00       = ------Sequential Create------ --------Random = Create--------
linux-blk        = ;   -Create-- --Read--- -Delete-- -Create-- --Read--- = -Delete--
          &= nbsp;   files  /sec %CP  = /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec = %CP
           &= nbsp;     16 +++++ +++ +++++ +++ = +++++ +++ +++++ +++ +++++ +++ +++++ = +++
Latency          =     7620us     = 126us    1648us     = 151us      15us     = 633us

--------------------------------- speed drop = ---------------------------------

Version  = 2.00       ------Sequential Output------ = --Sequential Input- = --Random-
          &= nbsp;         -Per Chr- --Block-- = -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size = etc        /sec %CP  /sec = %CP  /sec %CP  /sec %CP  /sec %CP  /sec = %CP
linux-blk    63640M  676k  99  = 451m  99  314m  93  951k  99  402m  = 99 15167 = 530
Latency          =    11902us    8959us   = 24711us   10185us   20884us    = 5831us
Version  2.00       = ------Sequential Create------ --------Random = Create--------
linux-blk        = ;   -Create-- --Read--- -Delete-- -Create-- --Read--- = -Delete--
          &= nbsp;   files  /sec %CP  = /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec = %CP
           &= nbsp;     16     = 0  96 +++++ +++ +++++ +++     0  96 +++++ = +++     0  = 75
Latency          &= nbsp;    343us     = 165us    1636us     = 113us      55us    = 1836us

In the example above, the benchmark test = repeated about 20 times with results that were similar to the = performance shown above the dotted line ( ~ 1.6g/s seq write and 1.3g/s = seq read ). After that, the performance dropped to what's shown below = the dotted line which is less than 1/4 the original speed ( ~ 451m/s seq = write and 402m/s seq read ). 

&nbs= p;2. What results expecting?

What I = expect is that, when I perform the same test with the same parameters, = the results would stay more or less consistent over time. This is = true when KVM is used as the hypervisor on the same hardware and guest = options. That said, I'm not worried about bhyve being consistently = slower than kvm or a FreeBSD guest being consistently slower than a = Linux guest. I'm concerned that the performance drop over time is = indicative of an issue with how bhyve interacts with non-freebsd = guests.

&nbs= p;3. VM configuration, virtio-blk disk size, etc.
 4. = Full command for tests (including size of test-set), bhyve, = etc.

I believe this was answered above. = Please let me know if you have additional questions.


=
 5. Did you pass virtio-blk as 512 or 4K ? If 512, = probably you should try 4K.

The = testing performed was not exclusively with = virtio-blk.

&nbs= p;6. Linux has several read-ahead options for IO schedule, and it could = be related too.

I suppose it's = possible that bhyve could be somehow causing the disk scheduler in the = Linux guest to act differently. I'll see if I can figure out how to = disable that in future tests.

Addi= tionally could also you play with =E2=80=9Csync=3Ddisabled=E2=80=9D = volume/zvol option? Of course it is only for write = testing.

The testing performed was not = exclusively with zvols.

Once I have more hardware = available, I'll try to report back with more testing. It may be = interesting to also see how a Windows guest performs compared to Linux = & FreeBSD. I suspect that this issue may only be triggered when a = fast disk array is in use on the host. My tests use a 16x SSD RAID 10 = array. It's also quite possible that the disk IO slowdown is only a = symptom of another issue that's triggered by the disk IO test ( please = see end of my last post related to scheduler priority observations ). = All I can say for sure is that ...

1) There is a problem and it's = reproducible across multiple hosts
2) It affects RHEL8 & RHEL9 = guests but not FreeBSD guests
3) It is not specific to any host or = guest storage = option

Thanks,

-Matthew


= --Apple-Mail=_8118B541-F589-4E79-AF6C-3E98D8AADC93-- From nobody Wed Feb 28 21:31:39 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlSGq4qzWz5CSNR for ; Wed, 28 Feb 2024 21:31:47 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlSGp5y0Tz4gFj for ; Wed, 28 Feb 2024 21:31:46 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=EOMbAPKp; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 41SLVeIj006909; Wed, 28 Feb 2024 15:31:40 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1709155900; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hIYOrKlheX1G4yxRxytbAE4RMn8PtinUlq28NWLbP+8=; b=EOMbAPKpfI9P0D8TptuM8jAgD0Lk82+3Dn2tWy5JZXVjtCmuKLmloG+Om0YMMbipBYvONX RVqjtbQE8Md6uwzp6dT5B6NK1kxF1JBMlRJ5YBhqPhhIxrUXjYB2hnxLaD8wAcdoQrKf7a gqjwo+v+eXi2MJ7/s9eJrDQFLhSqg0tJi3lLj487dlRAz5e9kFw24qI8uqo3tbGsQtymgY zQKO/jT0i8OjhOrOnph+sL2thSwNXH0IWUHQZHJmFeFSO8r1lkUhpzyw+cgy6imUFt4uYP ZrDQ3NxrgQoT/Drl/nirFWbF5d2Nb2dD+jHF5SpImT0Kz5qEVPHkSeknaDg1bg== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 0FE5F3AB37; Wed, 28 Feb 2024 15:31:40 -0600 (CST) Content-Type: multipart/alternative; boundary="------------uA3ah2p5aRXCMQ7fXwU07F9H" Message-ID: <3738b08a-7841-4d18-9439-5f1c73a5c9e1@shrew.net> Date: Wed, 28 Feb 2024 15:31:39 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: Vitaliy Gusev Cc: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> <1DAEB435-A613-4A04-B63F-D7AF7A0B7C0A@gmail.com> <3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com> From: Matthew Grooms In-Reply-To: <3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.47 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.979]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[shrew.net]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[shrew.net:+] X-Rspamd-Queue-Id: 4TlSGp5y0Tz4gFj This is a multi-part message in MIME format. --------------uA3ah2p5aRXCMQ7fXwU07F9H Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/28/24 15:02, Vitaliy Gusev wrote: > > >> On 28 Feb 2024, at 23:03, Matthew Grooms wrote: >> >> ... >> The virtual disks were provisioned with either a 128G disk image or a >> 1TB raw partition, so I don't think space was an issue. >> >> Trim is definitely not an issue. I'm using a tiny fraction of the >> 32TB array have tried both heavily under-provisioned HW RAID10 and SW >> RAID10 using GEOM. The latter was tested after sending full trim >> resets to all drives individually. >> > It could be then TRIM/UNMAP is not used, zvol (for the instance) > becomes full for the while. ZFS considers it as all blocks are used > and write operations could  have troubles. I believe it was recently > fixed. > > Also look at this one: > > GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk > > The problem of UNMAP that it could have unpredictable slowdown at any > time. So I would suggest to check results with enabled and disabled > UNMAP in a guest. > Yes. I'm aware of issues associated with TRIM/UNMAP, but that's not implemented by any hardware RAID vendor that I'm aware of. I tested with both hardware and software RAID10. The issue I'm reporting is present in both cases. I'm quite certain this has nothing to do with TRIM/UNMAP. -Matthew --------------uA3ah2p5aRXCMQ7fXwU07F9H Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/28/24 15:02, Vitaliy Gusev wrote:


On 28 Feb 2024, at 23:03, Matthew Grooms <mgrooms@shrew.net> wrote:

...
The virtual disks were provisioned with either a 128G disk image or a 1TB raw partition, so I don't think space was an issue.

Trim is definitely not an issue. I'm using a tiny fraction of the 32TB array have tried both heavily under-provisioned HW RAID10 and SW RAID10 using GEOM. The latter was tested after sending full trim resets to all drives individually.

It could be then TRIM/UNMAP is not used, zvol (for the instance) becomes full for the while. ZFS considers it as all blocks are used and write operations could  have troubles. I believe it was recently fixed.

Also look at this one:

    GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk

The problem of UNMAP that it could have unpredictable slowdown at any time. So I would suggest to check results with enabled and disabled UNMAP in a guest.

Yes. I'm aware of issues associated with TRIM/UNMAP, but that's not implemented by any hardware RAID vendor that I'm aware of. I tested with both hardware and software RAID10. The issue I'm reporting is present in both cases. I'm quite certain this has nothing to do with TRIM/UNMAP.

-Matthew

--------------uA3ah2p5aRXCMQ7fXwU07F9H-- From nobody Fri Mar 1 21:32:56 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TmhCD2sDvz5D5VG for ; Fri, 1 Mar 2024 21:32:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TmhCD1VGgz435Q for ; Fri, 1 Mar 2024 21:32:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709328776; a=rsa-sha256; cv=none; b=nRjlfSX0hG4qrk+RjNEZyTv8BPi2OlVy16t4TY84hUTDjrInAPllDN7iVrHBn1WR7NtHHE UeYyySlHYRIInVKisSxggqZa9s8xwoXUpfH361WVZmr0uQMMsceNc0XsVbLHd0h6uFwmUw TRdiae9uOw/yIg+9RofsBieYf4cYraMxR3+6kfYaCRTwsfb20iOzNAFT9T1Vny5d/tOrEF ZVXy0/amdAovrzR2OpHJ6MPPdahjEE45Leoe8kpNqqPkywex4NuOPhCORtiyXH0K3wpZsT 8+vZkFpUaXVif+8JeMaMH5bz1e1mVGB1Y6tpeVmQ0ayKUwsOVmmZUlskTuUuGw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709328776; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fsPTTcZlGfrohWVBCDUcUE3JGJjcVdoLuNnTkZWi7V4=; b=SUxb/3su+offWC6A2ZKRukTk+rocEhLPOyWjcMqSBnB95huMBJYm3k68YXVkEkByB1Gqc4 XHaedO3sdc1ByohGM/C0GCHxnorpiI4xIDTZRzRhpWJo9Z0/N6R6SgRnv2PTDUr9grYAPe GzKEl4izGbUOBT/QVsIOe/jB9dlKLvi5dDyM+vXl7VPo9MGNHo0h3u8KzoVI/h9kjvnSMC nEt3zgR2jnVPp2vVy5lqnfRM/xbpag8zbBq8WUE/Kaed6wv6XzwM7rIsAcG/hgFj4KE4lQ 4XUuaYQIM/f2pssh30uAfMMf2R9Ef8JMabdPZWAoSKsRoRNenKL9wJytso/oVw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TmhCD16JYzj4P for ; Fri, 1 Mar 2024 21:32:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 421LWu0m091729 for ; Fri, 1 Mar 2024 21:32:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 421LWu6Z091728 for virtualization@FreeBSD.org; Fri, 1 Mar 2024 21:32:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 237636] bhyve guest ZFS filesystem freezes in zcw->zcw_cv state Date: Fri, 01 Mar 2024 21:32:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd-bugs@morgandavis.net X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D237636 Morgan Davis changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |FIXED --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 1 22:29:21 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TmjSb3vhwz5D9km for ; Fri, 1 Mar 2024 22:29:35 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TmjSZ3mf6z4FGp for ; Fri, 1 Mar 2024 22:29:34 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nomadlogic.org header.s=04242021 header.b=EnLC06LZ; dmarc=pass (policy=quarantine) header.from=nomadlogic.org; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 66.165.241.226 as permitted sender) smtp.mailfrom=pete@nomadlogic.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1709332160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=EFdXTxYmkvTEe5h5G09IrBVgj+l6X4Od/2+DDHCUeQE=; b=EnLC06LZBkoNH0LuSZS0u33BsU1f8u8YknUSAEmQC96WsZTl40Up9VqxRsb/m9u/wOiIZx BPB7lYti4Hdv+m31XHQrjOVJya1sn+K84VGpU5U6qFnZI1ssTFXjZvR93kD6jIiTEGbgbd PkzXQpMmZNcEHAoGeF7sPzUVTGVTk+U= Received: from [192.168.1.160] ( [47.154.31.160]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id a839bdeb (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 1 Mar 2024 22:29:19 +0000 (UTC) Message-ID: Date: Fri, 1 Mar 2024 14:29:21 -0800 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: freebsd-virtualization@FreeBSD.org Content-Language: en-US From: Pete Wright Subject: Bhyve Boot Question Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[nomadlogic.org,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[nomadlogic.org:s=04242021]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@FreeBSD.org]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[nomadlogic.org:+] X-Rspamd-Queue-Id: 4TmjSZ3mf6z4FGp Hello, I was hoping someone could help me diagnose an issue I'm having porting some FreeBSD VM's that are currently running on FreeBSD+Bhyve to a SmartOS+Bhyve configuration. The VM's I'd like to port over use a file back VM raw disk image, this works fine on my FreeBSD hypervisor but SmartOS would prefer zvols.  One thing I should note is I'm using bhyveload(8) to boot my VMs.  The guest instances in question are all FreeBSD as well. To test migrating these VM's to SmartOS I've done the following: 1. copy the VM image file to the SmartOS hypervisor 2. use quemu-img convert to convert the raw disk image to a zvol I did this with one of the official memdisk USB images, and things worked great.  I used the UEFI bootrom flag to bhyve.  Yet when I try to take one of the VM's I want to port things don't work as expected.  When I start the VM the console (and VNC display output) just hang on a blank screen.  I suspect its related to the fact that I use bhyveload(8). Here is what the disk looks like for one of the VM's in question: $ gpart  show nda0 =>      40  41942960  nda0  GPT  (20G)        40       216        - free -  (108K)       256      1024     1  freebsd-boot  (512K)      1280  39844608     2  freebsd-ufs  (19G)  39845888   2097112     3  freebsd-swap  (1.0G) So my question is this - if I am reading the gpart output correctly, I should be able to use the "bios" bootrom option for Bhyve under SmartOS right and it'd find the "freebsd-boot" partition?  Or is something special happening to my VM when I use bhyveload(8) that would cause problems porting it over to a zvol under SmartOS? Thanks in advance for any insights! -pete -- Pete Wright pete@nomadlogic.org From nobody Sat Mar 2 09:31:17 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn07F2gbxz5CPMv for ; Sat, 2 Mar 2024 09:30:33 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Received: from mo4-p05-ob.smtp.rzone.de (mo4-p05-ob.smtp.rzone.de [81.169.146.181]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.smtp.rzone.de", Issuer "Telekom Security ServerID OV Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn07D1l0dz4Md6 for ; Sat, 2 Mar 2024 09:30:31 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0002 header.b=GsxkavCR; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0003 header.b=VHRZEefZ; dmarc=none; spf=none (mx1.freebsd.org: domain of Stephan.Althaus@Duedinghausen.eu has no SPF policy when checking 81.169.146.181) smtp.mailfrom=Stephan.Althaus@Duedinghausen.eu; arc=pass ("strato.com:s=strato-dkim-0002:i=1") ARC-Seal: i=1; a=rsa-sha256; t=1709371829; cv=none; d=strato.com; s=strato-dkim-0002; b=R3XfFIE4XWUULO5M+QM+uybJiM41DQa+rp3jnnTPqixXElfKEoHm6Z0fhenkpKm98r 8SWI/EgzH7DGxG/SfbNVEPwX3m1U+Pprcc8FybC68Ppy/dMdOuoNXunh9ZUjqh5pGePW KZbTIcrpY6bJ1vquG6u7w8kXXHdYhHCgNn8LTqbfBp5GNumhcqqE95s3iaHS0TOt77W0 KlxHyO7kRMTsDidkUK3smnE94suDJ1Zu3ICLKov4YYnvFLrBQptJPEHGW6jAOLMqWtfo s/ahYM7GUAEG5B9N9m7b7J3RGZYABSXa99yDnJYtBneoPO1T3RBzluHOI8sdcMZ5FsB/ HjzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1709371829; s=strato-dkim-0002; d=strato.com; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=utr9DgeUpMAbco5AiNRoOhFpmJhUBCQt49GPgJfu/NU=; b=ZsYJGW/H2THpK/xoQCokEc00cQbPLlrHr4ji32gsUEMMkS/PpkCbioQSqJshsSFxhb 0zsPfB7bkcF/dnuiVEmNIxbMtG0BScxO227qsu8UH2Xral2mMIR+Co33nnYQzisIw7ZO rj0cgkaexV03qiDrXbWEwQWHnhrccFcYlxYTQDU/kPlNGDx3t5GELDzKQKuSEfPFYHRb W9w/zLeMQZ6KSN+573pblAxt+6424XGoghWyD4xxmD+nWfHIAz8PRI+GWLDhII/8E5NP IS1cmg12Gh89uSgn4PvmutVWzWQmvoe9kbnoII3wbAn+mBmiu5hw2eZOchwoJv3UflgF D2mA== ARC-Authentication-Results: i=1; strato.com; arc=none; dkim=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1709371828; s=strato-dkim-0002; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=utr9DgeUpMAbco5AiNRoOhFpmJhUBCQt49GPgJfu/NU=; b=GsxkavCRHaz/QFDTK5ZTOXdBm2cHssyn5AsRM7mChledRgouqOaRKA45J+2RsLVKs9 dOsGTkA7/JOoZDpA4rOyrohSRPNSJTVy/EK/2p+pcOPH6Bm+tk1y0ZXADMrcABN+noFl oZBQevws3ByK29XatdKLA0PpCunqZiXMkj8bGaSryYnnb//f+lmDHj0T6tCQ5/vSvs8u hOqsspH4A+EJKurApr3xWzZ0UhPHyCHULXyOHEv/+F4TsG/W+/1Mqsk3sl1/9YVRDdP7 aRxNKJfnXx3P4l7c3qKB7HlI2e1Cy3A0p4KojzhtD6P2nQgOH+9uQtBC1xImDQb5SgAV 6NTw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1709371828; s=strato-dkim-0003; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=utr9DgeUpMAbco5AiNRoOhFpmJhUBCQt49GPgJfu/NU=; b=VHRZEefZ0RaNGbqoInx8tVI5LpBwLXCI4+b5wPEWU/P7hlgBBCz3E4iO0R3KhU6jtk ZhtONzhvIxwXG0RQUPCQ== X-RZG-AUTH: ":O2kGeEG7b/pS1EW2TmikjLDsYYueHLp2aWg0q38nsxN1mOntnRORP93PLpfReKNjbyctOge9YThV7Q==" X-RZG-CLASS-ID: mo05 Received: from www.duedinghausen.eu by smtp.strato.de (RZmta 50.2.0 DYNA|AUTH) with ESMTPSA id R5eff80229USDE6 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate) for ; Sat, 2 Mar 2024 10:30:28 +0100 (CET) Received: from [192.168.2.63] (speedport.ip [192.168.1.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) (Authenticated sender: steven) by www.duedinghausen.eu (Postfix) with ESMTPSA id 6CF4A11A199 for ; Sat, 2 Mar 2024 10:28:38 +0100 (CET) Message-ID: <054b7368-ff59-41c0-aafc-f489d6643acf@Duedinghausen.eu> Date: Sat, 2 Mar 2024 10:31:17 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Bhyve Boot Question To: virtualization@freebsd.org References: Content-Language: en-US From: Stephan Althaus In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.39 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_ALLOW(-1.00)[strato.com:s=strato-dkim-0002:i=1]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[hoewweken.de:s=strato-dkim-0002,hoewweken.de:s=strato-dkim-0003]; RCVD_IN_DNSWL_LOW(-0.10)[81.169.146.181:from]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:6724, ipnet:81.169.144.0/22, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[duedinghausen.eu]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[hoewweken.de:+] X-Rspamd-Queue-Id: 4Tn07D1l0dz4Md6 On 3/1/24 23:29, Pete Wright wrote: > Hello, > > I was hoping someone could help me diagnose an issue I'm having > porting some FreeBSD VM's that are currently running on FreeBSD+Bhyve > to a SmartOS+Bhyve configuration. > > The VM's I'd like to port over use a file back VM raw disk image, this > works fine on my FreeBSD hypervisor but SmartOS would prefer zvols.  > One thing I should note is I'm using bhyveload(8) to boot my VMs.  The > guest instances in question are all FreeBSD as well. > > To test migrating these VM's to SmartOS I've done the following: > > 1. copy the VM image file to the SmartOS hypervisor > > 2. use quemu-img convert to convert the raw disk image to a zvol > > I did this with one of the official memdisk USB images, and things > worked great.  I used the UEFI bootrom flag to bhyve.  Yet when I try > to take one of the VM's I want to port things don't work as expected.  > When I start the VM the console (and VNC display output) just hang on > a blank screen.  I suspect its related to the fact that I use > bhyveload(8). > > Here is what the disk looks like for one of the VM's in question: > $ gpart  show nda0 > =>      40  41942960  nda0  GPT  (20G) >        40       216        - free -  (108K) >       256      1024     1  freebsd-boot  (512K) >      1280  39844608     2  freebsd-ufs  (19G) >  39845888   2097112     3  freebsd-swap  (1.0G) > > So my question is this - if I am reading the gpart output correctly, I > should be able to use the "bios" bootrom option for Bhyve under > SmartOS right and it'd find the "freebsd-boot" partition?  Or is > something special happening to my VM when I use bhyveload(8) that > would cause problems porting it over to a zvol under SmartOS? > > Thanks in advance for any insights! > > -pete > Hello! If you have a vm with BIOS boot, you have to use an other bootrom file.. UEFI looks like bootrom,/usr/share/bhyve/firmware/BHYVE_RELEASE.fd BIOS would be bootrom,/usr/share/bhyve/firmware/BHYVE_RELEASE_CSM.fd Alas i am on OpenIndiana, on SmartOs it may differ.. You can get your infos via pkg commands like this: # pkg list|grep bhy system/bhyve 0.5.11-2024.0.0.22024      i-- system/bhyve/firmware 20230801-2024.0.0.0        i-- system/library/bhyve 0.5.11-2024.0.0.22024      i-- system/zones/brand/bhyve 0.5.11-2024.0.0.5618       i-- # pkg contents system/bhyve/firmware PATH usr/share/bhyve usr/share/bhyve/firmware usr/share/bhyve/firmware/BHYVE.fd usr/share/bhyve/firmware/BHYVE_CSM.fd usr/share/bhyve/firmware/BHYVE_DEBUG.fd usr/share/bhyve/firmware/BHYVE_DEBUG_CSM.fd usr/share/bhyve/firmware/BHYVE_RELEASE.fd usr/share/bhyve/firmware/BHYVE_RELEASE_CSM.fd usr/share/bhyve/firmware/BHYVE_VARS.fd HTH Regards, Stephan From nobody Sat Mar 2 21:14:02 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnHkz5r36z5CySw for ; Sat, 2 Mar 2024 21:14:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnHkz2MHXz4dk1 for ; Sat, 2 Mar 2024 21:14:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709414043; a=rsa-sha256; cv=none; b=vH8/U3wxgqPim5gGYtFpNiazmjdjC7FU2Udi/uy6+z9FD1xDxXC2arFnEMrrSNApwST7MM f8bpoE/81xHKG81OkskXezv/cmAiNiM7RGcJ9ELtqGuIfKFAaTbXQj3vppenTuAentV3hm qVi/fqIdipoiOlGYO7x6lJHcy1Y6PHShFc79LwKuYhmr02D2i41zQosPJwlKwohsKdYzLh KzZgTble4ajaMK/m+HF+iVpd9fmju65GT3Bpu8bxPYzQA8bffj3SlxRouQ6jqIQZyhKfst U6brb5amr0ZzWFbkvMwYLxjVclEtYCAGOw0Ofcr+676LlZ81gXH0G+edlCTbSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709414043; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=boWRFNewnZIodQt8vdHZwmtiSru5/W4Fe+LDYmAaqtM=; b=a5vUIX+ELx3+lbxjEcfBRUr26c+ltZfkboE+Pjp5EOrKKlsPB2nbMtbcOSV1W3I5NaQgeO EzwXOMXVqZgJkEHqkTiWkXKuoayKt3B5HuujZ6aDtThJ0KE7UeL2Et7b4bdQIVUn1arn9r UNIfEOKDZhHV7qRbVhnlzfIEl4G99eVDosRPI3DhIjy4pTlTnHk58u1ZBnpmMbWydZnAtd mRXVPOtFbunocs4IEf7T2/SJhS0Cuof40YBz1lgnBEn3KdnUMUa3DylnDG5sVJ2/qkVvjf Jy3lPrL7XfF8W+M4otNQg+57UQVuUz2SfP2B7ZZxgTTm46TKz/oKJGYKLSWZKg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TnHkz1sbRz1PQ3 for ; Sat, 2 Mar 2024 21:14:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 422LE3U6077260 for ; Sat, 2 Mar 2024 21:14:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 422LE39E077253 for virtualization@FreeBSD.org; Sat, 2 Mar 2024 21:14:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 264253] hyperv: ifconfig media autoselect on FreeBSD 13.0 displays error: ifconfig: SIOCSIFMEDIA (media): Operation not supported Date: Sat, 02 Mar 2024 21:14:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: needs-qa, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: mp@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? mfc-stable13? mfc-stable12? X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264253 Mark Peek changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mp@FreeBSD.org --- Comment #6 from Mark Peek --- I see the same error across 10.4, 13.0, and 15-current 10.4 =3D=3D=3D=3D # uname -a FreeBSD fbsd10 10.4-RELEASE FreeBSD 10.4-RELEASE #0 r324094: Fri Sep 29 01:45:44 UTC 2017 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GEN= ERIC amd64 # ifconfig -mv hn0 hn0: flags=3D8843 metric 0 mtu 1500 options=3D8051b =20=20=20=20=20=20=20 capabilities=3D48071b ether 00:15:5d:0a:28:00 hwaddr 00:15:5d:0a:28:00 inet6 fe80::215:5dff:fe0a:2800%hn0 prefixlen 64 scopeid 0x2 inet6 2601:601:4000:c4de:215:5dff:fe0a:2800 prefixlen 64 autoconf inet6 fd50:d9bc:e952:6040:215:5dff:fe0a:2800 prefixlen 64 detached autoconf inet 10.1.10.215 netmask 0xffffff00 broadcast 10.1.10.255 nd6 options=3D23 media: Ethernet autoselect (10Gbase-T ) status: active supported media: media autoselect # ifconfig hn0 media autoselect ifconfig: SIOCSIFMEDIA (media): Operation not supported 13.0 =3D=3D=3D=3D # uname -a FreeBSD bsd13 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:24:09 UTC 2021=20=20=20=20 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 # ifconfig -mv hn0 hn0: flags=3D8843 metric 0 mtu 1500 options=3D8051b =20=20=20=20=20=20=20 capabilities=3D48071b ether 00:15:5d:0a:28:01 inet 10.1.10.139 netmask 0xffffff00 broadcast 10.1.10.255 media: Ethernet autoselect (10Gbase-T ) status: active supported media: media autoselect nd6 options=3D29 # ifconfig hn0 media autoselect ifconfig: SIOCSIFMEDIA (media): Operation not supported 15-current =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # uname -a FreeBSD FreeBSD-current 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n268454-9097284b98be: Thu Feb 22 03:00:34 UTC 2024=20=20=20=20 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 # ifconfig -mv hn0 hn0: flags=3D1008843 metri= c 0 mtu 1500 options=3D8051b =20=20=20=20=20=20=20 capabilities=3D48071b ether 00:15:5d:0a:28:02 inet 10.1.10.125 netmask 0xffffff00 broadcast 10.1.10.255 inet6 fe80::215:5dff:fe0a:2802%hn0 prefixlen 64 scopeid 0x2 inet6 2601:601:4000:c4de:215:5dff:fe0a:2802 prefixlen 64 autoconf pltime 14400 vltime 86400 inet6 fd50:d9bc:e952:6040:215:5dff:fe0a:2802 prefixlen 64 detached autoconf pltime 1800 vltime 1800 media: Ethernet autoselect (10Gbase-T ) status: active supported media: media autoselect nd6 options=3D23 drivername: hn0 # ifconfig hn0 media autoselect ifconfig: SIOCSIFMEDIA (media): Operation not supported This is not surprising since the hn_ifmedia_upd() callback is the same and returns EOPNOTSUPP. The code for 10.4 is here: https://github.com/freebsd/freebsd-src/blob/releng/10.4/sys/dev/hyperv/netv= sc/if_hn.c#L1008 Since the only media supported is autoselect, it seems reasonable to just return 0 from hn_ifmedia_upd() instead of EOPNOTSUPP. Patched on 15-current, this results in no error for "autoselect" and will error for unregistered m= edia types. # ifconfig hn0 media autoselect # ifconfig hn0 media 100baseT ifconfig: SIOCSIFMEDIA (media): Device not configured An early review D4611 did have hn_ifmedia_upd() return 0 but was abandoned.= A subsequent review D4852 has this function returning EOPNOTSUPP. Allowing selection of the current (and only) registered media type seems reasonable rather than throwing an error. I'm going to try contacting a cou= ple of the contributors to the reviews to see if they agree. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 3 20:56:31 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnvK426Bpz5CJPr; Sun, 3 Mar 2024 20:57:12 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnvK16k9gz42yp; Sun, 3 Mar 2024 20:57:09 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=T3qKZ6uv; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::52e as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-ed1-x52e.google.com with SMTP id 4fb4d7f45d1cf-5643ae47cd3so5281106a12.3; Sun, 03 Mar 2024 12:57:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709499428; x=1710104228; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=QGMyxPYddGCloH1b9ZXYErmNwRbBWY6kp4i87YzBpNA=; b=T3qKZ6uve/7jfX2GBuAx4nC9UziqglHlX9bCuC1i70swki0L9VFliDApUmTexzvNKf eOhSAvdjhae1dY3Ij79/0UyuuiQK8zWbt9i84/YZpaGIcK3HRaLqof4vywEq8LBFrq70 tNrp5OiGDl3cqUOi89b25hDfT8shcLqudEuDmVPC1lN3tMTm27Z67dxIwKpS1aDza99K ocaw+oiuqzhzF6FeoW0sj4izkNdXpYahXVTMGnZfYNepufOX3QY5OSzigMMg3VvOgo2o msQeUDDRYMh0BU3yxjTaTm2gry0N/IBXo1lL/l2tyvPGWAD1zIqlzed03Ylvo9o73f/9 rDJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709499428; x=1710104228; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=QGMyxPYddGCloH1b9ZXYErmNwRbBWY6kp4i87YzBpNA=; b=oqmTCpY5r72A6ulm1dWBKsWEOjNgYUyKLIT0AUnGXYRfz5OlfS2C0qMHCq0gESB19B r7kS2HTy2lZasS/D7lfOIETtqza06Te4LeLQt0h/LuO+MDScGT873+hLaHXcWo3DS3oS mtApiLbv6oAM2KswH2/0xwX6/XW9mTPyTcJZWn8EhMU94VEFOoTKYtGi82hqJKVH8psK DLIGRc04j6tDY763Y50H1Z+F7lN2icmRJVoQh2vmsfukaM6lcVOWCSPaiYrglUTZOZer 9Egw2z/b7kCEq/8HHt/7uEeFBPhiNSJ9Bki4mAdaQmZYxjbWJfsU2J9Q2KMrf6PMWugz +MIg== X-Forwarded-Encrypted: i=1; AJvYcCUTFAy0kzDC/hSI5Hb7YH8xTxWzBjAnF5APznzfPwta8CcUU9ztVs+37TlJ2rm0WMpGE0wU1xDU2MazmSmqsPIsbW1aiqAZskT+iUrPrOLjsd6AGH00ylMmw6js9626yhHNEpH0reicbEOg X-Gm-Message-State: AOJu0YwdEtmF03cCSNbWHTHcFl0rnq5GQt0O5rPWuHK4gmgO4+sVIZWE lz3DF5PB6ldMJ4AI819PBOChodDwPQwbhqJIXcMJaIXSdZb2RoY4viejqBAmfb15odCx/4f6ivV SvNXmdpMtn2tPjTVZ98MwcHyh40/sdqtq9ZA= X-Google-Smtp-Source: AGHT+IGMp1xB8z/7knFJnFRAkW91HDXlrh0g91UttottUVIKNN52t72RpBGYQ7X3dv9r4nc9QzGNqcn5auZywbyRk3s= X-Received: by 2002:a17:906:37d6:b0:a45:f05:7e10 with SMTP id o22-20020a17090637d600b00a450f057e10mr1676790ejc.24.1709499427455; Sun, 03 Mar 2024 12:57:07 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 From: Mario Marietto Date: Sun, 3 Mar 2024 21:56:31 +0100 Message-ID: Subject: A project is to be able to pass through an nvidia gpu to a Windows vm virtualized with bhyve. To: FreeBSD virtualization , FreeBSD Mailing List , freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000001e06830612c7dafe" X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52e:from]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-questions@freebsd.org,freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+] X-Rspamd-Queue-Id: 4TnvK16k9gz42yp --0000000000001e06830612c7dafe Content-Type: text/plain; charset="UTF-8" Hello. My project is to be able to pass through an nvidia gpu to a Windows vm virtualized with bhyve. I know how to do that,but no one wants to take my idea into consideration. The problem is that Windows gives error 12 (interrupt conflicts) when it finds the nvidia gpu that has been passed through by bhyve. This is the patch needed : https://gitlab.com/Queuecumber/linux-acs-override/raw/master/workspaces/5.6.12/acso.patch Linux does not need this patch anymore because it has been added in the mainline kernel. Different story is Windows. No one can add that code inside the Windows code. But,maybe it can be added inside the bhyve code or that code can be converted in powershell code and run it under Windows. I don't know what's the best method. But I'm sure that it will work. Let me know what you think. -- Mario. --0000000000001e06830612c7dafe Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello.

My project is to be a= ble to pass through an nvidia gpu to a Windows vm virtualized with bhyve. I know how to do that,but no one wants to=20 take my idea into consideration.=C2=A0 The problem is that Windows gives=20 error 12 (interrupt conflicts) when it finds the nvidia gpu that has=20 been passed through by bhyve. This is the patch needed :

=

Linux does not need this patch anymore because it has been added in the=20 mainline kernel. Different story is Windows. No one can add that code=20 inside the Windows code. But,maybe it can be added inside the bhyve code or that code can be converted in powershell code and run it under=20 Windows. I don't know what's the best method. But I'm sure that= it will=20 work.

Let me know what you think.

--
Mario.
--0000000000001e06830612c7dafe-- From nobody Sun Mar 3 21:00:50 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnvPH4VBZz5CJC6 for ; Sun, 3 Mar 2024 21:00:51 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnvPH0phRz473r for ; Sun, 3 Mar 2024 21:00:51 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709499651; a=rsa-sha256; cv=none; b=IL8kz686fLxRsSs534jsLEoF2j0VZixBde3OwwIFgEGC9GcQWKa0J1AwGr6hVu38IAnBoS XZW1hzvcTzzkgzoYSqe6AJbVUWsCo0xH6vDN76Itl62lxCRi7lHRd2bCrKwUYa/qdoTLrn XscyfDLcRnvw+bRuhzVgXKqB11qVJRK6UZs/9C+jiYxUPLAp9CdhPaKt2tNySBVNu2W27j ZVAW6zC2Q0ZA4Ds1EOPZNOGcHfILloe7ZuFRzKflRpPCh2TMHtKdO/VBWeEld7HHefv2Qp eGyrzZgcjqcuLXkYdUhPZp+tiE1jxSIIYIrMynlj5MvxGtUjOP7116keHBfC+g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709499651; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tXCgJrVadneaC8tC/9LXIPUep9j4UaYDZN/1yl6oFvY=; b=EsBaTMwkYbIs9Ihmvf79/UlQY5Ib5DSCJ65v95MqcHlN1p08O1690XjXTfIbl5HSOVcbAA 8OMcgQnPgYdcD3hwWcIgNumO4Pa/0jA3axZKR96SVEsBn9u/lH8ygfl6uZwadhekALa/4C ejcqzFD6rWs45+/7V4DbSDr/6MghXDua/hohp6wLN6mQY5M3agyojfQvBr+dzYwOAAo1vS AiDHjo/c8e91QA1mcfeHLeY8UVz4l87nucP4Twy3X+fveDZEJELZbaWH90bbuhUXg2Ud21 rwYrhP3V0EhAQAlYSaKwOS2H9Nl2EF1JZAu30Ady4XGa9IvjZ7sFuBHy3V9POg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TnvPH0QtHzthH for ; Sun, 3 Mar 2024 21:00:51 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 423L0oKO075764 for ; Sun, 3 Mar 2024 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 423L0oSU075763 for virtualization@FreeBSD.org; Sun, 3 Mar 2024 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202403032100.423L0oSU075763@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 3 Mar 2024 21:00:50 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17094996506.cba5.66720" Content-Transfer-Encoding: 7bit --17094996506.cba5.66720 Date: Sun, 3 Mar 2024 21:00:50 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 264267 | UEFI Booting on Azure Generation 2 VMs crashes 1 problems total for which you should take action. --17094996506.cba5.66720 Date: Sun, 3 Mar 2024 21:00:50 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    264267 | UEFI Booting on Azure Generation 2 VMs crashes

1 problems total for which you should take action.
--17094996506.cba5.66720--