From nobody Tue Feb 20 18:54:54 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TfT9W3jRRz55R1S for ; Tue, 20 Feb 2024 18:54:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TfT9W024Mz4Tlt for ; Tue, 20 Feb 2024 18:54:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708455295; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vtmdO7ZVh4yGwFdc5TDWobzuyf/6w5Mt1clMjFcUF2Y=; b=Hl2OGSKh6PPH1XCn3v/qB8v/ntxadn9HkiokBvZGkrLP+FvW4m6RZb05FlNGnm51qcpLEh ViLTFD6X5Qo07XPZdKKI1OAIMv1QYA7s81C4JRcL5aBCKeoTL3SjTPYE6/Fs57dcB5C7gx UrNPoCP8vIRs/PUPvqQgMJQq8oZw0QGAGN2Wa8Gprf9n+LHvV5EuPUjM5/f1Tu3h9Zkpq3 2vIg9JKZS1dnl9NsKtV05iipuFRkKArAzM83e2Wgp8UY70cadOiQTHExXeEDemzp9lBX1K 4Wu0kDFfAtlmVA8xWBlVnaBWMrGEKz8dHPl0cIZpksU80fbMVH5xxnG4x5aGQQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708455295; a=rsa-sha256; cv=none; b=yOXALy5ylIS05BOu4vi35jIcm48td5NX7WgDJS4TzWnpGNysIUxMucCRqyGE8O1PZ+yYpn QK1A1waZ7CpgiCWUoN/bOAotIjbfAs5YW0GrRaxaE0hp72C4f3bPwFDRzAGcKk00shPbpT dV/Aq+zir4Nubveasywo4gLi0Fsp4NcZFlIdLGdOPCryACCRLAoRfksynE945oBIWAIlOT aQZf8srwQLtuRJvCil/N8z7LCw3ZgfbPf8rVFDtw92WV3CwtL43sgtCYP58hNuxEkAJkjP Ln0o19dWNXMZJayi3Py950ApRrck/SJq0lZc3cXWijAn/lL8XBDbNOYv3tzugQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TfT9V6DnMzWFd for ; Tue, 20 Feb 2024 18:54:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41KIss0F051067 for ; Tue, 20 Feb 2024 18:54:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41KIsslD051066 for virtualization@FreeBSD.org; Tue, 20 Feb 2024 18:54:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271134] bhyve(8), bhyvectl(8), bhyve(4), bhyve_config(5): manual pages cleanup Date: Tue, 20 Feb 2024 18:54:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Documentation X-Bugzilla-Component: Manual Pages X-Bugzilla-Version: Latest X-Bugzilla-Keywords: bhyve X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: freebsd@ny-central.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271134 --- Comment #8 from Chris Moerz --- Related to this topic, there's now a review for improving the bhyve man page further: https://reviews.freebsd.org/D43940 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Feb 22 16:32:16 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tgdw10Dqvz59wrH for ; Thu, 22 Feb 2024 16:32:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tgdw036ZZz4Lms for ; Thu, 22 Feb 2024 16:32:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708619536; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KfnL5KKCKagpct1ZLJqoXCtUocVt1R3pB0BCE86WYLw=; b=VoeR6yP3H1kMLS1gqT3DCy7cILx516/vYeg0nB01WOnOGk2L5Cu4cloCeb245WI7j43BuD 7vIH5O6V4IkwiWJ16Yjfk8D+KnFZIg9smImKCXS1SPeAU2AQxNeywt2ANwIntXeZVZLodR fiKJ0WihzJNlgBlgxh5m3STkfvVgaNtie5jTaEAGOuqd7V8roo4PINRQbdn6a0/8D1T8Du 82clZ0zHdtX66lTaAslINTRkhlYZGwwi9Jmt1ayn+i1GBAAA9LOOMJQYfpGW6JRj+QYHxu +YUO9PXYsWGltqtuvVRDFgV3FzxhNvEFbY9VITO8UEWzHokuoTwDs0XfuDpNgA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708619536; a=rsa-sha256; cv=none; b=n/SpQ9B2dqmpbj83yLMNZ4N/ig2PQPRCztXSXINlSMAQGyreEorK2A/9T6Rul0vQHAy8aD tmdxALU3iOxjHMYeFHSG51Rya5aqaBoygV0zOw/7pOH0LTtgoH9bDHjkx26VK0c48R68Xb pRNUCO9mu4AQPrfNO5m58hstwFNMa1DOceSmLofPaEmbY4ej0x4191MrB+qwmbX87x0hH3 n+uvaf07OyN/AxGp47iV4v32SrDHJmim1N1I16IVmdSFxBTbvjojxB9WX7b3xACOtgffFS 8p4ZGLl7+6BxSMNJ8YURdVj/JLmFvmy6L6MmNMdZa5f4QfQiQecmvWK3QCmMdw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tgdw02D7Czfds for ; Thu, 22 Feb 2024 16:32:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41MGWGrK096461 for ; Thu, 22 Feb 2024 16:32:16 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41MGWGZk096459 for virtualization@FreeBSD.org; Thu, 22 Feb 2024 16:32:16 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 258755] vmxnet3: low throughput Date: Thu, 22 Feb 2024 16:32:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: michal@maximale.sk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: version Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755 Michal Vanco changed: What |Removed |Added ---------------------------------------------------------------------------- Version|13.0-RELEASE |13.2-RELEASE --- Comment #4 from Michal Vanco --- This issue is still present in 13.2-RELEASE-p10. Is really nobody intereste= d in solving this? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Feb 22 16:32:36 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TgdwN2bTwz59wYc for ; Thu, 22 Feb 2024 16:32:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TgdwM75TYz4MWK for ; Thu, 22 Feb 2024 16:32:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708619556; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aoWQet3ytTFcdTHrRfRNsTiI2g6jNFFUlFyg6MG2yEc=; b=HpK02ElAE3BI9f7Nb1wGeq67EOLHQF/wvF7CszaN1EOWq/+EK63ue0IKFDdmqqhWsKcxJl voQRSCGJ8E7qFJPdNULl0kDT9V/KpWI3GJK9jwHRRRuExWn6Jnc8gx5dOWO1XWpEXSBXg/ Dlxb6Iti/cjRVNcFxf6PFPShM/S62DQl0T089EDyiLkRbTmk+kaeGGlxJRIRloCuQiKqr4 CxF1fyVNJeONAVpyEdZknvxWhAtkNC8s4LJ89SIlXRDK4TWQzzVSv2Ff5RDkAP+wwsc6XN +ZLxVIJFrBQLvwxgw3fKBo+iXevA81T3W/W73ncTFv+sQqjWYUD5ilJ2jtGEhg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708619556; a=rsa-sha256; cv=none; b=ZHpWREZ5+NpJ3DHTc3W9Zdpox7lVjwx6b7Hy3DWzgLKgMJFUjFk6ixEshOVX6IfrXjst5z aaDK1FyGC+jFEfXZsNsINcBLIXBmN16n3+Y01oAQ5/U/9arieGXMyB45A0Q60SuCcXvzTb fywfsPf4owdJXyJcN+YGvcSandRuwGtBiHpwiNikwhevrm+O0wqT4t/nJkULOzOdEUJ1mx WMfRPRM4/maRmHWLIudXQvunyMYLyMDA31nnEuSH7vJq0MxfUDOIxCn3P05BixHCG+ihTu BvZubNLM7s5QDxjZvbukvPKOBzx69oh4vBUJrI0l3DNuyaOPaylteGy8QtTHLA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TgdwM6CzTzfqY for ; Thu, 22 Feb 2024 16:32:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41MGWZHl000149 for ; Thu, 22 Feb 2024 16:32:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41MGWZVT000148 for virtualization@FreeBSD.org; Thu, 22 Feb 2024 16:32:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 258755] vmxnet3: low throughput Date: Thu, 22 Feb 2024 16:32:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: michal@maximale.sk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755 --- Comment #5 from Michal Vanco --- This issue is still present in 13.2-RELEASE-p10. Is really nobody intereste= d in solving this? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 23 02:01:45 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TgtYB5Qt8z5BQHD for ; Fri, 23 Feb 2024 02:01:50 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TgtY92zrGz4VqP for ; Fri, 23 Feb 2024 02:01:49 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm1 header.b=fD+TCUHw; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=OcinoV+o; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.25 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id EC7393200A3B for ; Thu, 22 Feb 2024 21:01:47 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Thu, 22 Feb 2024 21:01:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm1; t=1708653707; x=1708740107; bh=ekQLZgnRg5+/aNItF9l2xJ8eh6eUo74N rR4WvKTVf/A=; b=fD+TCUHwp/fl4UcbgUJfsXiebXc9bYDi/Y6E/R5F+bZO2BPH wIl8kFLQPWJzQWYuHhw6rLAEfgFCWN1J5G6gRYfyz+dtWTbVOdJT+cIA+gEzTb/r iR02zuWu+hhZlAzcssGlIjn1ufz52Qj91RrWScne1H8qdilocU3ODvHVd1onsivB GOs02vQvUEo1Pv/1fSkGc1FVieHSsaDoK454TtstgQachc1hH5bnKjW6a6HBDLkm HSKv8mJbzL2/0DZuI39/0npOu7hJ9LkGZGruZS6873dnDxUPYLDg2cgBdzqkUzwC JbdnKk+GibHd9zgEK3CUgjEQ9/h3MzjWsznz1w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708653707; x=1708740107; bh=ekQLZgnRg5+/aNItF9l2xJ8eh6eUo74NrR4 WvKTVf/A=; b=OcinoV+o/xuzJzYqEegnUPJVbmnLFofhFiSRo1fBXoPuFZi2ohE 6D2L9Wra9749vVDppatwkh4Pc9RbH1Sfcu2oX8KdfYjfZMAOHAjouTE5dBddkZ1L F2j/ls/aDLHe+M6s08fycSpIljZc+0WCalxbt2KP5PRoU3vKyEcfl6bRRsjfD/Jc xkJmLqLL4fPhlJwtUWAOCk4WHjPfGmPesHw/fJgFkCQEgqy+XWoPh8XgdaTPT7hC DRys+o16sEe0IP0w22mn9Gt2hLQQlqFDHcObEmWW63HELMfwiasO+6KfXzA3lxUn VwZVD0fn7UT+3SDm7KyDNzuUbRCP0yxY+DQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrfeehgdegtdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehttdertddttd dvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrghtthgv rhhnpeevudffiedvffffgffhgeefjeefffdtieetheetkeefhfdvfefgtedtueehgeffue enucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhi ugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 22 Feb 2024 21:01:46 -0500 (EST) Date: Fri, 23 Feb 2024 02:01:45 +0000 From: void To: freebsd-virtualization@freebsd.org Subject: problems booting netbsd guest on bhyve host Message-ID: Mail-Followup-To: freebsd-virtualization@freebsd.org List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Rspamd-Queue-Id: 4TgtY92zrGz4VqP X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.60 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.25]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm1,messagingengine.com:s=fm1]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.25:from]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm] Hi, I'm trying to boot most recent netbsd9 into bhyve. Trying to start the install like this # bhyve -AHP -s 0:0,amd_hostbridge -s 1:0,lpc \ -s 2:0,virtio-net,tap6 \ -s 3:0,virtio-blk,./netbsd9.img \ -s 4:0,ahci-cd,/isos/NetBSD/9/NetBSD-9.3-amd64.iso -c 4 -m 8G \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -l com1,/dev/nmdm0A \ netbsd9 then, in another terminal # cu -l /dev/nmdm0B result: >> NetBSD/x86 EFI Boot (x64), Revision 1.1 (Thu Aug 4 15:30:37 UTC 2022) (from NetBSD 9.3) >> Memory: 640/3137664 k 1. Boot normally 2. Boot single user 3. Drop to boot prompt Choose an option; RETURN for default; SPACE to stop countdown. Option 1 will be chosen in 0 seconds. command(s): rndseed /var/db/entropy-file;boot default boot twice, skipping... 18476680+668736+1428416 [869943+1252488+894273]=0x16e11b0 WARNING: couldn't open /var/db/entropy-file Loading /stand/amd64/9.3/modules/cd9660/cd9660.kmod WARNING: 1 module failed to load On a linux box I can create a netbsd vm in virtualbox. Why not in bhyve? thanks in advance for any help/assistance -- From nobody Fri Feb 23 05:51:06 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tgzdv6qNVz5BnVr for ; Fri, 23 Feb 2024 05:51:15 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tgzdt6ZDXz4vdN for ; Fri, 23 Feb 2024 05:51:14 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=ImiGNypP; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41N5p7fS020114 for ; Thu, 22 Feb 2024 23:51:07 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708667467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EfqeRVd2mX6zGTPKlW7xx50AEfGgEIjl1PJeAxD7aSg=; b=ImiGNypP6xRh0nGuVh4S4ypI7LBa52RV5eG3ZaezIiBRok799ucIxhifcPhJABTT+jz6TK it61+v4fRz9ijI+2sW6getkLSK/eyU1tror7PHGsZNjwyidNnYynk11XN8iNzkTWWyKdK5 WvICI2KFb9ShE5rsySHLQTJSDKu7jVCo77xrBaGq/ItDOlW7OrakpGrNlqbND0ndwVNezR xWy3L4qZI7LGIpHRZ5jHWDXzWkSQ3gmIfqvMfKG6qxUrjSXSVYchPEC01OfFnVyhacnqhY jhYwOKDXlFti6gXPx5FUwfM0TfV6pTZzcQsSRzDN+Pe3x83LbpzBtytL6zU1YA== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 663BB3B597 for ; Thu, 22 Feb 2024 23:51:07 -0600 (CST) Message-ID: <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> Date: Thu, 22 Feb 2024 23:51:06 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> From: Matthew Grooms In-Reply-To: <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Tgzdt6ZDXz4vdN X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] On 2/18/24 09:21, Matthew Grooms wrote: > On 2/17/24 15:53, Matthew Grooms wrote: >> >> Unfortunately same story with 13.2. I'm going to try and downgrade to >> 12.4 and see if it gets any better. >> >> ================================================================================ >> Begin @ Sat Feb 17 11:00:01 CST 2024 >> >> Version  2.00       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >> localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99 1.3g  68 >> +++++ +++ >> Latency             11759us   29114us    8098us    8649us 23413us    >> 4540us >> Version  2.00       ------Sequential Create------ --------Random >> Create-------- >> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ >> +++++ +++ >> Latency              7791us     131us    1671us     464us 15us    1811us >> >> End @ Sat Feb 17 11:03:13 CST 2024 >> ================================================================================ >> Begin @ Sat Feb 17 11:10:01 CST 2024 >> >> Version  2.00       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >> localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99 398m  99 >> 16204 563 >> Latency             12147us    1079us   24470us    8795us 17853us    >> 4384us >> Version  2.00       ------Sequential Create------ --------Random >> Create-------- >> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >>                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ >> +++++ +++ >> Latency               118us     159us     269us     164us 54us     844us >> >> End @ Sat Feb 17 11:18:43 CST 2024 >> > I wasn't able to get a working 12.4 system in place due to lack of > packages. However, I did fire up a FreeBSD 14 VM and let it run > overnight on the same SSD array. It consistently ran at a much higher > speed for 50+ runs @ 10m intervals ... > > ================================================================================ > Begin @ Sun Feb 18 15:00:00 UTC 2024 > > Version  1.98       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP > freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99 1.1g  42 > +++++ +++ > Latency             13447us   68490us     207ms    7187us 195ms   17665us > Version  1.98       ------Sequential Create------ --------Random > Create-------- > freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP >                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ > +++++ +++ > Latency             18225us      18us      28us   18812us 18us      25us > > End @ Sun Feb 18 15:03:11 UTC 2024 > > I used identical options to run both that VM and the Linux VM I've > been testing. The backing store for each VM has a 1TB partition and > the guest interfaces are NVME. Now I'm really scratching my head. > > Chuck, were you testing disk performance in Linux VMs or only FreeBSD? > > Anyone have ideas on why Linux disk performance would drop off a cliff > over time? > I took a detour trying out Xen but apparently that has it's own set of performance issues related to the FreeBSD port missing newer features. I did install KVM on the same hardware for comparison. I then tested a guest provisioned identically to the bhyve VM with a virtio blk device which ran for 2.5 hours with consistently impressive output ... ================================================================================ Begin @ Thu Feb 22 20:55:01 CST 2024 Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP linux2.shrew.pr 31G 2191k  99  1.4g  55  1.1g  59 3484k  99  2.5g 83  7817 127 Latency              4480us    2528us   17656us    2650us 102us    3568us Version  2.00       ------Sequential Create------ --------Random Create-------- linux2.shrew.prv    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency              9722us      90us     123us      61us 20us     998us End @ Thu Feb 22 20:56:11 CST 2024 ================================================================================ For comparison, here is the output from a recent run of a Linux VM on bhyve using the virtio blk device. Note the performance drop off between the first and second run ... ================================================================================ Begin @ Thu Feb 22 22:00:02 CST 2024 Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP linux.shrew. 63640M  694k  99  1.5g  97  772m  70  985k  99  1.4g 75  2302 115 Latency             11557us   28959us   27540us    8308us 25227us    8605us Version  2.00       ------Sequential Create------ --------Random Create-------- linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency              4058us     125us    1651us      96us 23us     627us End @ Thu Feb 22 22:03:07 CST 2024 ================================================================================ Begin @ Thu Feb 22 22:10:02 CST 2024 Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP linux.shrew. 63640M  676k  99  406m  99  300m  92  952k  99  373m 99  2145 158 Latency             11871us     154us   20673us    9926us 22765us   14034us Version  2.00       ------Sequential Create------ --------Random Create-------- linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16     0  95 +++++ +++     0  79     0  96 +++++ +++     0  76 Latency              4426us     125us    1687us     576us 72us     654us End @ Thu Feb 22 22:19:19 CST 2024 ================================================================================ It certainly feels like a deficiency in bhyve that isn't tied to any particular storage device model. I did notice a pattern in top that I thought was peculiar. When watching the bhyve threads while the benchmark test is running, I see several CPU threads running at the top of the list followed by what I assume to be block device threads. When the disk performance is high, it looks like this ...   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND  5628 root         68    0    32G    31G CPU24   24   0:53  88.90% bhyve{vcpu 5}  5628 root         36    0    32G    31G vmidle   7   0:18  17.86% bhyve{vcpu 15}  5628 root         34    0    32G    31G vmidle  26   1:06  16.76% bhyve{vcpu 8}  5628 root         21    0    32G    31G uwait   37   0:05   2.69% bhyve{blk-4:0-2}  5628 root         21    0    32G    31G uwait   60   0:04   2.64% bhyve{blk-4:0-0}  5628 root         21    0    32G    31G uwait   52   0:06   2.62% bhyve{blk-4:0-1}  5628 root         21    0    32G    31G uwait   14   0:05   2.58% bhyve{blk-4:0-6}  5628 root         21    0    32G    31G RUN     50   0:05   2.51% bhyve{blk-4:0-4}  5628 root         20    0    32G    31G uwait   38   0:05   2.51% bhyve{blk-4:0-5}  5628 root         21    0    32G    31G uwait   56   0:05   2.46% bhyve{blk-4:0-3}  5628 root         20    0    32G    31G uwait   22   0:06   2.45% bhyve{blk-4:0-7} When disk performance drops off, I see that one of the bhyve CPU threads shows it's PRI value climb quickly until it hits around 135. At that point, the one CPU thread basically is pegged at 100% WCPU until the test is done. Other bhyve threads are much less busy ...   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND  5518 root        135    0    32G    31G CPU59   59   6:49  99.97% bhyve{vcpu 2}  5518 root         26    0    32G    31G vmidle  36   5:40   8.80% bhyve{vcpu 15}  5518 root         23    0    32G    31G vmidle  18   0:41   4.74% bhyve{vcpu 13}  5518 root         20    0    32G    31G vmidle  37   0:09   0.85% bhyve{vcpu 10}  5518 root         20    0    32G    31G uwait    6   0:20   0.72% bhyve{blk-4:0-2}  5518 root         20    0    32G    31G uwait    8   0:20   0.71% bhyve{blk-4:0-7}  5518 root         20    0    32G    31G uwait   43   0:22   0.70% bhyve{blk-4:0-0}  5518 root         20    0    32G    31G uwait   12   0:20   0.70% bhyve{blk-4:0-5}  5518 root         20    0    32G    31G uwait   43   0:19   0.68% bhyve{blk-4:0-3}  5518 root         20    0    32G    31G uwait   36   0:21   0.68% bhyve{blk-4:0-6}  5518 root         20    0    32G    31G uwait   46   0:20   0.68% bhyve{blk-4:0-4}  5518 root         20    0    32G    31G uwait   32   0:20   0.64% bhyve{blk-4:0-1} This pattern in top repeats each time the benchmark is run unless the guest is rebooted. Does bhyve call pthread_attr_setschedparam at run time under certain circumstances or is that the scheduler doing it's thing? Anyone have any ideas? I'm pretty much out of them :/ Thanks in advance, -Matthew From nobody Fri Feb 23 06:25:39 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Th0Pr0SsZz5Br3q for ; Fri, 23 Feb 2024 06:25:52 +0000 (UTC) (envelope-from jason@tubnor.net) Received: from mail.tubnor.net (mail.tubnor.net [103.236.162.16]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Th0Pq1BJPz40L0 for ; Fri, 23 Feb 2024 06:25:50 +0000 (UTC) (envelope-from jason@tubnor.net) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tubnor.net; s=20220915; t=1708669539; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=64FQt5HOOw01uI5h5b76gxU+o/bhrMQPKX2/r48wb8M=; b=ncgOJ0iTQbTtBOj1rPpj/z0Fgo4kQwzfDWIuJZuZFmlgJZjvT9m+L+cIXePIVN7pHq9dD3 h6i4cVIxMUv65b47wDhe0XtyQ3pcmCDiIjEAjuy4Zz7BPu79mLiNFBclmoCliQ6q0Q9Czx OFZfjzmjaQV2NRbJ9n2hcbBg9yyRqVo= Received: from [IPV6:2403:5812:73e6:1:6023:4380:cf39:62a9] ( [2403:5812:73e6:1:6023:4380:cf39:62a9]) by mel01.ar18.net (OpenSMTPD) with ESMTPSA id a4abba43 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 23 Feb 2024 17:25:38 +1100 (AEDT) Message-ID: Date: Fri, 23 Feb 2024 17:25:39 +1100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: Matthew Grooms , virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> From: Jason Tubnor In-Reply-To: <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Th0Pq1BJPz40L0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:133159, ipnet:103.236.162.0/23, country:AU] On 23/02/2024 4:51 pm, Matthew Grooms wrote: > On 2/18/24 09:21, Matthew Grooms wrote: >> On 2/17/24 15:53, Matthew Grooms wrote: >>> >>> Unfortunately same story with 13.2. I'm going to try and downgrade >>> to 12.4 and see if it gets any better. >>> >>> ================================================================================ >>> >>> Begin @ Sat Feb 17 11:00:01 CST 2024 >>> >>> Version  2.00       ------Sequential Output------ --Sequential >>> Input- --Random- >>>                     -Per Chr- --Block-- -Rewrite- -Per Chr- >>> --Block-- --Seeks-- >>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>> %CP  /sec %CP >>> localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99 1.3g  68 >>> +++++ +++ >>> Latency             11759us   29114us    8098us    8649us 23413us    >>> 4540us >>> Version  2.00       ------Sequential Create------ --------Random >>> Create-------- >>> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- >>> --Read--- -Delete-- >>>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>> %CP  /sec %CP >>>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ >>> +++ +++++ +++ >>> Latency              7791us     131us    1671us     464us 15us    >>> 1811us >>> >>> End @ Sat Feb 17 11:03:13 CST 2024 >>> ================================================================================ >>> >>> Begin @ Sat Feb 17 11:10:01 CST 2024 >>> >>> Version  2.00       ------Sequential Output------ --Sequential >>> Input- --Random- >>>                     -Per Chr- --Block-- -Rewrite- -Per Chr- >>> --Block-- --Seeks-- >>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>> %CP  /sec %CP >>> localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99 398m  99 >>> 16204 563 >>> Latency             12147us    1079us   24470us    8795us 17853us    >>> 4384us >>> Version  2.00       ------Sequential Create------ --------Random >>> Create-------- >>> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- >>> --Read--- -Delete-- >>>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>> %CP  /sec %CP >>>                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ >>> +++ +++++ +++ >>> Latency               118us     159us     269us     164us 54us     >>> 844us >>> >>> End @ Sat Feb 17 11:18:43 CST 2024 >>> >> I wasn't able to get a working 12.4 system in place due to lack of >> packages. However, I did fire up a FreeBSD 14 VM and let it run >> overnight on the same SSD array. It consistently ran at a much higher >> speed for 50+ runs @ 10m intervals ... >> >> ================================================================================ >> >> Begin @ Sun Feb 18 15:00:00 UTC 2024 >> >> Version  1.98       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >> freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99 1.1g  42 >> +++++ +++ >> Latency             13447us   68490us     207ms    7187us 195ms   >> 17665us >> Version  1.98       ------Sequential Create------ --------Random >> Create-------- >> freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ >> +++++ +++ >> Latency             18225us      18us      28us   18812us 18us      25us >> >> End @ Sun Feb 18 15:03:11 UTC 2024 >> >> I used identical options to run both that VM and the Linux VM I've >> been testing. The backing store for each VM has a 1TB partition and >> the guest interfaces are NVME. Now I'm really scratching my head. >> >> Chuck, were you testing disk performance in Linux VMs or only FreeBSD? >> >> Anyone have ideas on why Linux disk performance would drop off a >> cliff over time? >> > I took a detour trying out Xen but apparently that has it's own set of > performance issues related to the FreeBSD port missing newer features. > I did install KVM on the same hardware for comparison. I then tested a > guest provisioned identically to the bhyve VM with a virtio blk device > which ran for 2.5 hours with consistently impressive output ... > > ================================================================================ > > Begin @ Thu Feb 22 20:55:01 CST 2024 > > Version  2.00       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  > /sec %CP > linux2.shrew.pr 31G 2191k  99  1.4g  55  1.1g  59 3484k  99  2.5g 83  > 7817 127 > Latency              4480us    2528us   17656us    2650us 102us 3568us > Version  2.00       ------Sequential Create------ --------Random > Create-------- > linux2.shrew.prv    -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  > /sec %CP >                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ > +++++ +++ > Latency              9722us      90us     123us      61us 20us 998us > > End @ Thu Feb 22 20:56:11 CST 2024 > ================================================================================ > > > > For comparison, here is the output from a recent run of a Linux VM on > bhyve using the virtio blk device. Note the performance drop off > between the first and second run ... > > > ================================================================================ > > Begin @ Thu Feb 22 22:00:02 CST 2024 > > Version  2.00       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  > /sec %CP > linux.shrew. 63640M  694k  99  1.5g  97  772m  70  985k  99  1.4g 75  > 2302 115 > Latency             11557us   28959us   27540us    8308us 25227us    > 8605us > Version  2.00       ------Sequential Create------ --------Random > Create-------- > linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  > /sec %CP >                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ > +++++ +++ > Latency              4058us     125us    1651us      96us 23us 627us > > End @ Thu Feb 22 22:03:07 CST 2024 > ================================================================================ > > Begin @ Thu Feb 22 22:10:02 CST 2024 > > Version  2.00       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  > /sec %CP > linux.shrew. 63640M  676k  99  406m  99  300m  92  952k  99  373m 99  > 2145 158 > Latency             11871us     154us   20673us    9926us 22765us   > 14034us > Version  2.00       ------Sequential Create------ --------Random > Create-------- > linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  > /sec %CP >                  16     0  95 +++++ +++     0  79     0  96 +++++ > +++     0  76 > Latency              4426us     125us    1687us     576us 72us 654us > > End @ Thu Feb 22 22:19:19 CST 2024 > ================================================================================ > > > It certainly feels like a deficiency in bhyve that isn't tied to any > particular storage device model. I did notice a pattern in top that I > thought was peculiar. When watching the bhyve threads while the > benchmark test is running, I see several CPU threads running at the > top of the list followed by what I assume to be block device threads. > When the disk performance is high, it looks like this ... > >   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU > COMMAND >  5628 root         68    0    32G    31G CPU24   24   0:53  88.90% > bhyve{vcpu 5} >  5628 root         36    0    32G    31G vmidle   7   0:18  17.86% > bhyve{vcpu 15} >  5628 root         34    0    32G    31G vmidle  26   1:06  16.76% > bhyve{vcpu 8} >  5628 root         21    0    32G    31G uwait   37   0:05   2.69% > bhyve{blk-4:0-2} >  5628 root         21    0    32G    31G uwait   60   0:04   2.64% > bhyve{blk-4:0-0} >  5628 root         21    0    32G    31G uwait   52   0:06   2.62% > bhyve{blk-4:0-1} >  5628 root         21    0    32G    31G uwait   14   0:05   2.58% > bhyve{blk-4:0-6} >  5628 root         21    0    32G    31G RUN     50   0:05   2.51% > bhyve{blk-4:0-4} >  5628 root         20    0    32G    31G uwait   38   0:05   2.51% > bhyve{blk-4:0-5} >  5628 root         21    0    32G    31G uwait   56   0:05   2.46% > bhyve{blk-4:0-3} >  5628 root         20    0    32G    31G uwait   22   0:06   2.45% > bhyve{blk-4:0-7} > > When disk performance drops off, I see that one of the bhyve CPU > threads shows it's PRI value climb quickly until it hits around 135. > At that point, the one CPU thread basically is pegged at 100% WCPU > until the test is done. Other bhyve threads are much less busy ... > >   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU > COMMAND >  5518 root        135    0    32G    31G CPU59   59   6:49  99.97% > bhyve{vcpu 2} >  5518 root         26    0    32G    31G vmidle  36   5:40   8.80% > bhyve{vcpu 15} >  5518 root         23    0    32G    31G vmidle  18   0:41   4.74% > bhyve{vcpu 13} >  5518 root         20    0    32G    31G vmidle  37   0:09   0.85% > bhyve{vcpu 10} >  5518 root         20    0    32G    31G uwait    6   0:20   0.72% > bhyve{blk-4:0-2} >  5518 root         20    0    32G    31G uwait    8   0:20   0.71% > bhyve{blk-4:0-7} >  5518 root         20    0    32G    31G uwait   43   0:22   0.70% > bhyve{blk-4:0-0} >  5518 root         20    0    32G    31G uwait   12   0:20   0.70% > bhyve{blk-4:0-5} >  5518 root         20    0    32G    31G uwait   43   0:19   0.68% > bhyve{blk-4:0-3} >  5518 root         20    0    32G    31G uwait   36   0:21   0.68% > bhyve{blk-4:0-6} >  5518 root         20    0    32G    31G uwait   46   0:20   0.68% > bhyve{blk-4:0-4} >  5518 root         20    0    32G    31G uwait   32   0:20   0.64% > bhyve{blk-4:0-1} > > This pattern in top repeats each time the benchmark is run unless the > guest is rebooted. > > Does bhyve call pthread_attr_setschedparam at run time under certain > circumstances or is that the scheduler doing it's thing? Anyone have > any ideas? I'm pretty much out of them :/ > > Thanks in advance, > > -Matthew > > Hi Matthew, Have you tried using UFS2 or even memory backed host storage for testing yet? This will rule out if it is a ZFS or bhyve issue. Cheers, Jason. From nobody Fri Feb 23 06:46:26 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Th0sf2Ghmz5Bslr for ; Fri, 23 Feb 2024 06:46:30 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Th0sf0GxFz41LL for ; Fri, 23 Feb 2024 06:46:29 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41N6kROZ020609; Fri, 23 Feb 2024 00:46:27 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708670787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tWOvb0J9O/6WfjGiuRgJUUlPhclrxOoGGv+01n4vulU=; b=aOL4hlcgqgLHaNsKGFnhJBUVVVg1SmsJn9HXR/bGziCR204O8JPHQvDVv6zxyQs4Bd0H9E PmK76pT2GzAUKIbm18Rza/2ifl74C04BuW4xMb3GXl4qLidLVpe8U3TTxBVq3CZ8jsphRb A69qK7UMywyD5T7WeBqiF+kt+627lX78ikysyWe8UKO/fhIJbx/2u5gLxTUA6W6yx/ePNV Lv+eBq5xjIQnjOHoAI46cbYy7Q/VNx0NG9zRkduwM7j2re1U4NnA80H4XtsSZzt9frVjKV R8oEfCjrYo8Vq47pM48FnnMfj+1r5jb0u1BlhigiQP4hjXWLbZKwRRdNuEUniw== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id EE2D63B58D; Fri, 23 Feb 2024 00:46:26 -0600 (CST) Message-ID: Date: Fri, 23 Feb 2024 00:46:26 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: Jason Tubnor , virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> From: Matthew Grooms In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Th0sf0GxFz41LL X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US] On 2/23/24 00:25, Jason Tubnor wrote: > > On 23/02/2024 4:51 pm, Matthew Grooms wrote: >> On 2/18/24 09:21, Matthew Grooms wrote: >>> On 2/17/24 15:53, Matthew Grooms wrote: >>>> >>>> Unfortunately same story with 13.2. I'm going to try and downgrade >>>> to 12.4 and see if it gets any better. >>>> >>>> ================================================================================ >>>> >>>> Begin @ Sat Feb 17 11:00:01 CST 2024 >>>> >>>> Version  2.00       ------Sequential Output------ --Sequential >>>> Input- --Random- >>>>                     -Per Chr- --Block-- -Rewrite- -Per Chr- >>>> --Block-- --Seeks-- >>>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>>> %CP  /sec %CP >>>> localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99 1.3g  >>>> 68 +++++ +++ >>>> Latency             11759us   29114us    8098us    8649us >>>> 23413us    4540us >>>> Version  2.00       ------Sequential Create------ --------Random >>>> Create-------- >>>> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- >>>> --Read--- -Delete-- >>>>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>>> %CP  /sec %CP >>>>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ >>>> +++ +++++ +++ >>>> Latency              7791us     131us    1671us     464us 15us    >>>> 1811us >>>> >>>> End @ Sat Feb 17 11:03:13 CST 2024 >>>> ================================================================================ >>>> >>>> Begin @ Sat Feb 17 11:10:01 CST 2024 >>>> >>>> Version  2.00       ------Sequential Output------ --Sequential >>>> Input- --Random- >>>>                     -Per Chr- --Block-- -Rewrite- -Per Chr- >>>> --Block-- --Seeks-- >>>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>>> %CP  /sec %CP >>>> localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99 398m  >>>> 99 16204 563 >>>> Latency             12147us    1079us   24470us    8795us >>>> 17853us    4384us >>>> Version  2.00       ------Sequential Create------ --------Random >>>> Create-------- >>>> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- >>>> --Read--- -Delete-- >>>>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>>> %CP  /sec %CP >>>>                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ >>>> +++ +++++ +++ >>>> Latency               118us     159us     269us     164us 54us     >>>> 844us >>>> >>>> End @ Sat Feb 17 11:18:43 CST 2024 >>>> >>> I wasn't able to get a working 12.4 system in place due to lack of >>> packages. However, I did fire up a FreeBSD 14 VM and let it run >>> overnight on the same SSD array. It consistently ran at a much >>> higher speed for 50+ runs @ 10m intervals ... >>> >>> ================================================================================ >>> >>> Begin @ Sun Feb 18 15:00:00 UTC 2024 >>> >>> Version  1.98       ------Sequential Output------ --Sequential >>> Input- --Random- >>>                     -Per Chr- --Block-- -Rewrite- -Per Chr- >>> --Block-- --Seeks-- >>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>> %CP  /sec %CP >>> freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99 1.1g  42 >>> +++++ +++ >>> Latency             13447us   68490us     207ms    7187us 195ms   >>> 17665us >>> Version  1.98       ------Sequential Create------ --------Random >>> Create-------- >>> freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- >>> --Read--- -Delete-- >>>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec >>> %CP  /sec %CP >>>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ >>> +++ +++++ +++ >>> Latency             18225us      18us      28us   18812us 18us      >>> 25us >>> >>> End @ Sun Feb 18 15:03:11 UTC 2024 >>> >>> I used identical options to run both that VM and the Linux VM I've >>> been testing. The backing store for each VM has a 1TB partition and >>> the guest interfaces are NVME. Now I'm really scratching my head. >>> >>> Chuck, were you testing disk performance in Linux VMs or only FreeBSD? >>> >>> Anyone have ideas on why Linux disk performance would drop off a >>> cliff over time? >>> >> I took a detour trying out Xen but apparently that has it's own set >> of performance issues related to the FreeBSD port missing newer >> features. I did install KVM on the same hardware for comparison. I >> then tested a guest provisioned identically to the bhyve VM with a >> virtio blk device which ran for 2.5 hours with consistently >> impressive output ... >> >> ================================================================================ >> >> Begin @ Thu Feb 22 20:55:01 CST 2024 >> >> Version  2.00       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >> linux2.shrew.pr 31G 2191k  99  1.4g  55  1.1g  59 3484k  99 2.5g 83  >> 7817 127 >> Latency              4480us    2528us   17656us    2650us 102us 3568us >> Version  2.00       ------Sequential Create------ --------Random >> Create-------- >> linux2.shrew.prv    -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ >> +++++ +++ >> Latency              9722us      90us     123us      61us 20us 998us >> >> End @ Thu Feb 22 20:56:11 CST 2024 >> ================================================================================ >> >> >> >> For comparison, here is the output from a recent run of a Linux VM on >> bhyve using the virtio blk device. Note the performance drop off >> between the first and second run ... >> >> >> ================================================================================ >> >> Begin @ Thu Feb 22 22:00:02 CST 2024 >> >> Version  2.00       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >> linux.shrew. 63640M  694k  99  1.5g  97  772m  70  985k  99 1.4g 75  >> 2302 115 >> Latency             11557us   28959us   27540us    8308us 25227us    >> 8605us >> Version  2.00       ------Sequential Create------ --------Random >> Create-------- >> linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ >> +++++ +++ >> Latency              4058us     125us    1651us      96us 23us 627us >> >> End @ Thu Feb 22 22:03:07 CST 2024 >> ================================================================================ >> >> Begin @ Thu Feb 22 22:10:02 CST 2024 >> >> Version  2.00       ------Sequential Output------ --Sequential Input- >> --Random- >>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >> linux.shrew. 63640M  676k  99  406m  99  300m  92  952k  99 373m 99  >> 2145 158 >> Latency             11871us     154us   20673us    9926us 22765us   >> 14034us >> Version  2.00       ------Sequential Create------ --------Random >> Create-------- >> linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  >> /sec %CP >>                  16     0  95 +++++ +++     0  79     0  96 +++++ >> +++     0  76 >> Latency              4426us     125us    1687us     576us 72us 654us >> >> End @ Thu Feb 22 22:19:19 CST 2024 >> ================================================================================ >> >> >> It certainly feels like a deficiency in bhyve that isn't tied to any >> particular storage device model. I did notice a pattern in top that I >> thought was peculiar. When watching the bhyve threads while the >> benchmark test is running, I see several CPU threads running at the >> top of the list followed by what I assume to be block device threads. >> When the disk performance is high, it looks like this ... >> >>   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME WCPU COMMAND >>  5628 root         68    0    32G    31G CPU24   24   0:53 88.90% >> bhyve{vcpu 5} >>  5628 root         36    0    32G    31G vmidle   7   0:18 17.86% >> bhyve{vcpu 15} >>  5628 root         34    0    32G    31G vmidle  26   1:06 16.76% >> bhyve{vcpu 8} >>  5628 root         21    0    32G    31G uwait   37   0:05 2.69% >> bhyve{blk-4:0-2} >>  5628 root         21    0    32G    31G uwait   60   0:04 2.64% >> bhyve{blk-4:0-0} >>  5628 root         21    0    32G    31G uwait   52   0:06 2.62% >> bhyve{blk-4:0-1} >>  5628 root         21    0    32G    31G uwait   14   0:05 2.58% >> bhyve{blk-4:0-6} >>  5628 root         21    0    32G    31G RUN     50   0:05 2.51% >> bhyve{blk-4:0-4} >>  5628 root         20    0    32G    31G uwait   38   0:05 2.51% >> bhyve{blk-4:0-5} >>  5628 root         21    0    32G    31G uwait   56   0:05 2.46% >> bhyve{blk-4:0-3} >>  5628 root         20    0    32G    31G uwait   22   0:06 2.45% >> bhyve{blk-4:0-7} >> >> When disk performance drops off, I see that one of the bhyve CPU >> threads shows it's PRI value climb quickly until it hits around 135. >> At that point, the one CPU thread basically is pegged at 100% WCPU >> until the test is done. Other bhyve threads are much less busy ... >> >>   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME WCPU COMMAND >>  5518 root        135    0    32G    31G CPU59   59   6:49 99.97% >> bhyve{vcpu 2} >>  5518 root         26    0    32G    31G vmidle  36   5:40 8.80% >> bhyve{vcpu 15} >>  5518 root         23    0    32G    31G vmidle  18   0:41 4.74% >> bhyve{vcpu 13} >>  5518 root         20    0    32G    31G vmidle  37   0:09 0.85% >> bhyve{vcpu 10} >>  5518 root         20    0    32G    31G uwait    6   0:20 0.72% >> bhyve{blk-4:0-2} >>  5518 root         20    0    32G    31G uwait    8   0:20 0.71% >> bhyve{blk-4:0-7} >>  5518 root         20    0    32G    31G uwait   43   0:22 0.70% >> bhyve{blk-4:0-0} >>  5518 root         20    0    32G    31G uwait   12   0:20 0.70% >> bhyve{blk-4:0-5} >>  5518 root         20    0    32G    31G uwait   43   0:19 0.68% >> bhyve{blk-4:0-3} >>  5518 root         20    0    32G    31G uwait   36   0:21 0.68% >> bhyve{blk-4:0-6} >>  5518 root         20    0    32G    31G uwait   46   0:20 0.68% >> bhyve{blk-4:0-4} >>  5518 root         20    0    32G    31G uwait   32   0:20 0.64% >> bhyve{blk-4:0-1} >> >> This pattern in top repeats each time the benchmark is run unless the >> guest is rebooted. >> >> Does bhyve call pthread_attr_setschedparam at run time under certain >> circumstances or is that the scheduler doing it's thing? Anyone have >> any ideas? I'm pretty much out of them :/ >> >> Thanks in advance, >> >> -Matthew >> >> > Hi Matthew, > > Have you tried using UFS2 or even memory backed host storage for > testing yet? This will rule out if it is a ZFS or bhyve issue. > Hi Jason, I appreciate the suggestion. But yes, I've tested with block devices, ZVOLs, ZFS and UFS image files on the host. I've also tried using both nvme and virtio in the guest. The issue isn't specific to any combination of these. Thanks, -Matthew From nobody Fri Feb 23 09:38:37 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Th4gS35PZz5C7Mm for ; Fri, 23 Feb 2024 09:37:56 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Received: from mo4-p05-ob.smtp.rzone.de (mo4-p05-ob.smtp.rzone.de [85.215.255.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.smtp.rzone.de", Issuer "Telekom Security ServerID OV Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Th4gR31xsz4Lqq for ; Fri, 23 Feb 2024 09:37:54 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0002 header.b=iBKEBR0G; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0003 header.b=r0LN00gu; dmarc=none; spf=none (mx1.freebsd.org: domain of Stephan.Althaus@Duedinghausen.eu has no SPF policy when checking 85.215.255.136) smtp.mailfrom=Stephan.Althaus@Duedinghausen.eu; arc=pass ("strato.com:s=strato-dkim-0002:i=1") ARC-Seal: i=1; a=rsa-sha256; t=1708681072; cv=none; d=strato.com; s=strato-dkim-0002; b=oA8Cbnke7ZrrRLCHDWsmXyDtxddXQTUk9c507RbTfktCvHWg2SgpR9PDROa1TAseuz z+NmyvMq7iL9d5ilS8DboDjOZbRa/oHBpKiiVrVok2kG/B/pGF88bsgxjbPigatr8VkT Ky31MTa3B8exeY8B4MAT2QxM8OfYmtnqOg+oK+sT7QvXCU5Yattu+UXkXJ+ObQDeuFRx 4G4V+CbTheGfuQtAc0/Z/l2MW3yJDHnAEiRbi2/eIvnQf7aDOffGW8+m9845ksI4Ps7q xnifQhSgvDwG4nNx4h3sYZMxyunfv3KY6CLNFHzDHDoI4oWlWZeN7wjFxghFUV+1/chC GF4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1708681072; s=strato-dkim-0002; d=strato.com; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=2di4rsHERC5bDmosof3JbotkC0gVxotW3RHdNhPodBQ=; b=FqtR7KzQX1M31fgw+efeDnnEVyl3DflI191KEIvNZWXDG/8Gx/VLrDi8kFH+KbuBVU dNtNx+vg2sb6jiOANfpA3yKHIt+o0wpHcmen8OEIVtZd8Vm5ze+Cdvnc+3ETdNjstjhk 3MFfc57kiJzWw6hjilQmGbatLjZv0UdApIcK+AG41OiSqfWuS1lroTApqfjoIIl97ZhK NZDqeH6YQfxs3MqRNOuc0oQ5PkFGBQxNIDeBmbwfTsxK/bFTG3q4R465cGW8ajUsqZrC WrusaFZbD5F/NvheJ/auNwYl+FHRAGSLTELKBUYW4LrA5Tcplgk6R/hDIQIHP4A4Rw4A 8W9Q== ARC-Authentication-Results: i=1; strato.com; arc=none; dkim=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1708681072; s=strato-dkim-0002; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=2di4rsHERC5bDmosof3JbotkC0gVxotW3RHdNhPodBQ=; b=iBKEBR0GKTFoPzwW4YT651DmTBenZlu+4ATymqJ7jeR07Tbz3eVIm/4WepDWgxudEb TNf18aHMIDV5pmxgJtDbuWsJDitrZ12k9WuvBxUc4u2Lb11CFeQx7/7a+3cYwiPyLDez aaH+W1L9JEN8Y601oCKf2pqd/vvHgHnJPdxIqE0WUZC2ljlYxnB9XhpEvnMZmlnEbCtv 1+Dig2Tyrs3HcDRpocz3m+u9iCqC8Lshh+D3JIE9MtCogruRGowEydANa4SQ1Xt+DS+j 0Ez6yvuoTcR5Lo5C+05GK/uzEY7LqJliOEhtSmrZsACpcUoVO3QuZ2oHnzjqzP34F56K p9iA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1708681072; s=strato-dkim-0003; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=2di4rsHERC5bDmosof3JbotkC0gVxotW3RHdNhPodBQ=; b=r0LN00guLWK8fschkinBgIhSZh7c426kfh+AQN6JrvGr2riOmaj2mnH8ti5VZ1MRYj 45zlmXQPWf0fST083QDQ== X-RZG-AUTH: ":O2kGeEG7b/pS1EW2TmikjLDsYYueHLp2aWg0q38nsxN1mOntnRORP93PLpfReKNjbyYtOwejfjvzauA=" X-RZG-CLASS-ID: mo05 Received: from www.duedinghausen.eu by smtp.strato.de (RZmta 50.0.3 DYNA|AUTH) with ESMTPSA id jfcd7001N9bq3S9 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate) for ; Fri, 23 Feb 2024 10:37:52 +0100 (CET) Received: from [192.168.2.63] (speedport.ip [192.168.1.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) (Authenticated sender: steven) by www.duedinghausen.eu (Postfix) with ESMTPSA id 12795118B88 for ; Fri, 23 Feb 2024 10:36:07 +0100 (CET) Message-ID: Date: Fri, 23 Feb 2024 10:38:37 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> Content-Language: en-US From: Stephan Althaus In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.29 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; ARC_ALLOW(-1.00)[strato.com:s=strato-dkim-0002:i=1]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[hoewweken.de:s=strato-dkim-0002,hoewweken.de:s=strato-dkim-0003]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6724, ipnet:85.215.255.0/24, country:DE]; DMARC_NA(0.00)[duedinghausen.eu]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[hoewweken.de:+] X-Rspamd-Queue-Id: 4Th4gR31xsz4Lqq On 2/23/24 07:46, Matthew Grooms wrote: > UFS image files Hello! Please don't mind asking stupid.. Was the UFS image file placed on a ZFS file system on the host ? You should place an image file on a different disk/filesystem on the host to rule out ZFS issues on the host side.. Regards, Stephan From nobody Fri Feb 23 15:37:47 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThDfl2jspz59xjm for ; Fri, 23 Feb 2024 15:37:51 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThDfk1Dp9z4MXx for ; Fri, 23 Feb 2024 15:37:50 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=kGvJeXVX; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41NFbmEX025729 for ; Fri, 23 Feb 2024 09:37:48 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708702668; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GUyXUFkN8MVQUXLYTqp7/XtYiWkxlMEr6l49bTDM1tA=; b=kGvJeXVXt+eXYMiWXLug/P83hGgr/wwEDlwWnW1mzgnKhB8XOnuEPXa9WPQrLr/w35YsdN qJTO8XPUgOJWJeXWHyxgoA31EZ/1QF2ooddhRmIo9oWUZuZ46V+j+njgq5odU+QeEne9DS rHrtKuKCH7mxb9EDauctMGmRlw7JmQjkd4HMFtfYD51aEUE8lxNcNA3hZPUvvjtGrh5taL fHZDSaDy1mVPmGVHxbMfNdyizNTuZYQgDQialcAMBATrsY7UF2ewVKm31YKqCXu9iLpHpJ Lc6VPtHY+7TLvvdXiwlcEBnRqogtWcLIEWItHAfAifBSwRg0JoDAcklokG5iyg== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id E739C3B592 for ; Fri, 23 Feb 2024 09:37:47 -0600 (CST) Message-ID: Date: Fri, 23 Feb 2024 09:37:47 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> From: Matthew Grooms In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] X-Rspamd-Queue-Id: 4ThDfk1Dp9z4MXx On 2/23/24 03:38, Stephan Althaus wrote: > On 2/23/24 07:46, Matthew Grooms wrote: >> UFS image files > > Hello! > > Please don't mind asking stupid.. > > Was the UFS image file placed on a ZFS file system on the host ? > > You should place an image file on a different disk/filesystem on the > host to rule out ZFS issues on the host side.. > The problem occurs when an image file is used on either ZFS or UFS. The problem also occurs when the virtual disk is backed by a raw disk partition or a ZVOL. This issue isn't related to a specific underlying filesystem. -Matthew From nobody Fri Feb 23 19:17:02 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThKWl4pHzz5BLQ3 for ; Fri, 23 Feb 2024 19:17:07 +0000 (UTC) (envelope-from jo@durchholz.org) Received: from www382.your-server.de (www382.your-server.de [78.46.146.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThKWj6WJfz5213 for ; Fri, 23 Feb 2024 19:17:05 +0000 (UTC) (envelope-from jo@durchholz.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=durchholz.org header.s=default2202 header.b=ppwS5QoG; dmarc=none; spf=pass (mx1.freebsd.org: domain of jo@durchholz.org designates 78.46.146.228 as permitted sender) smtp.mailfrom=jo@durchholz.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=durchholz.org; s=default2202; h=Content-Transfer-Encoding:Content-Type: Subject:From:To:MIME-Version:Date:Message-ID:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References; bh=1XkX2zpgDXTG8aQKIDESFF39Vs+B5uBqx/QygCeGDEQ=; b=ppwS5QoGSQ1ZO8SHAUcbtjdkWq tfji3lFxjZETidJ6Np4Sv/VAnpLt6/faP+3z1tFbNAbpWPRMzfwDFXtddvbXyvz6Wg6uvccy2CBNz 3oW5TNVPsAEjR67p0H9lm8+7Tu4AqDEx9sWKVJCTDDneFDRvsG3u7L0rCe3dAHSeFG1YQ5OrIipT5 y9xKQYMAYe1FRKsStM/+sb3m+/ec3FCfPc3VqneStwvjcFyT9wgQA5T4PXYUqy6cMSBtgk8OU64EZ 12XZKKQSZx4KyJgRJRders7MF0mj58+Sso/0Wg5L1ofxg2F0QDPaRYmRF7EqmZZ6fmoY4Fg2E1his fECnbzwA==; Received: from sslproxy05.your-server.de ([78.46.172.2]) by www382.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rdb2d-000JX1-Bh for freebsd-virtualization@freebsd.org; Fri, 23 Feb 2024 20:17:03 +0100 Received: from [81.221.201.210] (helo=[192.168.178.48]) by sslproxy05.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rdb2d-000Lbl-8C for freebsd-virtualization@freebsd.org; Fri, 23 Feb 2024 20:17:03 +0100 Message-ID: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> Date: Fri, 23 Feb 2024 20:17:02 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: "freebsd-virtualization@FreeBSD.org" From: Jo Durchholz Subject: Best way to have a FreeBSD VM for automated testing? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-Sender: jo@durchholz.org X-Virus-Scanned: Clear (ClamAV 0.103.10/27194/Fri Feb 23 10:32:16 2024) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.49 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[durchholz.org:s=default2202]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:24940, ipnet:78.46.0.0/15, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[durchholz.org]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; HAS_X_AS(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[durchholz.org:+] X-Rspamd-Queue-Id: 4ThKWj6WJfz5213 Hi all, I'm in repeatable build land, working in Linux and developing a FreeBSD appliance. For tests, I need to run a FreeBSD VM, put some Python code and test data into it, run the script, and get the test results back. Repeatability means: Everything done with the VM needs to be scriptable (using a GUI for exploring is okay but things have to translate). Which in turn means that every setup step for a FreeBSD image comes with a pretty high coding and maintenance cost. So my question is: What's the FreeBSD image that has the least number of steps to get the base system up and running? I suppose it's the VM-IMAGES section, but is this correct? Follow-up question: The startup time needs to be as fast as possible. Sub-second would be great ("don't disrupt the developer's thought stream"). I see the boot process from a vanilla VM-IMAGES image takes multiple seconds; can this be sped up to just a few seconds, or do I need to run the setup and create a VM snapshot at which the VM starts for each test run? Regards, Jo From nobody Fri Feb 23 18:41:27 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThL4c0nB3z5BNk4 for ; Fri, 23 Feb 2024 19:42:08 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThL4b6BgSz55rl for ; Fri, 23 Feb 2024 19:42:07 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-512db550e3fso1555594e87.2 for ; Fri, 23 Feb 2024 11:42:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708717324; x=1709322124; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5wo2j+YiZ+uughzTZf7K2Io3e1A0yLouYckRWdHcr0E=; b=KuyV+uAzDoEunnwHaw13BD2sltvySDnUUFYHlJCEAFrHGFAYEKa0Ms1cQNhqvIB4Na 3cjqHBV80XPdEPu5DmjG8rHOH8TglkCo4tdwjihIyubKNWqZqYPcNbC6eghUI+pehuKq 2dtZBdodsDevDdlqCmOlgETKqKMyluBqf2mtMW0J/ii5F2EbmmSV+BGMBqJ6PvaAay6r JwSz+pW6WNefr0mJrWihaXBGgBEzlbkRL1CXUVDlVRCwqzUeL82fclvkrEK5A4gmgmxQ BjnP2dyAmnC9w7iUxs+qfFkvRm/U7e1BxEwBIJccPdxDEREkfsITb9ic7LlEJZQB6mqE I6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708717324; x=1709322124; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5wo2j+YiZ+uughzTZf7K2Io3e1A0yLouYckRWdHcr0E=; b=h6ayYvLe6hIutABMzNDG5PuhVvk7AzWArrh+aY3u3AUi16Axdb6oC2ShCXC2fitNSP TU+6Lh566p0ViQMgi5eNEEOQmlbr4S7G7Y7jn1+JLKllHRzEurTtxK8yLTOXrnriLUHc 4xPi6SV52uygGfouIFjR4oCnkieW/MTP0J5caqrODEyNE+trWQMaB/jm6KnFa1hmHKMu GWzDl7xP9t01a20eABrS0vDlYPZAXFV/iIxLQJA6PLSiIbUnSWVLDTsXgTHnnXroi+I9 pAvHfy2ZgrA/zx49JrNLqDoHL5+y7v7fkZIw91AUNyXxX4pQRYgHYweGRtYFPKU90lya og8Q== X-Gm-Message-State: AOJu0YyyxaSWexc1UVyVZf2C+ZMQNMIvuFssy4tpNgPIpEG7cWAwywag 0OtdXd2BSWwQFtnIZsa83dATw/6j8QYuvCjC7w3Ajuy6/OdabV41PPBe6jA6Up++WOuSckxSq5Y YqlXNcK2WLgtxXNgc9slCI1jwmqNhWjmvrd8x8Q== X-Google-Smtp-Source: AGHT+IERhqOrWjOIq0q1A4d79IjCnEIMjNeshyTE6rNc9oY6zoS2suA+dl6e74CZqin94f5uyMmxNNvZuWwV93OCDJ8= X-Received: by 2002:a05:6512:ba2:b0:512:bfb7:6d2c with SMTP id b34-20020a0565120ba200b00512bfb76d2cmr500849lfv.53.1708717323436; Fri, 23 Feb 2024 11:42:03 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> In-Reply-To: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> From: Mario Marietto Date: Fri, 23 Feb 2024 19:41:27 +0100 Message-ID: Subject: Re: Best way to have a FreeBSD VM for automated testing? To: Jo Durchholz Cc: "freebsd-virtualization@FreeBSD.org" Content-Type: multipart/alternative; boundary="00000000000015c899061211c1d2" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4ThL4b6BgSz55rl --00000000000015c899061211c1d2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable To speed up the booting of a bhyve VM I'm using this method : nohup /usr/sbin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -= A \ -s 0,hostbridge \ -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex=3D= 1 \ -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ -s 13,virtio-net,tap19 \ -s 14,virtio-9p,sharename=3D/ \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:19/dev/null 2>&1 & if test -f nohup.out; then rm -r nohup.out fi I've installed a ssh server within the vm and I connect to it from FreeBSD using ssh -Y user@IP ; it's faster. But the project is not completed. I want to install VirGL to have the graphic acceleration without using the real GPU of the host inside the VM. On Fri, Feb 23, 2024 at 8:17=E2=80=AFPM Jo Durchholz wro= te: > Hi all, > > I'm in repeatable build land, working in Linux and developing a FreeBSD > appliance. > > For tests, I need to run a FreeBSD VM, put some Python code and test > data into it, run the script, and get the test results back. > > Repeatability means: Everything done with the VM needs to be scriptable > (using a GUI for exploring is okay but things have to translate). > Which in turn means that every setup step for a FreeBSD image comes with > a pretty high coding and maintenance cost. > > So my question is: > What's the FreeBSD image that has the least number of steps to get the > base system up and running? I suppose it's the VM-IMAGES section, but is > this correct? > > Follow-up question: > The startup time needs to be as fast as possible. Sub-second would be > great ("don't disrupt the developer's thought stream"). > I see the boot process from a vanilla VM-IMAGES image takes multiple > seconds; can this be sped up to just a few seconds, or do I need to run > the setup and create a VM snapshot at which the VM starts for each test > run? > > Regards, > Jo > > --=20 Mario. --00000000000015c899061211c1d2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
To speed up the booting= of a bhyve VM I'm using this method :

nohup /usr/sbin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -= w -H -A \
-s 0,hostbr= idge \
-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,booti= ndex=3D1 \
-s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
-s 13,virtio-net,tap19 \
-s 14,virtio-9p,sharename=3D/ \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19</dev/null >/dev/null 2>&1 &
if test -f nohup.out; then rm -r nohup.out
fi
<= span style=3D"font-family:monospace">
I've installed a ssh server within the vm and I c= onnect to it from FreeBSD using ssh -Y user@IP ; it's faster. But the p= roject is not completed. I want to install VirGL to have the graphic accele= ration without using the real GPU of the host inside the VM.

On Fri, Feb 23, 2024 at 8:17=E2=80=AFPM Jo Durchholz <jo@durchholz.org> wrote:
Hi all,

I'm in repeatable build land, working in Linux and developing a FreeBSD=
appliance.

For tests, I need to run a FreeBSD VM, put some Python code and test
data into it, run the script, and get the test results back.

Repeatability means: Everything done with the VM needs to be scriptable (using a GUI for exploring is okay but things have to translate).
Which in turn means that every setup step for a FreeBSD image comes with a pretty high coding and maintenance cost.

So my question is:
What's the FreeBSD image that has the least number of steps to get the =
base system up and running? I suppose it's the VM-IMAGES section, but i= s
this correct?

Follow-up question:
The startup time needs to be as fast as possible. Sub-second would be
great ("don't disrupt the developer's thought stream"). I see the boot process from a vanilla VM-IMAGES image takes multiple
seconds; can this be sped up to just a few seconds, or do I need to run the setup and create a VM snapshot at which the VM starts for each test run= ?

Regards,
Jo



--
Mario.
--00000000000015c899061211c1d2-- From nobody Fri Feb 23 21:37:34 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThNdt2Cv4z5BYfC for ; Fri, 23 Feb 2024 21:37:38 +0000 (UTC) (envelope-from jo@durchholz.org) Received: from www382.your-server.de (www382.your-server.de [78.46.146.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThNds3khHz44NW for ; Fri, 23 Feb 2024 21:37:37 +0000 (UTC) (envelope-from jo@durchholz.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=durchholz.org header.s=default2202 header.b=AofeCvOo; dmarc=none; spf=pass (mx1.freebsd.org: domain of jo@durchholz.org designates 78.46.146.228 as permitted sender) smtp.mailfrom=jo@durchholz.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=durchholz.org; s=default2202; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:References:To:Subject:From:MIME-Version:Date:Message-ID:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=YyX8rSwr+Q+LeuQCh/oYIT+Otwi0T7hukHXhrEcM+s0=; b=AofeCvOofxDrlSlrgH+OsscXnW h7AMkQLSHhAjEGrsIuD+sJyc4k49ZNh2jtXlC1Q4D1rLV0c0M0LJu/fyCSHu50jWo3wTw8ONW29aw mJFb8h2SKfr23osRE0ZflDZ6SflGahdTH0ls5mHlBIqtdoAHpIhbb04uWM1RfH3WIwiekyiKPKCTM dUT2RVc1ERrcAFL5TUKIWdwPufIWlO0UjsfsEP4hP1aM4ZFZWqdpj7r8t5ObSfSS+uDRKJ5vfCYsu M00cQh+0Vf0OVEfb9S8VR8zoUBcBrMSXo0qe9tTsxfOMIwXJfsfMuUmgTw5c5FAjXENqikAk/W0W7 0tfj3HEg==; Received: from sslproxy06.your-server.de ([78.46.172.3]) by www382.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rddEd-000CXq-3h for freebsd-virtualization@freebsd.org; Fri, 23 Feb 2024 22:37:35 +0100 Received: from [81.221.201.210] (helo=[192.168.178.48]) by sslproxy06.your-server.de with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rddEc-0002yr-WD for freebsd-virtualization@freebsd.org; Fri, 23 Feb 2024 22:37:35 +0100 Message-ID: <4073dc47-528e-493b-a28e-a21aff733225@durchholz.org> Date: Fri, 23 Feb 2024 22:37:34 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Jo Durchholz Subject: Re: Best way to have a FreeBSD VM for automated testing? Content-Language: en-US To: "freebsd-virtualization@FreeBSD.org" References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Authenticated-Sender: jo@durchholz.org X-Virus-Scanned: Clear (ClamAV 0.103.10/27194/Fri Feb 23 10:32:16 2024) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.45 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.956]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[durchholz.org:s=default2202]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:24940, ipnet:78.46.0.0/15, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[durchholz.org]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; HAS_X_AS(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[durchholz.org:+] X-Rspamd-Queue-Id: 4ThNds3khHz44NW On 23.02.24 19:41, Mario Marietto wrote: > To speed up the booting of a bhyve VM I'm using this method : > > nohup /usr/sbin/bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \ > -s 0,hostbridge \ > -s > 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex=1 \ > -s 11,hda,play=/dev/dsp,rec=/dev/dsp \ > -s 13,virtio-net,tap19 \ > -s 14,virtio-9p,sharename=/ \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ > vm0:19/dev/null 2>&1 & > if test -f nohup.out; then rm -r nohup.out > fi Hmmm... my knee-jerk reaction was that bhyve runs on FreeBSD and I want to use a Linux host, but to my surprise, yeah you can run byhve on Linux. However, it's an additional layer, which means more potential failure modes, and it requires extra configuration on the system level which means it's another small entry barrier for new contributors. So... no bhyve for me. (Note that this does not mean that there's no merit in the approach! Just that it does not fit my project's priorities.) Do you know how to translate all these bhyve options to a KVM or Vagrant incantation? I know too little about either to be confident in what I'd produce. Maybe a short explanation of each parameter would help? Why is it there, what problem does it solve, that kind of information, just to get me started so I can focus on those options that are actually relevant to the applicance I'm trying to set up. > I've installed a ssh server within the vm and I connect to it from > FreeBSD using ssh -Y user@IP ; it's faster. But the project is not > completed. I want to install VirGL to have the graphic acceleration > without using the real GPU of the host inside the VM. Heh. Yet another rabbit hole. Good luck with that! I have to say I'm lucky that my project is just a web server so I don't need that. Anyway: Thanks for the info! Knowing what you're doing tells everybody a lot about what's routine and what isn't, and that's valuable. Regards, Jo > On Fri, Feb 23, 2024 at 8:17 PM Jo Durchholz > wrote: > > Hi all, > > I'm in repeatable build land, working in Linux and developing a FreeBSD > appliance. > > For tests, I need to run a FreeBSD VM, put some Python code and test > data into it, run the script, and get the test results back. > > Repeatability means: Everything done with the VM needs to be scriptable > (using a GUI for exploring is okay but things have to translate). > Which in turn means that every setup step for a FreeBSD image comes > with > a pretty high coding and maintenance cost. > > So my question is: > What's the FreeBSD image that has the least number of steps to get the > base system up and running? I suppose it's the VM-IMAGES section, > but is > this correct? > > Follow-up question: > The startup time needs to be as fast as possible. Sub-second would be > great ("don't disrupt the developer's thought stream"). > I see the boot process from a vanilla VM-IMAGES image takes multiple > seconds; can this be sped up to just a few seconds, or do I need to run > the setup and create a VM snapshot at which the VM starts for each > test run? > > Regards, > Jo > > > > -- > Mario. From nobody Sat Feb 24 00:46:09 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThSr94Bf5z5BcFg for ; Sat, 24 Feb 2024 00:46:49 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThSr92SrKz4Sxv for ; Sat, 24 Feb 2024 00:46:49 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-a42e2944482so20546866b.1 for ; Fri, 23 Feb 2024 16:46:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708735606; x=1709340406; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QYyDDVjMTmmgos+d8LNfU6bAxFSGNpsxxul4GDOjQJo=; b=KEPQAhCnvrwqmim5GugdpAoN+OVpuWWNydsLkZ6FAXB+TQOIdxEQv2tbFhoRIUhHxb JHhWkj2W5Qi7dZ8ZFIOUzH3fbsAc1/2FsZGmm+mudBPjCZnQJgjdxFixlW9WQ1L8ySiq /crfEUMByPtdXqu/oJRKYerwlzZc717YbkRH0ujJefCNR+7esRTXUf6J69H9rD7UnMn1 qY9T7YqnQ3ef3meLT/ZW+Fcyz6EbA444hRYv3Y/+AgunTDyWbsWjN/FhdGjPMXGYtWUl GxaBPGNLUJQSWlcweDuF4H9XD6NExdDLfll4OPzko6Xk304tEtdYye4XKdsB7kbcvSrA doZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708735606; x=1709340406; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QYyDDVjMTmmgos+d8LNfU6bAxFSGNpsxxul4GDOjQJo=; b=PycRTCxESUGw0UQmdISkm1dHUAUNm47kvQD5bZEtS1ZfQakX5HMhewBq/he6SQs7R8 YAwIiCvHqpHgZaIuRiXey3ZhzaZm49pNUnSkWrlmFN4OXz1BmL/mlATGUPw4fCmGld6s 686GDJvfiJEjIVm4k1W06Xd/dGZKy4xxMtjrFmN82b4QeAZ2jae3FWjn/zsTs/ha3IUZ 93MJ10IJZGcY9QVIwSg7AhXMWxJhUSoB/ZccRcMN71QGdgqfNjqw9NB1kHQsz5FtQ+uv ExzZCpp6iR9tNlGsszuOEB/uuWpxcatOhnf3jpjwG+RtnIbaJ/o13h0AyhNa+Vp9mXc7 Wohw== X-Gm-Message-State: AOJu0Yw/DdTRnV2SRQQtlW81kACRJlTNdmA+DDuo8GfNr+1feBHuWWLr pf0mtQGd5o/92qYtyUk8ke1pD9S0VHvlBjGVsTTzqP6By0dP0AcrICkmjF2APL8m0gBQ/P9d0m0 W63xJIvQYN5QmD05RJ+BJF4LkBIMAVQbxxNw= X-Google-Smtp-Source: AGHT+IEQlHEcjS/pMVKiaHeojYA0+v6wuw6VVNLTI/md9iILRmH8N64j16ZOdlnh6xjE6mBK3W4yEyhBWgHX6gLMpqk= X-Received: by 2002:a17:906:289b:b0:a3f:d797:42ec with SMTP id o27-20020a170906289b00b00a3fd79742ecmr940937ejd.53.1708735605814; Fri, 23 Feb 2024 16:46:45 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> <4073dc47-528e-493b-a28e-a21aff733225@durchholz.org> In-Reply-To: <4073dc47-528e-493b-a28e-a21aff733225@durchholz.org> From: Mario Marietto Date: Sat, 24 Feb 2024 01:46:09 +0100 Message-ID: Subject: Re: Best way to have a FreeBSD VM for automated testing? To: Jo Durchholz Cc: "freebsd-virtualization@FreeBSD.org" Content-Type: multipart/alternative; boundary="000000000000ccba570612160246" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4ThSr92SrKz4Sxv --000000000000ccba570612160246 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I forgot to say that I want to create a different FreeBSD flavour that will have a lot of virtual machines already embedded and ready to work and that work transparently. On Fri, Feb 23, 2024 at 10:37=E2=80=AFPM Jo Durchholz wr= ote: > On 23.02.24 19:41, Mario Marietto wrote: > > To speed up the booting of a bhyve VM I'm using this method : > > > > nohup /usr/sbin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w = -H -A \ > > -s 0,hostbridge \ > > -s > > > 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootindex= =3D1 \ > > -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > > -s 13,virtio-net,tap19 \ > > -s 14,virtio-9p,sharename=3D/ \ > > -s 30,xhci,tablet \ > > -s 31,lpc \ > > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ > > vm0:19/dev/null 2>&1 & > > if test -f nohup.out; then rm -r nohup.out > > fi > > Hmmm... my knee-jerk reaction was that bhyve runs on FreeBSD and I want > to use a Linux host, but to my surprise, yeah you can run byhve on Linux. > > However, it's an additional layer, which means more potential failure > modes, and it requires extra configuration on the system level which > means it's another small entry barrier for new contributors. > > So... no bhyve for me. > (Note that this does not mean that there's no merit in the approach! > Just that it does not fit my project's priorities.) > > Do you know how to translate all these bhyve options to a KVM or Vagrant > incantation? I know too little about either to be confident in what I'd > produce. > Maybe a short explanation of each parameter would help? Why is it there, > what problem does it solve, that kind of information, just to get me > started so I can focus on those options that are actually relevant to > the applicance I'm trying to set up. > > > I've installed a ssh server within the vm and I connect to it from > > FreeBSD using ssh -Y user@IP ; it's faster. But the project is not > > completed. I want to install VirGL to have the graphic acceleration > > without using the real GPU of the host inside the VM. > > Heh. Yet another rabbit hole. Good luck with that! > I have to say I'm lucky that my project is just a web server so I don't > need that. > > Anyway: Thanks for the info! Knowing what you're doing tells everybody a > lot about what's routine and what isn't, and that's valuable. > > Regards, > Jo > > > On Fri, Feb 23, 2024 at 8:17=E2=80=AFPM Jo Durchholz > > wrote: > > > > Hi all, > > > > I'm in repeatable build land, working in Linux and developing a > FreeBSD > > appliance. > > > > For tests, I need to run a FreeBSD VM, put some Python code and tes= t > > data into it, run the script, and get the test results back. > > > > Repeatability means: Everything done with the VM needs to be > scriptable > > (using a GUI for exploring is okay but things have to translate). > > Which in turn means that every setup step for a FreeBSD image comes > > with > > a pretty high coding and maintenance cost. > > > > So my question is: > > What's the FreeBSD image that has the least number of steps to get > the > > base system up and running? I suppose it's the VM-IMAGES section, > > but is > > this correct? > > > > Follow-up question: > > The startup time needs to be as fast as possible. Sub-second would = be > > great ("don't disrupt the developer's thought stream"). > > I see the boot process from a vanilla VM-IMAGES image takes multipl= e > > seconds; can this be sped up to just a few seconds, or do I need to > run > > the setup and create a VM snapshot at which the VM starts for each > > test run? > > > > Regards, > > Jo > > > > > > > > -- > > Mario. > > > --=20 Mario. --000000000000ccba570612160246 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I forgot to say that I want to create a different FreeBSD = flavour that will have a lot of virtual machines already embedded and ready= to work and that work transparently.

On Fri, Feb 23, 2024 at 10:37=E2= =80=AFPM Jo Durchholz <jo@durchholz.= org> wrote:
On 23.02.24 19:41, Mario Marietto wrote:
> To speed up the booting of a bhyve VM I'm using this method :
>
> nohup /usr/sbin/bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w= -H -A \
> -s 0,hostbridge \
> -s
> 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/Ubuntu2310.img,bootind= ex=3D1 \
> -s 11,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
> -s 13,virtio-net,tap19 \
> -s 14,virtio-9p,sharename=3D/ \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
> vm0:19</dev/null >/dev/null 2>&1 &
> if test -f nohup.out; then rm -r nohup.out
> fi

Hmmm... my knee-jerk reaction was that bhyve runs on FreeBSD and I want to use a Linux host, but to my surprise, yeah you can run byhve on Linux.
However, it's an additional layer, which means more potential failure <= br> modes, and it requires extra configuration on the system level which
means it's another small entry barrier for new contributors.

So... no bhyve for me.
(Note that this does not mean that there's no merit in the approach! Just that it does not fit my project's priorities.)

Do you know how to translate all these bhyve options to a KVM or Vagrant incantation? I know too little about either to be confident in what I'd=
produce.
Maybe a short explanation of each parameter would help? Why is it there, what problem does it solve, that kind of information, just to get me
started so I can focus on those options that are actually relevant to
the applicance I'm trying to set up.

> I've installed a ssh server within the vm and I connect to it from=
> FreeBSD using ssh -Y user@IP ; it's faster. But the project is not=
> completed. I want to install VirGL to have the graphic acceleration > without using the real GPU of the host inside the VM.

Heh. Yet another rabbit hole. Good luck with that!
I have to say I'm lucky that my project is just a web server so I don&#= 39;t
need that.

Anyway: Thanks for the info! Knowing what you're doing tells everybody = a
lot about what's routine and what isn't, and that's valuable.
Regards,
Jo

> On Fri, Feb 23, 2024 at 8:17=E2=80=AFPM Jo Durchholz <jo@durchholz.org
> <mailto:jo@du= rchholz.org>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Hi all,
>
>=C2=A0 =C2=A0 =C2=A0I'm in repeatable build land, working in Linux = and developing a FreeBSD
>=C2=A0 =C2=A0 =C2=A0appliance.
>
>=C2=A0 =C2=A0 =C2=A0For tests, I need to run a FreeBSD VM, put some Pyt= hon code and test
>=C2=A0 =C2=A0 =C2=A0data into it, run the script, and get the test resu= lts back.
>
>=C2=A0 =C2=A0 =C2=A0Repeatability means: Everything done with the VM ne= eds to be scriptable
>=C2=A0 =C2=A0 =C2=A0(using a GUI for exploring is okay but things have = to translate).
>=C2=A0 =C2=A0 =C2=A0Which in turn means that every setup step for a Fre= eBSD image comes
>=C2=A0 =C2=A0 =C2=A0with
>=C2=A0 =C2=A0 =C2=A0a pretty high coding and maintenance cost.
>
>=C2=A0 =C2=A0 =C2=A0So my question is:
>=C2=A0 =C2=A0 =C2=A0What's the FreeBSD image that has the least num= ber of steps to get the
>=C2=A0 =C2=A0 =C2=A0base system up and running? I suppose it's the = VM-IMAGES section,
>=C2=A0 =C2=A0 =C2=A0but is
>=C2=A0 =C2=A0 =C2=A0this correct?
>
>=C2=A0 =C2=A0 =C2=A0Follow-up question:
>=C2=A0 =C2=A0 =C2=A0The startup time needs to be as fast as possible. S= ub-second would be
>=C2=A0 =C2=A0 =C2=A0great ("don't disrupt the developer's = thought stream").
>=C2=A0 =C2=A0 =C2=A0I see the boot process from a vanilla VM-IMAGES ima= ge takes multiple
>=C2=A0 =C2=A0 =C2=A0seconds; can this be sped up to just a few seconds,= or do I need to run
>=C2=A0 =C2=A0 =C2=A0the setup and create a VM snapshot at which the VM = starts for each
>=C2=A0 =C2=A0 =C2=A0test run?
>
>=C2=A0 =C2=A0 =C2=A0Regards,
>=C2=A0 =C2=A0 =C2=A0Jo
>
>
>
> --
> Mario.




--
Mario.
--000000000000ccba570612160246-- From nobody Sat Feb 24 01:51:17 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThVGk6Lrcz5BjSG for ; Sat, 24 Feb 2024 01:51:26 +0000 (UTC) (envelope-from freebsd@shaneware.biz) Received: from mail-relay2.dca2.superb.net (mail-relay2.dca2.superb.net [66.148.95.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThVGk2j2Rz4Zds for ; Sat, 24 Feb 2024 01:51:26 +0000 (UTC) (envelope-from freebsd@shaneware.biz) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=superb.net; s=mailrelay; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From: References:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To:Cc: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=Fg4QF+UHWX/jgsciLKURMKZUCM78vbrzOgk9kbeT+Nc=; b=lLa6f5OR2orxjx2Ty086IK+moU 8LIlJMBcytx2QOc5ARuLejQTPsWz3rtKbwJGBfCosZM5HMnLsVm83Di6ED9X/i/L1BHXn/w/GVBSb qs2RXABbRFBp4zBanI6f9gdKUedKpQQVg2PoeA8NmStKwVwpKIxGxJZ4zdfyhBGV/vf5uq9JHeg/G G9pXyJlQz7dK6+KKXMNOymrGxDhyzayZcpwwZ4j5/x7ZlleSO8pnCw7mi18+tuMHNZ9mBKzEurahC e+V8g1UDWfkd2RcvMxhaie6eKmFvlpHc6hZ7dUqLQHLCroHbAh3mYted3kpyfniQiZoi10RC/8JXn RTYhNP6g==; Received: from ppp230-11.static.internode.on.net ([203.122.230.11] helo=[192.168.9.155]) by mail-relay2.dca2.superb.net with esmtpsa (TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (envelope-from ) id 1rdhCF-00086j-E7; Fri, 23 Feb 2024 20:51:24 -0500 Message-ID: <01ccdc98-2b8e-331f-65df-5496f8d16ead@ShaneWare.Biz> Date: Sat, 24 Feb 2024 12:21:17 +1030 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: Best way to have a FreeBSD VM for automated testing? To: Jo Durchholz , "freebsd-virtualization@FreeBSD.org" References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> Content-Language: en-AU From: Shane Ambler In-Reply-To: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SA-Score: -3.5 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:14361, ipnet:66.148.95.0/24, country:US] X-Rspamd-Queue-Id: 4ThVGk2j2Rz4Zds On 24/2/24 05:47, Jo Durchholz wrote: > Hi all, > > I'm in repeatable build land, working in Linux and developing a FreeBSD > appliance. > > For tests, I need to run a FreeBSD VM, put some Python code and test > data into it, run the script, and get the test results back. > > Repeatability means: Everything done with the VM needs to be scriptable > (using a GUI for exploring is okay but things have to translate). > Which in turn means that every setup step for a FreeBSD image comes with > a pretty high coding and maintenance cost. > > So my question is: > What's the FreeBSD image that has the least number of steps to get the > base system up and running? I suppose it's the VM-IMAGES section, but is > this correct? > > Follow-up question: > The startup time needs to be as fast as possible. Sub-second would be > great ("don't disrupt the developer's thought stream"). > I see the boot process from a vanilla VM-IMAGES image takes multiple > seconds; can this be sped up to just a few seconds, or do I need to run > the setup and create a VM snapshot at which the VM starts for each test > run? Firecracker - firecracker-microvm.github.io It may add another level of setup but it provides sub-second boot times - and can run FreeBSD https://www.daemonology.net/blog/2022-10-18-FreeBSD-Firecracker.html Not sure if there are any pre-built images with the firecracker kernel, so may not meet your requirements. -- FreeBSD - the place to B...Software Developing Shane Ambler From nobody Sat Feb 24 06:41:40 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Thcjy38wJz5C8fY for ; Sat, 24 Feb 2024 06:41:58 +0000 (UTC) (envelope-from mark@exonetric.com) Received: from relay.exonetric.net (relay0.exonetric.net [178.250.72.161]) by mx1.freebsd.org (Postfix) with ESMTP id 4Thcjy0wM6z45pN for ; Sat, 24 Feb 2024 06:41:58 +0000 (UTC) (envelope-from mark@exonetric.com) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (22.pool92-172-210.dynamic.orange.es [92.172.210.22]) by relay.exonetric.net (Postfix) with ESMTPSA id 8075A2D47C; Sat, 24 Feb 2024 06:41:51 +0000 (GMT) From: Mark Blackman Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_6B35707E-D8C1-420E-AF5E-322444DCFFB5" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: Best way to have a FreeBSD VM for automated testing? Date: Sat, 24 Feb 2024 07:41:40 +0100 In-Reply-To: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> Cc: "freebsd-virtualization@FreeBSD.org" To: Jo Durchholz References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:12290, ipnet:178.250.72.0/21, country:GB] X-Rspamd-Queue-Id: 4Thcjy0wM6z45pN --Apple-Mail=_6B35707E-D8C1-420E-AF5E-322444DCFFB5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 23 Feb 2024, at 20:17, Jo Durchholz wrote: >=20 > I see the boot process from a vanilla VM-IMAGES image takes multiple = seconds; can this be sped up to just a few seconds, or do I need to run = the setup and create a VM snapshot at which the VM starts for each test = run? If an Amazon firecracker environment works for you, there=E2=80=99s = https://www.freebsd.org/status/report-2023-04-2023-06/boot-performance/ = of course, there=E2=80=99s more to booting than the kernel. I=E2=80=99d = guess some of those improvements could apply to more generic VM = hypervisors too. - Mark= --Apple-Mail=_6B35707E-D8C1-420E-AF5E-322444DCFFB5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 23 Feb = 2024, at 20:17, Jo Durchholz <jo@durchholz.org> wrote:

I see = the boot process from a vanilla VM-IMAGES image takes multiple seconds; = can this be sped up to just a few seconds, or do I need to run the setup = and create a VM snapshot at which the VM starts for each test = run?

If an Amazon firecracker = environment works for you, there=E2=80=99s https://www.freebsd.org/status/report-2023-04-2023-06/boot-performa= nce/ of course, there=E2=80=99s more to booting than the = kernel. I=E2=80=99d guess some of those improvements could apply to more = generic VM hypervisors too.

- = Mark
= --Apple-Mail=_6B35707E-D8C1-420E-AF5E-322444DCFFB5-- From nobody Sat Feb 24 08:56:33 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThgjK2M56z5CMwv for ; Sat, 24 Feb 2024 08:56:37 +0000 (UTC) (envelope-from jo@durchholz.org) Received: from www382.your-server.de (www382.your-server.de [78.46.146.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThgjJ3N6rz4QHf for ; Sat, 24 Feb 2024 08:56:36 +0000 (UTC) (envelope-from jo@durchholz.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=durchholz.org header.s=default2202 header.b=dAMfrM4B; dmarc=none; spf=pass (mx1.freebsd.org: domain of jo@durchholz.org designates 78.46.146.228 as permitted sender) smtp.mailfrom=jo@durchholz.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=durchholz.org; s=default2202; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:References:To:Subject:From:MIME-Version:Date:Message-ID:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=wb+tgFUg5Qoh5ff0WbHeqLgqRITNHufngtGUcNh10oY=; b=dAMfrM4Bt0eUO7inIps152UZkg YifbAePulI6Qzmjiq23Dxg7hQU4WNbqKGBZ2vczAF7t9nZZwS3djR7GTu1XQli4lIlThmuL3ynnFb HCrVeIshxPSpK6I8XrY17JI85lT+zW07KAZ5NQEdnBJVEux/Nalbi1CrwFJUc0dx7T/5uncleRDGH aWrzLp5WgOk7X9ibJBoPmlWc4yoROGpj/mQ613iIJQCHuH9+0HQkqpQstkfT8b6I+nqroG2o0YFEq mOBbVvfFK3AGL0yumimPE0/3SA13up57nYTwqK9akRhD8tnD3qZaSOwiKSIGIecHVRKJbLu22m6ez 9+weDcJg==; Received: from sslproxy01.your-server.de ([78.46.139.224]) by www382.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rdnph-000I7B-Ny for freebsd-virtualization@freebsd.org; Sat, 24 Feb 2024 09:56:33 +0100 Received: from [81.221.201.210] (helo=[192.168.178.48]) by sslproxy01.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1rdnph-00EfZN-1q for freebsd-virtualization@freebsd.org; Sat, 24 Feb 2024 09:56:33 +0100 Message-ID: <7e5bc32c-dddb-4d33-96e9-99a955eed572@durchholz.org> Date: Sat, 24 Feb 2024 09:56:33 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Jo Durchholz Subject: Re: Best way to have a FreeBSD VM for automated testing? Content-Language: en-US To: "freebsd-virtualization@FreeBSD.org" References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Authenticated-Sender: jo@durchholz.org X-Virus-Scanned: Clear (ClamAV 0.103.10/27194/Fri Feb 23 10:32:16 2024) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.58 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; R_DKIM_ALLOW(-0.20)[durchholz.org:s=default2202]; R_SPF_ALLOW(-0.20)[+a]; RCVD_IN_DNSWL_LOW(-0.10)[78.46.139.224:received]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:24940, ipnet:78.46.0.0/15, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[durchholz.org]; HAS_X_AS(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; DKIM_TRACE(0.00)[durchholz.org:+] X-Rspamd-Queue-Id: 4ThgjJ3N6rz4QHf On 24.02.24 07:41, Mark Blackman wrote: > > If an Amazon firecracker environment works for you, there’s > https://www.freebsd.org/status/report-2023-04-2023-06/boot-performance/ Oh. Thanks. 20 ms boot time sounds good enough to me :-D ... aww, the PDF link to the slides is broken :-( https://wiki.freebsd.org/BootTime (Colin Percival) works and reports ~8 seconds on FreeBSD 14.0. http://www.daemonology.net/blog/2021-08-12-EC2-boot-time-benchmarking.html (again, Colin Percival 2021) tells me that the best boot-to-TCP time was about 1.23 s, while typical boot times would be around 10-15 s. Those 8 seconds are actually good, unless the competition has sped up as well :-) (This benchmarks just TCP availability, I don't know or how much needs to be added for ssh availability.) There's a conclusion here: While those improvements are awesome, 8 seconds are nowhere near the do-not-disrupt-developer-workflow threshold, so the VM snapshot it will be. Oh. Wait. These links talk about FreeBSD's boot time. I don't know how this relates to Firecracker. OTOH Firecracker is very new, so I'm somewhat reluctant to jump that bandwagon anyway. > of course, there’s more to booting than the kernel. Definitely. In my own testing, something that vaguely sounded like a mail subsystem had a full minute-long wait. Some tweaking will be needed to get rid of that behaviour. I'm comfortable with that, though not with having to put those tweaks into a setup script and keeping it up-to-date with every new FreeBSD version. I haven't seen an OS or distro that does not have this kind of problem though, so I'll just have to live with that. > I’d guess some of those improvements could apply to more generic VM hypervisors too. Probably, but as much as I like exploring rabbit holes, I already have a too-long list of these to add yet another one :-) Regards, Jo From nobody Sat Feb 24 12:36:31 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Thnx539blz59nZk; Sat, 24 Feb 2024 13:37:13 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Thnx36sJbz42l0; Sat, 24 Feb 2024 13:37:11 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=K1f+V3tD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::62c as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-ej1-x62c.google.com with SMTP id a640c23a62f3a-a26fa294e56so291804266b.0; Sat, 24 Feb 2024 05:37:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708781828; x=1709386628; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=tzM+/d43wUL0e78Cjl9XHg1Vn6rxJudHflw0fGlng4U=; b=K1f+V3tD0qcn4SP//fjSezJi2P9YHBnI8s5LbqaoYKOvH8FAaHvtyQVNzQR7gyaMnh RwKM55k6kjQ5CQ2KE18jkXnl1FwA1LFXR6PgCdQQwaaurdbrBb2E0ycbtaRtnSmxAyG+ qyh/qKbnfqJhn615mtSmJ8eoM2qXyGmLqoBCO7AxlYPwriS/bN9Jzmc4jTsBXEPfYvNj ns7Ry8N59BaCpkxxF1CLx4CzrbeWwx2MyOFXvNVVU60K1aRciKG+rZKMSg/lR87FAHF+ g/buCpZbsmwY856vgSywDpYgdWOm3Vjw94Hnn7nVkSxbu6RgfW5E0ad0LfIKAGvFYg2j BDNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708781828; x=1709386628; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=tzM+/d43wUL0e78Cjl9XHg1Vn6rxJudHflw0fGlng4U=; b=dZ+I6zurVd/j3/3FagMgrzMOW6zlmMcExgn+aFF74OB3H9xxPWAfa7J1ly6HpjLdPE 4TIP4HEp0iT+4mbW4WuFHCr1HJAhte59HS4Ib9qtEn7OzKr5smATdi4FaMQ0LdmyNMI3 IdPQMO1WdLBmzjcy5KsL5dm6+UT6wDtMrgSGI/TiHJ8DJbIvi2kp49nNF6xB/716K0NF HBVdzUB4ddeHo8z8hWbVzp4dlVh5S6pO3HCtLHpgEiN78+rfVxuruPT9V/B6snDzhXbx BaqLJ2XXBiXoRyzOvvVNezsien1QZ+hfoKq7vrUdh/YHoKVI2dfHaHFhDhoFPvvoU1OC SYHQ== X-Forwarded-Encrypted: i=1; AJvYcCUR17gHOASWYiUuGT7YGVcOXiABm376KbC9bs0ncaXWCzTt7crORDxqqk9a9H5BVXR6GJP2n12ySixItSJYEVjRkUBEcQ4shjkB6U3sv8VD8vuLk5aflWJRYJpamobAJ1GW8hp32Uk3gEDlWdd+UQrGiTd2Uh7DqZGiMG0/HoLiDvSHNczlnKpGnC84L6BesuM3CQRoErKOvyx461+xgZzcZ//rKnGII9bzRfSzeh7Y+trXt9Y= X-Gm-Message-State: AOJu0YwnZsMWSMpfqhh+nqCCnRWsf+ZKpcGN36HSel4pIea61Cw45Euh IZm/2mro6bmompX16i2kXzcXlzkwqZp3FCJHMI0dC5AK3fHwoE0ermSl9wvNRbF4qMC+z6hmBFF GFd54nUQMi9FbObET6wB8iQhJ1mdd+IDz7p0= X-Google-Smtp-Source: AGHT+IHghReDG5UhxF6Bupatplxt+sUIu/Hj+pzaC+J4Az9KhY/ths897VUrscoQjJYrGbDlVWeSfrLiwJOf0xGw15w= X-Received: by 2002:a17:906:8313:b0:a43:7bc:2749 with SMTP id j19-20020a170906831300b00a4307bc2749mr201320ejx.13.1708781827369; Sat, 24 Feb 2024 05:37:07 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 From: Mario Marietto Date: Sat, 24 Feb 2024 13:36:31 +0100 Message-ID: Subject: drm-kmod-20220907_2 not supported for this configuration / NVIDIA : Failed to initialize the NVIDIA kernel module To: FreeBSD virtualization , FreeBSD Mailing List , freebsd-hackers , freebsd-x11@freebsd.org, freebsd-drivers@freebsd.org, Ronald Klop , Jake Freeland Content-Type: multipart/alternative; boundary="000000000000d1afdf061220c57a" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-questions@freebsd.org,freebsd-hackers@freebsd.org,freebsd-x11@freebsd.org,freebsd-drivers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62c:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+] X-Rspamd-Queue-Id: 4Thnx36sJbz42l0 --000000000000d1afdf061220c57a Content-Type: text/plain; charset="UTF-8" Hello to everyone. I need to conduct some tests on top of the FreeBSD 13.1. So,I've installed it on one of my disks. I've installed xfce4 and KDE,Xorg and from the ports (after having upgraded them),I've installed the nvidia-driver version. 535.146.02 ; I didn't have any problem. The problem arises when I tried to install the package drm-kmod from ports. Unfortunately I'm not able to compile it. This is what happened : root@marietto:/usr/ports/graphics/drm-kmod # make ====> drm-kmod-20220907_2 not supported for this configuration. ok. At this point I tried to install it from the packages : root@marietto:/usr/ports/graphics/drm-kmod # make clean ====> cleaning for drm-kmod-20220907_2 so : root@marietto:/usr/ports/graphics/drm-kmod # pkg install drm-kmod New packages to be INSTALLED : drm-kmod: 20220907_2 OK now,I have performed additional configuration to : nano /home/marietto/.xinitrc : exec ck-launch-session dbus-launch --exit-with-session startxfce4 nano /etc/rc.conf : kdm5_enable="YES" dbus_enable="YES" hald_enable="YES" kld_list="nvidia nvidia-modeset" rpcbind_enable="YES" dtcms_enable="YES" inetd_enable="YES" nano /boot/loader.conf vmm_load="YES" nmdm_load="YES" tmpfs_load="YES" cryptodev_load="YES" zfs_load="YES" kern.racct.enable="1" kern.vty=vt kern.cam.scsi_delay="10000" nano /etc/X11/xorg.conf Section "Device" Identifier "Card0" Driver "nvidia" BusID "PCI:1:0:0" EndSection root@marietto:/home/marietto # lspci 01:00.0 NVIDIA GP106 01:00.1 NVIDIA GP106 High Definition Audio Controller 02:00.0 NVIDIA TU102 02:00.1 NVIDIA TU102 High Definition Audio Controller 02:00.2 NVIDIA TU102 USB Controller 02:00.3 NVIDIA TU102 Serial BUS Controller Now,I want to startx to start xfce4 : marietto@marietto: $ startx Fatal server error : no screens found (EE) Check the log file at "/var/log/Xorg.0.log" nano /var/log/Xorg.0.log : NVIDIA dlloader X driver 535.146.02 NVIDIA Unified Driver for all Supported NVIDIA gpus NVIDIA : Failed to initialize the NVIDIA kernel module How to fix it ? -- Mario. --000000000000d1afdf061220c57a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello to everyone.

I need to= conduct some tests on top of the FreeBSD 13.1.
So,I've insta= lled it on one of my disks. I've installed xfce4 and KDE,Xorg and from = the ports (after having upgraded them),I've installed the nvidia-driver= version. 535.146.02 ; I didn't have any problem.=C2=A0

<= /div>
The problem arises when I tried to install the package drm-kmod f= rom ports.=C2=A0

Unfortunately I'm not able to= compile it. This is what happened :

root@marietto= :/usr/ports/graphics/drm-kmod # make
=3D=3D=3D=3D> drm-kmod-20= 220907_2 not supported for this configuration.

ok.= At this point I tried to install it from the packages :

root@marietto:/usr/ports/graphics/drm-kmod # make clean
=3D=3D=3D=3D> cleaning for drm-kmod-20220907_2

so :

root@marietto:/usr/ports/graphics/drm= -kmod # pkg install drm-kmod

New packages to be IN= STALLED :
drm-kmod: 20220907_2
OK

<= div>now,I have performed additional configuration to :

=
nano /home/marietto/.xinitrc :

exec ck-launch= -session dbus-launch --exit-with-session startxfce4

nano /etc/rc.conf :

kdm5_enable=3D"YES"= ;
dbus_enable=3D"YES"
hald_enable=3D"YES= "
kld_list=3D"nvidia nvidia-modeset"
rpc= bind_enable=3D"YES"
dtcms_enable=3D"YES"
inetd_enable=3D"YES"

na= no /boot/loader.conf

vmm_load=3D"YES"
nmdm_load=3D"YES"
tmpfs_load=3D"YES"<= /div>
cryptodev_load=3D"YES"
zfs_load=3D"YES&q= uot;
kern.racct.enable=3D"1"
kern.vty=3Dvt
kern.cam.scsi_delay=3D"10000"

na= no /etc/X11/xorg.conf

Section "Device"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 Identifier=C2=A0=C2=A0 "Card0"
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 Driver=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "nvidia&= quot;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 BusID=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 "PCI:1:0:0"
EndSection


root@marietto:/home/marietto # lspci
01:00.0 NVIDIA GP106
01:00.1 NVIDIA GP106 High Defin= ition Audio Controller
02:00.0 NVIDIA TU102
02:00.1 NVI= DIA TU102 High Definition Audio Controller
02:00.2 NVIDIA TU102 USB Con= troller
02:00.3 NVIDIA TU102 Serial BUS Controller

=
Now,I want to startx to start xfce4 :

m= arietto@marietto: $ startx

Fatal server error : no= screens found (EE)
Check the log file at "/var/log/Xorg.0.l= og"

nano /var/log/Xorg.0.log :

=
NVIDIA dlloader X driver 535.146.02
NVIDIA Unified Dri= ver for all Supported NVIDIA gpus
NVIDIA : Failed to initialize t= he NVIDIA kernel module

How to fix it ?

--
Mario.
--000000000000d1afdf061220c57a-- From nobody Sat Feb 24 14:08:55 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Thpdt25ksz59rF0; Sat, 24 Feb 2024 14:09:06 +0000 (UTC) (envelope-from semfly@fly777.net) Received: from mail.fly777.net (mail.fly777.net [37.228.91.242]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Thpds4PxYz47xM; Sat, 24 Feb 2024 14:09:05 +0000 (UTC) (envelope-from semfly@fly777.net) Authentication-Results: mx1.freebsd.org; none Received: by mail.fly777.net (Postfix, from userid 58) id 2B99B49712; Sat, 24 Feb 2024 17:08:56 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fly777.net; s=mailsrvserv7; t=1708783736; bh=jsOdfEwp7SRHDGk113VRYBDFta9pANzVj2x9VvpW84o=; h=Date:Subject:In-Reply-To:From:To; b=qsuo/7cY2qER4Opo+z+p/kifJhtCmGD4ELd221evGFSW8PMlHX61jWj5LEHcZf8M3 QhmJlQSnyV78kQpM9DHa7cwBbVnPCcSuI2FtqvUf5w3cXNPKQCBptDaXbYror8ydLC oNw/B1mvujHrjpZCG7LST5nx7doeJ8VvcpfEXGgg= X-Spam-Checker-Version: SpamAssassin 3.4.5 (2021-03-20) on mailsrv-serv7.lan X-Spam-Level: X-Spam-Status: No, score=-0.7 required=5.0 tests=ALL_TRUSTED,DKIM_INVALID, DKIM_SIGNED,HTML_MESSAGE,MISSING_MID,URIBL_BLOCKED, URIBL_DBL_BLOCKED_OPENDNS,URIBL_ZEN_BLOCKED_OPENDNS autolearn=no autolearn_force=no version=3.4.5 Received: from [192.168.177.123] (unknown [109.248.1.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.fly777.net (Postfix) with ESMTPSA id 592B74955C; Sat, 24 Feb 2024 17:08:55 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fly777.net; s=mailsrvserv7; t=1708783735; bh=jsOdfEwp7SRHDGk113VRYBDFta9pANzVj2x9VvpW84o=; h=Date:Subject:In-Reply-To:From:To; b=PJWfxbZeJDJctjms8YutqR7aqMAnFzaxIorBxsRV2xhDVr6SeAwlMwjx8Lb+ZhD34 INK9OSM1vhTzMMewuUGrwZnmT5Mali0FbfVrtlMo+uRM4HtAJFbH+Mr0t3F1/5BUBZ 1b+SV4KoFKFfpqYPNw3g5LyzlIE2vMqZvJFkGB+g= SavedFromEmail: semfly@fly777.net Date: Sat, 24 Feb 2024 17:08:55 +0300 Subject: RE: drm-kmod-20220907_2 not supported for this configuration / NVIDIA : Failed to initialize the NVIDIA kernel module In-Reply-To: Importance: normal From: Simon Gusev To: Mario Marietto , FreeBSD virtualization , FreeBSD Mailing List , freebsd-hackers , freebsd-x11@freebsd.org, freebsd-drivers@freebsd.org, Ronald Klop , Jake Freeland List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="--_com.samsung.android.email_7568670750559100" Message-Id: <20240224140856.2B99B49712@mail.fly777.net> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:48347, ipnet:37.228.88.0/22, country:RU] X-Rspamd-Queue-Id: 4Thpds4PxYz47xM ----_com.samsung.android.email_7568670750559100 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SGVsbG8gTWFyaW8uSSBoYXZlIHByb2JsZW0gdG9vLkEgZmV3IGRheXMgaSB0cnkgc29sdmUgdGhl IHNhbWUgcHJvYmxlbS5BbG9uZSB3YXMgYSB2ZXJ5IGRpZmZpY3VsdCwgYnV0IHNvbHZlZCBpdC5U cnkgdGhpcy7CoFJlbW92ZSBhbGwgZHJtIGFuZCBjbGVhci5Ucnkgc3RlcHMgYmVsb3cuSW5zdGFs bCB0aGlzcGtnIGlucyBudmlkaWEteGNvbmZpZ3BrZyBpbnMgbnZpZGlhLWRyaXZlciAtIHlvdSBj YXJkIGRyaXZlciBzZWxlY3Rwa2cgaW5zIG52aWRpYS1kcm0gLSB1c2UgcGtnIHNlYXJjaCBhbmQg c2VsZWN0IHlvdXIgdmFyaWFudCB2ZXJzaW9uQnV0IGRybSBub3QgbmVlZGVkIGZvciBudmlkaWFB ZnRlciBpbnN0YWxudmlkaWEteGNvbmZpZyAtIHdpbGwgY3JlYXRlIFhvcmcgY29uZiBmaWxlU3Rh cnQgYW5kIGxvYWTCoGRidXNrbGRsb2FkIG52aWRpYVRyeSB4ZmNlIG9yIEtERUJlc3QsU2ltb24g KS0gLSAtCi0tLS0tLS0tINCY0YHRhdC+0LTQvdC+0LUg0YHQvtC+0LHRidC10L3QuNC1IC0tLS0t LS0t0J7RgjogTWFyaW8gTWFyaWV0dG8gPG1hcmlldHRvMjAwOEBnbWFpbC5jb20+INCU0LDRgtCw OiAyNC4wMi4yMDI0ICAxNjozOCAgKEdNVCswMzowMCkg0JrQvtC80YM6IEZyZWVCU0QgdmlydHVh bGl6YXRpb24gPGZyZWVic2QtdmlydHVhbGl6YXRpb25AZnJlZWJzZC5vcmc+LCBGcmVlQlNEIE1h aWxpbmcgTGlzdCA8ZnJlZWJzZC1xdWVzdGlvbnNAZnJlZWJzZC5vcmc+LCBmcmVlYnNkLWhhY2tl cnMgPGZyZWVic2QtaGFja2Vyc0BmcmVlYnNkLm9yZz4sIGZyZWVic2QteDExQGZyZWVic2Qub3Jn LCBmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmcsIFJvbmFsZCBLbG9wIDxyb25hbGRAZnJlZWJz ZC5vcmc+LCBKYWtlIEZyZWVsYW5kIDxqYWtlQHRlY2hub2xvZ3lmcmllbmRzLm5ldD4g0KLQtdC8 0LA6IGRybS1rbW9kLTIwMjIwOTA3XzIgbm90IHN1cHBvcnRlZCBmb3IgdGhpcyBjb25maWd1cmF0 aW9uIC8gTlZJRElBIDogRmFpbGVkIHRvIGluaXRpYWxpemUgdGhlIE5WSURJQSBrZXJuZWwgbW9k dWxlIEhlbGxvIHRvIGV2ZXJ5b25lLkkgbmVlZCB0byBjb25kdWN0IHNvbWUgdGVzdHMgb24gdG9w IG9mIHRoZSBGcmVlQlNEIDEzLjEuU28sSSd2ZSBpbnN0YWxsZWQgaXQgb24gb25lIG9mIG15IGRp c2tzLiBJJ3ZlIGluc3RhbGxlZCB4ZmNlNCBhbmQgS0RFLFhvcmcgYW5kIGZyb20gdGhlIHBvcnRz IChhZnRlciBoYXZpbmcgdXBncmFkZWQgdGhlbSksSSd2ZSBpbnN0YWxsZWQgdGhlIG52aWRpYS1k cml2ZXIgdmVyc2lvbi4gNTM1LjE0Ni4wMiA7IEkgZGlkbid0IGhhdmUgYW55IHByb2JsZW0uwqBU aGUgcHJvYmxlbSBhcmlzZXMgd2hlbiBJIHRyaWVkIHRvIGluc3RhbGwgdGhlIHBhY2thZ2UgZHJt LWttb2QgZnJvbSBwb3J0cy7CoFVuZm9ydHVuYXRlbHkgSSdtIG5vdCBhYmxlIHRvIGNvbXBpbGUg aXQuIFRoaXMgaXMgd2hhdCBoYXBwZW5lZCA6cm9vdEBtYXJpZXR0bzovdXNyL3BvcnRzL2dyYXBo aWNzL2RybS1rbW9kICMgbWFrZT09PT0+IGRybS1rbW9kLTIwMjIwOTA3XzIgbm90IHN1cHBvcnRl ZCBmb3IgdGhpcyBjb25maWd1cmF0aW9uLm9rLiBBdCB0aGlzIHBvaW50IEkgdHJpZWQgdG8gaW5z dGFsbCBpdCBmcm9tIHRoZSBwYWNrYWdlcyA6cm9vdEBtYXJpZXR0bzovdXNyL3BvcnRzL2dyYXBo aWNzL2RybS1rbW9kICMgbWFrZSBjbGVhbj09PT0+IGNsZWFuaW5nIGZvciBkcm0ta21vZC0yMDIy MDkwN18yc28gOnJvb3RAbWFyaWV0dG86L3Vzci9wb3J0cy9ncmFwaGljcy9kcm0ta21vZCAjIHBr ZyBpbnN0YWxsIGRybS1rbW9kTmV3IHBhY2thZ2VzIHRvIGJlIElOU1RBTExFRCA6ZHJtLWttb2Q6 IDIwMjIwOTA3XzJPS25vdyxJIGhhdmUgcGVyZm9ybWVkIGFkZGl0aW9uYWwgY29uZmlndXJhdGlv biB0byA6bmFubyAvaG9tZS9tYXJpZXR0by8ueGluaXRyYyA6ZXhlYyBjay1sYXVuY2gtc2Vzc2lv biBkYnVzLWxhdW5jaCAtLWV4aXQtd2l0aC1zZXNzaW9uIHN0YXJ0eGZjZTRuYW5vIC9ldGMvcmMu Y29uZiA6a2RtNV9lbmFibGU9IllFUyJkYnVzX2VuYWJsZT0iWUVTImhhbGRfZW5hYmxlPSJZRVMi a2xkX2xpc3Q9Im52aWRpYSBudmlkaWEtbW9kZXNldCJycGNiaW5kX2VuYWJsZT0iWUVTImR0Y21z X2VuYWJsZT0iWUVTImluZXRkX2VuYWJsZT0iWUVTIm5hbm8gL2Jvb3QvbG9hZGVyLmNvbmZ2bW1f bG9hZD0iWUVTIm5tZG1fbG9hZD0iWUVTInRtcGZzX2xvYWQ9IllFUyJjcnlwdG9kZXZfbG9hZD0i WUVTInpmc19sb2FkPSJZRVMia2Vybi5yYWNjdC5lbmFibGU9IjEia2Vybi52dHk9dnRrZXJuLmNh bS5zY3NpX2RlbGF5PSIxMDAwMCJuYW5vIC9ldGMvWDExL3hvcmcuY29uZlNlY3Rpb24gIkRldmlj ZSLCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIElkZW50aWZpZXLCoMKgICJDYXJkMCLCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIERyaXZlcsKgwqDCoMKgwqDCoMKgwqAgIm52aWRpYSLC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIEJ1c0lEwqDCoMKgwqDCoMKgwqDCoMKgICJQQ0k6 MTowOjAiRW5kU2VjdGlvbnJvb3RAbWFyaWV0dG86L2hvbWUvbWFyaWV0dG8gIyBsc3BjaTAxOjAw LjAgTlZJRElBIEdQMTA2MDE6MDAuMSBOVklESUEgR1AxMDYgSGlnaCBEZWZpbml0aW9uIEF1ZGlv IENvbnRyb2xsZXIwMjowMC4wIE5WSURJQSBUVTEwMjAyOjAwLjEgTlZJRElBIFRVMTAyIEhpZ2gg RGVmaW5pdGlvbiBBdWRpbyBDb250cm9sbGVyMDI6MDAuMiBOVklESUEgVFUxMDIgVVNCIENvbnRy b2xsZXIwMjowMC4zIE5WSURJQSBUVTEwMiBTZXJpYWwgQlVTIENvbnRyb2xsZXJOb3csSSB3YW50 IHRvIHN0YXJ0eCB0byBzdGFydCB4ZmNlNCA6bWFyaWV0dG9AbWFyaWV0dG86ICQgc3RhcnR4RmF0 YWwgc2VydmVyIGVycm9yIDogbm8gc2NyZWVucyBmb3VuZCAoRUUpQ2hlY2sgdGhlIGxvZyBmaWxl IGF0ICIvdmFyL2xvZy9Yb3JnLjAubG9nIm5hbm8gL3Zhci9sb2cvWG9yZy4wLmxvZyA6TlZJRElB IGRsbG9hZGVyIFggZHJpdmVyIDUzNS4xNDYuMDJOVklESUEgVW5pZmllZCBEcml2ZXIgZm9yIGFs bCBTdXBwb3J0ZWQgTlZJRElBIGdwdXNOVklESUEgOiBGYWlsZWQgdG8gaW5pdGlhbGl6ZSB0aGUg TlZJRElBIGtlcm5lbCBtb2R1bGVIb3cgdG8gZml4IGl0ID8tLSBNYXJpby4K ----_com.samsung.android.email_7568670750559100 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPjwvaGVhZD48Ym9keSBkaXI9ImF1dG8iPjxkaXYgZGlyPSJh dXRvIj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8iPkhlbGxvIE1hcmlvLjwvZGl2PjxkaXYgZGly PSJhdXRvIj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8iPkkgaGF2ZSBwcm9ibGVtIHRvby48L2Rp dj48ZGl2IGRpcj0iYXV0byI+QSBmZXcgZGF5cyBpIHRyeSBzb2x2ZSB0aGUgc2FtZSBwcm9ibGVt LjwvZGl2PjxkaXYgZGlyPSJhdXRvIj5BbG9uZSB3YXMgYSB2ZXJ5IGRpZmZpY3VsdCwgYnV0IHNv bHZlZCBpdC48L2Rpdj48ZGl2IGRpcj0iYXV0byI+PGJyPjwvZGl2PjxkaXYgZGlyPSJhdXRvIj5U cnkgdGhpcy4mbmJzcDs8L2Rpdj48ZGl2IGRpcj0iYXV0byI+PGJyPjwvZGl2PjxkaXYgZGlyPSJh dXRvIj5SZW1vdmUgYWxsIGRybSBhbmQgY2xlYXIuPC9kaXY+PGRpdiBkaXI9ImF1dG8iPjxicj48 L2Rpdj48ZGl2IGRpcj0iYXV0byI+PGJyPjwvZGl2PjxkaXYgZGlyPSJhdXRvIj5Ucnkgc3RlcHMg YmVsb3cuPC9kaXY+PGRpdiBkaXI9ImF1dG8iPjxicj48L2Rpdj48ZGl2IGRpcj0iYXV0byI+SW5z dGFsbCB0aGlzPC9kaXY+PGRpdiBkaXI9ImF1dG8iPnBrZyBpbnMgbnZpZGlhLXhjb25maWc8L2Rp dj48ZGl2IGRpcj0iYXV0byI+PGJyPjwvZGl2PjxkaXYgZGlyPSJhdXRvIj5wa2cgaW5zIG52aWRp YS1kcml2ZXIgLSB5b3UgY2FyZCBkcml2ZXIgc2VsZWN0PC9kaXY+PGRpdiBkaXI9ImF1dG8iPjxi cj48L2Rpdj48ZGl2IGRpcj0iYXV0byI+cGtnIGlucyBudmlkaWEtZHJtIC0gdXNlIHBrZyBzZWFy Y2ggYW5kIHNlbGVjdCB5b3VyIHZhcmlhbnQgdmVyc2lvbjwvZGl2PjxkaXYgZGlyPSJhdXRvIj5C dXQgZHJtIG5vdCBuZWVkZWQgZm9yIG52aWRpYTwvZGl2PjxkaXYgZGlyPSJhdXRvIj48YnI+PC9k aXY+PGRpdiBkaXI9ImF1dG8iPkFmdGVyIGluc3RhbDwvZGl2PjxkaXYgZGlyPSJhdXRvIj5udmlk aWEteGNvbmZpZyAtIHdpbGwgY3JlYXRlIFhvcmcgY29uZiBmaWxlPC9kaXY+PGRpdiBkaXI9ImF1 dG8iPjxicj48L2Rpdj48ZGl2IGRpcj0iYXV0byI+U3RhcnQgYW5kIGxvYWQmbmJzcDs8L2Rpdj48 ZGl2IGRpcj0iYXV0byI+ZGJ1czwvZGl2PjxkaXYgZGlyPSJhdXRvIj5rbGRsb2FkIG52aWRpYTwv ZGl2PjxkaXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8iPlRyeSB4ZmNlIG9y IEtERTwvZGl2PjxkaXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8iPkJlc3Qs PC9kaXY+PGRpdiBkaXI9ImF1dG8iPlNpbW9uICk8L2Rpdj48ZGl2IGRpcj0iYXV0byI+PGJyPjwv ZGl2PjxkaXYgZGlyPSJhdXRvIiBpZD0iY29tcG9zZXJfc2lnbmF0dXJlIj4tIC0gLTwvZGl2Pjxk aXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8iIHN0 eWxlPSJmb250LXNpemU6MTAwJTtjb2xvcjojMDAwMDAwIj48L2Rpdj48ZGl2IHN0eWxlPSJmb250 LXNpemU6MTAwJTtjb2xvcjojMDAwMDAwIiBkaXI9ImF1dG8iIGFsaWduPSJsZWZ0Ij48ZGl2Pi0t LS0tLS0tINCY0YHRhdC+0LTQvdC+0LUg0YHQvtC+0LHRidC10L3QuNC1IC0tLS0tLS0tPC9kaXY+ PGRpdj7QntGCOiBNYXJpbyBNYXJpZXR0byAmbHQ7bWFyaWV0dG8yMDA4QGdtYWlsLmNvbSZndDsg PC9kaXY+PGRpdj7QlNCw0YLQsDogMjQuMDIuMjAyNCAgMTY6MzggIChHTVQrMDM6MDApIDwvZGl2 PjxkaXY+0JrQvtC80YM6IEZyZWVCU0QgdmlydHVhbGl6YXRpb24gJmx0O2ZyZWVic2QtdmlydHVh bGl6YXRpb25AZnJlZWJzZC5vcmcmZ3Q7LCBGcmVlQlNEIE1haWxpbmcgTGlzdCAmbHQ7ZnJlZWJz ZC1xdWVzdGlvbnNAZnJlZWJzZC5vcmcmZ3Q7LCBmcmVlYnNkLWhhY2tlcnMgJmx0O2ZyZWVic2Qt aGFja2Vyc0BmcmVlYnNkLm9yZyZndDssIGZyZWVic2QteDExQGZyZWVic2Qub3JnLCBmcmVlYnNk LWRyaXZlcnNAZnJlZWJzZC5vcmcsIFJvbmFsZCBLbG9wICZsdDtyb25hbGRAZnJlZWJzZC5vcmcm Z3Q7LCBKYWtlIEZyZWVsYW5kICZsdDtqYWtlQHRlY2hub2xvZ3lmcmllbmRzLm5ldCZndDsgPC9k aXY+PGRpdj7QotC10LzQsDogZHJtLWttb2QtMjAyMjA5MDdfMiBub3Qgc3VwcG9ydGVkIGZvciB0 aGlzIGNvbmZpZ3VyYXRpb24gLyBOVklESUEgOiBGYWlsZWQgdG8gaW5pdGlhbGl6ZSB0aGUgTlZJ RElBIGtlcm5lbCBtb2R1bGUgPC9kaXY+PGRpdj48YnI+PC9kaXY+PC9kaXY+PGRpdiBkaXI9Imx0 ciI+PGRpdj5IZWxsbyB0byBldmVyeW9uZS48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkkgbmVl ZCB0byBjb25kdWN0IHNvbWUgdGVzdHMgb24gdG9wIG9mIHRoZSBGcmVlQlNEIDEzLjEuPC9kaXY+ PGRpdj5TbyxJJ3ZlIGluc3RhbGxlZCBpdCBvbiBvbmUgb2YgbXkgZGlza3MuIEkndmUgaW5zdGFs bGVkIHhmY2U0IGFuZCBLREUsWG9yZyBhbmQgZnJvbSB0aGUgcG9ydHMgKGFmdGVyIGhhdmluZyB1 cGdyYWRlZCB0aGVtKSxJJ3ZlIGluc3RhbGxlZCB0aGUgbnZpZGlhLWRyaXZlciB2ZXJzaW9uLiA1 MzUuMTQ2LjAyIDsgSSBkaWRuJ3QgaGF2ZSBhbnkgcHJvYmxlbS4mbmJzcDs8L2Rpdj48ZGl2Pjxi cj48L2Rpdj48ZGl2PlRoZSBwcm9ibGVtIGFyaXNlcyB3aGVuIEkgdHJpZWQgdG8gaW5zdGFsbCB0 aGUgcGFja2FnZSBkcm0ta21vZCBmcm9tIHBvcnRzLiZuYnNwOzwvZGl2PjxkaXY+PGJyPjwvZGl2 PjxkaXY+VW5mb3J0dW5hdGVseSBJJ20gbm90IGFibGUgdG8gY29tcGlsZSBpdC4gVGhpcyBpcyB3 aGF0IGhhcHBlbmVkIDo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PnJvb3RAbWFyaWV0dG86L3Vz ci9wb3J0cy9ncmFwaGljcy9kcm0ta21vZCAjIG1ha2U8L2Rpdj48ZGl2Pj09PT0mZ3Q7IGRybS1r bW9kLTIwMjIwOTA3XzIgbm90IHN1cHBvcnRlZCBmb3IgdGhpcyBjb25maWd1cmF0aW9uLjwvZGl2 PjxkaXY+PGJyPjwvZGl2PjxkaXY+b2suIEF0IHRoaXMgcG9pbnQgSSB0cmllZCB0byBpbnN0YWxs IGl0IGZyb20gdGhlIHBhY2thZ2VzIDo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjxkaXY+cm9v dEBtYXJpZXR0bzovdXNyL3BvcnRzL2dyYXBoaWNzL2RybS1rbW9kICMgbWFrZSBjbGVhbjwvZGl2 PjxkaXY+PT09PSZndDsgY2xlYW5pbmcgZm9yIGRybS1rbW9kLTIwMjIwOTA3XzI8L2Rpdj48ZGl2 Pjxicj48L2Rpdj48ZGl2PnNvIDo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjxkaXY+cm9vdEBt YXJpZXR0bzovdXNyL3BvcnRzL2dyYXBoaWNzL2RybS1rbW9kICMgcGtnIGluc3RhbGwgZHJtLWtt b2Q8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pk5ldyBwYWNrYWdlcyB0byBiZSBJTlNUQUxMRUQg OjwvZGl2PjxkaXY+ZHJtLWttb2Q6IDIwMjIwOTA3XzI8L2Rpdj48ZGl2Pk9LPC9kaXY+PGRpdj48 YnI+PC9kaXY+PGRpdj5ub3csSSBoYXZlIHBlcmZvcm1lZCBhZGRpdGlvbmFsIGNvbmZpZ3VyYXRp b24gdG8gOjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+bmFubyAvaG9tZS9tYXJpZXR0by8ueGlu aXRyYyA6PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5leGVjIGNrLWxhdW5jaC1zZXNzaW9uIGRi dXMtbGF1bmNoIC0tZXhpdC13aXRoLXNlc3Npb24gc3RhcnR4ZmNlNDwvZGl2PjxkaXY+PGJyPjwv ZGl2PjxkaXY+bmFubyAvZXRjL3JjLmNvbmYgOjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+a2Rt NV9lbmFibGU9IllFUyI8L2Rpdj48ZGl2PmRidXNfZW5hYmxlPSJZRVMiPC9kaXY+PGRpdj5oYWxk X2VuYWJsZT0iWUVTIjwvZGl2PjxkaXY+a2xkX2xpc3Q9Im52aWRpYSBudmlkaWEtbW9kZXNldCI8 L2Rpdj48ZGl2PnJwY2JpbmRfZW5hYmxlPSJZRVMiPC9kaXY+PGRpdj5kdGNtc19lbmFibGU9IllF UyI8L2Rpdj48ZGl2PmluZXRkX2VuYWJsZT0iWUVTIjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+ PC9kaXY+PGRpdj5uYW5vIC9ib290L2xvYWRlci5jb25mPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRp dj52bW1fbG9hZD0iWUVTIjwvZGl2PjxkaXY+bm1kbV9sb2FkPSJZRVMiPC9kaXY+PGRpdj50bXBm c19sb2FkPSJZRVMiPC9kaXY+PGRpdj5jcnlwdG9kZXZfbG9hZD0iWUVTIjwvZGl2PjxkaXY+emZz X2xvYWQ9IllFUyI8L2Rpdj48ZGl2Pmtlcm4ucmFjY3QuZW5hYmxlPSIxIjwvZGl2PjxkaXY+a2Vy bi52dHk9dnQ8L2Rpdj48ZGl2Pmtlcm4uY2FtLnNjc2lfZGVsYXk9IjEwMDAwIjwvZGl2PjxkaXY+ PGJyPjwvZGl2PjxkaXY+bmFubyAvZXRjL1gxMS94b3JnLmNvbmY8L2Rpdj48ZGl2Pjxicj48L2Rp dj48ZGl2PlNlY3Rpb24gIkRldmljZSI8L2Rpdj48ZGl2PiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyBJZGVudGlmaWVyJm5ic3A7Jm5ic3A7ICJDYXJkMCI8L2Rpdj48ZGl2PiZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyBEcml2ZXImbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsgIm52aWRpYSI8L2Rpdj48ZGl2PiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyBCdXNJRCZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyAiUENJOjE6MDowIjwvZGl2PjxkaXY+RW5kU2VjdGlvbjwvZGl2PjxkaXY+PGJyPjwv ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PGRpdj5yb290QG1hcmlldHRvOi9ob21lL21hcmlldHRv ICMgbHNwY2k8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjAxOjAwLjAgTlZJRElBIEdQMTA2PC9k aXY+PGRpdj4wMTowMC4xIE5WSURJQSBHUDEwNiBIaWdoIERlZmluaXRpb24gQXVkaW8gQ29udHJv bGxlcjwvZGl2PjxkaXY+MDI6MDAuMCBOVklESUEgVFUxMDI8L2Rpdj48ZGl2PjAyOjAwLjEgTlZJ RElBIFRVMTAyIEhpZ2ggRGVmaW5pdGlvbiBBdWRpbyBDb250cm9sbGVyPGRpdj4wMjowMC4yIE5W SURJQSBUVTEwMiBVU0IgQ29udHJvbGxlcjwvZGl2PjxkaXY+MDI6MDAuMyBOVklESUEgVFUxMDIg U2VyaWFsIEJVUyBDb250cm9sbGVyPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Ob3csSSB3YW50 IHRvIHN0YXJ0eCB0byBzdGFydCB4ZmNlNCA6PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5tYXJp ZXR0b0BtYXJpZXR0bzogJCBzdGFydHg8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkZhdGFsIHNl cnZlciBlcnJvciA6IG5vIHNjcmVlbnMgZm91bmQgKEVFKTwvZGl2PjxkaXY+Q2hlY2sgdGhlIGxv ZyBmaWxlIGF0ICIvdmFyL2xvZy9Yb3JnLjAubG9nIjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+ bmFubyAvdmFyL2xvZy9Yb3JnLjAubG9nIDo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pk5WSURJ QSBkbGxvYWRlciBYIGRyaXZlciA1MzUuMTQ2LjAyPC9kaXY+PGRpdj5OVklESUEgVW5pZmllZCBE cml2ZXIgZm9yIGFsbCBTdXBwb3J0ZWQgTlZJRElBIGdwdXM8L2Rpdj48ZGl2Pk5WSURJQSA6IEZh aWxlZCB0byBpbml0aWFsaXplIHRoZSBOVklESUEga2VybmVsIG1vZHVsZTxicj48L2Rpdj48L2Rp dj48L2Rpdj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48L2Rpdj48ZGl2PkhvdyB0byBmaXggaXQgPzwv ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PHNwYW4gY2xhc3M9ImdtYWlsX3NpZ25hdHVyZV9wcmVm aXgiPi0tIDwvc3Bhbj48YnI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSIg ZGF0YS1zbWFydG1haWw9ImdtYWlsX3NpZ25hdHVyZSI+TWFyaW8uPGJyPjwvZGl2PjwvZGl2Pjwv ZGl2Pgo8L2JvZHk+PC9odG1sPg== ----_com.samsung.android.email_7568670750559100-- From nobody Sat Feb 24 13:46:58 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThqVK13XCz59w1Z; Sat, 24 Feb 2024 14:47:37 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThqVJ5vBsz4F2l; Sat, 24 Feb 2024 14:47:36 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-a3d484a58f6so247027066b.3; Sat, 24 Feb 2024 06:47:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708786055; x=1709390855; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=iGw3w3Fi4+6nJndBWEVb8v4HvFP6ZzSPMTQAa9roF+4=; b=HRfDgZTMOoO9WQr3wd94rRqKFsE9t8ZXCJ3yIFm6p7rVs9U6j4+OmoDuDBYIahAKoa 3p8zlezWVgYBDOLGLpii6S+d/6/CuXvZb2obZwgXbSIA9m/gFfkH9f5j8ighnDdme4cV SmpXXugDLLE8D+XXDqUt6OcgwJQvuLQs2sjT2b0oZgc4hBCcHTJZ10TzknQlIg0ErgXF NARr8Eh7LpeMwxpLAIlVbGs/8cKPs6UeBXSxOtx8wx0LUGMIcucZrbxqYSwsXSZzKbCD a1LCu14TRpG8FKEqXIGjhzPEOIX+RlD1Ojrq9qA1nCFWKDMWSQQdf3yv9Clkz/Qmu8WM bxvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708786055; x=1709390855; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=iGw3w3Fi4+6nJndBWEVb8v4HvFP6ZzSPMTQAa9roF+4=; b=SmwYsKCq/RGDkYROMpzaRRc0J1zu1HqZGvQ/vYSRit+2LL486yqlF3pYNpRi+H60DL 1llEz7zbAa0KvxNE/SY/NG170mxT5J/W0S67QfMeLRzD8fYiwp+5wlInWFlFCi+87Ox1 pP8kFsLW7UE+z6fLs/S0Heo3gyFmFiLcQJfT5+CmpdN5OU84jpFJUOf6Q2LamNSZi4ZD 3W/O5lfp++R07Zz+qTkSE+RPHojxjCE6eDgjaWlOiimdIMmtuB13iXzDhmI8bGXAjS8Z 5BPO6GVt27TtHuDTW3WJcE7XF3KeTEwWnc1DYeAcwzQlMYsNSeYGU73EM4Yn4hBg4xkz 2blw== X-Forwarded-Encrypted: i=1; AJvYcCVKFFbAQoXVbAsNeHJxWnapwwPDp06p4zKwlgWn3FwgcYKGybU2jnS899YDXpA+DbkxuwLCiKZE2oDHEV1ppGqZr6QpM8dUT0o85skLR74PBQzW1gd6Lco3I31PUTcQbZ9wPW32NIhMVhrEouWsS+DTl5MlQSHNonNl5hzFBXS++wD1c0m4leJ16U8IRyxRn97hbaxnOxftg+gqzjzihh0K3Q+FgLswjDzFbi0wpRtTgy2IN9I= X-Gm-Message-State: AOJu0YxKByiOR3jUeJzD9Ttg5rbF3V1F8yFRPh8z+dqmt2WF4bZsvTY2 wutpmU1laBi0IviFv4OrQHSQ/BQCndyOdfWjE+HhD4NkOCImRXnoG51Afy/9dtlDprUtClIIZxW k8VlBUnNt9dqZ3GjhE6i/fa3gUq8= X-Google-Smtp-Source: AGHT+IGFprjIxVIZUeHogwOO6ZwzK0D53MOp3txcH5BJtwSjdJ9I33b5UzcDxszH82YEgYRYo/hUuJtzYAAFp1Spztk= X-Received: by 2002:a17:906:ce30:b0:a3f:bd94:4d80 with SMTP id sd16-20020a170906ce3000b00a3fbd944d80mr1698545ejb.76.1708786054908; Sat, 24 Feb 2024 06:47:34 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <20240224140856.2B99B49712@mail.fly777.net> In-Reply-To: <20240224140856.2B99B49712@mail.fly777.net> From: Mario Marietto Date: Sat, 24 Feb 2024 14:46:58 +0100 Message-ID: Subject: Re: drm-kmod-20220907_2 not supported for this configuration / NVIDIA : Failed to initialize the NVIDIA kernel module To: Simon Gusev Cc: FreeBSD virtualization , FreeBSD Mailing List , freebsd-hackers , freebsd-x11@freebsd.org, freebsd-drivers@freebsd.org, Ronald Klop , Jake Freeland Content-Type: multipart/alternative; boundary="000000000000ccd18d061221c1c1" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4ThqVJ5vBsz4F2l --000000000000ccd18d061221c1c1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable :( KLD nvidia.ko : depends on kernel. Not available or version mismatch linker_load_file : /boot/modules/nvidia.ko : unsupported file type. Can you give me your "nvidia.ko" file ? thanks. On Sat, Feb 24, 2024 at 3:09=E2=80=AFPM Simon Gusev wro= te: > > Hello Mario. > > I have problem too. > A few days i try solve the same problem. > Alone was a very difficult, but solved it. > > Try this. > > Remove all drm and clear. > > > Try steps below. > > Install this > pkg ins nvidia-xconfig > > pkg ins nvidia-driver - you card driver select > > pkg ins nvidia-drm - use pkg search and select your variant version > But drm not needed for nvidia > > After instal > nvidia-xconfig - will create Xorg conf file > > Start and load > dbus > kldload nvidia > > Try xfce or KDE > > Best, > Simon ) > > - - - > > > -------- =D0=98=D1=81=D1=85=D0=BE=D0=B4=D0=BD=D0=BE=D0=B5 =D1=81=D0=BE=D0= =BE=D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D0=B5 -------- > =D0=9E=D1=82: Mario Marietto > =D0=94=D0=B0=D1=82=D0=B0: 24.02.2024 16:38 (GMT+03:00) > =D0=9A=D0=BE=D0=BC=D1=83: FreeBSD virtualization , > FreeBSD Mailing List , freebsd-hackers < > freebsd-hackers@freebsd.org>, freebsd-x11@freebsd.org, > freebsd-drivers@freebsd.org, Ronald Klop , Jake > Freeland > =D0=A2=D0=B5=D0=BC=D0=B0: drm-kmod-20220907_2 not supported for this conf= iguration / NVIDIA : > Failed to initialize the NVIDIA kernel module > > Hello to everyone. > > I need to conduct some tests on top of the FreeBSD 13.1. > So,I've installed it on one of my disks. I've installed xfce4 and KDE,Xor= g > and from the ports (after having upgraded them),I've installed the > nvidia-driver version. 535.146.02 ; I didn't have any problem. > > The problem arises when I tried to install the package drm-kmod from > ports. > > Unfortunately I'm not able to compile it. This is what happened : > > root@marietto:/usr/ports/graphics/drm-kmod # make > =3D=3D=3D=3D> drm-kmod-20220907_2 not supported for this configuration. > > ok. At this point I tried to install it from the packages : > > root@marietto:/usr/ports/graphics/drm-kmod # make clean > =3D=3D=3D=3D> cleaning for drm-kmod-20220907_2 > > so : > > root@marietto:/usr/ports/graphics/drm-kmod # pkg install drm-kmod > > New packages to be INSTALLED : > drm-kmod: 20220907_2 > OK > > now,I have performed additional configuration to : > > nano /home/marietto/.xinitrc : > > exec ck-launch-session dbus-launch --exit-with-session startxfce4 > > nano /etc/rc.conf : > > kdm5_enable=3D"YES" > dbus_enable=3D"YES" > hald_enable=3D"YES" > kld_list=3D"nvidia nvidia-modeset" > rpcbind_enable=3D"YES" > dtcms_enable=3D"YES" > inetd_enable=3D"YES" > > nano /boot/loader.conf > > vmm_load=3D"YES" > nmdm_load=3D"YES" > tmpfs_load=3D"YES" > cryptodev_load=3D"YES" > zfs_load=3D"YES" > kern.racct.enable=3D"1" > kern.vty=3Dvt > kern.cam.scsi_delay=3D"10000" > > nano /etc/X11/xorg.conf > > Section "Device" > Identifier "Card0" > Driver "nvidia" > BusID "PCI:1:0:0" > EndSection > > > root@marietto:/home/marietto # lspci > > 01:00.0 NVIDIA GP106 > 01:00.1 NVIDIA GP106 High Definition Audio Controller > 02:00.0 NVIDIA TU102 > 02:00.1 NVIDIA TU102 High Definition Audio Controller > 02:00.2 NVIDIA TU102 USB Controller > 02:00.3 NVIDIA TU102 Serial BUS Controller > > Now,I want to startx to start xfce4 : > > marietto@marietto: $ startx > > Fatal server error : no screens found (EE) > Check the log file at "/var/log/Xorg.0.log" > > nano /var/log/Xorg.0.log : > > NVIDIA dlloader X driver 535.146.02 > NVIDIA Unified Driver for all Supported NVIDIA gpus > NVIDIA : Failed to initialize the NVIDIA kernel module > > How to fix it ? > > -- > Mario. > --=20 Mario. --000000000000ccd18d061221c1c1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
:(

KLD nvidia.ko : depends o= n kernel. Not available or version mismatch
linker_load_file : /b= oot/modules/nvidia.ko : unsupported file type.

Can= you give me your "nvidia.ko" file ? thanks.

<= div class=3D"gmail_quote">
On Sat, Feb= 24, 2024 at 3:09=E2=80=AFPM Simon Gusev <semfly@fly777.net> wrote:

Hello Mario.

I have problem too.
A few days i try solve the sam= e problem.
Alone was a very difficult, but solved it= .

Try this.=C2=A0
<= div dir=3D"auto">
Remove all drm and clear.


= Try steps below.

Install= this
pkg ins nvidia-xconfig
=
pkg ins nvidia-driver - you card driver select<= /div>

pkg ins nvidia-drm - use= pkg search and select your variant version
But drm = not needed for nvidia

Af= ter instal
nvidia-xconfig - will create Xorg conf fi= le

Start and load=C2=A0<= /div>
dbus
kldload nvidia

Try xfce or KDE

Best,
Simon )

- - -


<= div dir=3D"auto" style=3D"font-size:100%;color:rgb(0,0,0)">
-----= --- =D0=98=D1=81=D1=85=D0=BE=D0=B4=D0=BD=D0=BE=D0=B5 =D1=81=D0=BE=D0=BE=D0= =B1=D1=89=D0=B5=D0=BD=D0=B8=D0=B5 --------
=D0=9E=D1=82: Mario Ma= rietto <mari= etto2008@gmail.com>
=D0=94=D0=B0=D1=82=D0=B0: 24.02.2024 = 16:38 (GMT+03:00)
=D0=9A=D0=BE=D0=BC=D1=83: FreeBSD virtualiza= tion <freebsd-virtualization@freebsd.org>, FreeBSD Mailing List <= freebsd-= questions@freebsd.org>, freebsd-hackers <freebsd-hackers@freebsd.org>= ;, freebsd-x11= @freebsd.org, freebsd-drivers@freebsd.org, Ronald Klop <ronald@freebsd.org>, Jake Free= land <ja= ke@technologyfriends.net>
=D0=A2=D0=B5=D0=BC=D0=B0: drm-k= mod-20220907_2 not supported for this configuration / NVIDIA : Failed to in= itialize the NVIDIA kernel module

Hello to everyone.

I need to conduct some = tests on top of the FreeBSD 13.1.
So,I've installed it on one= of my disks. I've installed xfce4 and KDE,Xorg and from the ports (aft= er having upgraded them),I've installed the nvidia-driver version. 535.= 146.02 ; I didn't have any problem.=C2=A0

The = problem arises when I tried to install the package drm-kmod from ports.=C2= =A0

Unfortunately I'm not able to compile it. = This is what happened :

root@marietto:/usr/ports/g= raphics/drm-kmod # make
=3D=3D=3D=3D> drm-kmod-20220907_2 not = supported for this configuration.

ok. At this poin= t I tried to install it from the packages :

r= oot@marietto:/usr/ports/graphics/drm-kmod # make clean
=3D=3D=3D= =3D> cleaning for drm-kmod-20220907_2

so :

root@marietto:/usr/ports/graphics/drm-kmod # pkg = install drm-kmod

New packages to be INSTALLED :
drm-kmod: 20220907_2
OK

now,I ha= ve performed additional configuration to :

nano /h= ome/marietto/.xinitrc :

exec ck-launch-session dbu= s-launch --exit-with-session startxfce4

nano /etc/= rc.conf :

kdm5_enable=3D"YES"
= dbus_enable=3D"YES"
hald_enable=3D"YES"
=
kld_list=3D"nvidia nvidia-modeset"
rpcbind_enable= =3D"YES"
dtcms_enable=3D"YES"
inetd= _enable=3D"YES"

nano /boot/lo= ader.conf

vmm_load=3D"YES"
nmd= m_load=3D"YES"
tmpfs_load=3D"YES"
c= ryptodev_load=3D"YES"
zfs_load=3D"YES"
<= div>kern.racct.enable=3D"1"
kern.vty=3Dvt
ker= n.cam.scsi_delay=3D"10000"

nano /etc/X11= /xorg.conf

Section "Device"
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Identifier=C2=A0=C2=A0 "Card0"
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Dr= iver=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "nvidia"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 BusID=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 "PCI:1:0:0"
EndSection

root@marietto:/home/marietto # lspci

<= div>01:00.0 NVIDIA GP106
01:00.1 NVIDIA GP106 High Definition Aud= io Controller
02:00.0 NVIDIA TU102
02:00.1 NVIDIA TU102= High Definition Audio Controller
02:00.2 NVIDIA TU102 USB Controller
02:00.3 NVIDIA TU102 Serial BUS Controller

Now,I want to startx to start xfce4 :

marietto@m= arietto: $ startx

Fatal server error : no screens = found (EE)
Check the log file at "/var/log/Xorg.0.log"<= /div>

nano /var/log/Xorg.0.log :

NVIDIA dlloader X driver 535.146.02
NVIDIA Unified Driver for a= ll Supported NVIDIA gpus
NVIDIA : Failed to initialize the NVIDIA= kernel module

How to = fix it ?

--=
Mario.


--
Mario.<= br>
--000000000000ccd18d061221c1c1-- From nobody Sat Feb 24 14:02:10 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Thqqx10K4z59xGy; Sat, 24 Feb 2024 15:02:53 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Thqqw2fPFz4Kxx; Sat, 24 Feb 2024 15:02:52 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=jwXaDDNz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::535 as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-ed1-x535.google.com with SMTP id 4fb4d7f45d1cf-565b86c9943so328310a12.1; Sat, 24 Feb 2024 07:02:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708786966; x=1709391766; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pmsXL5fxizsP6epAqqXA7dm6NR4aK2TOWnRY1roMNnc=; b=jwXaDDNzao2IjtWD58zGBRJDuTzjppZ16MvfFtMNhyINO8tkBtmhXMDsjOuXMOMPEB B49pfeqYcbDz7dRSzQYlj9KNxcXL2TuO0IZ3iWqPKCptKJhhyKoPVTBRpRY+R9XDSGG3 CtkHgRUbYkFUTZU1eJPdBMrSybcFXMP22LU36PovW1snPJUdANGlAROB6Y0dYsLib7b1 qn3L+VaFH+zy+2anXm4ByEkvs74cGAHt+RrG0sblz7bnXwJMkF4etvwi7al36EI6DHQV Fl8JDKIusywHQaL1n3wKzQb8xZUtFQeuymVayWVB9U2WyAc4Pf3Bx7bGX6mdY6A1qh0Z YPQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708786966; x=1709391766; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pmsXL5fxizsP6epAqqXA7dm6NR4aK2TOWnRY1roMNnc=; b=VRbwrlV3BX7FUePMkDm4CR75Y+YD2KHC0oi29GD9CyX17m6cao/v2o1XdJsZ92A9iG UdWrv6kpO2ys63hE4sgoiuqESMjxgr9P4HCyFHJc5uiXxZLEUJ4JDUWe08/QOHCohOpg CsBwzufWbwsU/EpevYMVeevFlqDSiY/HaWuhZgLFnCsFbKHtwl8+yYHeAVWaPN7Nxbjn DqZwTe7H9gZwYXoja4+7aPeCQygfkNcNIBbNbVm9SiS46/sAYE4DGKmg4avO/k19tHAA XpbKnvMZLeCf1zxt0Kvs01x3TuIWRvyfTnTHVzENyFl/H0IB2Wdx7Hrcw9jXryJglavi jYqA== X-Forwarded-Encrypted: i=1; AJvYcCV8EggSefTroSMpZyFQMFbsirO9+nVqAowWHk1RpaStbOoeROfs5/1Z7W6iF+H81lB+lJNpX43dVlSA+cDiCmqtxBVAE4S12VHLdZfwcyHvGWPQJXEFsh7IHIjbZE4aS2tdOjagamsHn1D3/qGZ/fe2J98mSGs4yP8dt/cBmeL39Qtmgw453EeZuBJNaHmJPp+7oBPnoVkKHbTbIU0wglEWhLTJjBQc+1AcHVQiNsFgjsnJ0Bs= X-Gm-Message-State: AOJu0Yyqvf7+kpVuThHHkAP+G9qILGYlQFDoy/tKak//tuR+2QBBqDfC zix6/K/TXZNx7qMArlYdeZAHheLtqWIb65P9nZmrs8oZeeJg+HTdAKcsx9eA/5Yo5ZU1+MP5Jh8 fv7uRRtEJD5z3ke8+pEEp3P7wD6I= X-Google-Smtp-Source: AGHT+IEXF72P3W/11Sr+O15eO8Bw/5Iu76DfjGwY3E2qOi+334EmW80nbVjD1nQr0yrxX5S3s0N5duWAvVvSUeV6OZg= X-Received: by 2002:a17:906:4104:b0:a3e:8b8e:8795 with SMTP id j4-20020a170906410400b00a3e8b8e8795mr2212709ejk.34.1708786966174; Sat, 24 Feb 2024 07:02:46 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <20240224140856.2B99B49712@mail.fly777.net> In-Reply-To: From: Mario Marietto Date: Sat, 24 Feb 2024 15:02:10 +0100 Message-ID: Subject: Re: drm-kmod-20220907_2 not supported for this configuration / NVIDIA : Failed to initialize the NVIDIA kernel module To: Simon Gusev Cc: FreeBSD virtualization , FreeBSD Mailing List , freebsd-hackers , freebsd-x11@freebsd.org, freebsd-drivers@freebsd.org, Ronald Klop , Jake Freeland Content-Type: multipart/alternative; boundary="0000000000001d9ecc061221f8f1" X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; RCPT_COUNT_SEVEN(0.00)[8]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-questions@freebsd.org,freebsd-hackers@freebsd.org,freebsd-x11@freebsd.org,freebsd-drivers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::535:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+] X-Rspamd-Queue-Id: 4Thqqw2fPFz4Kxx --0000000000001d9ecc061221f8f1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable It persists installing nvidia-driver-535.146.02,but I suspect that it is too now for FreeBSD 13.1. On Sat, Feb 24, 2024 at 2:46=E2=80=AFPM Mario Marietto wrote: > :( > > KLD nvidia.ko : depends on kernel. Not available or version mismatch > linker_load_file : /boot/modules/nvidia.ko : unsupported file type. > > Can you give me your "nvidia.ko" file ? thanks. > > On Sat, Feb 24, 2024 at 3:09=E2=80=AFPM Simon Gusev w= rote: > >> >> Hello Mario. >> >> I have problem too. >> A few days i try solve the same problem. >> Alone was a very difficult, but solved it. >> >> Try this. >> >> Remove all drm and clear. >> >> >> Try steps below. >> >> Install this >> pkg ins nvidia-xconfig >> >> pkg ins nvidia-driver - you card driver select >> >> pkg ins nvidia-drm - use pkg search and select your variant version >> But drm not needed for nvidia >> >> After instal >> nvidia-xconfig - will create Xorg conf file >> >> Start and load >> dbus >> kldload nvidia >> >> Try xfce or KDE >> >> Best, >> Simon ) >> >> - - - >> >> >> -------- =D0=98=D1=81=D1=85=D0=BE=D0=B4=D0=BD=D0=BE=D0=B5 =D1=81=D0=BE= =D0=BE=D0=B1=D1=89=D0=B5=D0=BD=D0=B8=D0=B5 -------- >> =D0=9E=D1=82: Mario Marietto >> =D0=94=D0=B0=D1=82=D0=B0: 24.02.2024 16:38 (GMT+03:00) >> =D0=9A=D0=BE=D0=BC=D1=83: FreeBSD virtualization , >> FreeBSD Mailing List , freebsd-hackers < >> freebsd-hackers@freebsd.org>, freebsd-x11@freebsd.org, >> freebsd-drivers@freebsd.org, Ronald Klop , Jake >> Freeland >> =D0=A2=D0=B5=D0=BC=D0=B0: drm-kmod-20220907_2 not supported for this con= figuration / NVIDIA : >> Failed to initialize the NVIDIA kernel module >> >> Hello to everyone. >> >> I need to conduct some tests on top of the FreeBSD 13.1. >> So,I've installed it on one of my disks. I've installed xfce4 and >> KDE,Xorg and from the ports (after having upgraded them),I've installed = the >> nvidia-driver version. 535.146.02 ; I didn't have any problem. >> >> The problem arises when I tried to install the package drm-kmod from >> ports. >> >> Unfortunately I'm not able to compile it. This is what happened : >> >> root@marietto:/usr/ports/graphics/drm-kmod # make >> =3D=3D=3D=3D> drm-kmod-20220907_2 not supported for this configuration. >> >> ok. At this point I tried to install it from the packages : >> >> root@marietto:/usr/ports/graphics/drm-kmod # make clean >> =3D=3D=3D=3D> cleaning for drm-kmod-20220907_2 >> >> so : >> >> root@marietto:/usr/ports/graphics/drm-kmod # pkg install drm-kmod >> >> New packages to be INSTALLED : >> drm-kmod: 20220907_2 >> OK >> >> now,I have performed additional configuration to : >> >> nano /home/marietto/.xinitrc : >> >> exec ck-launch-session dbus-launch --exit-with-session startxfce4 >> >> nano /etc/rc.conf : >> >> kdm5_enable=3D"YES" >> dbus_enable=3D"YES" >> hald_enable=3D"YES" >> kld_list=3D"nvidia nvidia-modeset" >> rpcbind_enable=3D"YES" >> dtcms_enable=3D"YES" >> inetd_enable=3D"YES" >> >> nano /boot/loader.conf >> >> vmm_load=3D"YES" >> nmdm_load=3D"YES" >> tmpfs_load=3D"YES" >> cryptodev_load=3D"YES" >> zfs_load=3D"YES" >> kern.racct.enable=3D"1" >> kern.vty=3Dvt >> kern.cam.scsi_delay=3D"10000" >> >> nano /etc/X11/xorg.conf >> >> Section "Device" >> Identifier "Card0" >> Driver "nvidia" >> BusID "PCI:1:0:0" >> EndSection >> >> >> root@marietto:/home/marietto # lspci >> >> 01:00.0 NVIDIA GP106 >> 01:00.1 NVIDIA GP106 High Definition Audio Controller >> 02:00.0 NVIDIA TU102 >> 02:00.1 NVIDIA TU102 High Definition Audio Controller >> 02:00.2 NVIDIA TU102 USB Controller >> 02:00.3 NVIDIA TU102 Serial BUS Controller >> >> Now,I want to startx to start xfce4 : >> >> marietto@marietto: $ startx >> >> Fatal server error : no screens found (EE) >> Check the log file at "/var/log/Xorg.0.log" >> >> nano /var/log/Xorg.0.log : >> >> NVIDIA dlloader X driver 535.146.02 >> NVIDIA Unified Driver for all Supported NVIDIA gpus >> NVIDIA : Failed to initialize the NVIDIA kernel module >> >> How to fix it ? >> >> -- >> Mario. >> > > > -- > Mario. > --=20 Mario. --0000000000001d9ecc061221f8f1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It persists installing nvidia-driver-535.146.02,but I susp= ect that it is too now for FreeBSD 13.1.

On Sat, Feb 24, 2024 at 2:46=E2= =80=AFPM Mario Marietto <marie= tto2008@gmail.com> wrote:
:(

KLD nvid= ia.ko : depends on kernel. Not available or version mismatch
link= er_load_file : /boot/modules/nvidia.ko : unsupported file type.
<= br>
Can you give me your "nvidia.ko" file ? thanks.
=

On Sat, Feb 24, 2024 at 3:09=E2=80=AFPM Simon Gusev <semfly@fly777.net> wrote:<= br>

Hello Mario.

I have problem too.
A few days i try solve the same problem.
Alone= was a very difficult, but solved it.

Try this.=C2=A0

Remove all drm and clear.


Try steps below.

Install this
pkg ins = nvidia-xconfig

pkg ins n= vidia-driver - you card driver select

pkg ins nvidia-drm - use pkg search and select your variant v= ersion
But drm not needed for nvidia

After instal
n= vidia-xconfig - will create Xorg conf file

Start and load=C2=A0
dbus
kldload nvidia

Try xfce or KDE

B= est,
Simon )

- - -


-------- =D0=98=D1= =81=D1=85=D0=BE=D0=B4=D0=BD=D0=BE=D0=B5 =D1=81=D0=BE=D0=BE=D0=B1=D1=89=D0= =B5=D0=BD=D0=B8=D0=B5 --------
=D0=9E=D1=82: Mario Marietto <<= a href=3D"mailto:marietto2008@gmail.com" target=3D"_blank">marietto2008@gma= il.com>
=D0=94=D0=B0=D1=82=D0=B0: 24.02.2024 16:38 (GMT= +03:00)
=D0=9A=D0=BE=D0=BC=D1=83: FreeBSD virtualization <freebs= d-virtualization@freebsd.org>, FreeBSD Mailing List <freebsd-questions@fr= eebsd.org>, freebsd-hackers <freebsd-hackers@freebsd.org>, freebsd-x11@freebsd.o= rg, fr= eebsd-drivers@freebsd.org, Ronald Klop <ronald@freebsd.org>, Jake Freeland <<= a href=3D"mailto:jake@technologyfriends.net" target=3D"_blank">jake@technol= ogyfriends.net>
=D0=A2=D0=B5=D0=BC=D0=B0: drm-kmod-202209= 07_2 not supported for this configuration / NVIDIA : Failed to initialize t= he NVIDIA kernel module

He= llo to everyone.

I need to conduct some tests on t= op of the FreeBSD 13.1.
So,I've installed it on one of my dis= ks. I've installed xfce4 and KDE,Xorg and from the ports (after having = upgraded them),I've installed the nvidia-driver version. 535.146.02 ; I= didn't have any problem.=C2=A0

The problem ar= ises when I tried to install the package drm-kmod from ports.=C2=A0

Unfortunately I'm not able to compile it. This is wha= t happened :

root@marietto:/usr/ports/graphics/drm= -kmod # make
=3D=3D=3D=3D> drm-kmod-20220907_2 not supported f= or this configuration.

ok. At this point I tried t= o install it from the packages :

root@mariett= o:/usr/ports/graphics/drm-kmod # make clean
=3D=3D=3D=3D> clea= ning for drm-kmod-20220907_2

so :

root@marietto:/usr/ports/graphics/drm-kmod # pkg install drm-= kmod

New packages to be INSTALLED :
drm-= kmod: 20220907_2
OK

now,I have performed= additional configuration to :

nano /home/marietto= /.xinitrc :

exec ck-launch-session dbus-launch --e= xit-with-session startxfce4

nano /etc/rc.conf :

kdm5_enable=3D"YES"
dbus_enable= =3D"YES"
hald_enable=3D"YES"
kld_li= st=3D"nvidia nvidia-modeset"
rpcbind_enable=3D"YES= "
dtcms_enable=3D"YES"
inetd_enable=3D&q= uot;YES"

nano /boot/loader.conf

vmm_load=3D"YES"
nmdm_load=3D&qu= ot;YES"
tmpfs_load=3D"YES"
cryptodev_loa= d=3D"YES"
zfs_load=3D"YES"
kern.rac= ct.enable=3D"1"
kern.vty=3Dvt
kern.cam.scsi_d= elay=3D"10000"

nano /etc/X11/xorg.conf

Section "Device"
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Id= entifier=C2=A0=C2=A0 "Card0"
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Driver=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "nvidia"
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 BusID=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "PC= I:1:0:0"
EndSection


=
root@marietto:/home/marietto # lspci

01:00.0 = NVIDIA GP106
01:00.1 NVIDIA GP106 High Definition Audio Controlle= r
02:00.0 NVIDIA TU102
02:00.1 NVIDIA TU102 High Defini= tion Audio Controller
02:00.2 NVIDIA TU102 USB Controller
02:= 00.3 NVIDIA TU102 Serial BUS Controller

Now,I want= to startx to start xfce4 :

marietto@marietto: $ s= tartx

Fatal server error : no screens found (EE)
Check the log file at "/var/log/Xorg.0.log"
nano /var/log/Xorg.0.log :

NVIDIA dll= oader X driver 535.146.02
NVIDIA Unified Driver for all Supported= NVIDIA gpus
NVIDIA : Failed to initialize the NVIDIA kernel modu= le

How to fix it ?

--
=
Mario.


--
Mario.<= br>


--
Mario.
--0000000000001d9ecc061221f8f1-- From nobody Sat Feb 24 19:19:45 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThxXW5mnDz5BQkQ for ; Sat, 24 Feb 2024 19:19:55 +0000 (UTC) (envelope-from paul@redbarn.org) Received: from util.redbarn.org (util.redbarn.org [24.104.150.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "*.redbarn.org", Issuer "RapidSSL TLS RSA CA G1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThxXW2F0Zz42J1 for ; Sat, 24 Feb 2024 19:19:55 +0000 (UTC) (envelope-from paul@redbarn.org) Authentication-Results: mx1.freebsd.org; none Received: from family.redbarn.org (family.redbarn.org [24.104.150.213]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "*.redbarn.org", Issuer "RapidSSL TLS RSA CA G1" (not verified)) by util.redbarn.org (Postfix) with ESMTPS id 7261F19CCAE; Sat, 24 Feb 2024 19:19:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=redbarn.org; s=util; t=1708802386; bh=5fyt3pBYk434xSp1baOZphbm1DGaCjmaYlQSMRxPXBA=; h=Date:Subject:In-Reply-To:References:From:To; b=XphAeq8twj44iTV/1ofWwMAQqfyWJM+MO+y2my6SO5L0m2RKutw/AwAkCbiosVT/s z9jAp+JbZbeKUp9r1APF5zr65BvykAq2bmwQCSiPlTUNef61mX5k8VCeufAeU7h5yJ gCp0XOYSqBNmIjkDnn/Q0yaIP65b4E+J7swy8xRQ= Received: from [24.104.150.167] (dhcp-167.access.rits.tisf.net [24.104.150.167]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id 411A2C3F1F; Sat, 24 Feb 2024 19:19:46 +0000 (UTC) Date: Sat, 24 Feb 2024 11:19:45 -0800 Subject: Re: Best way to have a FreeBSD VM for automated testing? Message-ID: In-Reply-To: <7e5bc32c-dddb-4d33-96e9-99a955eed572@durchholz.org> References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> <7e5bc32c-dddb-4d33-96e9-99a955eed572@durchholz.org> From: Paul Vixie To: Jo Durchholz , freebsd-virtualization@freebsd.org List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="--_com.boxer.email_139436356837275" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:33651, ipnet:24.104.150.0/24, country:US] X-Rspamd-Queue-Id: 4ThxXW2F0Zz42J1 ----_com.boxer.email_139436356837275 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 PDxPVE9IIEZpcmVjcmFja2VyIGlzIHZlcnkgbmV3LCBzbyBJJ20gc29tZXdoYXQgcmVsdWN0YW50 IHRvIGp1bXAgdGhhdCBiYW5kd2Fnb24gYW55d2F5Lj4+CgoKTm90IG5ldy4gQmVlbiBpbiBwcm9k dWN0aW9uIGZvciB5ZWFycyBub3cuCgoKaHR0cHM6Ly9naXRodWIuY29tL2ZpcmVjcmFja2VyLW1p Y3Jvdm0vZmlyZWNyYWNrZXIKCgpwIHZpeGllCgo= ----_com.boxer.email_139436356837275 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+PGJvZHk+PGRpdiBkaXI9Imx0ciI+Jmx0OyZsdDtPVE9IIEZpcmVjcmFja2VyIGlzIHZl cnkgbmV3LCBzbyBJJiMzOTttIHNvbWV3aGF0IHJlbHVjdGFudCB0byBqdW1wIHRoYXQgYmFuZHdh Z29uIGFueXdheS4mZ3Q7Jmd0OzwvZGl2Pgo8YnI+PGRpdiBkaXI9Imx0ciI+Tm90IG5ldy4gQmVl biBpbiBwcm9kdWN0aW9uIGZvciB5ZWFycyBub3cuPC9kaXY+Cjxicj48ZGl2IGRpcj0ibHRyIj5o dHRwczovL2dpdGh1Yi5jb20vZmlyZWNyYWNrZXItbWljcm92bS9maXJlY3JhY2tlcjwvZGl2Pgo8 YnI+PGRpdiBkaXI9Imx0ciI+cCB2aXhpZTwvZGl2Pgo8L2JvZHk+PC9odG1sPg== ----_com.boxer.email_139436356837275-- From nobody Sat Feb 24 20:35:55 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThzDP2MsLz5BYfP for ; Sat, 24 Feb 2024 20:36:05 +0000 (UTC) (envelope-from jo@durchholz.org) Received: from www382.your-server.de (www382.your-server.de [78.46.146.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThzDN1gcFz4KrF for ; Sat, 24 Feb 2024 20:36:04 +0000 (UTC) (envelope-from jo@durchholz.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=durchholz.org header.s=default2202 header.b=P7e6KMza; dmarc=none; spf=pass (mx1.freebsd.org: domain of jo@durchholz.org designates 78.46.146.228 as permitted sender) smtp.mailfrom=jo@durchholz.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=durchholz.org; s=default2202; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:References:To:Subject:From:MIME-Version:Date:Message-ID:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=bEp7gw1BKDNp+otHc5qxDkPyDpsCwgZjlbcYdYKzS2Y=; b=P7e6KMza+lGuvqDCsvYA5vXaAg b0Wgok/jldCya8A34z3yviwDwJVICDK6yp1Qn4JqMtg0IcTRQNwh4YbJgK95419giJDwBLpdQZdCn MxhrQU4lP7HSjnUg4HeOgDQL8ZExqq755RDkS+l5j8+eVnVGIxBxA6eLr7u93UiM+X+qg05vORfrl lTyJykzYGaO66fs3JqSTByqxo28NY1KsIo1Bkn+rHO+mOdLX7UhLB6/DmeU79Yeq+kuY45vvvixia pme8gl5dqhYbUlYtIKwJWc9OgT3DaFgcGJiclCP8T9jq8BSA5lGOInvSBHx3tRVDJdzbhl/tdqHX2 faUE4dvA==; Received: from sslproxy02.your-server.de ([78.47.166.47]) by www382.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rdykV-000AnO-P9 for virtualization@freebsd.org; Sat, 24 Feb 2024 21:35:55 +0100 Received: from [81.221.201.210] (helo=[192.168.178.48]) by sslproxy02.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1rdykV-00EKMS-1t for virtualization@freebsd.org; Sat, 24 Feb 2024 21:35:55 +0100 Message-ID: <9b115e8e-dd8d-491a-9493-ee1ad1a8cc75@durchholz.org> Date: Sat, 24 Feb 2024 21:35:55 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Jo Durchholz Subject: Re: Best way to have a FreeBSD VM for automated testing? Content-Language: en-US To: virtualization@freebsd.org References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> <7e5bc32c-dddb-4d33-96e9-99a955eed572@durchholz.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-Sender: jo@durchholz.org X-Virus-Scanned: Clear (ClamAV 0.103.10/27195/Sat Feb 24 10:28:55 2024) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.49 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[durchholz.org:s=default2202]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:24940, ipnet:78.46.0.0/15, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[durchholz.org]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; HAS_X_AS(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[durchholz.org:+] X-Rspamd-Queue-Id: 4ThzDN1gcFz4KrF On 24.02.24 20:19, Paul Vixie wrote: > < bandwagon anyway.>> > > Not new. Been in production for years now. I admit I was going by assumptions, based on the fact I had never heard about it before, despite having been in the CI/CD treadmill for a few years. > https://github.com/firecracker-microvm/firecracker Hm. V 1.0.0 came out in Jan/Feb 2022, so in that sense, it's somewhat new. Might have been in production use in pre-1.0.0 versions, of course. I tend to assume software exists starting with 1.0.0, pre-1.0 versions tend to be very experimental and/or require handholding by the engineers that are building it. Of course, for a shop like AWS, that's not always the most accurate assumption :-) Regards, Jo From nobody Sat Feb 24 21:47:52 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tj0sj6qrgz5Bg1Y for ; Sat, 24 Feb 2024 21:50:01 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [69.87.218.172]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tj0sf6fMKz4Vhk for ; Sat, 24 Feb 2024 21:49:58 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of lists@jnielsen.net designates 69.87.218.172 as permitted sender) smtp.mailfrom=lists@jnielsen.net Received: from smtpclient.apple (136-36-176-5.googlefiber.net [136.36.176.5] (may be forged)) (authenticated bits=0) by webmail5.jnielsen.net (8.17.2/8.17.1) with ESMTPSA id 41OLm6IM030217 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 24 Feb 2024 14:48:11 -0700 (MST) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host 136-36-176-5.googlefiber.net [136.36.176.5] (may be forged) claimed to be smtpclient.apple Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: John Nielsen List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (1.0) Subject: Re: Best way to have a FreeBSD VM for automated testing? Message-Id: Date: Sat, 24 Feb 2024 14:47:52 -0700 Cc: freebsd-virtualization@freebsd.org To: Jo Durchholz X-Mailer: iPhone Mail (21C66) X-Spamd-Bar: / X-Spamd-Result: default: False [0.73 / 15.00]; THREAD_HIJACKING_FROM_INJECTOR(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.973]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; APPLE_IOS_MAILER_COMMON(0.00)[]; DMARC_NA(0.00)[jnielsen.net]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:6364, ipnet:69.87.218.0/24, country:US]; HAS_XAW(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4Tj0sf6fMKz4Vhk =EF=BB=BF >=20 > On Feb 23, 2024, at 12:17=E2=80=AFPM, Jo Durchholz wrot= e: >=20 > I'm in repeatable build land, working in Linux and developing a FreeBSD ap= pliance. >=20 > For tests, I need to run a FreeBSD VM, put some Python code and test data i= nto it, run the script, and get the test results back. >=20 > Repeatability means: Everything done with the VM needs to be scriptable (u= sing a GUI for exploring is okay but things have to translate). > Which in turn means that every setup step for a FreeBSD image comes with a= pretty high coding and maintenance cost. >=20 > So my question is: > What's the FreeBSD image that has the least number of steps to get the bas= e system up and running? I suppose it's the VM-IMAGES section, but is this c= orrect? >=20 > Follow-up question: > The startup time needs to be as fast as possible. Sub-second would be grea= t ("don't disrupt the developer's thought stream"). > I see the boot process from a vanilla VM-IMAGES image takes multiple secon= ds; can this be sped up to just a few seconds, or do I need to run the setup= and create a VM snapshot at which the VM starts for each test run? I loved Colin=E2=80=99s presentation on Firecracker and reducing boot times.= If fast boot is your primary requirement then for sure pursue that as other= s have suggested. I wanted to mention two alternatives that might require le= ss lift up front: 1) AWS EC2 + cloud-init. The FreeBSD Project publishes AMIs for every releas= e (again, thanks cperciva). You can launch an EC2 instance from one of those= with a few clicks. You can additionally assign an IAM role to the instance t= o give it permission to write to an S3 bucket and drop a script in the user-= data for the instance to run when it boots. The script could download the Py= thon code to be run, run it, and upload the results to S3 all non-interactiv= ely. With a couple hours=E2=80=99 work you could adapt the stock image to add any= missing dependencies and disable unneeded drivers and services to speed the= boot, then snapshot that and use your custom image for future runs. You could also easily manage the whole thing with Terraform. Copy/paste the i= nfrastructure as code templates and tweak the variables to get an easily rep= eatable system. 2) KVM + libvirt on whatever Linux host(s) you have available. The VM-IMAGES= released by the Project include qcow volumes. You might need to tweak the i= mage to do your custom script(s) at startup (though the image may also have c= loud-init if your environment supports it) but qcow does support snapshots a= s well and the whole thing should have low boot times. JN From nobody Sat Feb 24 23:34:25 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tj3BG0Wnmz5BqDd for ; Sat, 24 Feb 2024 23:34:30 +0000 (UTC) (envelope-from jo@durchholz.org) Received: from www382.your-server.de (www382.your-server.de [78.46.146.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tj3BD6FyVz4jby for ; Sat, 24 Feb 2024 23:34:28 +0000 (UTC) (envelope-from jo@durchholz.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=durchholz.org header.s=default2202 header.b=VILbB4Al; dmarc=none; spf=pass (mx1.freebsd.org: domain of jo@durchholz.org designates 78.46.146.228 as permitted sender) smtp.mailfrom=jo@durchholz.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=durchholz.org; s=default2202; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:References:To:Subject:From:MIME-Version:Date:Message-ID:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=8mZ7q57XK1/rsGYwaishqqNafm/6BBQnyKgcGU8+R/A=; b=VILbB4AlCHgLM5K8HfvCFspUPW HEDapGaJ6lPPvWP2rEma/oGBTjZbHxUPb08Kz77yag7dx/GkXYpwshgLwep6WWNbHZIomuzWFPBnA wbWq56DuHj+rQUP8DgpsLYYBPi1rh/L0D6nQE+ukL0eZy9uSpGqtNfKyIPGMHUYYsFuWd8cIlnEYR jkGF3ouvgpPXEEs6fnIdSfs3/IfHAO20eb4lNJq9fTxUyqm52U/OzdPiXCwmbZ3WkWiXk2E8Az8GA OrFBomj/tcMoyDzC6h3UIp6woGbHrT4g/VQ/EXMEHnZ/JlUV6/1GG1/Kug1MKz7Q729qFajf1yE1C GUHxlHcA==; Received: from sslproxy02.your-server.de ([78.47.166.47]) by www382.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1re1XG-0009Wl-Bu for freebsd-virtualization@freebsd.org; Sun, 25 Feb 2024 00:34:26 +0100 Received: from [81.221.201.210] (helo=[192.168.178.48]) by sslproxy02.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1re1XG-00380V-0Z for freebsd-virtualization@freebsd.org; Sun, 25 Feb 2024 00:34:26 +0100 Message-ID: <84e1634c-1d23-4d2a-867b-4482351da572@durchholz.org> Date: Sun, 25 Feb 2024 00:34:25 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Jo Durchholz Subject: Re: Best way to have a FreeBSD VM for automated testing? Content-Language: en-US To: "freebsd-virtualization@FreeBSD.org" References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Authenticated-Sender: jo@durchholz.org X-Virus-Scanned: Clear (ClamAV 0.103.10/27195/Sat Feb 24 10:28:55 2024) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.48 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.989]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[durchholz.org:s=default2202]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:24940, ipnet:78.46.0.0/15, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[durchholz.org]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; HAS_X_AS(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[durchholz.org:+] X-Rspamd-Queue-Id: 4Tj3BD6FyVz4jby TL;DR: Just my current status. On 24.02.24 22:47, John Nielsen wrote: > 1) AWS EC2 + cloud-init. The FreeBSD Project publishes AMIs for every release (again, thanks cperciva). You can launch an EC2 instance from one of those with a few clicks. You can additionally assign an IAM role to the instance to give it permission to write to an S3 bucket and drop a script in the user-data for the instance to run when it boots. The script could download the Python code to be run, run it, and upload the results to S3 all non-interactively. > > With a couple hours’ work you could adapt the stock image to add any missing dependencies and disable unneeded drivers and services to speed the boot, then snapshot that and use your custom image for future runs. > > You could also easily manage the whole thing with Terraform. Copy/paste the infrastructure as code templates and tweak the variables to get an easily repeatable system. Ah, I didn't mention that. The tests are for running locally. The overall project is for a home appliance, so no cloud. > 2) KVM + libvirt on whatever Linux host(s) you have available. The VM-IMAGES released by the Project include qcow volumes. You might need to tweak the image to do your custom script(s) at startup (though the image may also have cloud-init if your environment supports it) but qcow does support snapshots as well and the whole thing should have low boot times. That was my second approach :-) It turned out to have various roadbumps (mostly lack of knowledge/experience on my side, no doubts about that). I'm currently going back to my first approach, Vagrant, but diving into writing my own Vagrantfile. I understand that's easy enough for somebody without deep knowledge of KVM/libvirt, so I'll see. Regards, Jo From nobody Sun Feb 25 00:24:25 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tj4J81tmHz5Btyg for ; Sun, 25 Feb 2024 00:24:40 +0000 (UTC) (envelope-from freebsd@shaneware.biz) Received: from mail-relay3.dca2.superb.net (mail-relay3.dca2.superb.net [66.148.95.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tj4J805jZz4qSN for ; Sun, 25 Feb 2024 00:24:40 +0000 (UTC) (envelope-from freebsd@shaneware.biz) Authentication-Results: mx1.freebsd.org; none Received: from ppp230-11.static.internode.on.net ([203.122.230.11] helo=[192.168.9.155]) by mail-relay3.dca2.superb.net with esmtpsa (TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (envelope-from ) id 1re2Jh-00038o-QJ; Sat, 24 Feb 2024 19:24:31 -0500 Message-ID: Date: Sun, 25 Feb 2024 10:54:25 +1030 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: Best way to have a FreeBSD VM for automated testing? To: Jo Durchholz , virtualization@freebsd.org References: <163e57a9-0b61-414c-a8f7-109f5ac90f69@durchholz.org> <7e5bc32c-dddb-4d33-96e9-99a955eed572@durchholz.org> <9b115e8e-dd8d-491a-9493-ee1ad1a8cc75@durchholz.org> Content-Language: en-AU From: Shane Ambler In-Reply-To: <9b115e8e-dd8d-491a-9493-ee1ad1a8cc75@durchholz.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SA-Score: -1.6 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:14361, ipnet:66.148.95.0/24, country:US] X-Rspamd-Queue-Id: 4Tj4J805jZz4qSN On 25/2/24 07:05, Jo Durchholz wrote: > On 24.02.24 20:19, Paul Vixie wrote: >> <> bandwagon anyway.>> >> >> Not new. Been in production for years now. > > I admit I was going by assumptions, based on the fact I had never heard > about it before, despite having been in the CI/CD treadmill for a few > years. > >> https://github.com/firecracker-microvm/firecracker > > Hm. > V 1.0.0 came out in Jan/Feb 2022, so in that sense, it's somewhat new. > > Might have been in production use in pre-1.0.0 versions, of course. > I tend to assume software exists starting with 1.0.0, pre-1.0 versions > tend to be very experimental and/or require handholding by the engineers > that are building it. > Of course, for a shop like AWS, that's not always the most accurate > assumption :-) Took me a while, but I have let go of waiting for 1.0 releases. If it works - use it, if it breaks - help make it better, otherwise you just re-invent the wheel to get what you want. How long was openssl in wide use before 1.0 ? There is always something to fix and something to add. Tagging a release or build as v1.0 is only a decision, it isn't a certainty of stability. I have about 20% of installed pkgs with a version starting with 0 -- FreeBSD - the place to B...Software Developing Shane Ambler From nobody Sun Feb 25 21:01:36 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TjblP1Wdvz5C4BV for ; Sun, 25 Feb 2024 21:01:37 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TjblN59dSz4Vhy for ; Sun, 25 Feb 2024 21:01:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708894896; a=rsa-sha256; cv=none; b=tfmBR1VHEdg0q4U8prEZtUY6ghn/j3//gzuzLclT/f7ZJXPW13c1VmPWLEHqMDTMjqoJKa ahK0le9gsWQ55s/tSxV+IovovPdFfkdBOXIwJY3L/wMivN7wOUTA9AjE4Aj61yXz/VzBT3 4wp8jAFRUXX6OW2Km14ROqnITBQNNSg3rIjivCaSE/IyNdjQqNmF9mIWqK0V9mJxoemNu4 vBpIDIhHXJ1g4ZLpI99G67A/8TywOa6vAcYNHs+5FDxDUr1YCoF6tiBOat5iLMT0TUH1JR Vd90d/X5aNBZ4p4nxM6ripof4zBha8+soaPU6ye/upk5sym/4oX+SjRJuOLukw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708894896; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=38U4/P7EVc7oo8pVsuuJgzsxLM7iGNgTXEXDL5fTRos=; b=X7gNuJSw8pMbwEUr1LV7MAHiDUcv707DEFm3lOPhx6Gll5ph4JrM9MwkeCasPAGYiMf7hh M+ueYKDeNYJzXvqo1BQGpCVP+sTRS4CahKaDtsUMGuyBC1cipd1cn12cgXs1S/zcjURx8b tWMd1pFMT03pX/UvEFhcvz+WqAwFY5cTO+iTfCbKBDLrHQ7ZDHqPVFWccwC6GPXiOtKdLd pZOhD4fV6FxTehtaT58DWCffB/52xvnedE1g4P0S0p/W4jR6FJp/lMkc9/sxq0E6zg+jQG cjBlSiU84UjVJ45/yRyhu/UjbX4VzVjdOdVpygMrsZ6f7WGf4ZI2EiTea3gMTg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TjblN4Hc3z10XF for ; Sun, 25 Feb 2024 21:01:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41PL1aML068883 for ; Sun, 25 Feb 2024 21:01:36 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41PL1a65068882 for virtualization@FreeBSD.org; Sun, 25 Feb 2024 21:01:36 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202402252101.41PL1a65068882@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 25 Feb 2024 21:01:36 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17088948962.f6aa4.53378" Content-Transfer-Encoding: 7bit --17088948962.f6aa4.53378 Date: Sun, 25 Feb 2024 21:01:36 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 264253 | hyperv: ifconfig media autoselect on FreeBSD 13.0 Open | 264267 | UEFI Booting on Azure Generation 2 VMs crashes 2 problems total for which you should take action. --17088948962.f6aa4.53378 Date: Sun, 25 Feb 2024 21:01:36 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    264253 | hyperv: ifconfig media autoselect on FreeBSD 13.0
Open        |    264267 | UEFI Booting on Azure Generation 2 VMs crashes

2 problems total for which you should take action.
--17088948962.f6aa4.53378--