From nobody Mon Nov 21 20:30:57 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NGJts2FRnz4hSN9 for ; Mon, 21 Nov 2022 20:31:01 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NGJts0KPvz3xKs; Mon, 21 Nov 2022 20:31:01 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x832.google.com with SMTP id cg5so8023071qtb.12; Mon, 21 Nov 2022 12:31:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=Wu+PaxPZE10JjWlddd6SC/LeePG0sUs/7EJce0fCLT0=; b=UdkHjKNG3tDvla1oEzXB1wsk1szVDasIiwttIdYMeLX98PwEq65chi625/4erG1t9D Qm9Oz1ED3Riki50RKbyx5gdcqIslNUYet68oBymcp2UjanxFwMj+PR77ky4IVvFSGp/s pPwiMtFY5Jmot/bhDMhDpeFZhWuHL9nbrm245LR/gcMUd/tzHimI7rMSBmZLWAU1pE3z k652AoW8niNUsVEHJtTD71b+qLaAiLcxhcmo/skH3iNYL9D9t3/o68I7qibQvFy3EJbI niziOl3NGqu2lUaGCZqd8/toA2NqUUZIPKv3e4geDNtDLOOrKyrnxjzPaazOSYOoAQ3+ PMcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wu+PaxPZE10JjWlddd6SC/LeePG0sUs/7EJce0fCLT0=; b=2A47ZUiXBIzDU3rLdasymyqcIgYuI1gLr2nGL0Gt6x0yoyCl4oUkBQGeHDFWnZDxx/ PCaGNL8LfsMjOUOqzwffOcYrxa3OUpYZ79G4DDRSouPjboRYlBCZ9+VQvfPlxLRn6SVp r9mkoSp0YWT9E6zGxcbscptYAowjfxf+bNJygx1Z0HLsUsZ+nhgXCofQuf6nchmyarVU zguMWBRxv3Lz083vPHwANMeQUQOLVWdeJeaKf/kzW9BSXBd3Ikrec0ibxDDUhlt8gOW4 kmbIFX/qsxxxa9bYFWPlLR0QxI23yxK1uQs6+7c7jQ+RS5+myFE05HI42xGi/mvBT8ZX 5eQQ== X-Gm-Message-State: ANoB5pnXXvc3ix+kfRGc16o2G4DMjF45mc3Ly8ccznyOAvug5xdfWhwp RIHKQnOirCBzoQrsbDdk6vRO7BZKr/s= X-Google-Smtp-Source: AA0mqf4H880+sKv/GcgwBwKwmG2AlgtGEVf46XeWVlyHc3VFug7A9eopn0hMucgnzgqL7IX1HiOa+g== X-Received: by 2002:ac8:44c9:0:b0:3a5:7f82:b083 with SMTP id b9-20020ac844c9000000b003a57f82b083mr1979224qto.561.1669062660213; Mon, 21 Nov 2022 12:31:00 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id i23-20020ac87657000000b0038d9555b580sm7134401qtr.44.2022.11.21.12.30.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Nov 2022 12:30:59 -0800 (PST) Date: Mon, 21 Nov 2022 15:30:57 -0500 From: Mark Johnston To: Konstantin Belousov Cc: "Bjoern A. Zeeb" , freebsd-hackers@freebsd.org Subject: Re: Can we "pause" in loader and ddb>? Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4NGJts0KPvz3xKs X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mon, Nov 21, 2022 at 09:37:36PM +0200, Konstantin Belousov wrote: > On Mon, Nov 21, 2022 at 02:06:47PM -0500, Mark Johnston wrote: > > On Mon, Nov 21, 2022 at 06:42:44PM +0000, Bjoern A. Zeeb wrote: > > > Hi, > > > > > > having a VM sitting (a) in loader prompt or (b) in ddb> makes the fans > > > go loud on a test-laptop which makes one wonder if we can avoid that. > > > Note -- I assume also on real HW though that's less likely observered > > > here. > > > > > > For example I am sitting in a 4 vCPU bhyve current in ddb> and two > > > threads are at 100% on the base system. Do we need to heat up the > > > planet doing that or are there alternatives? I forgot to mention: in this situation my "solution" is to attach gdb to the running VM, which I usually want to do anyway. This causes all vCPU threads to pause. > > > I haven't looke dta the > > > code in ages and I assume we need to poll in these situations on the > > > console and for interactivity often enough? Can we "pause"? And why > > > would two [v]CPUs be at 100% and not just one as I would expect all but > > > one to be stopped? > > > > When DDB is active, one CPU is going to be spinning in cncheckc() > > waiting for input, and the others will be in cpustop_handler(). For the > > latter, there is an option to use MWAIT/MONITOR to pause the CPU instead > > of calling pause in a loop, but bhyve hides the MWAIT/MONITOR CPU > > capability from guests (I'm not sure exactly why), so they just execute > > "pause" in a loop, which won't help the apparent CPU usage. > We do not want the guest thread eating CPU time in MWAIT, it is the host > scheduler that should decide that there is nothing to do and schedule > idle thread. > > In other words, MWAIT should be emulated, perhaps by revoking the monitoring > page on MONITOR, and waking up the MWAIT-ing thread on write to the armed > range. > > Additional complication is that the range to arm is usually equal to the cache > line, doing dumb emulation by interpreting any write to the monitor page > might be too harsh. > > There is also non-trivial cooperation with atomic interrupt delivery. > > These complexities are perhaps the reason to not enable M* for guests. I see. Maybe instead we could allow stopped CPUs to execute "hlt" with interrupts disabled, and when exiting DDB, the last CPU resumes all of the others by raising NMIs. Then only one vCPU thread will be consuming host CPU time when in DDB. > > Note that when the kernel panics, cpustop_handler() will execute "hlt", > > which should result in a vmexit if you started bhyve with -H, and you'll > > see only one vCPU thread consuming 100% when sitting at the DDB prompt. > > But when you enter DDB without a panic, we need some mechanism to resume > > the stopped CPUs when exiting DDB. >