From nobody Thu Mar 2 00:17:37 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PRsB91W5hz3vYpY for ; Thu, 2 Mar 2023 00:17:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PRsB86jDlz3jdS for ; Thu, 2 Mar 2023 00:17:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677716256; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xJsOsSlUyhOCaXVACFrchV6TMjpTnaT00Ppy46+tRBk=; b=Fl/SB41nVAoWcDfZVaISKiGc5xu7m8+qEB9Hb5SNemMZ3fkDOtsnliyfXoUx3Fbe161Xnt 7YoVaTLmTUZtsyPG3L8NmD6DKjbxKJrmNuQ/tq/TaDaDnvFEP9oYjBU0e8bYVYd/KaC5ff ahdlu7T/dIoabsAF7Zl89l+iFGlaWdv7UU4Efbo/t3rE7sN8eJlF7Dce6CX1prfDbCdlZL DTn8WQRSkjnVxF61Bq/ITv5RbXx1nE6BkZVT8v3S0Oz9Jv1zlUa6jg5cPLgGMlVBLYuXpz g/0KYe16Sv6+Gdw/J2LUEHv/97kAAh3UbNntAZ0UmH9ho3AjhOZE3fTN8T1ZWg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677716256; a=rsa-sha256; cv=none; b=TW3vt4EnxB7wRn4Hw22LamqA5NiC57Y7+IfMqRsv/FLsd+qJTQ6WGhgMPIWHuzx86JxPvK 7/mZY3lcNUpWF1l4pFeTwkNeDhVLPd5PXnz+GStqLhm57dpAPkw74O2o/mMBNRudDkOeF4 Zfi9Ylms88HkwDt71PRqrljZCK/5erp91FwNw5TIVY5wo2Btr5/WJO9bYtNY0L5XfDHrQz p2uMS6Gb5m2MRLB8rHcy6yKPYyrt64zvxqLiy1hb547DaIehK0Q6wpXsB3Mq8LmGfmsWHa cKC7FD2FVFWfXLwb2xyJ4TbCVXY8i/2/G6Ih6DBjjYycFu0UDS8BGNkpIWl+gw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PRsB85n5HzvRx for ; Thu, 2 Mar 2023 00:17:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3220HaBD010225 for ; Thu, 2 Mar 2023 00:17:36 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3220HakU010224 for virtualization@FreeBSD.org; Thu, 2 Mar 2023 00:17:36 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 269823] rand_harvest produces 100%CPU on 1 CPU with virtio_random.ko loaded Date: Thu, 02 Mar 2023 00:17:37 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd@igalic.co X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D269823 --- Comment #5 from Mina Gali=C4=87 --- been debugging this on and off for the past 2 days. at first I thought that dtrace -n 'kinst::virtqueue_poll:' output was not not very useful, until I realized that it constantly repeats: 1 73355 virtqueue_poll:112=20 1 73356 virtqueue_poll:115=20 1 73357 virtqueue_poll:118=20 1 73358 virtqueue_poll:124=20 1 73359 virtqueue_poll:129=20 1 73360 virtqueue_poll:133=20 1 73361 virtqueue_poll:135=20 1 73362 virtqueue_poll:137=20 1 73363 virtqueue_poll:141=20 1 73364 virtqueue_poll:144=20 1 73365 virtqueue_poll:151=20 1 73366 virtqueue_poll:155=20 1 73367 virtqueue_poll:158=20 1 73355 virtqueue_poll:112=20 1 73356 virtqueue_poll:115=20 1 73357 virtqueue_poll:118=20 1 73358 virtqueue_poll:124=20 1 73359 virtqueue_poll:129=20 1 73360 virtqueue_poll:133=20 1 73361 virtqueue_poll:135=20 1 73362 virtqueue_poll:137=20 1 73363 virtqueue_poll:141=20 1 73364 virtqueue_poll:144=20 1 73365 virtqueue_poll:151=20 1 73366 virtqueue_poll:155=20 1 73367 virtqueue_poll:158=20 etc=E2=80=A6 This function never returns, we don't even get (back) into vtrnd_read: dtrace -n 'kinst::vtrnd_read:' dtrace: description 'kinst::vtrnd_read:' matched 88 probes =E2=80=A6 # this never prints anything on this machine So let's look at those instructions: 0xffffffff80a113e0 <+112>: 48 89 df mov %rbx,%rdi 0xffffffff80a113e3 <+115>: ff 50 08 call *0x8(%rax) 0xffffffff80a113e6 <+118>: 41 0f b7 4c 24 56 movzwl 0x56(%r12),%ecx 0xffffffff80a113ec <+124>: 49 8b 44 24 48 mov 0x48(%r12),%rax 0xffffffff80a113f1 <+129>: 66 3b 48 02 cmp 0x2(%rax),%cx 0xffffffff80a113f5 <+133>: 75 32 jne 0xffffffff80a11429 0xffffffff80a113f7 <+135>: f3 90 pause=20=20 0xffffffff80a113f9 <+137>: 49 8b 1c 24 mov (%r12),%rbx 0xffffffff80a113fd <+141>: 48 8b 0b mov (%rbx),%rcx 0xffffffff80a11400 <+144>: 0f b6 15 59 c0 0c 01 movzbl # 0x10cc059(%rip),%edx # 0xffffffff81add460 0xffffffff80a11407 <+151>: 48 8b 04 d1 mov (%rcx,%rdx,8),%rax 0xffffffff80a1140b <+155>: 4c 39 38 cmp %r15,(%rax) 0xffffffff80a1140e <+158>: 74 d0 je 0xffffffff80a113e0 in other, C words: void * virtqueue_poll(struct virtqueue *vq, uint32_t *len) { void *cookie; VIRTIO_BUS_POLL(vq->vq_dev); while ((cookie =3D virtqueue_dequeue(vq, len)) =3D=3D NULL) { cpu_spinwait(); VIRTIO_BUS_POLL(vq->vq_dev); } return (cookie); } (if *0x8(%rax) is an obfuscated call to virtqueue_dequeue() then=E2=80=A6) This is literally just the inner loop here. That means: virtqueue_dequeue() never returns anything other than NULL. Why, i don't know yet. There are several virtqueue_dequeue()s running on the system, according to dtrace, but none of them seem to be running on CPU1, l= ike our virtqueue_poll(). my assumption here is rand_harvest running CPU1 (true) =E2=86=92 virtqueue_poll() running on CPU1= (true) Therefore, our virtqueue_dequeue() should be running on CPU1 as well. But either i'm not seeing it on CPU1, in fact it's running on all other CPUs exccept CPU1. Either that, or I don't know how to filter for it, but this doesn't work: root@freebsd:~ # dtrace -n 'kinst::virtqueue_dequeue: /cpu =3D=3D 1/' dtrace: invalid probe specifier kinst::virtqueue_dequeue: /cpu =3D=3D 1/: s= yntax error near end of input --=20 You are receiving this mail because: You are on the CC list for the bug.=