From nobody Fri Jun 9 06:28:31 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qcrkp1lcCz4bvFF for ; Fri, 9 Jun 2023 06:28:50 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qcrkp0JHtz3MYT for ; Fri, 9 Jun 2023 06:28:50 +0000 (UTC) (envelope-from otis@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686292130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Fh07cMMNiO5KY7YB7FTFFZF6tyN5nKlpPeJX3EMLcYA=; b=s6etibK5yJu+W9gqPO1gqUF3VY8irYThJ/6DqiiY6iCR6Qlq4WuX4z6fUSQoLUbSXZYMrE Npphz7tuBoXMackD7t8xUqs47WjP3uwLoewhBaXjMOi3iD6HP0UDGHVw4CdMN+ciYIcXDJ rquQI9s9cIHwwj2cag7IbOW/Dbd/oOs5yEIoCHyacsNOECO3Wnqek1MVo401asH+crMQfL yX5gGAqE96Y59c7rkhTrMMq6fT/zqC6eytho/cj7x+tYtOATz/0DGf4OKvHegH5oTkV/Ka BwoK/nurAz6ae6Y//+A3mAVuSdI09NuvSMUST/4W1A4bkUIE5gO2et95S0QJig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1686292130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Fh07cMMNiO5KY7YB7FTFFZF6tyN5nKlpPeJX3EMLcYA=; b=KMEk8meY4FPZsfvlqa2gxoN0OBowrhrGvZYaB+dD57MziCAJ9H6jiARDf/DlJcxEN+oU9i CqAkU7KoHEs//SWURs2uDtfRG2gbBP17Xo+Q/DIUfiIA1MFSAXHh+54xHVY3+6Ekgj3QlH 9S1KGM7Qz0h3/BcnR0oHkuJ8m5+pRrjLHTxd/58rV4vjArVTQRZtPzRL0y6LHg0XdqS9mF KSSwQRYpaR3+cdDxdOx4deVzxyh89YmyZ4imZ26Z/KxL8bkybo58w6wCpGci1bd3hx88M8 2rGwnNlES3r+8X3hmqXpeT4yuN9Z8cAwNn8Zldu3sK6Y39xiR/YSSlUGTPrcoQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1686292130; a=rsa-sha256; cv=none; b=rZvaBhqcY4sQAdXB1dlCqDMNXbPu5JlcBf3uksDmefoYQTU4LG0oWs9cZ5NPDY0nwo4A52 /XluR2UUEQF0AqnwcJ8kxjqKO3GTJW4TDQWyRNbxazrSd0mAYitYjuEbQlrFcbAvwe0d8x Cvunlq1DDxLJLhseSTOgiHsZhcdbIsjq3FRJY7MasYnMFB2Pd4axIbfIlckfYQ5In2WaOd tGcAoXlbWZ0/MDV0vy+qZils260EUXiAnn+sokKVA4vstPbizELMPU5d/+WEJL+raFoO/x I0pxfd/4ncN3kcThB6NVst9Rilgpe6MMn9skv1hnX/9kNTZth3MWKMvShRTjRQ== Received: from ns2.wilbury.net (ns2.wilbury.net [92.60.51.55]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) (Authenticated sender: otis) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Qcrkn5gzhz14vT for ; Fri, 9 Jun 2023 06:28:49 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtpclient.apple (gw-upc.owhome.net [188.167.168.254]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id 4C15B61F8D for ; Fri, 9 Jun 2023 08:28:46 +0200 (CEST) From: Juraj Lutter Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: CPU hog on -current (pfSense 23.05) Message-Id: <50D90C0F-4C92-4A6A-B1FD-D56609434AF4@FreeBSD.org> Date: Fri, 9 Jun 2023 08:28:31 +0200 To: FreeBSD CURRENT X-Mailer: Apple Mail (2.3731.600.7) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on ns2.wilbury.net X-ThisMailContainsUnwantedMimeParts: N Hi, since cca 7 days I=E2=80=99ve started to observe a CPU hog of one CPU = core on APU2 box running pfSense 23.05 (if that matters). mjg@ suggested to run a dtrace oneliner that showed: ~~~~~~ kernel`pmap_copy 33 kernel`amd64_syscall 33 kernel`vm_radix_insert 35 kernel`vm_map_pmap_enter 37 kernel`vm_radix_lookup_unlocked 38 kernel`memcpy_std 38 kernel`vm_object_deallocate 39 kernel`pmap_enter_quick_locked 41 kernel`em_update_stats_counters 43 kernel`copyout_nosmap_std 43 kernel`ck_epoch_poll_deferred 44 kernel`sbuf_put_bytes 46 kernel`vm_page_pqbatch_submit 48 kernel`pmap_remove_pte 51 kernel`pmap_pvh_remove 53 kernel`vm_pqbatch_process_page 54 kernel`cpu_search_highest 56 kernel`get_pv_entry 57 kernel`pmap_try_insert_pv_entry 59 kernel`vm_map_lookup_entry 65 kernel`epoch_call_task 92 kernel`pmap_enter 101 kernel`vm_fault 110 kernel`pagecopy 110 kernel`0xffffffff81 133 kernel`lock_delay 145 kernel`pmap_remove_pages 203 kernel`_thread_lock 415 kernel`pagezero_std 490 kernel`assert_rw 532 kernel`acpi_cpu_c1 600 kernel`callout_lock 641 kernel`kern_yield 1206 kernel`_callout_stop_safe 2010 kernel`spinlock_enter 2032 kernel`tcp_timer_stop 2703 0x0 5927 kernel`spinlock_exit 40964 kernel`cpu_idle 61943 kernel`sched_idletd 76722 ~~~~~~~ The symptom is that the kernel thread "kernel{if_io_tqg_1}=E2=80=9D = consumes 100% of the CPU core. Can this be debugged any further? I also suspect the hardware problem (I have one spare box where I=E2=80=99= ll put my XML config and test whether the problem will persist). I=E2=80=99d be very thankful for any pointers. Thanks! otis =E2=80=94 Juraj Lutter otis@FreeBSD.org