From nobody Sun Jun 29 11:16:42 2025 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bVRZN5kQNz605v0 for ; Sun, 29 Jun 2025 11:16:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bVRZN4kN4z49yg for ; Sun, 29 Jun 2025 11:16:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1751195804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CFHCAlI425kEgISBep0ySGqZSCCYZyR98z2uVN1VVCU=; b=ci1qYcVsxCUJNjyZdRqjGHya1j2Rsb0XMY6hlxeoYz5ApBoQ+5Oyf4vSrUiuP0r0bbQb3o nw7XdnUN9P2rTq2PE5Wm4Zct8LMmT/xUZsDD+E83inIGZ/jvtk450oLuGmaLK6qpka/QmB x4E/oAiSPAK9jVH5THqazgNiMCCzWShax31/1w8FLyvjYdTlF5erEHkn2S9aFZDhZVPvno WHH08Rk+CXn1jtzCdSetmGoBu5zOEZAyHjo7jjri4eg82TC23mR9rgfyPouVRDUCHeel5r QpfOba/Th+YkF0SR0T7tJZLNBNeVMtppGX7pHpJJk4pSX08YeKM+DuVuxDEhXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1751195804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CFHCAlI425kEgISBep0ySGqZSCCYZyR98z2uVN1VVCU=; b=G9pLsFjtNAFebhP1BWnWUkPRQxYDS0rgU/gd3QbbAFi4IR8ravncnqctK3VekW5gCytV3F LtfUzuJtpoxSPXxoQjtSRKIONCG99hTRwJ44Tvse7mkAR3vFuCB8O9b/2yCg3GpUQk2O7W rFIMkenAi+sWaZtgrP+u7nNX6g/2ok0fiRPr276ybwLroct+jXLONRIp/CMU+f8CnKvuPs lQSEc61tConjJDUb1LM/wWZqx5SEflwQkeJdGpKDR6+bWKfSjRScKMYBdk1XtM4nY2LAS+ xSOKsTRMfJ3mhakt3DPS+EDlMisGVagvpXUL3aat9EmDfUxyU+FzpeY2bG8ewg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1751195804; a=rsa-sha256; cv=none; b=c47xL99XyytpNa5u7cfR2PYjLlZ30y7qwU9nezbBJV8HP75Ex6jeadoApDgrLS3jUxxf1k 4Ym3TuNDdPMiWMMPSiSexVD9vv8ZGHQwMs8cfboZvQS2aQ2DpTDgAGTS8T8enRnw722sDf WjkHUdqjpkGdTkUlss3pazG56EbSWAsfZwd52WJcX+SONsMyPsCRzSz/xraT0EnhkFqBoy 46KSTYiZ0xxCHtGH7Gdb/n2mj21qkIiOq1SZ5Fd1nNU1Zxsim1XuvaKbSYO0nNJfyLZJLm ViS0U2O3mw4wO8YHguf0ZwY4Tec7u3mULPu9dNOpVizt2K+Wii31U5NR3XicqQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bVRZN4KpQz2Fm for ; Sun, 29 Jun 2025 11:16:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 55TBGit1086847 for ; Sun, 29 Jun 2025 11:16:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 55TBGinl086845 for net@FreeBSD.org; Sun, 29 Jun 2025 11:16:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 287872] 14.3-STABLE boot time regression related to netlink and devd Date: Sun, 29 Jun 2025 11:16:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.3-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: eugen@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D287872 --- Comment #8 from Eugene Grosbein --- I tried to use hwpmc(4) to profile the kernel: # kenv kern.hwpmc.logbuffersize=3D64 # kenv kern.hwpmc.nbuffers_pcpu=3D512 # kldload hwpmc Then removed all vlans, stopped devd, recreated vlans. Then: # pmcstat -S inst_retired.any_p -O devd.out & # service devd start # kill -INT %1 # pmcstat -R devd.out -g And "gprof -l /boot/kernel/kernel inst_retired.any_p/kernel.gmon" shows the following: % cumulative self self total time seconds seconds calls ms/call ms/call name 44.5 131231.00 131231.00 28705 4571.71 4571.71 lock_delay [2] 22.6 197818.00 66587.00 67341 988.80 988.80 DELAY [19] 3.1 207038.00 9220.00 2875 3206.96 3807.34 pmap_remove_pages [36] 1.4 211181.00 4143.00 1134 3653.44 7948.42 pmap_enter_quick_lock= ed [38] 1.3 215064.00 3883.00 956 4061.72 4138.80 get_pv_entry [48] 1.3 218829.00 3765.00 1053 3575.50 4107.77 pmap_enter [45] 1.0 221869.00 3040.00 1639 1854.79 4229.11 pmap_try_insert_pv_en= try [39] Above profiling for stock /sbin/ifconfig using netlink. Below for /sbin/ifconfig rebuilt without netlink and same profiling procedure: % cumulative self self total time seconds seconds calls ms/call ms/call name 11.1 581.00 581.00 422 1376.78 1376.78 counter_u64_fetch [16] 8.7 1037.00 456.00 226 2017.70 2017.70 DELAY [25] 6.6 1383.00 346.00 375 922.67 1037.36 pmap_remove_pages [26] 6.2 1709.00 326.00 187 1743.32 1743.32 strncmp [28] 3.2 1875.00 166.00 10 16600.00 21475.00 rtsock_msg_buffer [49] 2.6 2013.00 138.00 86 1604.65 1889.35 pmap_enter [52] 2.3 2134.00 121.00 512 236.33 2146.13 sysctl_iflist [7] 2.3 2253.00 119.00 124 959.68 1009.98 get_pv_entry [57] 2.1 2361.00 108.00 220 490.91 1000.49 pmap_try_insert_pv_en= try [48] --=20 You are receiving this mail because: You are the assignee for the bug.=