From nobody Fri Feb 23 23:45:55 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThRV20VZKz5BllW for ; Fri, 23 Feb 2024 23:46:02 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE Root Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThRV14blkz4MN5 for ; Fri, 23 Feb 2024 23:46:01 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Authentication-Results: mx1.freebsd.org; none Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 5DBD08D4A228; Fri, 23 Feb 2024 23:45:58 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 8CA382D029D8; Fri, 23 Feb 2024 23:45:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id MwQtQ9TfgP42; Fri, 23 Feb 2024 23:45:56 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 108CB2D029D2; Fri, 23 Feb 2024 23:45:55 +0000 (UTC) Date: Fri, 23 Feb 2024 23:45:55 +0000 (UTC) From: "Bjoern A. Zeeb" To: Warner Losh cc: FreeBSD FS Subject: Re: really slow problem with nvme In-Reply-To: Message-ID: References: <5n117579-8259-3on3-90pn-79o8n52q4q32@yvfgf.mnoonqbm.arg> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:195.201.0.0/16, country:DE] X-Rspamd-Queue-Id: 4ThRV14blkz4MN5 On Fri, 23 Feb 2024, Warner Losh wrote: >> How does this even work? Do we poll? >> > > Yes. We poll, and poll slowly. You have an interrupt problem. > > On an ARM platform. Fun. ITS and I are old.... foes? Friends? frenemies? > > As for why, I don't know. I've been fortunate never to have to chase > interrupts not working on arm problems.... I do now. Someone's been clever and loaded a dtb from loader after we had done changes and apparently firmware hadn't picked it up. And that doesn't go well if your firmware runs fixups and we do rely on these in FreeBSD. Makes me wonder if these FDT regions end up in the excluded memory list or if that late fixup as we leave boot services could possibly cause some memory changes we don't want but that's not for here... >>> Oh, and what's its temperature? Any message in dmesg? >> >> Nothing in dmesg, temp seems not too bad. Took a while to get >> smartmontools; >> we have no way to see this in nvmecontrol in human readable form, do we? >> >> Temperature Sensor 1: 51 Celsius >> Temperature Sensor 2: 48 Celsius >> > > A little warm, but not terrible. 50 is where I start to worry a bit, but > the card won't thermal > throttle until more like 60. Yeah, haven't checked but closing the box and pushing it back into the rack probably helped. And was it's running in power state 0, which makes me wonder how helpful that is in the 2-lane setup... # nvmecontrol power nvme0 Current Power State is 0 Current Workload Hint is 0 # nvmecontrol power -l nvme0 Power States Supported: 5 # Max pwr Enter Lat Exit Lat RT RL WT WL Idle Pwr Act Pwr Workloadd -- -------- --------- --------- -- -- -- -- -------- -------- -- 0: 7.8000W 0.000ms 0.000ms 0 0 0 0 0.0000W 0.0000W 0 1: 6.0000W 0.000ms 0.000ms 1 1 1 1 0.0000W 0.0000W 0 2: 3.4000W 0.000ms 0.000ms 2 2 2 2 0.0000W 0.0000W 0 3: 0.0700W* 0.210ms 1.200ms 3 3 3 3 0.0000W 0.0000W 0 4: 0.0100W* 2.000ms 8.000ms 4 4 4 4 0.0000W 0.0000W 0 > We don't currently have a nvmecontrol identify > field to tell you this > (I should add it, this is the second time in as many weeks I've wanted it). Would be awesome :) -- Bjoern A. Zeeb r15:7