Date: Fri, 23 Feb 2024 19:03:00 +0000 (UTC) From: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> To: Warner Losh <imp@bsdimp.com> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: really slow problem with nvme Message-ID: <rn497rrr-2893-7460-r39q-29sq3snnop0r@yvfgf.mnoonqbm.arg> In-Reply-To: <CANCZdfoBc7bW6so8H4=dD-XogPp0uQKr1pZjRvSH5%2B8HEGMTCg@mail.gmail.com> References: <5n117579-8259-3on3-90pn-79o8n52q4q32@yvfgf.mnoonqbm.arg> <CANCZdfoBc7bW6so8H4=dD-XogPp0uQKr1pZjRvSH5%2B8HEGMTCg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
--1098556516-1544092989-1708712493=:2366
Content-Type: text/plain; CHARSET=UTF-8; format=flowed
Content-Transfer-Encoding: 8BIT
Content-ID: <on3r1r32-ro4o-6s68-2p0s-9q3nsrq3r9r6@mnoonqbm.arg>
On Fri, 23 Feb 2024, Warner Losh wrote:
> On Fri, Feb 23, 2024, 10:46 AM Bjoern A. Zeeb <
> bzeeb-lists@lists.zabbadoz.net> wrote:
>
>> Hi,
>>
>> this is a Samsung SSD 970 EVO Plus 1TB nvme and gpart and newfs
>> were already slow (it took like two hours for newfs).
>>
>> Here's another example now:
>>
>> # /usr/bin/time mkdir foo
>> 1.82 real 0.00 user 0.00 sys
>>
>> How does one debug this?
>>
>
> What filesystem? Sounds like UFS but just making sure. .
yes, ufs
> So what's the link speed and number of lanes? If it's bad i might reseat
> (though that might not help) that looks good...
pciconf I had checked:
nvme0@pci4:1:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa808 subvendor=0x144d subdevice=0xa801
class = mass storage
subclass = NVM
bar [10] = type Memory, range 64, base 0x40000000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit
cap 10[70] = PCI-Express 2 endpoint max data 128(256) FLR RO NS
max read 512
link x2(x4) speed 8.0(8.0) ASPM disabled(L1) ClockPM disabled
cap 11[b0] = MSI-X supports 33 messages, enabled
Table in map 0x10[0x3000], PBA in map 0x10[0x2000]
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
ecap 0003[148] = Serial 1 0000000000000000
ecap 0004[158] = Power Budgeting 1
ecap 0019[168] = PCIe Sec 1 lane errors 0
ecap 0018[188] = LTR 1
ecap 001e[190] = L1 PM Substates 1
> Though I'd bet money that this is an interrupt issue. I'd do a vmstat. -i
> to watch how quickly they accumulate...
That I am waiting for a full world to get onto it. I wish I could have
netbooted but not possible there currently.
Only took 15 minutes to extract the tar now. Should have used ddb...
hadn't thought of that before...
# vmstat -ai | grep nvme
its0,0: nvme0:admin 0 0
its0,1: nvme0:io0 0 0
its0,2: nvme0:io1 0 0
its0,3: nvme0:io2 0 0
its0,4: nvme0:io3 0 0
its0,5: nvme0:io4 0 0
its0,6: nvme0:io5 0 0
its0,7: nvme0:io6 0 0
its0,8: nvme0:io7 0 0
How does this even work? Do we poll?
And before you ask:
[1.000407] nvme0: <Generic NVMe Device> mem 0x40000000-0x40003fff at device 0.0 on pci5
[1.000409] nvme0: attempting to allocate 9 MSI-X vectors (33 supported)
[1.000410] nvme0: using IRQs 106-114 for MSI-X
[1.000411] nvme0: CapLo: 0x3c033fff: MQES 16383, CQR, AMS WRRwUPC, TO 60
[1.000412] nvme0: CapHi: 0x00000030: DSTRD 0, NSSRS, CSS 1, CPS 0, MPSMIN 0, MPSMAX 0
[1.000413] nvme0: Version: 0x00010300: 1.3
> How old is the drive? Fresh install? Do other drives have this same issue
> in the same slot? Dies this drive have issues in other maxhines or slots?
The drive is a few months old but only in the box until it went on this
board.
I checked nvmecontrol for anything obvious but didn't see.
> Oh, and what's its temperature? Any message in dmesg?
Nothing in dmesg, temp seems not too bad. Took a while to get smartmontools;
we have no way to see this in nvmecontrol in human readable form, do we?
Temperature Sensor 1: 51 Celsius
Temperature Sensor 2: 48 Celsius
Ok I got a 2nd identical machine netbooted remotely (pressue with
problems often helps) -- slightly different freebsd version and kernel,
same baord, same type of nvme bought together:
# /usr/bin/time dd if=/dev/zero of=/dev/nda0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 1.657316 secs (647879880 bytes/sec)
1.66 real 0.00 user 0.94 sys
and ddb> show intrcnt
..
its0,0: nvme0:admin 24
its0,1: nvme0:io0 126
its0,2: nvme0:io1 143
its0,3: nvme0:io2 131
its0,4: nvme0:io3 128
its0,5: nvme0:io4 135
its0,6: nvme0:io5 147
its0,7: nvme0:io6 143
its0,8: nvme0:io7 144
..
I'll try to make sure I can safely access both over the weekend remotely
from a more comforting place and I know where to start looking now...
Thanks!
/bz
--
Bjoern A. Zeeb r15:7
--1098556516-1544092989-1708712493=:2366--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?rn497rrr-2893-7460-r39q-29sq3snnop0r>