Date: Mon, 23 Sep 2024 11:10:15 +0100 From: Warner Losh <imp@bsdimp.com> To: Alexey Sukhoguzov <mail@eseipi.net> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: nvme(4): some non-operational power states are broken Message-ID: <CANCZdfp0g118BecYsUjXOj1stMURsDfb-OMK7XT=3i=tCQMBtw@mail.gmail.com> In-Reply-To: <CANCZdfpv6EbB_HSnZG4Sy4crOJBbX16PGo3iwjCdkBzvTE7SVA@mail.gmail.com> References: <ZvE3f_gTC4D6TqBe@eseipi.net> <CANCZdfpv6EbB_HSnZG4Sy4crOJBbX16PGo3iwjCdkBzvTE7SVA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000000bcc730622c699f5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Sep 23, 2024, 10:47=E2=80=AFAM Warner Losh <imp@bsdimp.com> wrote: > > > On Mon, Sep 23, 2024, 10:41=E2=80=AFAM Alexey Sukhoguzov <mail@eseipi.net= > wrote: > >> Hi, >> >> My NVMe controller is Toshiba XG5, and it has 6 power states: the >> first three (0-2) are normal and the last three (3-5) are NOPS. >> Here is 'nvmecontrol power -l nvme0' output: >> >> # Max pwr Enter Lat Exit Lat RT RL WT WL Idle Pwr Act Pwr Workload= d >> -- -------- --------- --------- -- -- -- -- -------- -------- -- >> 0: 8.0000W 0.000ms 0.000ms 0 0 0 0 0.0000W 0.0000W 0 >> 1: 3.9000W 0.000ms 0.000ms 1 1 1 1 0.0000W 0.0000W 0 >> 2: 2.0000W 0.000ms 0.000ms 2 2 2 2 0.0000W 0.0000W 0 >> 3: 0.0500W* 1.500ms 1.500ms 3 3 3 3 0.0000W 0.0000W 0 >> 4: 0.0050W* 6.000ms 14.000ms 4 4 4 4 0.0000W 0.0000W 0 >> 5: 0.0030W* 50.000ms 80.000ms 5 5 5 5 0.0000W 0.0000W 0 >> >> The problem is that only one of the NOPS is working as expected >> (state 3). Another two (states 4-5) skyrocket the controller's power >> consumption far beyond normal (0-2) power states do, and far beyond >> reasonable. For example, when the controller is in state 3, my >> system consumes about 3-3.5 W at idle (according to acpiconf with >> laptop power cable unplugged), in states 0-2 - about 4 W, and in >> states 4-5 consumption is approaching 6 W. Thus, the NVMe becomes >> the hottest part of the system (>50C, still idle), and it eats up >> almost half of the battery alone. >> >> Linux doesn't have this issue, so it seems to be nvme(4) related. >> All the above data is collected on 14.1-RELEASE Live USB with no >> filesystem mounted. 15-CURRENT has the same problem. >> >> Any ideas what it might be? >> > > Does Linux have active power state management? > And what's the workload? What performance are you seeing? And what's the reported model number? What form factor? I have an m.2 XG5 hanging around. We used these at work years ago, but in only one hw spin. We measured no power diffs btn the states in the system with our streaming workload. They also had a higher latency more quickly than other vendors, but not enough to matter so far (the machines they were in are nearing EOL) and it was only during high write loads iirc. I never looked at the temperature since they never got above our limits. Warner > Warner > > >> Regards, >> Alexey >> >> --0000000000000bcc730622c699f5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Mon, Sep 23, 2024, 10:47=E2=80=AFAM Warner Losh <= ;<a href=3D"mailto:imp@bsdimp.com">imp@bsdimp.com</a>> wrote:<br></div><= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex"><div dir=3D"auto"><div><br><br><div class=3D"= gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Sep 23, 2024, 10= :41=E2=80=AFAM Alexey Sukhoguzov <<a href=3D"mailto:mail@eseipi.net" tar= get=3D"_blank" rel=3D"noreferrer">mail@eseipi.net</a>> wrote:<br></div><= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">Hi,<br> <br> My NVMe controller is Toshiba XG5, and it has 6 power states: the<br> first three (0-2) are normal and the last three (3-5) are NOPS.<br> Here is 'nvmecontrol power -l nvme0' output:<br> <br> =C2=A0#=C2=A0 =C2=A0Max pwr=C2=A0 Enter Lat=C2=A0 Exit Lat RT RL WT WL Idle= Pwr=C2=A0 Act Pwr Workloadd<br> --=C2=A0 --------=C2=A0 --------- --------- -- -- -- -- -------- -------- -= -<br> =C2=A00:=C2=A0 8.0000W=C2=A0 =C2=A0 0.000ms=C2=A0 =C2=A00.000ms=C2=A0 0=C2= =A0 0=C2=A0 0=C2=A0 0=C2=A0 0.0000W=C2=A0 0.0000W 0<br> =C2=A01:=C2=A0 3.9000W=C2=A0 =C2=A0 0.000ms=C2=A0 =C2=A00.000ms=C2=A0 1=C2= =A0 1=C2=A0 1=C2=A0 1=C2=A0 0.0000W=C2=A0 0.0000W 0<br> =C2=A02:=C2=A0 2.0000W=C2=A0 =C2=A0 0.000ms=C2=A0 =C2=A00.000ms=C2=A0 2=C2= =A0 2=C2=A0 2=C2=A0 2=C2=A0 0.0000W=C2=A0 0.0000W 0<br> =C2=A03:=C2=A0 0.0500W*=C2=A0 =C2=A01.500ms=C2=A0 =C2=A01.500ms=C2=A0 3=C2= =A0 3=C2=A0 3=C2=A0 3=C2=A0 0.0000W=C2=A0 0.0000W 0<br> =C2=A04:=C2=A0 0.0050W*=C2=A0 =C2=A06.000ms=C2=A0 14.000ms=C2=A0 4=C2=A0 4= =C2=A0 4=C2=A0 4=C2=A0 0.0000W=C2=A0 0.0000W 0<br> =C2=A05:=C2=A0 0.0030W*=C2=A0 50.000ms=C2=A0 80.000ms=C2=A0 5=C2=A0 5=C2=A0= 5=C2=A0 5=C2=A0 0.0000W=C2=A0 0.0000W 0<br> <br> The problem is that only one of the NOPS is working as expected<br> (state 3). Another two (states 4-5) skyrocket the controller's power<br= > consumption far beyond normal (0-2) power states do, and far beyond<br> reasonable. For example, when the controller is in state 3, my<br> system consumes about 3-3.5 W at idle (according to acpiconf with<br> laptop power cable unplugged), in states 0-2 - about 4 W, and in<br> states 4-5 consumption is approaching 6 W. Thus, the NVMe becomes<br> the hottest part of the system (>50C, still idle), and it eats up<br> almost half of the battery alone.<br> <br> Linux doesn't have this issue, so it seems to be nvme(4) related.<br> All the above data is collected on 14.1-RELEASE Live USB with no<br> filesystem mounted. 15-CURRENT has the same problem.<br> <br> Any ideas what it might be?<br></blockquote></div></div><div dir=3D"auto"><= br></div><div dir=3D"auto">Does Linux have active power state management?</= div></div></blockquote></div></div><div dir=3D"auto"><br></div><div dir=3D"= auto">And what's the workload? What performance are you seeing? And wha= t's the reported model number? What form factor? I have an m.2 XG5 hang= ing around. We used these at work years ago, but in only one hw spin. We me= asured no power diffs btn the states in the system with our streaming workl= oad. They also had a higher latency more quickly than other vendors, but no= t enough to matter so far (the machines they were in are nearing EOL) and i= t was only during high write loads iirc. I never looked at the temperature = since they never got above our limits.</div><div dir=3D"auto"><br></div><di= v dir=3D"auto">Warner</div><div dir=3D"auto"><br></div><div dir=3D"auto"><d= iv class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin:= 0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto"><= div dir=3D"auto"><br></div><div dir=3D"auto">Warner</div><div dir=3D"auto">= <br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquote class=3D= "gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding= -left:1ex"> <br> Regards,<br> Alexey<br> <br> </blockquote></div></div></div> </blockquote></div></div></div> --0000000000000bcc730622c699f5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfp0g118BecYsUjXOj1stMURsDfb-OMK7XT=3i=tCQMBtw>