Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 13 Jul 2026 18:28:04 -0400
From:      Ali Mashtizadeh <mashtizadeh@gmail.com>
To:        Alexander Leidinger <Alexander@leidinger.net>
Cc:        Current <current@freebsd.org>, "M. Warner Losh" <imp@bsdimp.com>
Subject:   Re: Crash in pmc: [pmc,1762] negative increment cpu=1 ri=17 newvalue=45d4906d saved=fffff616cee9 incr=ffff00004fbdc184
Message-ID:  <CAFL8=thKSsgFHshK6Jb=t0vT6eAmd=cF3g7Tq-AOZRiNxC4qkw@mail.gmail.com>
In-Reply-To: <67d8cfdda25c26d1bee79d23254e2768@Leidinger.net>
index | next in thread | previous in thread | raw e-mail 

[-- Attachment #1 --]
Hello Alexander,

I believe this is related to a bug I'm looking into, otherwise you may be
using a counter that's not really a counter.  We probably need to remove
that assert for some of the memory counters.

With process mode counters we eventually don't re-enable the pmc on a
context switch and you will eventually just see zeros for the incremental
counts.  That's probably what's happening here is that when it queries the
PMC it goes from some positive number to zero in the next sample.  Also
explains why you don't see this problem with sleep because it takes a few
context switches for it to show up.

Not sure when this bug was introduced but it seems to be fairly old and I
wasn't readily hitting it in some cases.

Do you mind sharing the pmcstat command you tested with?

Best,
Ali


On Sun, Jul 12, 2026 at 9:19 AM Alexander Leidinger <Alexander@leidinger.net>
wrote:

> Hi,
>
> I wanted to do some pmc based sampling in a jail and this crashed the
> kernel. Not directly, a "sleep 1" test was ok. The real application
> resulted in:
> ---snip---
> __curthread () at /space/system/usr_src/sys/amd64/include/pcpu_aux.h:57
> 57              __asm("movq %%gs:%c1,%0" : "=r" (td)
> (kgdb) #0  __curthread () at
> /space/system/usr_src/sys/amd64/include/pcpu_aux.h:57
>          td = <optimized out>
> #1  doadump (textdump=textdump@entry=1)
>      at /space/system/usr_src/sys/kern/kern_shutdown.c:399
>          error = 0
>          coredump = <optimized out>
> #2  0xffffffff8056183c in kern_reboot (howto=260)
>      at /space/system/usr_src/sys/kern/kern_shutdown.c:519
>          once = 0
>          __pc = 0x0
> #3  0xffffffff80561d5d in vpanic (fmt=<optimized out>,
>      ap=ap@entry=0xfffffe0797916c90)
>      at /space/system/usr_src/sys/kern/kern_shutdown.c:974
>          buf = "[pmc,1762] negative increment cpu=1 ri=17
> newvalue=45d4906d saved=fffff616cee9 incr=ffff00004fbdc184", '\000'
> <repeats 155 times>
>          __pc = 0x0
>          __pc = 0x0
>          __pc = 0x0
>          other_cpus = {__bits = {16777213, 0 <repeats 15 times>}}
>          td = 0xfffff80a4c988000
>          bootopt = <unavailable>
>          newpanic = <optimized out>
> #4  0xffffffff80561ba3 in panic (fmt=<unavailable>)
>      at /space/system/usr_src/sys/kern/kern_shutdown.c:887
>          ap = {{gp_offset = 48, fp_offset = 48,
>              overflow_arg_area = 0xfffffe0797916cc8,
>              reg_save_area = 0xfffffe0797916c60}}
> #5  0xffffffff8455ae47 in pmc_hook_handler (td=0xfffff80a4c988000,
>      function=<optimized out>, arg=<optimized out>)
>      at /space/system/usr_src/sys/dev/hwpmc/hwpmc_mod.c:1759
>          __pc = 0x0
>          __pc = 0x0
>          __pc = 0x0
>          __pc = 0x0
>          cpu = <optimized out>
> #6  0xffffffff80596d8d in sched_ule_sswitch (td=0xfffff80a4c988000,
>      flags=<optimized out>) at
> /space/system/usr_src/sys/kern/sched_ule.c:2389
>          __pc = 0x0
>          __pc = 0x0
>          __pc = 0x0
>          cpuid = <optimized out>
>          tdq = <optimized out>
>          ts = 0xfffff80a4c988720
>          preempted = <optimized out>
>          mtx = 0xfffffe0065b291c0
>          srqflag = <optimized out>
>          newtd = 0xfffff80df74e4780
>          pickcpu = <optimized out>
> #7  0xffffffff8056e8c8 in mi_switch (flags=flags@entry=517)
>      at /space/system/usr_src/sys/kern/kern_synch.c:526
>          __pc = 0x0
>          __pc = 0x0
>          __pc = 0x0
>          __pc = 0x0
>          __pc = 0x0
>          td = 0xfffff80a4c988000
>          new_switchtime = <optimized out>
>          runtime = <optimized out>
> ---snip---
>
> Core available.
>
> Bye,
> Alexander.
>
> --
> http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
> http://www.FreeBSD.org    netchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF
>

[-- Attachment #2 --]
<div dir="ltr"><div>Hello Alexander,</div><div><br></div><div>I believe this is related to a bug I&#39;m looking into, otherwise you may be using a counter that&#39;s not really a counter.  We probably need to remove that assert for some of the memory counters.</div><div><br></div><div>With process mode counters we eventually don&#39;t re-enable the pmc on a context switch and you will eventually just see zeros for the incremental counts.  That&#39;s probably what&#39;s happening here is that when it queries the PMC it goes from some positive number to zero in the next sample.  Also explains why you don&#39;t see this problem with sleep because it takes a few context switches for it to show up.</div><div><br></div><div>Not sure when this bug was introduced but it seems to be fairly old and I wasn&#39;t readily hitting it in some cases.</div><div><div></div><div><br></div><div>Do you mind sharing the pmcstat command you tested with?</div><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Best,<br></div>Ali<br></div></div></div><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sun, Jul 12, 2026 at 9:19 AM Alexander Leidinger &lt;<a href="mailto:Alexander@leidinger.net">Alexander@leidinger.net</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I wanted to do some pmc based sampling in a jail and this crashed the <br>
kernel. Not directly, a &quot;sleep 1&quot; test was ok. The real application <br>
resulted in:<br>
---snip---<br>
__curthread () at /space/system/usr_src/sys/amd64/include/pcpu_aux.h:57<br>
57              __asm(&quot;movq %%gs:%c1,%0&quot; : &quot;=r&quot; (td)<br>
(kgdb) #0  __curthread () at <br>
/space/system/usr_src/sys/amd64/include/pcpu_aux.h:57<br>
         td = &lt;optimized out&gt;<br>
#1  doadump (textdump=textdump@entry=1)<br>
     at /space/system/usr_src/sys/kern/kern_shutdown.c:399<br>
         error = 0<br>
         coredump = &lt;optimized out&gt;<br>
#2  0xffffffff8056183c in kern_reboot (howto=260)<br>
     at /space/system/usr_src/sys/kern/kern_shutdown.c:519<br>
         once = 0<br>
         __pc = 0x0<br>
#3  0xffffffff80561d5d in vpanic (fmt=&lt;optimized out&gt;,<br>
     ap=ap@entry=0xfffffe0797916c90)<br>
     at /space/system/usr_src/sys/kern/kern_shutdown.c:974<br>
         buf = &quot;[pmc,1762] negative increment cpu=1 ri=17 <br>
newvalue=45d4906d saved=fffff616cee9 incr=ffff00004fbdc184&quot;, &#39;\000&#39; <br>
&lt;repeats 155 times&gt;<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         other_cpus = {__bits = {16777213, 0 &lt;repeats 15 times&gt;}}<br>
         td = 0xfffff80a4c988000<br>
         bootopt = &lt;unavailable&gt;<br>
         newpanic = &lt;optimized out&gt;<br>
#4  0xffffffff80561ba3 in panic (fmt=&lt;unavailable&gt;)<br>
     at /space/system/usr_src/sys/kern/kern_shutdown.c:887<br>
         ap = {{gp_offset = 48, fp_offset = 48,<br>
             overflow_arg_area = 0xfffffe0797916cc8,<br>
             reg_save_area = 0xfffffe0797916c60}}<br>
#5  0xffffffff8455ae47 in pmc_hook_handler (td=0xfffff80a4c988000,<br>
     function=&lt;optimized out&gt;, arg=&lt;optimized out&gt;)<br>
     at /space/system/usr_src/sys/dev/hwpmc/hwpmc_mod.c:1759<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         cpu = &lt;optimized out&gt;<br>
#6  0xffffffff80596d8d in sched_ule_sswitch (td=0xfffff80a4c988000,<br>
     flags=&lt;optimized out&gt;) at <br>
/space/system/usr_src/sys/kern/sched_ule.c:2389<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         cpuid = &lt;optimized out&gt;<br>
         tdq = &lt;optimized out&gt;<br>
         ts = 0xfffff80a4c988720<br>
         preempted = &lt;optimized out&gt;<br>
         mtx = 0xfffffe0065b291c0<br>
         srqflag = &lt;optimized out&gt;<br>
         newtd = 0xfffff80df74e4780<br>
         pickcpu = &lt;optimized out&gt;<br>
#7  0xffffffff8056e8c8 in mi_switch (flags=flags@entry=517)<br>
     at /space/system/usr_src/sys/kern/kern_synch.c:526<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         __pc = 0x0<br>
         td = 0xfffff80a4c988000<br>
         new_switchtime = &lt;optimized out&gt;<br>
         runtime = &lt;optimized out&gt;<br>
---snip---<br>
<br>
Core available.<br>
<br>
Bye,<br>
Alexander.<br>
<br>
-- <br>
<a href="http://www.Leidinger.net" rel="noreferrer" target="_blank">http://www.Leidinger.net</a>; Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF<br>
<a href="http://www.FreeBSD.org" rel="noreferrer" target="_blank">http://www.FreeBSD.org</a>    netchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF<br>
</blockquote></div>
home | help

Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFL8=thKSsgFHshK6Jb=t0vT6eAmd=cF3g7Tq-AOZRiNxC4qkw>