Skip site navigation (1)Skip section navigation (2) 
Date:      Fri, 8 Apr 2022 19:02:49 +0000
From:      Dave Baukus <daveb@spectralogic.com>
To:        "freebsd-fs@FreeBSD.org" <freebsd-fs@FreeBSD.org>
Subject:   ZFS: OoM with very large kmem_cache_create("abd_chunk", ...) USED count
Message-ID:  <0b29483d-114f-c48c-343d-b25e6374290d@spectralogic.com>
index | next in thread | raw e-mail 

[-- Attachment #1 --]
On FreeBSD_stable13 I'm seeing multiple OoM crash dumps with very large adb_chunk
used count as reported by vmstat -z -M <crash dump> -N <kern>:

For example on a system with 128G memory:
vmstat -z -M <crash dump> -N <kern>m reports:

ITEM                   SIZE  LIMIT     USED     FREE      REQ     FAILSLEEP XDOMAIN
...
abd_chunk:             4096,      0,22967817,    5657,7737623005,   0,   0,3345348454
...

Meanwhile ARC stats are reasonable:
arc_stats.arcstat_size:  68955842224
arc_stats.arcstat_c:     68107622992
arc_stats.arcstat_c_max: 68643903488

But, adbstat_struct_size is impossibly large:

  abdstat_struct_size = {
    name = "struct_size", '\000' <repeats 243 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 207162941552,
      l = 207162941552,
      ul = 207162941552,
    }
  },
  abdstat_scatter_cnt = {
    name = "scatter_cnt", '\000' <repeats 243 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 76786,
      l = 76786,
    }
  },
  abdstat_scatter_data_size = {
    name = "scatter_data_size", '\000' <repeats 237 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 65734410752,
      l = 65734410752,
    }
  },
  abdstat_scatter_chunk_waste = {
    name = "scatter_chunk_waste", '\000' <repeats 235 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 482816,
      l = 482816,
    }
  },
  abdstat_linear_cnt = {
    name = "linear_cnt", '\000' <repeats 244 times>,
    data_type = 4 '\004',
    value = {
      l = 9822,
      ul = 9822,
    }
  },
  abdstat_linear_data_size = {
    name = "linear_data_size", '\000' <repeats 238 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 34973696,
      l = 34973696,
    }
  }

=========

anyone seen anything similar ?
--

Dave Baukus


[-- Attachment #2 --]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<font face="monospace">On FreeBSD_stable13 I'm seeing multiple OoM crash dumps with very large adb_chunk<br>
used count as reported by vmstat -z -M &lt;crash dump&gt; -N &lt;kern&gt;:<br>
<br>
For example on a system with 128G memory:<br>
</font><font face="monospace"><font face="monospace">vmstat -z -M &lt;crash dump&gt; -N &lt;kern&gt;m reports:<br>
<br>
</font>ITEM&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SIZE&nbsp; LIMIT&nbsp;&nbsp;&nbsp;&nbsp; USED&nbsp;&nbsp;&nbsp;&nbsp; FREE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; REQ&nbsp;&nbsp;&nbsp;&nbsp; FAILSLEEP XDOMAIN<br>
...<br>
abd_chunk:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4096,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0,22967817,&nbsp;&nbsp;&nbsp; 5657,7737623005,&nbsp;&nbsp; 0,&nbsp;&nbsp; 0,3345348454<br>
...<br>
<br>
Meanwhile ARC stats are reasonable:<br>
arc_stats.arcstat_size:&nbsp; 68955842224<br>
arc_stats.arcstat_c:&nbsp;&nbsp;&nbsp;&nbsp; 68107622992<br>
arc_stats.arcstat_c_max: 68643903488<br>
<br>
But, adbstat_struct_size is impossibly large:<br>
<br>
&nbsp; abdstat_struct_size = {<br>
&nbsp;&nbsp;&nbsp; name = &quot;struct_size&quot;, '\000' &lt;repeats 243 times&gt;,<br>
&nbsp;&nbsp;&nbsp; data_type = 4 '\004',<br>
&nbsp;&nbsp;&nbsp; value = {<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ui64 = 207162941552,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; l = 207162941552,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ul = 207162941552,<br>
&nbsp;&nbsp;&nbsp; }<br>
&nbsp; },<br>
&nbsp; abdstat_scatter_cnt = {<br>
&nbsp;&nbsp;&nbsp; name = &quot;scatter_cnt&quot;, '\000' &lt;repeats 243 times&gt;,<br>
&nbsp;&nbsp;&nbsp; data_type = 4 '\004',<br>
&nbsp;&nbsp;&nbsp; value = {<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ui64 = 76786,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; l = 76786,<br>
&nbsp;&nbsp;&nbsp; }<br>
&nbsp; },<br>
&nbsp; abdstat_scatter_data_size = {<br>
&nbsp;&nbsp;&nbsp; name = &quot;scatter_data_size&quot;, '\000' &lt;repeats 237 times&gt;,<br>
&nbsp;&nbsp;&nbsp; data_type = 4 '\004',<br>
&nbsp;&nbsp;&nbsp; value = {<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ui64 = 65734410752,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; l = 65734410752,<br>
&nbsp;&nbsp;&nbsp; }<br>
&nbsp; },<br>
&nbsp; abdstat_scatter_chunk_waste = {<br>
&nbsp;&nbsp;&nbsp; name = &quot;scatter_chunk_waste&quot;, '\000' &lt;repeats 235 times&gt;,<br>
&nbsp;&nbsp;&nbsp; data_type = 4 '\004',<br>
&nbsp;&nbsp;&nbsp; value = {<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ui64 = 482816,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; l = 482816,<br>
&nbsp;&nbsp;&nbsp; }<br>
&nbsp; },<br>
&nbsp; abdstat_linear_cnt = {<br>
&nbsp;&nbsp;&nbsp; name = &quot;linear_cnt&quot;, '\000' &lt;repeats 244 times&gt;,<br>
&nbsp;&nbsp;&nbsp; data_type = 4 '\004',<br>
&nbsp;&nbsp;&nbsp; value = {<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; l = 9822,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ul = 9822,<br>
&nbsp;&nbsp;&nbsp; }<br>
&nbsp; },<br>
&nbsp; abdstat_linear_data_size = {<br>
&nbsp;&nbsp;&nbsp; name = &quot;linear_data_size&quot;, '\000' &lt;repeats 238 times&gt;,<br>
&nbsp;&nbsp;&nbsp; data_type = 4 '\004',<br>
&nbsp;&nbsp;&nbsp; value = {<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ui64 = 34973696,<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; l = 34973696,<br>
&nbsp;&nbsp;&nbsp; }<br>
&nbsp; }<br>
<br>
=========<br>
<br>
anyone seen anything similar ?<br>
</font>--
<pre class="moz-signature" cols="10000">Dave Baukus
</pre>
</body>
</html>
help

Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0b29483d-114f-c48c-343d-b25e6374290d>