Date: Tue, 20 Nov 2018 15:45:37 -0500 From: Mark Johnston <markj@freebsd.org> To: Lev Serebryakov <lev@freebsd.org> Cc: FreeBSD stable <freebsd-stable@freebsd.org>, FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. Message-ID: <20181120204537.GB2630@raichu> In-Reply-To: <e2422965-0890-2d15-4716-0a31426b0a66@FreeBSD.org> References: <e2422965-0890-2d15-4716-0a31426b0a66@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 20, 2018 at 03:42:24PM +0300, Lev Serebryakov wrote:
>
> I have server which is mostly torrent box. It uses ZFS and equipped
> with 16GiB of physical memory. It is running 11-STABLE (r339914 now).
>
> I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago.
>
> I was used to see 13-14GiB of memory in ZFS ARC and it was Ok.
> Sometimes it "locks" under heavy disk load due to ARC memory pressure,
> but it was bearable, and as ZFS is main reason this server exists, I
> didn't limit ARC.
>
> But new revision (r339914) shows very strange behaivor: ARC is no more
> than 4GiB, but kernel has 15GiB wired:
>
> Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free
> ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other
> 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio
>
> It is typical numbers for last week: 15G wired, 237M Free, but only
> 4252M ARC!
>
> Where is other 11G of memory?!
>
> I've checked USED and FREE in "vmstat -z" output and got this:
>
> $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n |
> tail -20
> 23001088 9171456 MAP ENTRY
> 29680800 8404320 VM OBJECT
> 34417408 10813952 256
> 36377964 2665656 S VFS Cache
> 50377392 53856 sa_cache
> 50593792 622985216 zio_buf_131072
> 68913152 976896 mbuf_cluster
> 73543680 7225344 mbuf_jumbo_page
> 92358552 67848 zfs_znode_cache
> 95731712 51761152 4096
> 126962880 159581760 dmu_buf_impl_t
> 150958080 233920512 mbuf_jumbo_9k
> 165164600 92040 VNODE
> 192701120 30350880 UMA Slabs
> 205520896 291504128 zio_data_buf_1048576
> 222822400 529530880 zio_data_buf_524288
> 259143168 293476864 zio_buf_512
> 352485376 377061376 zio_buf_16384
> 376109552 346474128 dnode_t
> 2943016960 5761941504 abd_chunk
> $
>
> And total USED/FREE numbers is very strange for me:
>
> $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}'
> 5717965420 9328951088
> $
>
> So, only ~5.7G is used and 9.3G is free! But why this memory is not
> used by ARC anymore and why is it wired and not free?
Could you show the output of "vmstat -s" when in this state?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20181120204537.GB2630>