From owner-freebsd-fs@freebsd.org Tue Nov 20 12:42:36 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C766C1127E25; Tue, 20 Nov 2018 12:42:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3C2CF6C308; Tue, 20 Nov 2018 12:42:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 8B29817C6; Tue, 20 Nov 2018 15:42:33 +0300 (MSK) To: FreeBSD stable , FreeBSD FS Reply-To: lev@FreeBSD.org From: Lev Serebryakov Subject: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Tue, 20 Nov 2018 15:42:24 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu" X-Rspamd-Queue-Id: 3C2CF6C308 X-Spamd-Result: default: False [0.35 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.54)[0.535,0]; NEURAL_HAM_LONG(-0.22)[-0.218,0]; NEURAL_SPAM_MEDIUM(0.03)[0.028,0]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 12:42:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu Content-Type: multipart/mixed; boundary="3NXEgeGscIkTvx5g8diqcZbnH3iAg3lgP"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: FreeBSD stable , FreeBSD FS Message-ID: Subject: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. --3NXEgeGscIkTvx5g8diqcZbnH3iAg3lgP Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable I have server which is mostly torrent box. It uses ZFS and equipped with 16GiB of physical memory. It is running 11-STABLE (r339914 now). I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago. I was used to see 13-14GiB of memory in ZFS ARC and it was Ok. Sometimes it "locks" under heavy disk load due to ARC memory pressure, but it was bearable, and as ZFS is main reason this server exists, I didn't limit ARC. But new revision (r339914) shows very strange behaivor: ARC is no more than 4GiB, but kernel has 15GiB wired: Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other= 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio It is typical numbers for last week: 15G wired, 237M Free, but only 4252M ARC! Where is other 11G of memory?! I've checked USED and FREE in "vmstat -z" output and got this: $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n | tail -20 23001088 9171456 MAP ENTRY 29680800 8404320 VM OBJECT 34417408 10813952 256 36377964 2665656 S VFS Cache 50377392 53856 sa_cache 50593792 622985216 zio_buf_131072 68913152 976896 mbuf_cluster 73543680 7225344 mbuf_jumbo_page 92358552 67848 zfs_znode_cache 95731712 51761152 4096 126962880 159581760 dmu_buf_impl_t 150958080 233920512 mbuf_jumbo_9k 165164600 92040 VNODE 192701120 30350880 UMA Slabs 205520896 291504128 zio_data_buf_1048576 222822400 529530880 zio_data_buf_524288 259143168 293476864 zio_buf_512 352485376 377061376 zio_buf_16384 376109552 346474128 dnode_t 2943016960 5761941504 abd_chunk $ And total USED/FREE numbers is very strange for me: $ vmstat -z | tr : , | awk -F , '1{u+=3D$2*$4; f+=3D$2*$5} END{print u,f}= ' 5717965420 9328951088 $ So, only ~5.7G is used and 9.3G is free! But why this memory is not used by ARC anymore and why is it wired and not free? --=20 // Lev Serebryakov --3NXEgeGscIkTvx5g8diqcZbnH3iAg3lgP-- --2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlv0AThfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R48fbg//RdXbTqLsT62Wmp+HSbl8JMu2TYAyXJ10elr7nhm8wptE+xNoczvvXq9W 2iy8tFBB8vcpdUaMf4MKlioI3c/+IK8aFw7QhoPJi+C1TQ4OaDfMZ6nuwRS4uHF1 /j8x6HYxBEpO4TCacOM7tjzcKVkXdFnsKWyhbqJ2y6iycQTAGzqc+xbAiaHId62x wIo1s0Y4lGpLZigWGKz90Z4IytwmUQYlQeTx14PTH8v9xf6ftpERKvzptirTRsyd MeLL67VzOEKBPOg8RVpNAr9Ti/atc7ZmmbgVfo8MsisKgn9uJEAQSrS7GGebAx19 9Ss4aNbo4EYFlsPR3tULR/pO8mnN+3pMsVqUSEbML9Mu20QTsjHhQ2AP5MWzSDGA 471WHJIPJczOh7qwyx6K0ujQMrD+t/0PDWx8G5ulX0IJxRenNfFWRqfg4gU49jB5 NjyvfOzpcqRniZ+g/M0lJpHk6ZiDXTRSqT8gULPPx+ZJitFvhJHaD6pKr1Qcwu49 TMRTtcysnLxUuV3kXBhYCllGOc0Jxg3HkWkGj7HL7v7PJABGEUyD/FyF/Qjj0Bed 7DkVkGKCb/FqcYSLP3AI8St/KJgM06t7KiTXwIZZvfEdtjVuVWIDy6yU+a6ek99h j2hn9EhsED8y2or5dGhV/jQOdoiuQcSAe+F3Q7yunjOXm4zHDv4= =QkSG -----END PGP SIGNATURE----- --2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu--