From owner-freebsd-fs@freebsd.org Wed Nov 21 11:34:49 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E00531135142; Wed, 21 Nov 2018 11:34:48 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 6A5A87B88E; Wed, 21 Nov 2018 11:34:48 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:310b:1d84:68df:8590]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 58A901A33; Wed, 21 Nov 2018 14:34:46 +0300 (MSK) Date: Wed, 21 Nov 2018 14:34:45 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <123493949.20181121143445@serebryakov.spb.ru> To: Mark Johnston CC: FreeBSD FS , FreeBSD stable Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. In-Reply-To: <1999292077.20181121135618@serebryakov.spb.ru> References: <20181120204537.GB2630@raichu> <1999292077.20181121135618@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 6A5A87B88E X-Spamd-Result: default: False [0.52 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.26)[0.259,0]; NEURAL_HAM_LONG(-0.19)[-0.194,0]; NEURAL_SPAM_MEDIUM(0.46)[0.458,0]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 11:34:49 -0000 Hello Lev, Wednesday, November 21, 2018, 1:56:18 PM, you wrote: >> Could you show the output of "vmstat -s" when in this state? > Now ARC is 1.9GB (!!!) and 15G is still Wired: > % vmstat -s > 1311134466 cpu context switches > 2126950424 device interrupts > 45414140 software interrupts > 103215017 traps > 2910067754 system calls > 25 kernel threads created > 108756 fork() calls > 50914 vfork() calls > 111 rfork() calls > 0 swap pager pageins > 0 swap pager pages paged in > 0 swap pager pageouts > 0 swap pager pages paged out > 4201 vnode pager pageins > 51719 vnode pager pages paged in > 31597 vnode pager pageouts > 63028 vnode pager pages paged out > 7 page daemon wakeups > 1212387275 pages examined by the page daemon > 0 clean page reclamation shortfalls > 0 pages reactivated by the page daemon > 8281789 copy-on-write faults > 7909 copy-on-write optimized faults > 44440180 zero fill pages zeroed > 24762 zero fill pages prezeroed > 2 intransit blocking page faults > 63838217 total VM faults taken > 3816 page faults requiring I/O > 0 pages affected by kernel thread creation > 3996512 pages affected by fork() > 1842784 pages affected by vfork() > 5499 pages affected by rfork() > 88771375 pages freed > 0 pages freed by daemon > 21591789 pages freed by exiting processes > 4936 pages active > 161234 pages inactive > 16674 pages in the laundry queue > 3848989 pages wired down > 39951 pages free > 4096 bytes per page > 612681203 total name lookups > cache hits (95% pos + 0% neg) system 0% per-directory > deletions 0%, falsehits 0%, toolong 0% > % And here top consumers. I don't like that most of te memory is in FREE abd_chunks - 8GB! USED (b) FREE (b) NAME 28785840 9299280 VM OBJECT 33817600 11413760 256 36598392 2486808 S VFS Cache 38157856 246224 arc_buf_hdr_t_l2only 44302336 629276672 zio_buf_131072 50377824 53424 sa_cache 72323072 85626880 mbuf_jumbo_page 74094592 55296 mbuf_cluster 90320896 63102976 4096 92359344 67056 zfs_znode_cache 126912480 159632160 dmu_buf_impl_t 150967296 233911296 mbuf_jumbo_9k 165164600 92040 VNODE 199124480 23927520 UMA Slabs 209715200 8388608 zio_data_buf_1048576 218103808 48758784 zio_data_buf_524288 259242496 293377536 zio_buf_512 346030080 383516672 zio_buf_16384 375592672 346991008 dnode_t 531902464 8392536064 abd_chunk -- Best regards, Lev mailto:lev@FreeBSD.org