From owner-freebsd-current@freebsd.org Fri Mar 16 23:46:12 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D846FF577A4 for ; Fri, 16 Mar 2018 23:46:11 +0000 (UTC) (envelope-from bryan-lists@shatow.net) Received: from mail.xzibition.com (mail.xzibition.com [52.11.127.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 598D87AD09 for ; Fri, 16 Mar 2018 23:46:11 +0000 (UTC) (envelope-from bryan-lists@shatow.net) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id C0B99738 for ; Fri, 16 Mar 2018 23:46:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id KD3mTsyzvRDd for ; Fri, 16 Mar 2018 23:46:00 +0000 (UTC) Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com AC46D732 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=shatow.net; s=mxc204805312015; t=1521243960; bh=hKcesEZSxKHDF0emAr0/AAcU1gA6Kn6BxDZwC6s+2Mk=; h=Subject:To:References:From:Date:In-Reply-To; b=LUFHpFqQvwGAbMOrS71Hch2sKsWssXP2eKWD7gqZfuGJfQAfMxob8YYcwUZgCMlNG ZrC8DFsKiC7iYl+dqIc5tomWnJKRT7S+LOhH5BHnpqWqqs0Wc+kbPZLe9Nr98aSzMZ D2MZYSnkEAFmueg8xrE6YOeonSCV7FuLgiK0a/b3gCnnm5wwadmffqibmXSzTGvqFp w9o5QatVPK3AWS1zUmO4RNlxPxIcBC9ig9Rf3zlV94HBk25kAK74qxGd7Fi+euDVsp 0ImcAj4L56ERlMuf35rdl3f7vBbDaD1AALui++qH7HNmFc3N97vWfP8Y1RsbSGFqsQ iTqEZqS2YFEyA== To: freebsd-current@freebsd.org References: <20180306173455.oacyqlbib4sbafqd@ler-imac.lerctr.org> <201803061816.w26IGaW5050053@pdx.rh.CN85.dnsmgr.net> <20180306193645.vv3ogqrhauivf2tr@ler-imac.lerctr.org> <20180306221554.uyshbzbboai62rdf@dx240.localdomain> <20180307103911.GA72239@kloomba> <20180311004737.3441dbf9@thor.intern.walstatt.dynvpn.de> From: Bryan Drewery Openpgp: preference=signencrypt Message-ID: Date: Fri, 16 Mar 2018 16:45:59 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Mar 2018 23:46:12 -0000 On 3/11/2018 1:43 PM, Jeff Roberson wrote: > Hi Folks, > > This could be my fault from recent NUMA and concurrency related work.  I > did touch some of the arc back-pressure mechanisms.  First, I would like > to identify whether the wired memory is in the buffer cache.  Can those > of you that have a repro look at sysctl vfs.bufspace and tell me if that > accounts for the bulk of your wired memory usage?  I'm wondering if a > job ran that pulled in all of the bufs from your root disk and filled up > the buffer cache which doesn't have a back-pressure mechanism.  Then arc > didn't respond appropriately to lower its usage. > > Also, if you could try going back to r328953 or r326346 and let me know > if the problem exists in either.  That would be very helpful.  If anyone > is willing to debug this with me contact me directly and I will send > some test patches or debugging info after you have done the above steps. > > Thank you for the reports. > > Jeff It's like there is no ZFS ARC backpressure at all anymore... Mem: 8783M Active, 2851M Inact, 3716M Laundry, 53G Wired, 2449M Free ARC: 40G Total, 27G MFU, 11G MRU, 13M Anon, 631M Header, 955M Other 37G Compressed, 53G Uncompressed, 1.43:1 Ratio Swap: 116G Total, 18G Used, 98G Free, 15% Inuse, 4K In vfs.bufspace: 0 I ran out of swap space around 16GB and had to add a larger device on to keep a package test build going. Was there ever backpressure? Having to limit the arc size has been a constant workaround for package building. I have some large tmpfs at the moment totaling around ~28GB. > # df -ah|egrep '(Used|tmpfs)' > Filesystem Size Used Avail Capacity Mounted on > tmpfs 99G 520K 99G 0% /tmp > tmpfs 10G 8.0K 10G 0% /poudriere/data/.m/exp-11amd64-commit-test > tmpfs 10G 373M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/ref > tmpfs 1.0G 18M 1.0G 2% /poudriere/data/.m/exp-11amd64-commit-test/ref/.p > tmpfs 10G 176K 10G 0% /poudriere/data/.m/exp-11amd64-commit-test/ref/var/db/ports > tmpfs 10G 405M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/01 > tmpfs 10G 1.1G 8.9G 11% /poudriere/data/.m/exp-11amd64-commit-test/05 > tmpfs 10G 2.4G 7.6G 24% /poudriere/data/.m/exp-11amd64-commit-test/04 > tmpfs 10G 578M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/03 > tmpfs 10G 1.0G 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/02 > tmpfs 10G 936M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/06 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/07 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/09 > tmpfs 10G 875M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/08 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/10 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/12 > tmpfs 10G 390M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/11 > tmpfs 10G 992M 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/14 > tmpfs 10G 2.9G 7.1G 29% /poudriere/data/.m/exp-11amd64-commit-test/13 > tmpfs 1.0G 16K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/01/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/12/.p > tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/05/.p > tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/06/.p > tmpfs 1.0G 396K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/03/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/09/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/10/.p > tmpfs 1.0G 732K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/07/.p > tmpfs 1.0G 28K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/08/.p > tmpfs 1.0G 12K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/11/.p > tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/02/.p > tmpfs 1.0G 852K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/14/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/13/.p > tmpfs 1.0G 688K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/04/.p > tmpfs 10G 125M 9.9G 1% /poudriere/data/.m/exp-11amd64-commit-test/03/usr/local > tmpfs 10G 270M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/04/usr/local > tmpfs 10G 414M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/07/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/09/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/10/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/12/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/13/usr/local > tmpfs 10G 575M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/05/usr/local > tmpfs 10G 575M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/02/usr/local > tmpfs 10G 306M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/14/usr/local > tmpfs 10G 570M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/06/usr/local > tmpfs 10G 36M 10G 0% /poudriere/data/.m/exp-11amd64-commit-test/01/usr/local > tmpfs 10G 283M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/11/usr/local > tmpfs 10G 301M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/08/usr/local And some large processes. > # ps aux|sort -rnk 6|head -n 20 > root 2481 100.0 4.3 3334388 3209728 15 RJ 14:40 119:54.18 /usr/local/bin/python2.7 ../../rpython/bin/rpython --source -Ojit targetpypystandalone.py > root 46466 97.7 0.8 620692 581304 15 RJ 16:44 0:36.07 /wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib/bin/ghc-stage1 -B/wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib -hisuf hi -osuf o -hcsuf hc -static -H32m -O -I/usr/include -L/usr/lib -I/usr/loc > root 44667 100.0 0.2 187664 161444 15 RJ 16:43 1:04.16 /wrkdirs/usr/ports/lang/gcc6/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc/c-family - > root 47917 76.5 0.2 173572 140580 15 RJ 16:44 0:12.45 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name MemorySanitizer.cpp -mrelocation-model pic -pic-level 2 -mthread-model > root 47918 75.9 0.2 164784 136976 15 RJ 16:44 0:12.50 /wrkdirs/usr/ports/lang/gcc7-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20180308/gcc -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20 > root 47938 71.5 0.2 159952 134260 15 RJ 16:44 0:11.17 /wrkdirs/usr/ports/lang/gcc6-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6-devel/work/gcc-6-20180307/gcc -I /wrkdirs/usr/ports/lang/gcc6-devel/work/g > root 48090 48.1 0.2 159420 125748 15 RJ 16:44 0:06.02 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name AlignmentFromAssumptions.cpp -mrelocation-model pic -pic-level 2 -mthre > root 48110 38.4 0.2 153988 116220 15 RJ 16:44 0:04.35 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name property-type.cxx -mrelocation-model static -mthread-model posix -mdisa > root 48139 35.4 0.1 131636 101608 15 RJ 16:44 0:03.70 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name InstrInfoEmitter.cpp -mrelocation-model pic -pic-level 2 -mthread-model > root 48119 34.2 0.1 126396 93768 15 RJ 16:44 0:03.79 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name NaryReassociate.cpp -mrelocation-model pic -pic-level 2 -mthread-model -- Regards, Bryan Drewery