Date: Fri, 16 Mar 2018 16:45:59 -0700 From: Bryan Drewery <bryan-lists@shatow.net> To: freebsd-current@freebsd.org Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT Message-ID: <a2bf20a3-b5a4-f6e3-3692-4c2783b03b4d@shatow.net> In-Reply-To: <alpine.BSF.2.21.1803111038460.1232@desktop> References: <20180306173455.oacyqlbib4sbafqd@ler-imac.lerctr.org> <201803061816.w26IGaW5050053@pdx.rh.CN85.dnsmgr.net> <20180306193645.vv3ogqrhauivf2tr@ler-imac.lerctr.org> <20180306221554.uyshbzbboai62rdf@dx240.localdomain> <20180307103911.GA72239@kloomba> <20180311004737.3441dbf9@thor.intern.walstatt.dynvpn.de> <alpine.BSF.2.21.1803111038460.1232@desktop>
next in thread | previous in thread | raw e-mail | index | archive | help
On 3/11/2018 1:43 PM, Jeff Roberson wrote:
> Hi Folks,
>
> This could be my fault from recent NUMA and concurrency related work. I
> did touch some of the arc back-pressure mechanisms. First, I would like
> to identify whether the wired memory is in the buffer cache. Can those
> of you that have a repro look at sysctl vfs.bufspace and tell me if that
> accounts for the bulk of your wired memory usage? I'm wondering if a
> job ran that pulled in all of the bufs from your root disk and filled up
> the buffer cache which doesn't have a back-pressure mechanism. Then arc
> didn't respond appropriately to lower its usage.
>
> Also, if you could try going back to r328953 or r326346 and let me know
> if the problem exists in either. That would be very helpful. If anyone
> is willing to debug this with me contact me directly and I will send
> some test patches or debugging info after you have done the above steps.
>
> Thank you for the reports.
>
> Jeff
It's like there is no ZFS ARC backpressure at all anymore...
Mem: 8783M Active, 2851M Inact, 3716M Laundry, 53G Wired, 2449M Free
ARC: 40G Total, 27G MFU, 11G MRU, 13M Anon, 631M Header, 955M Other
37G Compressed, 53G Uncompressed, 1.43:1 Ratio
Swap: 116G Total, 18G Used, 98G Free, 15% Inuse, 4K In
vfs.bufspace: 0
I ran out of swap space around 16GB and had to add a larger device on to
keep a package test build going.
Was there ever backpressure? Having to limit the arc size has been a
constant workaround for package building.
I have some large tmpfs at the moment totaling around ~28GB.
> # df -ah|egrep '(Used|tmpfs)'
> Filesystem Size Used Avail Capacity Mounted on
> tmpfs 99G 520K 99G 0% /tmp
> tmpfs 10G 8.0K 10G 0% /poudriere/data/.m/exp-11amd64-commit-test
> tmpfs 10G 373M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/ref
> tmpfs 1.0G 18M 1.0G 2% /poudriere/data/.m/exp-11amd64-commit-test/ref/.p
> tmpfs 10G 176K 10G 0% /poudriere/data/.m/exp-11amd64-commit-test/ref/var/db/ports
> tmpfs 10G 405M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/01
> tmpfs 10G 1.1G 8.9G 11% /poudriere/data/.m/exp-11amd64-commit-test/05
> tmpfs 10G 2.4G 7.6G 24% /poudriere/data/.m/exp-11amd64-commit-test/04
> tmpfs 10G 578M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/03
> tmpfs 10G 1.0G 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/02
> tmpfs 10G 936M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/06
> tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/07
> tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/09
> tmpfs 10G 875M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/08
> tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/10
> tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/12
> tmpfs 10G 390M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/11
> tmpfs 10G 992M 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/14
> tmpfs 10G 2.9G 7.1G 29% /poudriere/data/.m/exp-11amd64-commit-test/13
> tmpfs 1.0G 16K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/01/.p
> tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/12/.p
> tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/05/.p
> tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/06/.p
> tmpfs 1.0G 396K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/03/.p
> tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/09/.p
> tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/10/.p
> tmpfs 1.0G 732K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/07/.p
> tmpfs 1.0G 28K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/08/.p
> tmpfs 1.0G 12K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/11/.p
> tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/02/.p
> tmpfs 1.0G 852K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/14/.p
> tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/13/.p
> tmpfs 1.0G 688K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/04/.p
> tmpfs 10G 125M 9.9G 1% /poudriere/data/.m/exp-11amd64-commit-test/03/usr/local
> tmpfs 10G 270M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/04/usr/local
> tmpfs 10G 414M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/07/usr/local
> tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/09/usr/local
> tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/10/usr/local
> tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/12/usr/local
> tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/13/usr/local
> tmpfs 10G 575M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/05/usr/local
> tmpfs 10G 575M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/02/usr/local
> tmpfs 10G 306M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/14/usr/local
> tmpfs 10G 570M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/06/usr/local
> tmpfs 10G 36M 10G 0% /poudriere/data/.m/exp-11amd64-commit-test/01/usr/local
> tmpfs 10G 283M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/11/usr/local
> tmpfs 10G 301M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/08/usr/local
And some large processes.
> # ps aux|sort -rnk 6|head -n 20
> root 2481 100.0 4.3 3334388 3209728 15 RJ 14:40 119:54.18 /usr/local/bin/python2.7 ../../rpython/bin/rpython --source -Ojit targetpypystandalone.py
> root 46466 97.7 0.8 620692 581304 15 RJ 16:44 0:36.07 /wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib/bin/ghc-stage1 -B/wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib -hisuf hi -osuf o -hcsuf hc -static -H32m -O -I/usr/include -L/usr/lib -I/usr/loc
> root 44667 100.0 0.2 187664 161444 15 RJ 16:43 1:04.16 /wrkdirs/usr/ports/lang/gcc6/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc/c-family -
> root 47917 76.5 0.2 173572 140580 15 RJ 16:44 0:12.45 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name MemorySanitizer.cpp -mrelocation-model pic -pic-level 2 -mthread-model
> root 47918 75.9 0.2 164784 136976 15 RJ 16:44 0:12.50 /wrkdirs/usr/ports/lang/gcc7-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20180308/gcc -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20
> root 47938 71.5 0.2 159952 134260 15 RJ 16:44 0:11.17 /wrkdirs/usr/ports/lang/gcc6-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6-devel/work/gcc-6-20180307/gcc -I /wrkdirs/usr/ports/lang/gcc6-devel/work/g
> root 48090 48.1 0.2 159420 125748 15 RJ 16:44 0:06.02 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name AlignmentFromAssumptions.cpp -mrelocation-model pic -pic-level 2 -mthre
> root 48110 38.4 0.2 153988 116220 15 RJ 16:44 0:04.35 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name property-type.cxx -mrelocation-model static -mthread-model posix -mdisa
> root 48139 35.4 0.1 131636 101608 15 RJ 16:44 0:03.70 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name InstrInfoEmitter.cpp -mrelocation-model pic -pic-level 2 -mthread-model
> root 48119 34.2 0.1 126396 93768 15 RJ 16:44 0:03.79 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name NaryReassociate.cpp -mrelocation-model pic -pic-level 2 -mthread-model
--
Regards,
Bryan Drewery
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a2bf20a3-b5a4-f6e3-3692-4c2783b03b4d>