Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 16 Mar 2018 16:45:59 -0700
From:      Bryan Drewery <bryan-lists@shatow.net>
To:        freebsd-current@freebsd.org
Subject:   Re: Strange ARC/Swap/CPU on yesterday's -CURRENT
Message-ID:  <a2bf20a3-b5a4-f6e3-3692-4c2783b03b4d@shatow.net>
In-Reply-To: <alpine.BSF.2.21.1803111038460.1232@desktop>
References:  <20180306173455.oacyqlbib4sbafqd@ler-imac.lerctr.org> <201803061816.w26IGaW5050053@pdx.rh.CN85.dnsmgr.net> <20180306193645.vv3ogqrhauivf2tr@ler-imac.lerctr.org> <20180306221554.uyshbzbboai62rdf@dx240.localdomain> <20180307103911.GA72239@kloomba> <20180311004737.3441dbf9@thor.intern.walstatt.dynvpn.de> <alpine.BSF.2.21.1803111038460.1232@desktop>

next in thread | previous in thread | raw e-mail | index | archive | help
On 3/11/2018 1:43 PM, Jeff Roberson wrote:
> Hi Folks,
> 
> This could be my fault from recent NUMA and concurrency related work.  I
> did touch some of the arc back-pressure mechanisms.  First, I would like
> to identify whether the wired memory is in the buffer cache.  Can those
> of you that have a repro look at sysctl vfs.bufspace and tell me if that
> accounts for the bulk of your wired memory usage?  I'm wondering if a
> job ran that pulled in all of the bufs from your root disk and filled up
> the buffer cache which doesn't have a back-pressure mechanism.  Then arc
> didn't respond appropriately to lower its usage.
> 
> Also, if you could try going back to r328953 or r326346 and let me know
> if the problem exists in either.  That would be very helpful.  If anyone
> is willing to debug this with me contact me directly and I will send
> some test patches or debugging info after you have done the above steps.
> 
> Thank you for the reports.
> 
> Jeff

It's like there is no ZFS ARC backpressure at all anymore...

Mem: 8783M Active, 2851M Inact, 3716M Laundry, 53G Wired, 2449M Free
ARC: 40G Total, 27G MFU, 11G MRU, 13M Anon, 631M Header, 955M Other
     37G Compressed, 53G Uncompressed, 1.43:1 Ratio
Swap: 116G Total, 18G Used, 98G Free, 15% Inuse, 4K In
vfs.bufspace: 0

I ran out of swap space around 16GB and had to add a larger device on to
keep a package test build going.

Was there ever backpressure? Having to limit the arc size has been a
constant workaround for package building.

I have some large tmpfs at the moment totaling around ~28GB.

> # df -ah|egrep '(Used|tmpfs)'
> Filesystem                                                              Size    Used   Avail Capacity  Mounted on
> tmpfs                                                                    99G    520K     99G     0%    /tmp
> tmpfs                                                                    10G    8.0K     10G     0%    /poudriere/data/.m/exp-11amd64-commit-test
> tmpfs                                                                    10G    373M    9.6G     4%    /poudriere/data/.m/exp-11amd64-commit-test/ref
> tmpfs                                                                   1.0G     18M    1.0G     2%    /poudriere/data/.m/exp-11amd64-commit-test/ref/.p
> tmpfs                                                                    10G    176K     10G     0%    /poudriere/data/.m/exp-11amd64-commit-test/ref/var/db/ports
> tmpfs                                                                    10G    405M    9.6G     4%    /poudriere/data/.m/exp-11amd64-commit-test/01
> tmpfs                                                                    10G    1.1G    8.9G    11%    /poudriere/data/.m/exp-11amd64-commit-test/05
> tmpfs                                                                    10G    2.4G    7.6G    24%    /poudriere/data/.m/exp-11amd64-commit-test/04
> tmpfs                                                                    10G    578M    9.4G     6%    /poudriere/data/.m/exp-11amd64-commit-test/03
> tmpfs                                                                    10G    1.0G    9.0G    10%    /poudriere/data/.m/exp-11amd64-commit-test/02
> tmpfs                                                                    10G    936M    9.1G     9%    /poudriere/data/.m/exp-11amd64-commit-test/06
> tmpfs                                                                    10G    2.7G    7.3G    27%    /poudriere/data/.m/exp-11amd64-commit-test/07
> tmpfs                                                                    10G    2.7G    7.3G    27%    /poudriere/data/.m/exp-11amd64-commit-test/09
> tmpfs                                                                    10G    875M    9.1G     9%    /poudriere/data/.m/exp-11amd64-commit-test/08
> tmpfs                                                                    10G    2.7G    7.3G    27%    /poudriere/data/.m/exp-11amd64-commit-test/10
> tmpfs                                                                    10G    2.7G    7.3G    27%    /poudriere/data/.m/exp-11amd64-commit-test/12
> tmpfs                                                                    10G    390M    9.6G     4%    /poudriere/data/.m/exp-11amd64-commit-test/11
> tmpfs                                                                    10G    992M    9.0G    10%    /poudriere/data/.m/exp-11amd64-commit-test/14
> tmpfs                                                                    10G    2.9G    7.1G    29%    /poudriere/data/.m/exp-11amd64-commit-test/13
> tmpfs                                                                   1.0G     16K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/01/.p
> tmpfs                                                                   1.0G    540K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/12/.p
> tmpfs                                                                   1.0G    1.3M    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/05/.p
> tmpfs                                                                   1.0G    1.3M    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/06/.p
> tmpfs                                                                   1.0G    396K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/03/.p
> tmpfs                                                                   1.0G    540K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/09/.p
> tmpfs                                                                   1.0G    540K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/10/.p
> tmpfs                                                                   1.0G    732K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/07/.p
> tmpfs                                                                   1.0G     28K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/08/.p
> tmpfs                                                                   1.0G     12K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/11/.p
> tmpfs                                                                   1.0G    1.3M    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/02/.p
> tmpfs                                                                   1.0G    852K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/14/.p
> tmpfs                                                                   1.0G    540K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/13/.p
> tmpfs                                                                   1.0G    688K    1.0G     0%    /poudriere/data/.m/exp-11amd64-commit-test/04/.p
> tmpfs                                                                    10G    125M    9.9G     1%    /poudriere/data/.m/exp-11amd64-commit-test/03/usr/local
> tmpfs                                                                    10G    270M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/04/usr/local
> tmpfs                                                                    10G    414M    9.6G     4%    /poudriere/data/.m/exp-11amd64-commit-test/07/usr/local
> tmpfs                                                                    10G    332M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/09/usr/local
> tmpfs                                                                    10G    332M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/10/usr/local
> tmpfs                                                                    10G    332M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/12/usr/local
> tmpfs                                                                    10G    332M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/13/usr/local
> tmpfs                                                                    10G    575M    9.4G     6%    /poudriere/data/.m/exp-11amd64-commit-test/05/usr/local
> tmpfs                                                                    10G    575M    9.4G     6%    /poudriere/data/.m/exp-11amd64-commit-test/02/usr/local
> tmpfs                                                                    10G    306M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/14/usr/local
> tmpfs                                                                    10G    570M    9.4G     6%    /poudriere/data/.m/exp-11amd64-commit-test/06/usr/local
> tmpfs                                                                    10G     36M     10G     0%    /poudriere/data/.m/exp-11amd64-commit-test/01/usr/local
> tmpfs                                                                    10G    283M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/11/usr/local
> tmpfs                                                                    10G    301M    9.7G     3%    /poudriere/data/.m/exp-11amd64-commit-test/08/usr/local

And some large processes.

> # ps aux|sort -rnk 6|head -n 20
> root        2481 100.0  4.3 3334388 3209728 15  RJ   14:40     119:54.18 /usr/local/bin/python2.7 ../../rpython/bin/rpython --source -Ojit targetpypystandalone.py
> root       46466  97.7  0.8  620692  581304 15  RJ   16:44       0:36.07 /wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib/bin/ghc-stage1 -B/wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib -hisuf hi -osuf o -hcsuf hc -static -H32m -O -I/usr/include -L/usr/lib -I/usr/loc
> root       44667 100.0  0.2  187664  161444 15  RJ   16:43       1:04.16 /wrkdirs/usr/ports/lang/gcc6/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc/c-family -
> root       47917  76.5  0.2  173572  140580 15  RJ   16:44       0:12.45 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name MemorySanitizer.cpp -mrelocation-model pic -pic-level 2 -mthread-model
> root       47918  75.9  0.2  164784  136976 15  RJ   16:44       0:12.50 /wrkdirs/usr/ports/lang/gcc7-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20180308/gcc -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20
> root       47938  71.5  0.2  159952  134260 15  RJ   16:44       0:11.17 /wrkdirs/usr/ports/lang/gcc6-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6-devel/work/gcc-6-20180307/gcc -I /wrkdirs/usr/ports/lang/gcc6-devel/work/g
> root       48090  48.1  0.2  159420  125748 15  RJ   16:44       0:06.02 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name AlignmentFromAssumptions.cpp -mrelocation-model pic -pic-level 2 -mthre
> root       48110  38.4  0.2  153988  116220 15  RJ   16:44       0:04.35 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name property-type.cxx -mrelocation-model static -mthread-model posix -mdisa
> root       48139  35.4  0.1  131636  101608 15  RJ   16:44       0:03.70 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name InstrInfoEmitter.cpp -mrelocation-model pic -pic-level 2 -mthread-model
> root       48119  34.2  0.1  126396   93768 15  RJ   16:44       0:03.79 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name NaryReassociate.cpp -mrelocation-model pic -pic-level 2 -mthread-model


-- 
Regards,
Bryan Drewery




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a2bf20a3-b5a4-f6e3-3692-4c2783b03b4d>