From owner-freebsd-fs@freebsd.org Tue Jul 2 14:58:43 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE4DE15D5E1F for ; Tue, 2 Jul 2019 14:58:43 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from dg.fsn.hu (dg.fsn.hu [84.2.225.196]) by mx1.freebsd.org (Postfix) with ESMTP id 48CAB89911 for ; Tue, 2 Jul 2019 14:58:42 +0000 (UTC) (envelope-from bra@fsn.hu) Received: by dg.fsn.hu (Postfix, from userid 1003) id A88EF7640; Tue, 2 Jul 2019 16:58:33 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 14.8305] X-CRM114-CacheID: sfid-20190702_16583_0D8A80A5 X-CRM114-Status: Good ( pR: 14.8305 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Tue Jul 2 16:58:32 2019 X-DSPAM-Confidence: 0.9899 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 5d1b7118409731295823851 X-DSPAM-Factors: 27, To*fs+FreeBSD.org, 0.01000, 13G, 0.01000, import, 0.01000, import, 0.01000, after+it, 0.01000, it+came, 0.01000, Received*(localhost+[127.0.0.1]), 0.01000, with+ZFS), 0.01000, 2+35, 0.01000, right, 0.01000, 5+1, 0.01000, zpool, 0.01000, zpool, 0.01000, 20+0, 0.01000, 20+0, 0.01000, solve, 0.01000, 24%25, 0.01000, 00%25, 0.01000, 00%25, 0.01000, and+not, 0.01000, 01%25+sshd, 0.01000, 01%25+sshd, 0.01000, Received*Tue, 0.01000, 10, 0.01000, 10, 0.01000, Date*2+Jul, 0.01000, X-Spambayes-Classification: ham; 0.00 Received: from [127.0.0.1] (localhost [127.0.0.1]) by dg.fsn.hu (Postfix) with ESMTP id 38028763E for ; Tue, 2 Jul 2019 16:58:30 +0200 (CEST) To: freebsd-fs@FreeBSD.org From: "Nagy, Attila" Subject: ZFS exhausts kernel memory just by importing zpools Message-ID: Date: Tue, 2 Jul 2019 16:58:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 48CAB89911 X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of bra@fsn.hu designates 84.2.225.196 as permitted sender) smtp.mailfrom=bra@fsn.hu X-Spamd-Result: default: False [4.26 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.96)[0.957,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[fsn.hu]; NEURAL_SPAM_SHORT(0.76)[0.764,0]; MX_GOOD(-0.01)[dg.fsn.hu]; NEURAL_SPAM_LONG(1.00)[0.999,0]; IP_SCORE(1.75)[ipnet: 84.2.224.0/21(4.92), asn: 15555(3.81), country: HU(0.02)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15555, ipnet:84.2.224.0/21, country:HU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jul 2019 14:58:43 -0000 Hi, Running latest stable/12 on amd64 with 64 GiB memory on a machine with 44 4T disks. Each disks have its own zpool on it (because I solve the redundancy between machines and not locally with ZFS). One example zpool holds 2.2 TiB of data (according to df) and have around 75 million files in hashed directories, this is the typical usage on them. When I import these zpools, top says around 50 GiB wired memory (ARC is minimal, files weren't yet touched) and after I start to use (heavy reads/writes) the pools, the free memory quickly disappears (ARC grows) until all memory is gone and the machine starts to kill processes, ends up in a deadlock, where nothing helps. If I import the pools one by one, each of them adds around 1-1.5 GiB of wired memory. Top shows this, right after it came to a halt and nothing else works (I can't log in even on the console): last pid: 61878;  load averages:  5.05,  4.42,  2.50    up 0+01:07:23  15:45:17 171 processes: 1 running, 162 sleeping, 1 stopped, 1 zombie, 6 waiting CPU:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle Mem: 7716K Active, 8192 Inact, 84K Laundry, 57G Wired, 180M Buf, 14M Free ARC: 21G Total, 10G MFU, 4812M MRU, 4922M Anon, 301M Header, 828M Other      5739M Compressed, 13G Uncompressed, 2.35:1 Ratio Swap:   PID USERNAME    THR PRI NICE   SIZE    RES STATE    C TIME    WCPU COMMAND 61412 root          1  20    0    14M  3904K CPU14   14 0:06   1.55% top 57569 redis        57  20    0  1272M    64M uwait   22 4:28   0.24% consul  5574 root          1  20    0    13M  3440K nanslp  10 0:02   0.05% gstat  5557 root          1  20    0    20M  7808K select  20 0:00   0.01% sshd  5511 root          1  20    0    20M  7808K select   4 0:01   0.01% sshd  4955 root          1  20    0    10M  1832K select   9 0:00   0.01% supervis  5082 root          1  20    0    25M    14M select   0 0:00   0.00% perl  4657 _pflogd       1  20    0    12M  2424K bpf      1 0:00   0.00% pflogd  5059 elasticsea    2  20  -20  6983M   385M STOP     5 1:29   0.00% java 61669 root          1  26    0    23M      0 pfault   4 0:14   0.00%