From owner-freebsd-fs@freebsd.org Wed Jul 3 14:34:16 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9916F15D5876 for ; Wed, 3 Jul 2019 14:34:16 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from dg.fsn.hu (dg.fsn.hu [84.2.225.196]) by mx1.freebsd.org (Postfix) with ESMTP id 4065476177 for ; Wed, 3 Jul 2019 14:34:15 +0000 (UTC) (envelope-from bra@fsn.hu) Received: by dg.fsn.hu (Postfix, from userid 1003) id 7EA997B9F; Wed, 3 Jul 2019 16:34:13 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 27.7999] X-CRM114-CacheID: sfid-20190703_16340_61A47AA6 X-CRM114-Status: Good ( pR: 27.7999 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Wed Jul 3 16:34:11 2019 X-DSPAM-Confidence: 0.9899 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 5d1cbce3556461935321310 X-DSPAM-Factors: 27, 13G, 0.01000, 13G, 0.01000, underlying, 0.01000, underlying, 0.01000, import, 0.01000, import, 0.01000, nanslp%c2%a0+10, 0.01000, nanslp%c2%a0+10, 0.01000, after+it, 0.01000, after+it, 0.01000, it+came, 0.01000, it+came, 0.01000, To*, 0.01000, 23M%c2%a0%c2%a0%c2%a0+16K, 0.01000, 23M%c2%a0%c2%a0%c2%a0+16K, 0.01000, >+(or, 0.01000, 5+1, 0.01000, 5+1, 0.01000, X-Spambayes-Classification: ham; 0.00 Received: from [127.0.0.1] (localhost [127.0.0.1]) by dg.fsn.hu (Postfix) with ESMTP id A1FED7B9D; Wed, 3 Jul 2019 16:34:09 +0200 (CEST) Subject: Re: ZFS exhausts kernel memory just by importing zpools To: "Sam Fourman Jr." Cc: FreeBSD FS References: From: "Nagy, Attila" Message-ID: <21b04b21-8850-c3c3-36c9-a0d0ede4dc22@fsn.hu> Date: Wed, 3 Jul 2019 16:34:08 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Rspamd-Queue-Id: 4065476177 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of bra@fsn.hu designates 84.2.225.196 as permitted sender) smtp.mailfrom=bra@fsn.hu X-Spamd-Result: default: False [3.89 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_SPAM_SHORT(0.39)[0.389,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[fsn.hu]; NEURAL_SPAM_MEDIUM(0.98)[0.975,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: dg.fsn.hu]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(1.00)[0.996,0]; IP_SCORE(1.74)[ipnet: 84.2.224.0/21(4.90), asn: 15555(3.78), country: HU(0.02)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:15555, ipnet:84.2.224.0/21, country:HU]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jul 2019 14:34:16 -0000 Hi, Oh, should've written about that: no I don't use (and never used) dedup. On 2019. 07. 02. 18:13, Sam Fourman Jr. wrote: > Hello, > > My initial guess is that you may have de-duplication enabled on one > (or more) of the underlying datasets. > **if** this is the case, a simple solution is to add more memory to > the machine. (64GB of memory is not sufficient for dedup to be enabled ) > > -- Sam Fourman Jr. > > On Tue, Jul 2, 2019 at 10:59 AM Nagy, Attila > wrote: > > Hi, > > Running latest stable/12 on amd64 with 64 GiB memory on a machine > with > 44 4T disks. Each disks have its own zpool on it (because I solve the > redundancy between machines and not locally with ZFS). > > One example zpool holds 2.2 TiB of data (according to df) and have > around 75 million files in hashed directories, this is the typical > usage > on them. > > When I import these zpools, top says around 50 GiB wired memory > (ARC is > minimal, files weren't yet touched) and after I start to use (heavy > reads/writes) the pools, the free memory quickly disappears (ARC > grows) > until all memory is gone and the machine starts to kill processes, > ends > up in a deadlock, where nothing helps. > > If I import the pools one by one, each of them adds around 1-1.5 > GiB of > wired memory. > > Top shows this, right after it came to a halt and nothing else > works (I > can't log in even on the console): > > last pid: 61878;  load averages:  5.05,  4.42,  2.50    up 0+01:07:23 > 15:45:17 > 171 processes: 1 running, 162 sleeping, 1 stopped, 1 zombie, 6 waiting > CPU:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle > Mem: 7716K Active, 8192 Inact, 84K Laundry, 57G Wired, 180M Buf, > 14M Free > ARC: 21G Total, 10G MFU, 4812M MRU, 4922M Anon, 301M Header, 828M > Other >       5739M Compressed, 13G Uncompressed, 2.35:1 Ratio > Swap: > >    PID USERNAME    THR PRI NICE   SIZE    RES STATE    C TIME    WCPU > COMMAND > 61412 root          1  20    0    14M  3904K CPU14   14 0:06 1.55% top > 57569 redis        57  20    0  1272M    64M uwait   22 4:28 0.24% > consul >   5574 root          1  20    0    13M  3440K nanslp  10 0:02   > 0.05% gstat >   5557 root          1  20    0    20M  7808K select  20 0:00   > 0.01% sshd >   5511 root          1  20    0    20M  7808K select   4 0:01   > 0.01% sshd >   4955 root          1  20    0    10M  1832K select   9 0:00   0.01% > supervis >   5082 root          1  20    0    25M    14M select   0 0:00   > 0.00% perl >   4657 _pflogd       1  20    0    12M  2424K bpf      1 0:00   > 0.00% pflogd >   5059 elasticsea    2  20  -20  6983M   385M STOP     5 1:29   > 0.00% java > 61669 root          1  26    0    23M      0 pfault   4 0:14 0.00% > 61624 root          1  20  -20    24M    14M buf_ha   9 0:09 0.00% > python3. > 61626 root          1  20  -20    23M    16K pfault   0 0:08 0.00% > python3. > 61651 root          1  20  -20    23M    14M buf_ha  10 0:08 0.00% > python3. > 61668 root          1  20  -20    23M    13M buf_ha  20 0:08 0.00% > python3. > > I've already tried to shrink ARC and vm.kmem_size without too much > success. > > Any ideas what causes this? > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to > "freebsd-fs-unsubscribe@freebsd.org > " > > > > -- > > Sam Fourman Jr.