From owner-freebsd-fs@freebsd.org Wed Jul 3 14:37:50 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09E3F15D5AD1 for ; Wed, 3 Jul 2019 14:37:50 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from dg.fsn.hu (dg.fsn.hu [84.2.225.196]) by mx1.freebsd.org (Postfix) with ESMTP id 360B6763B1 for ; Wed, 3 Jul 2019 14:37:49 +0000 (UTC) (envelope-from bra@fsn.hu) Received: by dg.fsn.hu (Postfix, from userid 1003) id 0FBE37BA4; Wed, 3 Jul 2019 16:37:49 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 12.8040] X-CRM114-CacheID: sfid-20190703_16374_5A5B0BD2 X-CRM114-Status: Good ( pR: 12.8040 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Wed Jul 3 16:37:48 2019 X-DSPAM-Confidence: 0.9899 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 5d1cbdbc557181928486798 X-DSPAM-Factors: 27, To*fs+FreeBSD.org, 0.01000, import, 0.01000, import, 0.01000, run+>, 0.01000, but, 0.01000, Received*(localhost+[127.0.0.1]), 0.01000, with+ZFS), 0.01000, zpool, 0.01000, zpool, 0.01000, AM, 0.01000, To*>, 0.01000, 02+18, 0.01000, of, 0.01000, of, 0.01000, zpool+on, 0.01000, To*Mike, 0.01000, 75+million, 0.01000, From*"Nagy, Attila" , 0.01000, scales, 0.01000, X-Spambayes-Classification: ham; 0.00 Received: from [127.0.0.1] (localhost [127.0.0.1]) by dg.fsn.hu (Postfix) with ESMTP id 1F6407BA2; Wed, 3 Jul 2019 16:37:47 +0200 (CEST) Subject: Re: ZFS exhausts kernel memory just by importing zpools To: Mike Tancsa , freebsd-fs@FreeBSD.org References: <820ceee3-95aa-9925-066d-5d22884ce001@sentex.net> From: "Nagy, Attila" Message-ID: <78882cea-c1aa-0d08-d2e8-7f7ae7131bb6@fsn.hu> Date: Wed, 3 Jul 2019 16:37:46 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 In-Reply-To: <820ceee3-95aa-9925-066d-5d22884ce001@sentex.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 360B6763B1 X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of bra@fsn.hu designates 84.2.225.196 as permitted sender) smtp.mailfrom=bra@fsn.hu X-Spamd-Result: default: False [4.39 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_SPAM_SHORT(0.91)[0.908,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[fsn.hu]; NEURAL_SPAM_MEDIUM(0.96)[0.960,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: dg.fsn.hu]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(1.00)[0.998,0]; IP_SCORE(1.73)[ipnet: 84.2.224.0/21(4.88), asn: 15555(3.75), country: HU(0.02)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15555, ipnet:84.2.224.0/21, country:HU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jul 2019 14:37:50 -0000 On 2019. 07. 02. 18:13, Mike Tancsa wrote: > On 7/2/2019 10:58 AM, Nagy, Attila wrote: >> Hi, >> >> Running latest stable/12 on amd64 with 64 GiB memory on a machine with >> 44 4T disks. Each disks have its own zpool on it (because I solve the >> redundancy between machines and not locally with ZFS). >> >> One example zpool holds 2.2 TiB of data (according to df) and have >> around 75 million files in hashed directories, this is the typical >> usage on them. >> >> When I import these zpools, top says around 50 GiB wired memory (ARC >> is minimal, files weren't yet touched) and after I start to use (heavy >> reads/writes) the pools, the free memory quickly disappears (ARC >> grows) until all memory is gone and the machine starts to kill >> processes, ends up in a deadlock, where nothing helps. >> >> If I import the pools one by one, each of them adds around 1-1.5 GiB >> of wired memory. > Hi, > >     You mean you have 44 different zpools ?  75mil files per pool sounds > like a lot. I wonder for testing purposes, you made 1 or two zpools with > 44 (or 22) different datasets and had 3.3billion files, would you run > into the same memory exhaustion ? > Yes, 44 different pools. I think this is related to how ZFS stores pool metadata in memory. I don't think these scales with the number of the files, but maybe with the number of stored blocks. Sadly, I can't put the same amount of data to a machine with a different setup ATM.