From owner-freebsd-fs@freebsd.org Tue Jul 2 16:13:41 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBE9F15D7822 for ; Tue, 2 Jul 2019 16:13:41 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 435888C606 for ; Tue, 2 Jul 2019 16:13:40 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: by mail-ed1-x529.google.com with SMTP id r12so27839528edo.5 for ; Tue, 02 Jul 2019 09:13:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9HqRpJfo40SzQSqcyGhaWMBfhZlssY00Ore4kRiwK6Q=; b=IID3vQXVLpApeS2hV4IvFt5fe5VVhlsgRPdTCR1jKpx1FAZXM4UC5elhRVDmbO57du OiTp4583292i7RWT798CPs1IMQZiPeCBtCjFeSLpX6Ao0NUmoOXjb6AQhB4f2Yg04PIm /CeeXC2llAe+ucckPzZ+ps+4qIFEF4sNtIIz102BVRKtoAPziTSLpTBOGlHE79pfWNCp xwD3g73UCxWDY6qFZ9uBwO/uz/9RO8hUYVxKz/jeFxKPnDfxTdnJHFMgdDT8KUUOni0n M6zj+AnCE3QvBojLsCVm4MZ5Uy1zlVxrd1QFJEqrDbHqO3uxKqrWkzvBFvcBgrDFOV9o RtBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9HqRpJfo40SzQSqcyGhaWMBfhZlssY00Ore4kRiwK6Q=; b=gSPfJDjTWS3RufgV6ehcbX0YC/ZONJ2ngjA8fyfDRJkKrY7eZZ9FPERsEHx0RQBewk +5qhMO0lZ9MkFejffKX4HtaCN+jjgcswiM+b5c/WcHEhAznD2xd8oMwCTdKUamAUcMos MmUhWm4hgnx6NxnjY6UfLikrBda5vO8LP1dr3mPUsPv7QHXjhMjr3RaO1SYIY4H8K/Q4 TateFGbwxKVlnFOxNB/Xdr3uR2Ru082/GQkCc8AaqJFxtTaGRGvW8bnGVJHE+IDkTM9d 4wlzVtI8ui2dKz4VWJbtCqcT0tPTol7e99Uf8R/G2nmB3IqkDRoJNxDmACF7MeSh4btU sEFQ== X-Gm-Message-State: APjAAAUG8iwhBaP6G6TmgzJ8bTr2MEk9LknLURY+RaGbjzrWLxYS1rGc Lw4NQqBncCsbMG3uAfLrR9j9ovg7im1YyBBiRWUjSiLV X-Google-Smtp-Source: APXvYqw6a5HOLK5Ul0RCpfAZCIqShsqXEuPc3pC8RAWs5Bskz+0WlIxK+ZsY8qUhMmNMBhDskrKZz3wauXZHP5Eh9ZU= X-Received: by 2002:a17:906:6055:: with SMTP id p21mr29971071ejj.35.1562084019129; Tue, 02 Jul 2019 09:13:39 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: "Sam Fourman Jr." Date: Tue, 2 Jul 2019 12:13:28 -0400 Message-ID: Subject: Re: ZFS exhausts kernel memory just by importing zpools To: "Nagy, Attila" Cc: FreeBSD FS X-Rspamd-Queue-Id: 435888C606 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=IID3vQXV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sfourman@gmail.com designates 2a00:1450:4864:20::529 as permitted sender) smtp.mailfrom=sfourman@gmail.com X-Spamd-Result: default: False [-6.87 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.970,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[9.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.89)[ip: (-9.39), ipnet: 2a00:1450::/32(-2.65), asn: 15169(-2.36), country: US(-0.06)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jul 2019 16:13:42 -0000 Hello, My initial guess is that you may have de-duplication enabled on one (or more) of the underlying datasets. **if** this is the case, a simple solution is to add more memory to the machine. (64GB of memory is not sufficient for dedup to be enabled ) -- Sam Fourman Jr. On Tue, Jul 2, 2019 at 10:59 AM Nagy, Attila wrote: > Hi, > > Running latest stable/12 on amd64 with 64 GiB memory on a machine with > 44 4T disks. Each disks have its own zpool on it (because I solve the > redundancy between machines and not locally with ZFS). > > One example zpool holds 2.2 TiB of data (according to df) and have > around 75 million files in hashed directories, this is the typical usage > on them. > > When I import these zpools, top says around 50 GiB wired memory (ARC is > minimal, files weren't yet touched) and after I start to use (heavy > reads/writes) the pools, the free memory quickly disappears (ARC grows) > until all memory is gone and the machine starts to kill processes, ends > up in a deadlock, where nothing helps. > > If I import the pools one by one, each of them adds around 1-1.5 GiB of > wired memory. > > Top shows this, right after it came to a halt and nothing else works (I > can't log in even on the console): > > last pid: 61878; load averages: 5.05, 4.42, 2.50 up 0+01:07:23 > 15:45:17 > 171 processes: 1 running, 162 sleeping, 1 stopped, 1 zombie, 6 waiting > CPU: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle > Mem: 7716K Active, 8192 Inact, 84K Laundry, 57G Wired, 180M Buf, 14M Free > ARC: 21G Total, 10G MFU, 4812M MRU, 4922M Anon, 301M Header, 828M Other > 5739M Compressed, 13G Uncompressed, 2.35:1 Ratio > Swap: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 61412 root 1 20 0 14M 3904K CPU14 14 0:06 1.55% top > 57569 redis 57 20 0 1272M 64M uwait 22 4:28 0.24% consul > 5574 root 1 20 0 13M 3440K nanslp 10 0:02 0.05% gstat > 5557 root 1 20 0 20M 7808K select 20 0:00 0.01% sshd > 5511 root 1 20 0 20M 7808K select 4 0:01 0.01% sshd > 4955 root 1 20 0 10M 1832K select 9 0:00 0.01% > supervis > 5082 root 1 20 0 25M 14M select 0 0:00 0.00% perl > 4657 _pflogd 1 20 0 12M 2424K bpf 1 0:00 0.00% > pflogd > 5059 elasticsea 2 20 -20 6983M 385M STOP 5 1:29 0.00% java > 61669 root 1 26 0 23M 0 pfault 4 0:14 0.00% > 61624 root 1 20 -20 24M 14M buf_ha 9 0:09 0.00% > python3. > 61626 root 1 20 -20 23M 16K pfault 0 0:08 0.00% > python3. > 61651 root 1 20 -20 23M 14M buf_ha 10 0:08 0.00% > python3. > 61668 root 1 20 -20 23M 13M buf_ha 20 0:08 0.00% > python3. > > I've already tried to shrink ARC and vm.kmem_size without too much success. > > Any ideas what causes this? > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Sam Fourman Jr.