Date: Tue, 2 Jul 2019 12:13:28 -0400 From: "Sam Fourman Jr." <sfourman@gmail.com> To: "Nagy, Attila" <bra@fsn.hu> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: ZFS exhausts kernel memory just by importing zpools Message-ID: <CAOFF%2BZ1rja=ALCJG9Mk7dycRqwErk7uVvBoE%2B3TYxS8qgkLAUw@mail.gmail.com> In-Reply-To: <e542dfd4-9534-1ec7-a269-89c3c20cca1d@fsn.hu> References: <e542dfd4-9534-1ec7-a269-89c3c20cca1d@fsn.hu>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, My initial guess is that you may have de-duplication enabled on one (or more) of the underlying datasets. **if** this is the case, a simple solution is to add more memory to the machine. (64GB of memory is not sufficient for dedup to be enabled ) -- Sam Fourman Jr. On Tue, Jul 2, 2019 at 10:59 AM Nagy, Attila <bra@fsn.hu> wrote: > Hi, > > Running latest stable/12 on amd64 with 64 GiB memory on a machine with > 44 4T disks. Each disks have its own zpool on it (because I solve the > redundancy between machines and not locally with ZFS). > > One example zpool holds 2.2 TiB of data (according to df) and have > around 75 million files in hashed directories, this is the typical usage > on them. > > When I import these zpools, top says around 50 GiB wired memory (ARC is > minimal, files weren't yet touched) and after I start to use (heavy > reads/writes) the pools, the free memory quickly disappears (ARC grows) > until all memory is gone and the machine starts to kill processes, ends > up in a deadlock, where nothing helps. > > If I import the pools one by one, each of them adds around 1-1.5 GiB of > wired memory. > > Top shows this, right after it came to a halt and nothing else works (I > can't log in even on the console): > > last pid: 61878; load averages: 5.05, 4.42, 2.50 up 0+01:07:23 > 15:45:17 > 171 processes: 1 running, 162 sleeping, 1 stopped, 1 zombie, 6 waiting > CPU: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle > Mem: 7716K Active, 8192 Inact, 84K Laundry, 57G Wired, 180M Buf, 14M Free > ARC: 21G Total, 10G MFU, 4812M MRU, 4922M Anon, 301M Header, 828M Other > 5739M Compressed, 13G Uncompressed, 2.35:1 Ratio > Swap: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 61412 root 1 20 0 14M 3904K CPU14 14 0:06 1.55% top > 57569 redis 57 20 0 1272M 64M uwait 22 4:28 0.24% consul > 5574 root 1 20 0 13M 3440K nanslp 10 0:02 0.05% gstat > 5557 root 1 20 0 20M 7808K select 20 0:00 0.01% sshd > 5511 root 1 20 0 20M 7808K select 4 0:01 0.01% sshd > 4955 root 1 20 0 10M 1832K select 9 0:00 0.01% > supervis > 5082 root 1 20 0 25M 14M select 0 0:00 0.00% perl > 4657 _pflogd 1 20 0 12M 2424K bpf 1 0:00 0.00% > pflogd > 5059 elasticsea 2 20 -20 6983M 385M STOP 5 1:29 0.00% java > 61669 root 1 26 0 23M 0 pfault 4 0:14 0.00% > <python3 > 61624 root 1 20 -20 24M 14M buf_ha 9 0:09 0.00% > python3. > 61626 root 1 20 -20 23M 16K pfault 0 0:08 0.00% > python3. > 61651 root 1 20 -20 23M 14M buf_ha 10 0:08 0.00% > python3. > 61668 root 1 20 -20 23M 13M buf_ha 20 0:08 0.00% > python3. > > I've already tried to shrink ARC and vm.kmem_size without too much success. > > Any ideas what causes this? > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Sam Fourman Jr.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOFF%2BZ1rja=ALCJG9Mk7dycRqwErk7uVvBoE%2B3TYxS8qgkLAUw>