From owner-freebsd-fs@freebsd.org Tue Jul 2 16:14:04 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60E8C15D7850 for ; Tue, 2 Jul 2019 16:14:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene.sentex.ca (unknown [IPv6:2607:f3e0:0:3::18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B6B818C614 for ; Tue, 2 Jul 2019 16:13:53 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [192.168.43.26] ([192.168.43.26]) by pyroxene.sentex.ca (8.15.2/8.15.2) with ESMTP id x62GDoiY061907; Tue, 2 Jul 2019 12:13:51 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: ZFS exhausts kernel memory just by importing zpools To: "Nagy, Attila" , freebsd-fs@FreeBSD.org References: From: Mike Tancsa Openpgp: preference=signencrypt Autocrypt: addr=mike@sentex.net; prefer-encrypt=mutual; keydata= mQENBEzcA24BCACpwI/iqOrs0GfQSfhA1v6Z8AcXVeGsRyKEKUpxoOYxXWc2z3vndbYlIP6E YJeifzKhS/9E+VjhhICaepLHfw865TDTUPr5D0Ed+edSsKjlnDtb6hfNJC00P7eoiuvi85TW F/gAxRY269A5d856bYrzLbkWp2lKUR3Bg6NnORtflGzx9ZWAltZbjYjjRqegPv0EQNYcHqWo eRpXilEo1ahT6nmOU8V7yEvT2j4wlLcQ6qg7w+N/vcBvyd/weiwHU+vTQ9mT61x5/wUrQhdw 2gJHeQXeDGMJV49RT2EEz+QVxaf477eyWsdQzPVjAKRMT3BVdK8WvpYAEfBAbXmkboOxABEB AAG0HG1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5jYT6JATgEEwECACIFAkzcA24CGwMGCwkI BwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEJXHwM2kc8rX+sMH/2V6pTBKsQ5mpWWLgs6wVP2k BC+6r/YKNXv9Rw/PrC6+9hTbgA+sSjJ+8gxsCbJsOQXZrxF0x3l9oYdYfuKcwdwXFX1/FS8p HfBeDkmlH+dI709xT9wgrR4dS5aMmKp0scPrXPIAKiYVOHjOlNItcLYTEEWEFBepheEVsgmk GrNbcrHwOx/u4igUQ8vcpyXPyUki+BsftPw8ZQvBU887igh0OxaCR8AurJppQ5UQd63r81cX E1ZjoFoWCaGK/SjPb/OhpYpu5swoZIhOxQbn7OtakYPsDd5t2A5KhvjI8BMTnd5Go+2xsCmr jlIEq8Bi29gCcfQUvNiClevi13ifmnm5AQ0ETNwDbgEIALWGNJHRAhpd0A4vtd3G0oRqMBcM FGThQr3qORmEBTPPEomTdBaHcn+Xl+3YUvTBD/67/mutWBwgp2R5gQOSqcM7axvgMSHbKqBL 9sd1LsLw0UT2O5AYxv3EwzhG84pwRg3XcUqvWA4lA8tIj/1q4Jzi5qOkg1zxq4W9qr9oiYK5 bBR638JUvr3eHMaz/Nz+sDVFgwHmXZj3M6aE5Ce9reCGbvrae7H5D5PPvtT3r22X8SqfVAiO TFKedCf/6jbSOedPN931FJQYopj9P6b3m0nI3ZiCDVSqeyOAIBLzm+RBUIU3brzoxDhYR8pz CJc2sK8l6YjqivPakrD86bFDff8AEQEAAYkBHwQYAQIACQUCTNwDbgIbDAAKCRCVx8DNpHPK 1+iQB/99aqNtez9ZTBWELj269La8ntuRx6gCpzfPXfn6SDIfTItDxTh1hrdRVP5QNGGF5wus N4EMwXouskva1hbFX3Pv72csYSxxEJXjW16oV8WK4KjKXoskLg2RyRP4uXqL7Mp2ezNtVY5F 9nu3fj4ydpHCSaqKy5xd70A8D50PfZsFgkrsa5gdQhPiGGEdxhq/XSeAAnZ4uVLJKarH+mj5 MEhgZPEBWkGrbDZpezl9qbFcUem/uT9x8FYT/JIztMVh9qDcdP5tzANW5J7nvgXjska+VFGY ryZK4SPDczh74mn6GI/+RBi7OUzXXPgpPBrhS5FByjwCqjjsSpTjTds+NGIY Organization: Sentex Communications Message-ID: <820ceee3-95aa-9925-066d-5d22884ce001@sentex.net> Date: Tue, 2 Jul 2019 12:13:48 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: B6B818C614 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::18 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-1.19 / 15.00]; ARC_NA(0.00)[]; RDNS_NONE(1.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.840,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[sentex.net]; HAS_ORG_HEADER(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[smtp.sentex.ca]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.82)[-0.818,0]; IP_SCORE(-1.72)[ipnet: 2607:f3e0::/32(-4.95), asn: 11647(-3.59), country: CA(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; HFILTER_HOSTNAME_UNKNOWN(2.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jul 2019 16:14:04 -0000 On 7/2/2019 10:58 AM, Nagy, Attila wrote: > Hi, > > Running latest stable/12 on amd64 with 64 GiB memory on a machine with > 44 4T disks. Each disks have its own zpool on it (because I solve the > redundancy between machines and not locally with ZFS). > > One example zpool holds 2.2 TiB of data (according to df) and have > around 75 million files in hashed directories, this is the typical > usage on them. > > When I import these zpools, top says around 50 GiB wired memory (ARC > is minimal, files weren't yet touched) and after I start to use (heavy > reads/writes) the pools, the free memory quickly disappears (ARC > grows) until all memory is gone and the machine starts to kill > processes, ends up in a deadlock, where nothing helps. > > If I import the pools one by one, each of them adds around 1-1.5 GiB > of wired memory. Hi,     You mean you have 44 different zpools ?  75mil files per pool sounds like a lot. I wonder for testing purposes, you made 1 or two zpools with 44 (or 22) different datasets and had 3.3billion files, would you run into the same memory exhaustion ?     ---Mike