From nobody Thu Feb 1 18:26:48 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TQnS71PVGz58NqQ for ; Thu, 1 Feb 2024 18:27:03 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TQnS63MPCz4CvN for ; Thu, 1 Feb 2024 18:27:01 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; none Received: from kalamity.joker.local (123-1-91-49.area1b.commufa.jp [123.1.91.49]) (authenticated bits=0) by www121.sakura.ne.jp (8.17.1/8.17.1/[SAKURA-WEB]/20201212) with ESMTPA id 411IQnk5029266; Fri, 2 Feb 2024 03:26:49 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Fri, 2 Feb 2024 03:26:48 +0900 From: Tomoaki AOKI To: Mark Millard Cc: kpielorz_lst@tdx.co.uk, FreeBSD Hackers Subject: Re: ... was killed: a thread waited too long to allocate a page [actually: was killed: failed to reclaim memory problem] Message-Id: <20240202032648.917edb3172db93881b3dd67f@dec.sakura.ne.jp> In-Reply-To: References: Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4TQnS63MPCz4CvN X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP] On Thu, 1 Feb 2024 08:30:19 -0800 Mark Millard wrote: > Karl Pielorz wrote on > Date: Thu, 01 Feb 2024 14:47:44 UTC : > > > --On 28 December 2023 11:38 +0200 Daniel Braniss > > wrote: > > > > > hi, > > > I'm running 13.2 Stable on this particular host, which has about 200TB of > > > zfs storage the host also has some 132Gb of memory, > > > lately, mountd is getting killed: > > > kernel: pid 3212 (mountd), jid 0, uid 0, was killed: a thread waited > > > too long to allocate a page > > > > > > rpcinfo shows it's still there, but > > > service mountd restart > > > fails. > > > > > > only solution is to reboot. > > > BTW, the only 'heavy' stuff that I can see are several rsync > > > processes. > > > > Hi, > > > > I seem to have run into something similar. I recently upgraded a 12.4 box > > to 13.2p9. The box has 32G of RAM, and runs ZFS. We do a lot of rsync work > > to it monthly - the first month we've done this with 13.2p9 we get a lot of > > processes killed, all with a similar (but not identical) message, e.g. > > > > pid 11103 (ssh), jid 0, uid 0, was killed: failed to reclaim memory > > pid 10972 (local-unbound), jid 0, uid 59, was killed: failed to reclaim > > memory > > pid 3223 (snmpd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3243 (mountd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3251 (nfsd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 10996 (sshd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3257 (sendmail), jid 0, uid 0, was killed: failed to reclaim memory > > pid 8562 (csh), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3363 (smartd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 8558 (csh), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3179 (ntpd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 8555 (tcsh), jid 0, uid 1001, was killed: failed to reclaim memory > > pid 3260 (sendmail), jid 0, uid 25, was killed: failed to reclaim memory > > pid 2806 (devd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3156 (rpcbind), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3252 (nfsd), jid 0, uid 0, was killed: failed to reclaim memory > > pid 3377 (getty), jid 0, uid 0, was killed: failed to reclaim memory > > > > This 'looks' like 'out of RAM' type situation - but at the time, top showed: > > > > last pid: 12622; load averages: 0.10, 0.24, 0.13 > > > > 7 processes: 1 running, 6 sleeping > > CPU: 0.1% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.7% idle > > Mem: 4324K Active, 8856K Inact, 244K Laundry, 24G Wired, 648M Buf, 7430M > > Free > > ARC: 20G Total, 8771M MFU, 10G MRU, 2432K Anon, 161M Header, 920M Other > > 15G Compressed, 23G Uncompressed, 1.59:1 Ratio > > Swap: 8192M Total, 5296K Used, 8187M Free > > > > Rebooting it recovers it, and it completed the rsync after the reboot - > > which left us with: > > > > last pid: 12570; load averages: 0.07, 0.14, 0.17 > > up 0+00:15:06 14:43:56 > > 26 processes: 1 running, 25 sleeping > > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > > Mem: 39M Active, 5640K Inact, 17G Wired, 42M Buf, 14G Free > > ARC: 15G Total, 33M MFU, 15G MRU, 130K Anon, 32M Header, 138M Other > > 14G Compressed, 15G Uncompressed, 1.03:1 Ratio > > Swap: 8192M Total, 8192M Free > > > > > > I've not seen any bug reports along this line, in fact very little coverage > > at all of the specific error. > > > > My only thought is to set a sysctl to limit ZFS ARC usage, i.e. to leave > > more free RAM floating around the system. During the rsync it was > > 'swapping' occasionally (few K in, few K out) - but it never ran out of > > swap that I saw - and it certainly didn't look like an complete out of > > memory scenario/box (which is what it felt like with everything getting > > killed). > > > One direction of control is . . . > > What do you have for ( copied from my /boot/loader.conf ): > > # > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=120 > > The default is 12 (last I knew, anyway). > > The 120 figure has allowed me and others to do buildworld, > buildkernel, and poudriere bulk runs on small arm boards > using all cores that otherwise got "failed to reclaim > memory" (to use the modern, improved [not misleading] > message text). Similarly for others that had other kinds > of contexts that got the message. > > (The units for the 120 are not time units: more like a > number of (re)tries to gain at least a target amount of > Free RAM before failure handling starts. The comment > wording is based on a consequence of the assignment.) > > The 120 is not a maximum, just a figure that has proved > useful in various contexts. > > But see the notes below based as well. > > Notes: > > "failed to reclaim memory" can happen even with swap > space enabled but no swap in use: sufficiently active > pages are just not paged out to swap space so if most > non-wired pages are classified as active, the kills > can start. > > (There are some other parameters of possible use for some > other modern "was killed" reason texts.) > > Wired pages are pages that can not be swapped out, even > if classified as inactive. > > Your report indicates: 24G Wired with 20G of that being > from ARC use. This likely was after some processes had > already been killed. So likely more was wired and less > was free at the start of the kills. > > That 24G+ of wired meant that only 8GiBytes- were > available everything else. Avoiding that by limiting > the ARC (tuning ZFS) or adjusting how the work load > is spread over time or some combination also looks > appropriate. > > I've no clue why ARC use would be signifcantly > different for 12.4 vs. 13.2p9 . > > === > Mark Millard > marklmi at yahoo.com Possibly not related, but codebase of ZFS on FreeBSD is switched from legacy one (aka ZoF) to OpenZFS on 13.0. -- Tomoaki AOKI