From owner-freebsd-current@freebsd.org Wed Apr 4 01:47:24 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B01A6F84C47 for ; Wed, 4 Apr 2018 01:47:24 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-so.shaw.ca (smtp-out-so.shaw.ca [64.59.136.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1867680CCE; Wed, 4 Apr 2018 01:47:23 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id 3XWDfPPihLkoz3XWEfBBmf; Tue, 03 Apr 2018 19:47:22 -0600 X-Authority-Analysis: v=2.3 cv=OeS28CbY c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=Kd1tUaAdevIA:10 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=8fqvCxzDAAAA:8 a=7Ku_9JyV4qsk_o-Ohk0A:9 a=gwJYkg3bk64PVVus:21 a=RnkS0W7mmbt9ixsp:21 a=CjuIK1q_8ugA:10 a=yJeIQVEa5b5VMVsl9ZYA:9 a=qtpcrbx-0IhQCRVI:21 a=s0mIYAx8frL3NTph:21 a=kKMX0Zkevfrez679:21 a=_W_S_7VecoQA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=xrUD0wEA3vlMLTxm4nS9:22 Received: from [25.85.100.44] (unknown [24.244.29.245]) by spqr.komquats.com (Postfix) with ESMTPSA id A64BC1101; Tue, 3 Apr 2018 18:47:20 -0700 (PDT) MIME-Version: 1.0 From: Cy Schubert Subject: RE: Strange ARC/Swap/CPU on yesterday's -CURRENT Date: Tue, 3 Apr 2018 19:47:21 -0600 To: Andriy Gapon , Bryan Drewery , Peter Jeremy , Jeff Roberson CC: FreeBSD current Message-Id: <20180404014720.A64BC1101@spqr.komquats.com> X-CMAE-Envelope: MS4wfJ/HXm6UQchj3AbYQXru1sYwqQ3hmeKpLamkPjkF8sbm4bn4Pke4Iwa5cATYDQZzdSldHQWoQbcVBTBqIN/FWyeRqG18+GL9TK72mKMj2FJNSzuKN8MB MCjmaPg8yALWfYmTwFcRF/roFOeodb65tWMS+FJvvxtTk5CUkCTEfjljF4XPdzSApGDkh4sj4AMENcYnN10pbU1lfsmnPoRPrSxS33qUgeQeEpSJD04W1MhR mDTWkNbJ5vmdLQ9rNAayEctl1faKsTv9NoO4pKcJYspTu3M9VDEe3rcZ1puUNZnc Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Apr 2018 01:47:24 -0000 Agreed. I've come to the same conclusion that there may be multiple issues. --- Sent using a tiny phone keyboard. Apologies for any typos and autocorrect. Also, this old phone only supports top post. Apologies. Cy Schubert or The need of the many outweighs the greed of the few. --- -----Original Message----- From: Andriy Gapon Sent: 27/03/2018 08:02 To: Bryan Drewery; Peter Jeremy; Jeff Roberson Cc: FreeBSD current Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT On 24/03/2018 01:21, Bryan Drewery wrote: > On 3/20/2018 12:07 AM, Peter Jeremy wrote: >> >> On 2018-Mar-11 10:43:58 -1000, Jeff Roberson w= rote: >>> Also, if you could try going back to r328953 or r326346 and let me know= if=20 >>> the problem exists in either. That would be very helpful. If anyone i= s=20 >>> willing to debug this with me contact me directly and I will send some= =20 >>> test patches or debugging info after you have done the above steps. >> >> I ran into this on 11-stable and tracked it to r326619 (MFC of r325851). >> I initially got around the problem by reverting that commit but either >> it or something very similar is still present in 11-stable r331053. >> >> I've seen it in my main server (32GB RAM) but haven't managed to reprodu= ce >> it in smaller VBox guests - one difficulty I faced was artificially fill= ing >> ARC. First, it looks like maybe several different issues are being discussed and possibly conflated in this thread. I see reports related to ZFS, I see rep= orts where ZFS is not used at all. Some people report problems that appeared ve= ry recently while others chime in with "yes, yes, I've always had this problem= ". This does not help to differentiate between problems and to analyze them. > Looking at the ARC change you referred to from r325851 > https://reviews.freebsd.org/D12163, I am convinced that ARC backpressure > is completely broken. Does your being convinced come from the code review or from experiments? If the former, could you please share your analysis? > On my 78GB RAM system with ARC limited to 40GB and > doing a poudriere build of all LLVM and GCC packages at once in tmpfs I > can get swap up near 50GB and yet the ARC remains at 40GB through it > all. It's always been slow to give up memory for package builds but it > really seems broken right now. Well, there are multiple angles. Maybe it's ARC that does not react proper= ly, or maybe it's not being asked properly. It would be useful to monitor the system during its transition to the state= that you reported. There are some interesting DTrace probes in ARC, specificall= y arc-available_memory and arc-needfree are first that come to mind. Their parameters and how frequently they are called are of interest. VM paramete= rs could be of interest as well. A rant. Basically, posting some numbers and jumping to conclusions does not help at= all. Monitoring, graphing, etc does help. ARC is a complex dynamic system. VM (pagedaemon, UMA caches) is a complex dynamic system. They interact in a complex dynamic ways. Sometimes a change in ARC is incorrect and requires a fix. Sometimes a change in VM is incorrect and requires a fix. Sometimes a change in VM requires a change in ARC. These three kinds of problems can all appear as a "problem with ARC". For instance, when vm.lowmem_period was introduced you wouldn't find any me= ntion of ZFS/ARC. But it does affect period between arc_lowmem() calls. Also, pin-pointing a specific commit requires proper bisecting and proper testing to correctly attribute systemic behavior changes to code changes. --=20 Andriy Gapon _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"