From owner-freebsd-current@freebsd.org  Thu Sep 13 15:11:41 2018
Return-Path: <owner-freebsd-current@freebsd.org>
Delivered-To: freebsd-current@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D6731090A11
 for <freebsd-current@mailman.ysv.freebsd.org>;
 Thu, 13 Sep 2018 15:11:41 +0000 (UTC)
 (envelope-from eric@vangyzen.net)
Received: from smtp.vangyzen.net (hotblack.vangyzen.net
 [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id C2E81752E8;
 Thu, 13 Sep 2018 15:11:40 +0000 (UTC)
 (envelope-from eric@vangyzen.net)
Received: from hammy.vangyzen.net (unknown [70.97.188.230])
 by smtp.vangyzen.net (Postfix) with ESMTPSA id 3DFA55646F;
 Thu, 13 Sep 2018 10:11:33 -0500 (CDT)
Subject: Re: arc_reclaim_thread running hot
From: Eric van Gyzen <eric@vangyzen.net>
To: freebsd-current <freebsd-current@freebsd.org>, markj@FreeBSD.org
References: <0cdba37f-6b47-0929-fb72-dcca7e70a05a@vangyzen.net>
Message-ID: <064f770e-e035-d3a3-c138-54e3f667a9be@vangyzen.net>
Date: Thu, 13 Sep 2018 10:11:26 -0500
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101
 Thunderbird/60.0
MIME-Version: 1.0
In-Reply-To: <0cdba37f-6b47-0929-fb72-dcca7e70a05a@vangyzen.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.27
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Sep 2018 15:11:41 -0000

On 9/13/18 8:18 AM, Eric van Gyzen wrote:
> This morning, I found the arc_reclaim_thread running hot on my laptop 
> running 12.0-ALPHA5 r338572.
> 
> vfs.zfs.arc_max="4294967296"  <-- 4 GiB
> 
> last pid: 13288;  load averages:  1.32,  1.26,  1.16
> Mem: 456M Active, 3837M Inact, 743M Laundry, 2563M Wired, 167M Free
> ARC: 1131M Total, 304M MFU, 145M MRU, 1344K Anon, 9116K Header, 671M Other
>       89M Compressed, 361M Uncompressed, 4.03:1 Ratio
> 
>     22 root         -8    -      0   256K CPU2     2 309:20  99.75% 
> zfskern{arc_reclaim_thread}
> 
> zfs_arc_meta_strategy is still the default of 1.
> 
> I sampled the thread's stacks with
> 
> for N in `jot 1000`; do procstat -kk 100101; done | grep 100101
> 
> and put the results here:
> 
> https://people.freebsd.org/~vangyzen/arc_reclaim_thread_stacks.txt
> 
> I'm happy to help debug this.  Just let me know what you need.

The thread eventually returned to normal, and I've rebooted the system 
since then, so I no longer have any state.  The thread started running 
hot at 03:00, so maybe the daily periodic run triggered it.  I have to 
work now, but I'll try running the periodic stuff manually when I have time.

Eric