From owner-freebsd-current@FreeBSD.ORG  Mon Jan  6 13:03:45 2014
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 91030548
 for <freebsd-current@freebsd.org>; Mon,  6 Jan 2014 13:03:45 +0000 (UTC)
Received: from frv199.fwdcdn.com (frv199.fwdcdn.com [212.42.77.199])
 (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 488781F1E
 for <freebsd-current@freebsd.org>; Mon,  6 Jan 2014 13:03:45 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net;
 s=ffe; 
 h=Content-Transfer-Encoding:Content-Type:MIME-Version:References:In-Reply-To:Message-Id:Cc:To:Subject:From:Date;
 bh=W43s5jQxspALSVM7LOGj4SFo/lYgkv2scsS5xxilNl4=; 
 b=Hp5XtV1fQPTtsV6h0F6ld5F7kFfS8ZsqvlTqiqjj0ieCjAM0fsG+6XnM5qIrRCdwiROZ1aDf2tImSUieRMgfa1YmvjcnBESFEYWBtZHCN42ucsfKaZQbtP4Ob91lIZQpCMtwikHBMspkP71F0bHadumc2eYazPibM7fwsTHpwKQ=;
Received: from [10.10.10.45] (helo=frv45.ukr.net)
 by frv199.fwdcdn.com with smtp ID 1W09qE-0003lF-77
 for freebsd-current@freebsd.org; Mon, 06 Jan 2014 15:03:38 +0200
Date: Mon, 06 Jan 2014 15:03:37 +0200
From: Vladimir Sharun <atz@ukr.net>
Subject: Re[2]: ARC "pressured out", how to control/stabilize ? (reformatted
 to text/plain)
To: Andriy Gapon <avg@freebsd.org>
X-Mailer: mail.ukr.net 5.0
Message-Id: <1389013106.234649040.eb8ieitz@frv45.ukr.net>
In-Reply-To: <52CA9963.1050507@FreeBSD.org>
References: <1388839805.123581691.q97ijp8l@frv45.ukr.net>
 <52C93E4D.1050100@FreeBSD.org>
 <1389005433.815055146.2dcjke36@frv45.ukr.net>
 <52CA9963.1050507@FreeBSD.org>
MIME-Version: 1.0
Received: from atz@ukr.net by frv45.ukr.net; Mon, 06 Jan 2014 15:03:38 +0200
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: binary
Content-Disposition: inline
Cc: Current FreeBSD <freebsd-current@freebsd.org>
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 06 Jan 2014 13:03:45 -0000

Dear Andriy, FreeBSD community,

Thank you for your suggestion, so we will turn off the L2ARC and will report if the issue persist or not.
For now the test server rebooted with L2ARC turned off and there's no allocations done in 
l2arc_feed_thread according to dtrace hook you provided.

Let's imagine the situation, we found l2arc_feed_thread allocations is the cause, what shall our next step ?

The feedback from us will be here within few days (can't reproduce faster)
 
> on 06/01/2014 13:14 Vladimir Sharun said the following:
> > Dear Andriy and FreeBSD community,
> > 
> > I got the few minutes run for this dtrace hook; here's the output for 15 minutes run:
> > 
> > http://pastebin.com/pKm9kLwa
> > 
> > Does it explain something ?
> 
> The following makes me suspect a problem with L2ARC compression code.
> 
> zfs.ko`l2arc_feed_thread+0x7d9
> kernel`fork_exit+0x9a
> kernel`0xffffffff8069ad6e
> 95131
> 
> I am not sure if the buffers are leaked somehow or if they are actually in use.
> It's one of the very few places where data buffers are allocated without
> charging ARC.  In all other places it's quite easy to match allocations and
> deallocations.  But in L2ARC it is not obvious that all buffers get freed or
> when that happens.