From owner-freebsd-fs@FreeBSD.ORG Thu May 5 08:19:52 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 816471065672 for ; Thu, 5 May 2011 08:19:52 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: from mail.bytecamp.net (mail.bytecamp.net [212.204.60.9]) by mx1.freebsd.org (Postfix) with ESMTP id 0F05D8FC1A for ; Thu, 5 May 2011 08:19:51 +0000 (UTC) Received: (qmail 39916 invoked by uid 89); 5 May 2011 10:19:50 +0200 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with CAMELLIA256-SHA encrypted SMTP; 5 May 2011 10:19:50 +0200 Message-ID: <4DC25DA6.3060009@bytecamp.net> Date: Thu, 05 May 2011 10:19:50 +0200 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: zfs l2arc issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2011 08:19:52 -0000 Hi, we are running an NFS server with the following pool setup: home ONLINE 0 0 0 raidz2 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 logs ONLINE 0 0 0 mirror ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 cache ad4 ONLINE 0 0 0 ad8 ONLINE 0 0 0 All drives except the caching SSDs are attached to a LSI 9690SA-8I. The system is equipped with 32 GB RAM, and runs with a load of <1, please note: we are running 8.0, yet, since there was one issue with ZFS which blocked the upgrade to 8-STABLE. After about 100d uptime, we had a sudden large increase in load of about 5-7, nfsd had 100-400% WCPU. Also an rsync downloading files from that machine was very slow. We didn't really narrow down the problem, we had to reboot the machine because performance was nearly completely absent. After reboot, system performance became normal. Could this problem be related to the caching SSDs beeing full? Cache consists of two 76 GB SSDs, after warming up, only 8 MB are free on each disk. Is ZFS supposed to fill arbitrary large caches? I think of doubling the cache and then ending up with fully filled SSDs again. For if, could l2arc be limited somehow, so that SSDs don't get written full? Could this behaviour also appear in 8-STABLE? With kind regards, Robert Schulze