From owner-freebsd-questions@FreeBSD.ORG Fri Mar 1 20:25:55 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D7828F73; Fri, 1 Mar 2013 20:25:55 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-la0-x22f.google.com (mail-la0-x22f.google.com [IPv6:2a00:1450:4010:c03::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 189501608; Fri, 1 Mar 2013 20:25:54 +0000 (UTC) Received: by mail-la0-f47.google.com with SMTP id fj20so3287721lab.6 for ; Fri, 01 Mar 2013 12:25:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=pZl3yfNTX6/p2gWRMQ0blnQ7e95MAg2FI+7TLJyVess=; b=PJ2bPROMjuvYQ8MHC4oQqXG8fm2FtClSYfBUAGOWnZby8eZbtApEbDGpl8QyVzfynH BaLcvCfFf//XzSIfQ+D4X/Q+sT5k/vPfb70Rx9vbWiigRo667KaXCPUt3fwRwjTD5Xxj Fuj6LanVm8wSEiJUO+f7j8vpyBZwKyeoLYO94WyIE2VzkyRSwW72Y6vxzO28v6p1Cx2V 3IdxtuuK0+z20X12W7oTaEb7CRQ+O+ZMPIopn2XQXycnuypPV/iNpGvmEnR5WkwZIpnZ kWlmUvA16ACkGfvWWkuT9M0r9WZiWnbzIvY17El+dyBv9gmS3p9rneA4DYVpcT9MicT5 eQLw== X-Received: by 10.152.109.84 with SMTP id hq20mr10122350lab.48.1362169554060; Fri, 01 Mar 2013 12:25:54 -0800 (PST) Received: from localhost ([178.150.115.244]) by mx.google.com with ESMTPS id t17sm7264102lam.9.2013.03.01.12.25.52 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Fri, 01 Mar 2013 12:25:53 -0800 (PST) Date: Fri, 1 Mar 2013 22:25:50 +0200 From: Mikolaj Golub To: Chad M Stewart Subject: Re: hast, zfs, unable to flush disk cache Message-ID: <20130301202549.GA2874@gmail.com> References: <3367E416-493C-44EC-9259-89877EDF6DE8@balius.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3367E416-493C-44EC-9259-89877EDF6DE8@balius.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Pawel Jakub Dawidek , freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Mar 2013 20:25:55 -0000 On Fri, Mar 01, 2013 at 11:39:23AM -0600, Chad M Stewart wrote: > I've setup a 2 node cluster using HAST. I'd previously had this > running 9.0 and then rebuilt the nodes using 9.1. Under 9.0 I used > vfs.zfs.vdev.bio_flush_disable=1 but that setting does not appear to > work in 9.1. The other difference, previous build had root disk on > UFS, while this build has only ZFS based file systems. > > FreeBSD node1.san 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243826: Tue Dec 4 06:55:39 UTC 2012 root@obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 > > > Mar 1 17:07:25 node1 hastd[1446]: [disk5] (primary) Unable to flush disk cache on activemap update: Operation not supported by device. > Mar 1 17:07:28 node1 hastd[1440]: [disk3] (primary) Unable to flush disk cache on activemap update: Operation not supported by device. > Mar 1 17:07:28 node1 hastd[1437]: [disk2] (primary) Unable to flush disk cache on activemap update: Operation not supported by device. > Mar 1 17:07:28 node1 hastd[1434]: [disk1] (primary) Unable to flush disk cache on activemap update: Operation not supported by device. > Mar 1 17:07:28 node1 hastd[1446]: [disk5] (primary) Unable to flush disk cache on activemap update: Operation not supported by device. > Mar 1 17:07:28 node1 hastd[1443]: [disk4] (primary) Unable to flush disk cache on activemap update: Operation not supported by device. > > I tried setting zfs set zfs:zfs_nocacheflush=1 but that > diddn't appear to work either. I get a lot of lines in my log file > because of this. I have also tried zfs set sync=disabled but > HAST still outputs those lines in the log file. These flushes were generated by HAST itself when it tried to flush activemap updates, that is why disabling BIO_FLUSH for ZFS dis not help much. Setting "metaflush off" in hast.conf should help though. BTW, hastd tries to detect devices that do not support BIO_FLUSH, checking for the returned errno, and automatically disable flushes if the errno is EOPNOTSUPP (Operation not supported). Unfortunately, your device returned ENODEV (Operation not supported by device). What device do you have? Pawel, do you think it would be a good idea to automatically disable activemap flush for ENODEV case too? -- Mikolaj Golub