From owner-freebsd-fs@FreeBSD.ORG Wed May 30 07:58:46 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 45AE0106566C; Wed, 30 May 2012 07:58:46 +0000 (UTC) (envelope-from resdev80@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6E5548FC1A; Wed, 30 May 2012 07:58:45 +0000 (UTC) Received: by lbon10 with SMTP id n10so3746211lbo.13 for ; Wed, 30 May 2012 00:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:subject:from:reply-to:to:cc:date:in-reply-to:references :content-type:x-mailer:content-transfer-encoding:mime-version; bh=pJjlhXKTrbjjuABc35bXQRWp4Bfh4uyhAUplG96FWDQ=; b=tdL2xBadHE0saUe9xPTNx2saxodXyEhp9uV/g3ttgPXg7ffATbStOHW3OQ+5lJn2ZR TxhHvqZ0p+39eOik+LBhX4wlp6GBrMsJF2tDpUlLvktAiofuTs3JcobdMfbwk6QmBpQC 35Gw8F9BYD2Rrmvmyt8kDhG4d3rFIu2JUccJvW/bpdgFh/gtIiAjOUAamZwPkuTCTK59 Q4AIgAb+v5nd91LyYShMIHyrirMM4kHwChE0yR6Hpwyk0fTtAdMwx5JWRBBY5pqT3qQu iYApKza41Goo66FG16MVJ4WxuJmZhIdbE8WcDPtncYMnLtPU1AWeiYbXqctS56CnjJNc nd+w== Received: by 10.112.86.132 with SMTP id p4mr6136518lbz.22.1338364724376; Wed, 30 May 2012 00:58:44 -0700 (PDT) Received: from [217.21.170.21] ([217.21.170.21]) by mx.google.com with ESMTPS id gv8sm26772889lab.14.2012.05.30.00.58.42 (version=SSLv3 cipher=OTHER); Wed, 30 May 2012 00:58:43 -0700 (PDT) Message-ID: <1338364726.2417.10.camel@brainbug-nb.office.ctco.lv> From: Juris Krumins To: Mikolaj Golub Date: Wed, 30 May 2012 10:58:46 +0300 In-Reply-To: <86ehq2nn35.fsf@in138.ua3> References: <1338295953.17714.12.camel@brainbug-nb.office.ctco.lv> <86ehq2nn35.fsf@in138.ua3> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.3 (3.2.3-3.fc16) Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Cc: freebsd-fs@freebsd.org Subject: Re: Hastd activemap cache flush and "request failed: FLUSH.: Operation not supported by device" messages. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: resdev80@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2012 07:58:46 -0000 Thanks for your answer Mikolaj. I'v found later last night that BIO_FLUSH request comes from ZFS and You prove it (thank You). More than that, I've also found that it's possible to disable BIO_FLUSH requests as ZFS level by setting following sysctl setting: vfs.zfs.vdev.bio_flush_disable: 1 The only concert I have right now is how this setting can potentially effect ZFS stability and ability to recover in case of power failures and/or other issues ? P.S. Underlying hardware is IBM EXS3000 disk bay with 12 disks combined into 3 RAID5 volumes (4 disks in each) with writeback, disk cache enabled and BBU. On Wed, 2012-05-30 at 10:29 +0300, Mikolaj Golub wrote: > On Tue, 29 May 2012 15:52:33 +0300 Juris Krumins wrote: > > JK> Hi everybody. > JK> I've recently setup two FreeBSD nodes to use them as a storage. > JK> So I'm using hast over "MegaRAID SAS 1078" and ZFS filesystem created on > JK> HAST resource devices from /dev/hast/*. > JK> Everything looks good, but when I execute iozone tests, I'm constantly > JK> getting following messages in /var/log/message: > > JK> primary node: > JK> May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Local request failed > JK> (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Local request failed > JK> (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Local request failed > JK> (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2312]: [mfid1] (primary) Remote request > JK> failed (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2313]: [mfid2] (primary) Remote request > JK> failed (Operation not supported by device): FLUSH. > JK> May 29 15:44:15 rook hastd[2311]: [mfid0] (primary) Remote request > JK> failed (Operation not supported by device): FLUSH. > > JK> secondary node: > JK> May 29 15:43:41 rook2 hastd[2458]: [mfid1] (secondary) Request failed: > JK> FLUSH.: Operation not supported by device. > JK> May 29 15:43:41 rook2 hastd[2457]: [mfid0] (secondary) Request failed: > JK> FLUSH.: Operation not supported by device. > JK> May 29 15:43:41 rook2 hastd[2462]: [mfid2] (secondary) Request failed: > JK> FLUSH.: Operation not supported by device. > > JK> I'v also found following topic > JK> http://lists.freebsd.org/pipermail/freebsd-fs/2012-January/013385.html. > JK> I've updated src, rebuild world and kernel and rebooted. I've also added > JK> metaflush "off" options to /etc/hast.conf configuration file. But still > JK> getting previously mentioned messages. > > Note, there are two sources of BIO_FLUSH operations in HAST: > > 1) BIO_FLUSH operations generated by HAST on metadata (activemap) update, if > enabled (not disabled) in config. > > 2) BIO_FLUSH operations sent by a HAST user (filesystem). > > If the underlying provider doesn't support BIO_FLUSH, in the first case a warning > "The provider doesn't support flushing write cache. Disabling it." will be > generated and there will be no more attempt to flush on activemap update. > > In the second case the error 'Request failed: FLUSH.: Operation not supported > by device' will be observed (as in your case). Before r225832 it logged on > every BIO_FLUSH coming. Since r225832 HAST will try only once and only one > error in logs should be observed after hastd start. > > JK> Can somebody point me to the right direction how to configure hastd to > JK> throw this warning only once. > > JK> I'm using: > JK> FreeBSD primary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 > JK> EEST 2012 root@primary.:/usr/obj/usr/src/sys/GENERIC amd64 > > JK> FreeBSD secondary. 9.0-STABLE FreeBSD 9.0-STABLE #0: Tue May 29 12:53:44 > JK> EEST 2012 root@secondary.:/usr/obj/usr/src/sys/GENERIC amd64 > > r225832 has been merged to 9-STABLE on 4 Jan 2012 as r229509 and you shouldn't > observe 'FLUSH: Operation not supported by device' constantly repeating. If > you do this is very strange and you should recheck your build. Please provide > the output of > > ident /sbin/hastd > > command. > > JK> and hast.conf configuration from both machines: > JK> I've decided to put metaflush option on every single level just to be > JK> sure. No effect. > > No need in disabling metaflush in your case. It will disable automatically > after the first attempt failed. >